Client cannot send with gunicorn gevent worker

[faith0811]

Using latest version of zerorpc, I cannot send message to zerorpc server. The client is hosted in a web server (flask) running upon gunicorn with gevent worker.
It looks like that the thread is just blocked, and there is no response at all.
The client works after I change to gunicorn sync worker ( with no -k gevent ).
is this a problem of zerorpc(i saw zerorpc has implemented a gevent version of zeromq) or it is something wrong with gunicorn or i messed up with something?

Merci!

[bombela]

Can you double check that gevent is working properly in your gunicorn+flask setup? zerorpc should definitively work in this case. zerorpc has its own wrapper integrating zmq with gevent, because of some sync/bug problems that were never resolved. But its very small and zerorpc definitively work with gevent.

de rien :)

[faith0811]

I wrote a demo for this
https://github.com/faith0811/zerorpc_demo
I run the command

$ PYTHONPATH=. gunicorn client:app

and visit the localhost website, and the app works fine. Here is the log:

[2015-08-07 18:18:44 +0800] [28067] [INFO] Starting gunicorn 19.3.0
[2015-08-07 18:18:44 +0800] [28067] [INFO] Listening at: http://127.0.0.1:8000 (28067)
[2015-08-07 18:18:44 +0800] [28067] [INFO] Using worker: sync
[2015-08-07 18:18:44 +0800] [28070] [INFO] Booting worker with pid: 28070
ping

And I switched to gevent worker by changing command to

$ PYTHONPATH=. gunicorn -k gevent client:app

when I visit the localhost website, the thread is blocked, and I didn't get the excepted response ping. After like 30s later, a critical message printed told me that the worker was timeout.
Here is the log:

[2015-08-07 18:18:55 +0800] [28074] [INFO] Starting gunicorn 19.3.0
[2015-08-07 18:18:55 +0800] [28074] [INFO] Listening at: http://127.0.0.1:8000 (28074)
[2015-08-07 18:18:55 +0800] [28074] [INFO] Using worker: gevent
[2015-08-07 18:18:55 +0800] [28077] [INFO] Booting worker with pid: 28077
[2015-08-07 18:19:30 +0800] [28074] [CRITICAL] WORKER TIMEOUT (pid:28077)
[2015-08-07 18:19:31 +0800] [28078] [INFO] Booting worker with pid: 28078

[faith0811]

Update:
It turns out that gunicorn gevent worker will do gevent.monkey.patch_all() at the beginning. The patch_thread() function will patch the threading module and eventually breaks the pyzmq.
pyzmq seems call the built-in threading module inside its own c code, and gevent monkey patch cannot patch it.
I tried to replace gevent_zmq writen in zerorpc with zmq.green which provided by offical pyzmq, and the problem is still there.
Surprisingly, zmq.green(gevent_zmq) works with simple send and recv even with gevent.monkey.patch_all() is called before.
This issue can be simply fixed by downgrading pyzmq to 13.1.0. After downgrading, everything works in both server and client side.
I think it might be some function zerorpc used here triggers the pyzmq threading module and sadly pyzmq cannot handle it with monkey patch.
Do you have any idea which part of code will trigger something like that?

thanks!

[bombela]

Hi, thanks for the throughout explanation. So it looks like monkey patching the threading module is not doing good on pyzmq. But hey its monkey patching, what did you expect ;) zerorpc itself is doing nothing wrong here, it uses pyzmq the way it should be used, and wraps it to work with the gevent loop. that's it.

The problem lies definitively with monkey patching everything and pyzmq is then failing. pyzmq probably requires real native thread to perform some tasks because it likely blocks the threads. If it suddenly uses greenthreads, it will lock up the main thread ie: the whole process. That's what I bet, at least.

Best,
fx

[lordnynex]

Is there a possible fix here? This issue is certainly a bug in pyzmq directly and not zerorpc, but given the way pyzmq.context is implemented, I don't see that this module can ever work with gevent and a modern version of pyzmq.

[faith0811]

@lordnynex see zeromq/pyzmq#765 for detail, it seems fixed in 15.2.0. I tested myself, in simple cases, it works with gevent monkey patch. So I think there might be something else in your code triggers the bug. I haven't use XREP before, does pyzmq.green support this?

[bombela]

@faith0811 thank you for the links, very interesting. The latest version of pyzmq should handle the gevent monkey patching of the threading API.

Few miscellaneous points:

• At some point XREP was called ROUTER, and XREQ was called DEALER.
• pyzmq.green or the equivalent in zerorpc are simply wrapping pyzmq to work with the gevent loop.
• gevent doesn't monkey patch by default.