Investigate adding gevent support to 0.9 tree #102

Closed
gmr opened this Issue Nov 9, 2011 · 9 comments

6 participants

@gmr gmr was assigned Nov 9, 2011
@dsuch

I can't help you out with it directly but would like to let you know that there's a ZeroMQ/gevent glue layer at https://github.com/traviscline/gevent-zeromq - who knows it, maybe it will serve as a source of inspiration.

Cheers!

@pacemkr

Any news on this? Since pika is a pure python implementation. Would gevent.monkey do the job?

@gmr
Pika member

No, I've not spent any time with gevent yet and want to get the growing buglist tackled before doing so.

@steeve

I've tried it with Celery, and on a Gevent reactor, there indeed seem to be concurency issues!

@daf
daf commented Sep 4, 2012

We've been running a project on it for nearly a year now, using gevent's normal monkey patching mechanisms. It mostly works ok, but we've built a layer on top of pika that has to go to lengths to make sure you don't do something we've noticed as bad.

We manually patch in some fixes for #97 (my bug) that might be related to gevent.

I've noticed sometimes we get a:

[snip]/eggs/pika-0.9.5-py2.7.egg/pika/callback.py:70: UserWarning: CallbackManager.add: Duplicate callback found for "0:Connection.CloseOk"

Which definetely appears related to the monkey patching - there's a race between channel closing (as called by the connection.close method) and the call at the end of connection.close which checks if there are any channels and if it should call the same method. I think it's mostly harmless but is one of the alarms when using this with gevent.

@steeve Can you elaborate more on your concurrency issues?

@steeve

I'd have to test it again, but I had some framing errors if I recall correctly

@daf
daf commented Sep 5, 2012

Ah, that reminds me. I had to plug a fix in for a similar sounding issue - pika/gevent would sometimes try to ack in the middle of a publish (which is a surprising three AMQP commands) and Rabbit would panic close everything.

Had to use Wireshark to figure out what was happening. Wish Rabbit would at least tell you which channel caused the panic, that would've helped narrow it down.

@sergedomk

Another developer and I wrote a gevent adapter for my current employer a while back. For the most part, it has been working flawlessly with no monkey patching. I will be updating it to work with the latest pika version. We've only seen one issue so far and it looks like it might be issue #97.

After I make the update, I'll submit a pull request for adding the adapter to pika.

@gmr
Pika member

Closing due to lack of time.

@gmr gmr closed this May 20, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment