gevent monkeypatching breaks celery

[Mihara]

I've been hunting for this issue for the better part of a year, but I've finally isolated it.

If gevent is installed, and django-mama-cas is installed and you're using celery, very bizarre things start happening eventually. Most importantly, gevent monkeypatches os.fork. Celery, in the default worker configuration, is dependent on using os.fork, and initially nothing seems out of the ordinary, until, after a few hours, it simply mysteriously freezes and workers stop accepting new jobs. Switching to threaded workers suddenly works.

Initially, I blamed it on myself not understanding celery properly. Finally, I decided to swap it out in favor of django-rq, which I found much less arcane to correctly set up. Django-rq started freezing immediately, because it, likewise, uses os.fork. Switching to SimpleWorker class, which just executes the job immediately, fixes it.

Now, removing gevent, so that django-mama-cas can't load it, immediately gets it all to work again.

I'm not sure what can be done about it, frankly, but I'm suspecting it needs to be used in a more explicit manner than that.

P.S. See spyoungtech/grequests#8 for a related problem caused by using gevent.
P.P.S. No, it is not sufficient to monkey.patch_all(thread=False, select=False, os=False) -- at least, it is not sufficient to get django-rq to work.
P.P.P.S. The problem appears to go away with gevent 1.1rc5, however...

[jbittel]

Interesting...I don't currently run these two together, so I've not encountered this before. Did you try running celery with the "-P gevent" option? Is that what you mean by switching to threaded workers?

On the plus side, gevent isn't required for django-mama-cas operation, but it can certainly make SLO requests happen faster.

[Mihara]

Threaded workers is -P thread. -P gevent just crashes.

Django-rq exhibits a similar problem, which also persists on gevent 1.1rc5 when I use memcached for session storage, which is why I caught this much later.

The problem isn't that gevent is required or not required, it's that if it is required by anything else, django-mama-cas will use it, and monkeypatch it in, (whether that anything else monkeypatches it in or not) with pretty unpredictable results...

I'm going to investigate what happens if I monkeypatch everything globally all the way up in gunicorn, which will ensure, at least, that everything is monkeypatched early, but I don't really have high hopes for this. I'd have much better hopes for https://github.com/i-trofimtschuk/frequests but django-mama-cas isn't structured to take advantage of that easily.

[jbittel]

Did you have any success monkeypatching earlier in the process?

The simplest idea to sidestep this would be to add a "don't use gevent" setting to forcibly ignore it even if it's present because of some other requirement.

[Mihara]

Haven't had the opportunity to do that yet, and probably won't for a week more. :(

A setting to disable gevent even if present would be a workable stopgap measure, though I think that actually using gevent without monkeypatching would be the preferable solution.