SocketIO makes Celery worker hang

[JanBenes]

I had an application that uses Celery that was working fine. I wanted to add Flask-SocketIO to implement a Celery unrelated feature. However, this made Celery worker hang.

More specifically, I invoke the celery worker by calling

celery worker -A app.celery_app -l info

in a directory which has the package app as a subdirectory and that initializes celery_app in its __init__.py. This would normally show me some info and eventually say

[2014-10-17 17:02:35,394: INFO/MainProcess] mingle: all alone
[2014-10-17 17:02:35,410: WARNING/MainProcess] celery@mymachine ready.

However, by adding

from flask.ext.socketio import SocketIO

to the top of the app/__init__.py file, these two last lines would never show and Celery couldn't be killed by pressing Ctrl-C once (which would make it wait for tasks) or twice (which would make it quit immediately). The Celery Worker will hang. Literally, commenting out this one import makes all the difference.

I think this is relatively unexpected. Is there a way around? Is that a problem with Flask-SocketIO?

For completeness, the rest of app\__init__.py looks like this:

from flask import Flask
from flask_login import LoginManager, login_required, login_user, logout_user
from flask.ext.sqlalchemy import SQLAlchemy
# from flask.ext.socketio import SocketIO  # uncommenting this causes celery worker to hang

# as per http://flask.pocoo.org/docs/0.10/patterns/celery/
# but running with RabbitMQ as a broker, namely
# celery = Celery(app.import_name, broker=app.config['CELERY_BROKER_URL'])
# becomes
# celery = Celery(app.import_name)
from app.celery import make_celery  

# create the flask application itself
app = Flask(__name__)
app.config.from_object('app.config')  # no Flask-SocketIO or Celery config here

# initialize login manager
login_manager = LoginManager()
login_manager.init_app(app)
login_manager.login_view = '/login'

# database
db = SQLAlchemy(app)

from app.celery import make_celery
celery_app = make_celery(app)

from app import tasks  # Celery tasks
from app import models  # sqlalchemy

I am relatively new to Python/Flask/Celery/SocketIO, so maybe I'm just missing something obvious.

I'm running Python 2.7 on Windows. Also, for what it's worth, it seems socket io also causes a problem with debugging the __init__ and the main files in pycharm, which I think might be related. Perhaps the immediate server restart is a problem? Later in the run, I can debug views just fine.

[miguelgrinberg]

This is probably due to the gevent monkey patching, see #55. I plan to remove monkey patching, the extension does not need it anyway. If you want to check if this does it, remove the first two lines in __init__.py.

[JanBenes]

Reading #55 and the linked spyoungtech/grequests#8, it seems this is likely the problem.

Yes, commenting out the first two lines, that is

from gevent import monkey
monkey.patch_all()

removes the problem. I assume this makes Flask-SocketIO dis-functional (haven't tried yet, will tomorrow). I think this is basically the same problem as #55. Also, sorry for the late reply.

[miguelgrinberg]

@JanBenes: monkey patching is currently not needed by Flask-SocketIO, that is a left-over from an old version. Applications that depend on monkey-patching can always monkey-patch from their side. I'll go ahead and push an update.

[miguelgrinberg]

Version 0.4.1 released. This version removes the monkey patching.

[JanBenes]

This works. Thanks!