Celery worker with gevent pool + Sentry logger = Deathlock #305

Closed
amezhenin opened this Issue May 12, 2013 · 8 comments

Projects

None yet

4 participants

@amezhenin

Sorry for repost from celery/celery#1341 , I just don't know where get help.

I'm using Celery with Django integration. I discovered some troubles with new commit to my current project: Celery worker with gevent pool refused to handle new tasks. After short investigation, I found that 'sentry' log handler cause a problem:
settings.py:

 LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'filters': {
        'require_debug_false': {
            '()': 'django.utils.log.RequireDebugFalse'
        }
    },
    'formatters': {
        'verbose': {
            'format': '%(levelname)s %(asctime)s %(module)s '\
                      '%(process)d %(thread)d %(message)s'
        },
        'gunicorn_style': {
            'format': CELERYD_TASK_LOG_FORMAT,
            },
    },
    'handlers': {
        'console':{
            'level': 'DEBUG',
            'class': 'logging.StreamHandler',
            'formatter': 'gunicorn_style'
        },
        'mail_admins': {
            'level': 'ERROR',
            'filters': ['require_debug_false'],
            'class': 'django.utils.log.AdminEmailHandler'
        },
        'sentry': {
            'level': 'WARNING',
            'class': 'raven.contrib.django.raven_compat.handlers.SentryHandler',
        }
},
'loggers': {
    'django.request': {
        'handlers': ['mail_admins'],
        'level': 'ERROR',
        'propagate': True,
    },
    'raven': {
        'level': 'INFO',
        'handlers': ['console'],
        'propagate': False,
    },
    'sentry.errors': {
        'level': 'INFO',
        'handlers': ['console'],
        'propagate': False,
        },
    }
}
...
#I want verbose logs only for my apps
for i in MY_APPS:
    LOGGING['loggers'][i] = {
                             'handlers': ['console', 'sentry'],
                             'level': CONSOLE_LOGLEVEL,
                             'propagate': False,
                            }
LOGGING['loggers']['celery'] = {
                           'handlers': ['sentry'],
                           'level': CONSOLE_LOGLEVEL,
                           'propagate': True,
                          }
...

With 'handlers': ['console'] all works fine, but when I add 'sentry' handler celer+gevent worker start to behave as follows: take N tasks from broker, where N is concurrency level and then stops.

I run celery worker with this command:

python manage.py celery worker -Q celery_gevent -P gevent -c 20    

Note: deathlock shows with concurrency >= 3

$ pip freeze
Django==1.5
Fabric==1.6.0
South==0.7.6
amqp==1.0.9
anyjson==0.3.3
argparse==1.2.1
billiard==2.7.3.22
celery==3.0.16
cssselect==0.8
distribute==0.6.24
django-appconf==0.6
django-celery==3.0.11
django-geoip==0.3
django-nose==1.1
django-redis==3.2
flower==0.5.0
gevent==0.13.8
greenlet==0.4.0
gunicorn==0.17.2
ipython==0.13.1
kombu==2.5.7
logilab-astng==0.24.2
logilab-common==0.59.0
lxml==3.1.1
nose==1.2.1
paramiko==1.10.0
progressbar==2.3dev
psycopg2==2.4.6
pycrypto==2.6
pylint==0.27.0
pymongo==2.4.2
python-dateutil==1.5
pytz==2013b
raven==3.2.0
redis==2.7.2
requests==1.0.4
six==1.3.0
tornado==3.0.1
wsgiref==0.1.2

I'm using RabbitMQ as broker and redis as result backend.
Thank you.

@dcramer
Member
dcramer commented May 12, 2013

You likely want to use the gevent+http transport, but otherwise I have absolutely no idea why this would be broken.

That said, Celery barely works with gevent.

@amezhenin

Is there any correct way to setup sentry logging for celery+gevent?
Should I try eventlet pool for celery?

I play around with logging settings and this configuration works better:

LOGGING = {
    'version': 1,
    'disable_existing_loggers': True,
    'root': {
        'level': 'WARNING',
        'handlers': ['sentry'],
    },
...

LOGGING['loggers']['celery'] = {
                           'handlers': ['sentry'],
                           'level': CONSOLE_LOGLEVEL,
                           'propagate': False,
                          }

Celery+gevent worker receives tasks for several hours before hang.
Is there any way for to debug such things?

@dcramer
Member
dcramer commented May 12, 2013

Did you try changing to gevent+http for the transport like I said?

@amezhenin

no, how can I do this?

@amezhenin

oh, I found this. I will try.

@amezhenin

@dcramer, thank you!. Now it seems to work.

@thedrow
thedrow commented Nov 3, 2014

So this issue should be closed no?

@xordoquy
xordoquy commented Nov 3, 2014

sounds fair.

@xordoquy xordoquy closed this Nov 3, 2014
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment