Unrecoverable error in celery worker

[holm]

After the update to Celery 3.0.3 (from 2.4.6) today, we had all workers crash after running for roughly 12 hours. The traceback was the following:

Jul 27 19:12:40 78.108.125.18 celery[19503:140072151082752]: [celery.worker] Unrecoverable error: TypeError('init() takes exactly 3 arguments (1 given)', <class 'hoist.services.citrix.CitrixError'>, ()) Traceback (most recent call last): File "/mnt/podio/api/shared/virtualenv/local/lib/python2.7/site-packages/celery/worker/init.py", line 350, in start component.start() File "/mnt/podio/api/shared/virtualenv/local/lib/python2.7/site-packages/celery/worker/consumer.py", line 360, in start self.consume_messages() File "/mnt/podio/api/shared/virtualenv/local/lib/python2.7/site-packages/celery/worker/consumer.py", line 430, in consume_messages readers[fileno](fileno, event) File "/mnt/podio/api/shared/virtualenv/local/lib/python2.7/site-packages/billiard/pool.py", line 635, in handle_event self._it.next() File "/mnt/podio/api/shared/virtualenv/local/lib/python2.7/site-packages/billiard/pool.py", line 608, in _process_result ready, task = poll(timeout) File "/mnt/podio/api/shared/virtualenv/local/lib/python2.7/site-packages/billiard/pool.py", line 997, in _poll_result return True, self._quick_get() TypeError: ('init() takes exactly 3 arguments (1 given)', <class 'hoist.services.citrix.CitrixError'>, ())

It seems it is trying to initiate an exception of ours without any arguments (besides self), which then fails and subsequently stop the worker.

billard 2.7.3.10 was installed.

[ask]

Have you been able to reproduce this? I'm not quite sure where this is failing

[public]

This looks like what happens when you raise an exception that doesnt pickle correctly.

http://packages.python.org/celery/userguide/tasks.html#creating-pickleable-exceptions

[holm]

Yes, that could be the case. I will look into that.

[holm]

This seems to have been the case, sorry about that. Quite surprising information about the exceptions.

[lsemel]

Experiencing the same error since upgrading to 3.0 It seems that anything that raises an unpickleable exception can take the Celery workers.

I see the issue was closed -- was this fixed?

[holm]

We are being bitten by this. We have fixed all our own exceptions, but it is quite hard to manage all possible exceptions from libraries. In this case it is the "requests" package that has exceptions that cannot be pickled, however I think celery should handle this better so it doesn't crash the worker.

[lsemel]

Same here -- there's no way we can go through every library we use to check that all its exceptions are pickleable, so it'd be great if Celery itself could gracefully handle them.

[public]

It used to jam stuff into an UnpickleableExceptionWrapper iirc so this probably is a bug :)

[lsemel]

Good to know!

[ask]

Fixed by the commit above, please upgrade to celery 3.0.6

[holm]

Thanks a lot Ask