Poll Result error causes celeryd to crash

[jjmalina]

Unrecoverable error: 'TypeError(\'__init__() takes exactly 3 arguments (2 given)\', <class \'requests.packages.urllib3.exceptions.MaxRetryError\'>, ("HTTPConnectionPool(host=\'www.timesfreepress.com\', port=80): Max retries exceeded with url: /news/2012/sep/08/dade-county-gets-new-extension-agriculture-agent/?utm_source=twitterfeed&utm_medium=twitter",))'

Stacktrace (most recent call last):

  File "celery/worker/__init__.py", line 353, in start
    component.start()
  File "celery/worker/consumer.py", line 369, in start
    self.consume_messages()
  File "celery/worker/consumer.py", line 453, in consume_messages
    readers[fileno](fileno, event)
  File "billiard/pool.py", line 660, in handle_event
    self._it.next()
  File "billiard/pool.py", line 633, in _process_result
    ready, task = poll(timeout)
  File "billiard/pool.py", line 1034, in _poll_result
    return True, self._quick_get()

This happened in a celery task that was making an HTTP request with requests, which raised an exception and it caused celeryd to stop running entirely. You have to manually restart after this. Versions are billiard==2.7.3.12 and celery==3.0.9

[ask]

The exception that is raised is not pickleable:

>>> from requests.packages.urllib3.exceptions import MaxRetryError
>>> x = MaxRetryError(None, "foo")
>>> import pickle
>>> pickle.loads(pickle.dumps(x))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 1382, in loads
    return Unpickler(file).load()
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 858, in load
    dispatch[key](self)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 1133, in load_reduce
    value = func(*args)
TypeError: __init__() takes exactly 3 arguments (2 given)

Celery have some elaborate mechanisms to deal with these non-standard exceptions though,
and it works fine for me:

@task
def unpick():
    from requests.packages.urllib3.exceptions import MaxRetryError
    raise MaxRetryError(None, "http://foo")

Does that not work for you? If it does, then at what point does it raise this exception?

[myusuf3]

I can confirm by upgrading to django-celery 3.0.11 and billard 2.7.3.19 that comes with it; stops the crashing of celery when a Exception that cannot be pickled is returned.

[ask]

Closing

[pjwerneck]

I'm still having the exact same problem with celery 3.0.18 and billard 2.7.3.19:

[2013-04-15 20:23:59,174: ERROR/MainProcess] Unrecoverable error: TypeError('__init__() takes exactly 3 arguments (2 given)', <class 'requests.packages.urllib3.exceptions.MaxRetryError'>, ("HTTPConnectionPool(hos
t='xxx', port=80): Max retries exceeded with url: xxx,))
Traceback (most recent call last):
  File "/home/deployer/titans-mobile-learning/.venv/lib/python2.6/site-packages/celery/worker/__init__.py", line 347, in start
    component.start()
  File "/home/deployer/titans-mobile-learning/.venv/lib/python2.6/site-packages/celery/worker/consumer.py", line 390, in start
    self.consume_messages()
  File "/home/deployer/titans-mobile-learning/.venv/lib/python2.6/site-packages/celery/worker/consumer.py", line 474, in consume_messages
    readers[fileno](fileno, event)
  File "/home/deployer/titans-mobile-learning/.venv/lib/python2.6/site-packages/billiard/pool.py", line 672, in handle_event
    self._it.next()
  File "/home/deployer/titans-mobile-learning/.venv/lib/python2.6/site-packages/billiard/pool.py", line 645, in _process_result
    ready, task = poll(timeout)
  File "/home/deployer/titans-mobile-learning/.venv/lib/python2.6/site-packages/billiard/pool.py", line 1062, in _poll_result
    return True, self._quick_get()
TypeError: ('__init__() takes exactly 3 arguments (2 given)', <class 'requests.packages.urllib3.exceptions.MaxRetryError'>, ("HTTPConnectionPool(host='x', port=80): Max retries exceeded with url: x",))