TypeError when retrying tasks on 3.1.11

[wonderb0lt]

Hi,

we're running Celery 3.1.11 with Redis 2.8.6 as the broker. Python 2.7.3 on Amazon Ubuntu 12.04.4 machines.

When retrying tasks, we get the following error:

[2014-04-23 08:19:41,425: WARNING/Worker-2] /usr/local/lib/python2.7/dist-packages/celery/app/trace.py:364: RuntimeWarning: Exception raised outside body: reject requeue=True: argument must be an int, or have a fileno() method.:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/celery/app/trace.py", line 253, in trace_task
    I, R, state, retval = on_error(task_request, exc, uuid)
  File "/usr/local/lib/python2.7/dist-packages/celery/app/trace.py", line 201, in on_error
    R = I.handle_error_state(task, eager=eager)
  File "/usr/local/lib/python2.7/dist-packages/celery/app/trace.py", line 85, in handle_error_state
    }[self.state](task, store_errors=store_errors)
  File "/usr/local/lib/python2.7/dist-packages/celery/app/trace.py", line 120, in handle_failure
    task.on_failure(exc, req.id, req.args, req.kwargs, einfo)
  File "/usr/local/lib/python2.7/dist-packages/pipes/tasks/pipes_task.py", line 185, in on_failure
    countdown=self.TASK_RETRY_COUNTDOWN_SECONDS)
  File "/usr/local/lib/python2.7/dist-packages/celery/app/task.py", line 673, in retry
    raise Reject(exc, requeue=True)
Reject: (TypeError('argument must be an int, or have a fileno() method.',), True)

We retry tasks with the following call:

raise self.retry(args=args, kwargs=kwargs, max_retries=self.MAX_TASK_RETRIES,
                                 countdown=self.TASK_RETRY_COUNTDOWN_SECONDS)

Reverting back to 3.1.10 corrects this issue.

Celery project output:
The project report output is the following:

software -> celery:3.1.11 (Cipater) kombu:3.0.15 py:2.7.3
            billiard:3.3.0.17 redis:2.9.1
platform -> system:Linux arch:64bit, ELF imp:CPython
loader   -> celery.loaders.app.AppLoader
settings -> transport:redis results:disabled

CELERY_ACKS_LATE: False
CELERY_RESULT_SERIALIZER: 'json'
CELERY_ACCEPT_CONTENT: ['json', 'msgpack', 'yaml']
CELERY_ENABLE_REMOTE_CONTROL: True
CELERYBEAT_SCHEDULE: {
    'tracking': {   'options': {   'priority': 6},
                    'schedule': datetime.timedelta(0, 1800),
                    'task': '***redacted***'}}
ENV_DEV: 'dev'
CELERY_IMPORTS: ['***REDACTED***']
BROKER_POOL_LIMIT: None
BROKER_URL: 'redis://***REDACTED***'
CELERY_RESULT_PERSISTENT: False
CELERY_TASK_SERIALIZER: 'json'
CELERY_SEND_EVENTS: True

[ask]

Thanks! Can you modify the retry method to reraise the original exception instead? Hard to see what's going on without the original traceback, but I guess this could be coming from the Redis transport implementation of Channel.basic_reject

But also, note that retry raising Reject means it could not send the retry message in the first place. On AMQP you have the ack transaction so you can actually recover from that if the server is down. If the server is down on Redis you cannot be able to retry the task (you would have to increase the CELERY_TASK_PUBLISH_RETRY_POLICY so that it doesn't back off too quickly, but that is not fail proof)

[wonderb0lt]

While trying to recreate the issue I've come across the same error message while calling "apply_async". I know that deviates a bit from the initial error report, but I think the issues are, if anything, closely related. Here's the stack trace:

>>> from pipes.tasks.backoffice import TestTaskWhichFails
>>> TestTaskWhichFails().apply_async()
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "/home/stegmann/.virtualenvs/celery1997/local/lib/python2.7/site-packages/celery-3.1.11-py2.7.egg/celery/app/task.py", line 555, in apply_async
    **dict(self._get_exec_options(), **options)
  File "/home/stegmann/.virtualenvs/celery1997/local/lib/python2.7/site-packages/celery-3.1.11-py2.7.egg/celery/app/base.py", line 351, in send_task
    reply_to=reply_to or self.oid, **options
  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
    self.gen.next()
  File "/home/stegmann/.virtualenvs/celery1997/local/lib/python2.7/site-packages/celery-3.1.11-py2.7.egg/celery/app/base.py", line 408, in producer_or_acquire
    yield producer
  File "/home/stegmann/.virtualenvs/celery1997/local/lib/python2.7/site-packages/kombu-3.0.15-py2.7.egg/kombu/messaging.py", line 224, in __exit__
    self.release()
  File "/home/stegmann/.virtualenvs/celery1997/local/lib/python2.7/site-packages/kombu-3.0.15-py2.7.egg/kombu/connection.py", line 907, in release
    self.release(R)
  File "/home/stegmann/.virtualenvs/celery1997/local/lib/python2.7/site-packages/kombu-3.0.15-py2.7.egg/kombu/pools.py", line 73, in release
    resource.__connection__.release()
  File "/home/stegmann/.virtualenvs/celery1997/local/lib/python2.7/site-packages/kombu-3.0.15-py2.7.egg/kombu/connection.py", line 907, in release
    self.release(R)
  File "/home/stegmann/.virtualenvs/celery1997/local/lib/python2.7/site-packages/kombu-3.0.15-py2.7.egg/kombu/connection.py", line 934, in release
    self.close_resource(resource)
  File "/home/stegmann/.virtualenvs/celery1997/local/lib/python2.7/site-packages/kombu-3.0.15-py2.7.egg/kombu/connection.py", line 1016, in close_resource
    resource._close()
  File "/home/stegmann/.virtualenvs/celery1997/local/lib/python2.7/site-packages/kombu-3.0.15-py2.7.egg/kombu/connection.py", line 302, in _close
    self._do_close_self()
  File "/home/stegmann/.virtualenvs/celery1997/local/lib/python2.7/site-packages/kombu-3.0.15-py2.7.egg/kombu/connection.py", line 292, in _do_close_self
    self.maybe_close_channel(self._default_channel)
  File "/home/stegmann/.virtualenvs/celery1997/local/lib/python2.7/site-packages/kombu-3.0.15-py2.7.egg/kombu/connection.py", line 284, in maybe_close_channel
    channel.close()
  File "/home/stegmann/.virtualenvs/celery1997/local/lib/python2.7/site-packages/kombu-3.0.15-py2.7.egg/kombu/transport/redis.py", line 735, in close
    self._pool.disconnect()
  File "build/bdist.linux-x86_64/egg/redis/connection.py", line 478, in disconnect
    connection.disconnect()
  File "/home/stegmann/.virtualenvs/celery1997/local/lib/python2.7/site-packages/kombu-3.0.15-py2.7.egg/kombu/transport/redis.py", line 800, in disconnect
    channel._on_connection_disconnect(self)
  File "/home/stegmann/.virtualenvs/celery1997/local/lib/python2.7/site-packages/kombu-3.0.15-py2.7.egg/kombu/transport/redis.py", line 461, in _on_connection_disconnect
    self.connection.cycle._on_connection_disconnect(connection)
  File "/home/stegmann/.virtualenvs/celery1997/local/lib/python2.7/site-packages/kombu-3.0.15-py2.7.egg/kombu/transport/redis.py", line 259, in _on_connection_disconnect
    self.poller.unregister(connection._sock)
  File "/home/stegmann/.virtualenvs/celery1997/local/lib/python2.7/site-packages/kombu-3.0.15-py2.7.egg/kombu/utils/eventio.py", line 85, in unregister
    self._epoll.unregister(fd)
TypeError: argument must be an int, or have a fileno() method.

[junqed]

I sometime get this error too (after calling an async celery task :() And still haven't found the reason. Version 3.1.10

[martinhoefling]

Have you tried to clear the redis queues? In my case, deleting all celery* keys in the redis db did the trick.

[junqed]

You mean "celery purge"?

[nimrodm]

Could it be related to the fact that Redis timeout's value is 0?

[ask]

Probably related to celery/kombu#342? Could you try that patch and report back and I can release a bugfix version of kombu.

[wonderb0lt]

Good news, this issue does not appear any more for us. I tried it out with Celery 3.1.11 + Kombu from its master branch. Neither of the cases I reported (retrying, starting) throw this any more. Thanks!

[ask]

Fixed in kombu 3.0.16 on PyPI now!