Celery: redis backend Unrecoverable error: KeyError(9,)

[linar-jether]

On Ubuntu 14.10, using current development versions of celery, kombu and billiard, and the redis backend starting a worker leads to 'Unrecoverable error: KeyError(9,)'

Even the most simple app fails to start,

from celery import Celery

app = Celery('hello', broker='redis://127.0.0.1//)

app.conf.update(CELERY_ACCEPT_CONTENT = ['json'])

@app.task
def hello():
    return 'hello world'

Seems related to celery/kombu#379

Here is the debug output and stack trace

[2015-09-29 11:03:56,332: DEBUG/MainProcess] | Worker: Preparing bootsteps.
[2015-09-29 11:03:56,332: DEBUG/MainProcess] | Worker: Building graph...
[2015-09-29 11:03:56,333: DEBUG/MainProcess] | Worker: New boot order: {Beat, Timer, Hub, Queues (intra), Pool, Autoreloader, StateDB, Autoscaler, Consumer}
[2015-09-29 11:03:56,337: DEBUG/MainProcess] | Consumer: Preparing bootsteps.
[2015-09-29 11:03:56,337: DEBUG/MainProcess] | Consumer: Building graph...
[2015-09-29 11:03:56,343: DEBUG/MainProcess] | Consumer: New boot order: {Connection, Agent, Events, Mingle, Tasks, Control, Gossip, Heart, event loop}

 -------------- celery@jether5 v3.2.0a2 (DEV)
---- **** ----- 
--- * ***  * -- Linux-3.16.0-38-generic-x86_64-with-debian-jessie-sid
-- * - **** --- 
- ** ---------- [config]
- ** ---------- .> app:         hello:0x7fd71df6b410
- ** ---------- .> transport:   redis://127.0.0.1:6379//
- ** ---------- .> results:     disabled
- *** --- * --- .> concurrency: 4 (prefork)
-- ******* ---- 
--- ***** ----- [queues]
 -------------- .> celery           exchange=celery(direct) key=celery


[tasks]
  . celery.accumulate
  . celery.backend_cleanup
  . celery.chain
  . celery.chord
  . celery.chord_unlock
  . celery.chunks
  . celery.group
  . celery.map
  . celery.starmap
  . test_task.hello

[2015-09-29 11:03:56,347: DEBUG/MainProcess] | Worker: Starting Hub
[2015-09-29 11:03:56,347: DEBUG/MainProcess] ^-- substep ok
[2015-09-29 11:03:56,347: DEBUG/MainProcess] | Worker: Starting Pool
[2015-09-29 11:03:56,350: DEBUG/MainProcess] ^-- substep ok
[2015-09-29 11:03:56,350: DEBUG/MainProcess] | Worker: Starting Consumer
[2015-09-29 11:03:56,350: DEBUG/MainProcess] | Consumer: Starting Connection
[2015-09-29 11:03:56,356: INFO/MainProcess] Connected to redis://127.0.0.1:6379//
[2015-09-29 11:03:56,356: DEBUG/MainProcess] ^-- substep ok
[2015-09-29 11:03:56,356: DEBUG/MainProcess] | Consumer: Starting Events
[2015-09-29 11:03:56,360: DEBUG/MainProcess] ^-- substep ok
[2015-09-29 11:03:56,360: DEBUG/MainProcess] | Consumer: Starting Mingle
[2015-09-29 11:03:56,360: INFO/MainProcess] mingle: searching for neighbors
[2015-09-29 11:03:57,364: INFO/MainProcess] mingle: all alone
[2015-09-29 11:03:57,365: DEBUG/MainProcess] ^-- substep ok
[2015-09-29 11:03:57,365: DEBUG/MainProcess] | Consumer: Starting Tasks
[2015-09-29 11:03:57,373: DEBUG/MainProcess] ^-- substep ok
[2015-09-29 11:03:57,374: DEBUG/MainProcess] | Consumer: Starting Control
[2015-09-29 11:03:57,378: DEBUG/MainProcess] ^-- substep ok
[2015-09-29 11:03:57,378: DEBUG/MainProcess] | Consumer: Starting Gossip
[2015-09-29 11:03:57,382: DEBUG/MainProcess] ^-- substep ok
[2015-09-29 11:03:57,382: DEBUG/MainProcess] | Consumer: Starting Heart
[2015-09-29 11:03:57,384: DEBUG/MainProcess] ^-- substep ok
[2015-09-29 11:03:57,384: DEBUG/MainProcess] | Consumer: Starting event loop
[2015-09-29 11:03:57,385: WARNING/MainProcess] celery@jether5 ready.
[2015-09-29 11:03:57,385: DEBUG/MainProcess] | Worker: Hub.register Pool...
[2015-09-29 11:03:57,387: CRITICAL/MainProcess] Unrecoverable error: KeyError(9,)
Traceback (most recent call last):
  File "/home/jether5/anaconda/lib/python2.7/site-packages/celery-3.2.0a2-py2.7.egg/celery/worker/__init__.py", line 208, in start
    self.blueprint.start(self)
  File "/home/jether5/anaconda/lib/python2.7/site-packages/celery-3.2.0a2-py2.7.egg/celery/bootsteps.py", line 123, in start
    step.start(parent)
  File "/home/jether5/anaconda/lib/python2.7/site-packages/celery-3.2.0a2-py2.7.egg/celery/bootsteps.py", line 374, in start
    return self.obj.start()
  File "/home/jether5/anaconda/lib/python2.7/site-packages/celery-3.2.0a2-py2.7.egg/celery/worker/consumer.py", line 278, in start
    blueprint.start(self)
  File "/home/jether5/anaconda/lib/python2.7/site-packages/celery-3.2.0a2-py2.7.egg/celery/bootsteps.py", line 123, in start
    step.start(parent)
  File "/home/jether5/anaconda/lib/python2.7/site-packages/celery-3.2.0a2-py2.7.egg/celery/worker/consumer.py", line 855, in start
    c.loop(*c.loop_args())
  File "/home/jether5/anaconda/lib/python2.7/site-packages/celery-3.2.0a2-py2.7.egg/celery/worker/loops.py", line 53, in asynloop
    connection.drain_events()
  File "/home/jether5/anaconda/lib/python2.7/site-packages/kombu-3.1.0a1-py2.7.egg/kombu/connection.py", line 275, in drain_events
    return self.transport.drain_events(self.connection, **kwargs)
  File "/home/jether5/anaconda/lib/python2.7/site-packages/kombu-3.1.0a1-py2.7.egg/kombu/transport/virtual/__init__.py", line 863, in drain_events
    item, channel = get(timeout=timeout)
  File "/home/jether5/anaconda/lib/python2.7/site-packages/kombu-3.1.0a1-py2.7.egg/kombu/transport/redis.py", line 343, in get
    ret = self.handle_event(fileno, event)
  File "/home/jether5/anaconda/lib/python2.7/site-packages/kombu-3.1.0a1-py2.7.egg/kombu/transport/redis.py", line 326, in handle_event
    return self.on_readable(fileno), self
  File "/home/jether5/anaconda/lib/python2.7/site-packages/kombu-3.1.0a1-py2.7.egg/kombu/transport/redis.py", line 320, in on_readable
    chan, type = self._fd_to_chan[fileno]
KeyError: 9
[2015-09-29 11:03:57,391: DEBUG/MainProcess] | Worker: Closing Hub...
[2015-09-29 11:03:57,391: DEBUG/MainProcess] | Worker: Closing Pool...
[2015-09-29 11:03:57,391: DEBUG/MainProcess] | Worker: Closing Consumer...
[2015-09-29 11:03:57,391: DEBUG/MainProcess] | Worker: Stopping Consumer...
[2015-09-29 11:03:57,392: DEBUG/MainProcess] | Consumer: Closing Connection...
[2015-09-29 11:03:57,392: DEBUG/MainProcess] | Consumer: Closing Events...
[2015-09-29 11:03:57,392: DEBUG/MainProcess] | Consumer: Closing Mingle...
[2015-09-29 11:03:57,393: DEBUG/MainProcess] | Consumer: Closing Tasks...
[2015-09-29 11:03:57,393: DEBUG/MainProcess] | Consumer: Closing Control...