Skip to content


Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP


Crashes with broken pipe #16

spulec opened this Issue · 16 comments

8 participants


This makes our workers crash/hang every hour or so.

Traceback (most recent call last):
File "../lib/python2.7/site-packages/billiard/", line 273, in _bootstrap
File "../lib/python2.7/site-packages/billiard/", line 122, in run
self._target(*self._args, **self._kwargs)
File "../lib/python2.7/site-packages/billiard/", line 302, in worker
put((ACK, (job, i, time.time(), pid)))
File "../lib/python2.7/site-packages/billiard/", line 377, in put
return send(obj)
IOError: [Errno 32] Broken pipe


We're using the SQS backend.


We are observing similar problems on:

software -> celery:3.0.10 (Chiastic Slide) kombu:2.4.7 py:2.7.3
            billiard: pymongo:2.3
platform -> system:Linux arch:64bit, ELF imp:CPython
loader   -> djcelery.loaders.DjangoLoader
settings -> transport:mongodb results:mongodb

When running:

python celery worker --loglevel=info --concurrency=4 --maxtasksperchild=10

But it does not happen when run as:

python celery worker --loglevel=info --concurrency=1 --maxtasksperchild=10


python celery worker --loglevel=info --concurrency=4 --maxtasksperchild=1

OK, it seems that with --concurrency=4 --maxtasksperchild=10 errors are more common, but they also happen with --concurrency=4 --maxtasksperchild=1.


I am also seeing this error with the latest stable release where celery crashes after a few hours of long running tasks. I'm seeing this happen after an hour long job has timed out.

[2012-10-22 03:22:13,673: WARNING/PoolWorker-1] Process PoolWorker-1:
[2012-10-22 03:22:13,674: WARNING/PoolWorker-1] Traceback (most recent call last):
[2012-10-22 03:22:13,675: WARNING/PoolWorker-1] File "/home/bhurley/.virtualenvs/celery/lib/python2.6/site-packages/billiard-", line 248, in _bootstrap
[2012-10-22 03:22:13,676: WARNING/PoolWorker-1]
[2012-10-22 03:22:13,677: WARNING/PoolWorker-1] File "/home/bhurley/.virtualenvs/celery/lib/python2.6/site-packages/billiard-", line 97, in run
[2012-10-22 03:22:13,678: WARNING/PoolWorker-1] self._target(self._args, *self._kwargs)
[2012-10-22 03:22:13,678: WARNING/PoolWorker-1] File "/home/bhurley/.virtualenvs/celery/lib/python2.6/site-packages/billiard-", line 308, in worker
[2012-10-22 03:22:13,679: WARNING/PoolWorker-1] put((READY, (job, i, (False, einfo))))
[2012-10-22 03:22:13,679: WARNING/PoolWorker-1] File "/home/bhurley/.virtualenvs/celery/lib/python2.6/site-packages/billiard-", line 352, in put
[2012-10-22 03:22:13,680: WARNING/PoolWorker-1] return send(obj)
[2012-10-22 03:22:13,680: WARNING/PoolWorker-1] IOError: [Errno 32] Broken pipe

software -> celery:3.0.11 (Chiastic Slide) kombu:2.4.7 py:2.6.6
billiard: amqplib:N/A
platform -> system:Linux arch:64bit, ELF imp:CPython
loader -> celery.loaders.default.Loader
settings -> transport:amqp results:disabled

BROKER_URL: 'amqp://something/myvhost'


@9thbit How did it time out? And do you think it's relevant?


@ask I was simply monitoring the tasks using top and just based on how long the job was running, it seemed to crash after one of the tasks exceeded time_limit.

I have added CELERYD_FORCE_EXECV = True to my config and I'll report back if that resolved the issue. Also, if it is related to using python 2.6 -- which is on the cluster I am using -- I can try 2.7.


I am using 2.7 and I am getting this.


Python 2.7 here with CELERYD_FORCE_EXECV = True.


Seeing this too with --concurrency=4 and kombu. Ugh.


Same issue for us.
It's not stable at all in production!


Same issue here.

ask commented

Could you please include the version your are using?

Also what broker are you using? (kombu is not a transport, it's the messaging framework we use)

The latest celery version disabled force_execv by default, it could be a culprit.


If you're using RabbitMQ/redis, could you please try running with CELERY_DISABLE_RATE_LIMITS=True ?


I'm are also seeing this crash. I also tried the CELERY_DISABLE_RATE_LIMITS=True suggestion but that didn't fix the error.

Unrecoverable error: IOError(32, 'Broken pipe')

Stacktrace (most recent call last):

  File "celery/worker/", line 351, in start
  File "celery/worker/", line 393, in start
  File "celery/worker/", line 483, in consume_messages
    handlermap[fileno](fileno, event)
  File "billiard/", line 1039, in maintain_pool
  File "billiard/", line 1034, in _maintain_pool
  File "billiard/", line 1020, in _repopulate_pool
  File "billiard/", line 904, in _create_worker_process
  File "billiard/", line 120, in start
    self._popen = Popen(self)
  File "billiard/", line 192, in __init__
    code = process_obj._bootstrap()
  File "billiard/", line 276, in _bootstrap

Here's our sys.argv:

['', 'celeryd', '--loglevel=INFO', '-c', '4', '--queues=celery', '--maxtasksperchild=100']

@jdp what celery/billiard version is this? please include the output of celery report (but make sure to remove sensitive information)

@ask ask closed this
@ask ask reopened this

If this happens while flushing stdout, then it may be worth a try to simply ignore this error.
It does not have a stdout at this point anyway.


Closing this issue now, please open a new issue if this happen again. Chances are it's not the same issue as the previous one.

@ask ask closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.