Error when starting a celery worker

[mpaf]

I create a new python environment (Python 2.7) and install django 1.4.3 and django-celery 3.0.11 . I then cd into my django project where the first two lines of settings.py have:

import djcelery
djcelery.setup_loader()

INSTALLED_APPS contains 'djcelery'

When I run:

 python manage.py celery worker

I get the following error:

-------------- celery@MIGUEL-PC v3.0.15 (Chiastic Slide)
---- **** -----
--- * *** * -- [Configuration]
-- * - **** --- . broker: amqp://guest@localhost:5672//

• ** ---------- . app: default:0x34fd9e8 (djcelery.loaders.DjangoLoader)
• ** ---------- . concurrency: 8 (processes)
• ** ---------- . events: OFF (enable -E to monitor this worker)
• ** ----------
• *** --- * --- [Queues]
  -- ******* ---- . celery: exchange:celery(direct) binding:celery
  --- ***** -----

[2013-02-13 15:40:58,119: ERROR/MainProcess] Unrecoverable error: TypeError("'LazySettings' object is not iterable",)
Traceback (most recent call last):
File "C:\Python27\Lib\site-packages\celery\worker__init__.py", line 351, in start
component.start()
File "C:\Python27\Lib\site-packages\celery\concurrency\base.py", line 111, in start
self.on_start()
File "C:\Python27\Lib\site-packages\celery\concurrency\processes__init__.py", line 80, in on_start
**self.options)
File "C:\Python27\Lib\site-packages\billiard\pool.py", line 845, in init
self._create_worker_process(i)
File "C:\Python27\Lib\site-packages\billiard\pool.py", line 915, in _create_worker_process
w.start()
File "C:\Python27\Lib\site-packages\billiard\process.py", line 119, in start
self._popen = Popen(self)
File "C:\Python27\Lib\site-packages\billiard\forking.py", line 336, in init
dump(process_obj, to_child, HIGHEST_PROTOCOL)
File "C:\Python27\Lib\site-packages\billiard\forking.py", line 127, in dump
ForkingPickler(file, protocol).dump(obj)
File "C:\Python27\lib\pickle.py", line 224, in dump
self.save(obj)
File "C:\Python27\lib\pickle.py", line 331, in save
self.save_reduce(obj=obj, *rv)
File "C:\Python27\lib\pickle.py", line 419, in save_reduce
save(state)
File "C:\Python27\lib\pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "C:\Python27\lib\pickle.py", line 649, in save_dict
self._batch_setitems(obj.iteritems())
File "C:\Python27\lib\pickle.py", line 681, in _batch_setitems
save(v)
File "C:\Python27\lib\pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "C:\Python27\lib\pickle.py", line 562, in save_tuple
save(element)
File "C:\Python27\lib\pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "C:\Python27\lib\pickle.py", line 548, in save_tuple
save(element)
File "C:\Python27\lib\pickle.py", line 306, in save
rv = reduce(self.proto)
File "C:\Python27\Lib\site-packages\celery\app\base.py", line 409, in reduce
(self.class, self.Pickler) + self.reduce_args(),
File "C:\Python27\Lib\site-packages\celery\app\base.py", line 419, in reduce_args
else self.conf._pickleable_changes())
File "C:\Python27\Lib\site-packages\celery\app\utils.py", line 77, in _pickleable_changes
R.update(d)
TypeError: 'LazySettings' object is not iterable

c:\Users\MIGUEL\Projects\django_atf>Traceback (most recent call last):
File "", line 1, in
File "C:\Python27\Lib\site-packages\billiard\forking.py", line 530, in main
self = load(from_parent)
File "C:\Python27\lib\pickle.py", line 1378, in load
return Unpickler(file).load()
File "C:\Python27\lib\pickle.py", line 858, in load
dispatchkey
File "C:\Python27\lib\pickle.py", line 880, in load_eof
raise EOFError
EOFError"

[orf]

I'm also getting this since I updated celery and billard. I run Windows, if that helps.

[ask]

@mpaf, are you also using Windows?

[mpaf]

Yes, sorry I forgot to mention. Windows 7 64 bit

[strayer]

Happens for me too. Win7 64bit.

[ask]

I don't have access to Windows at the moment,
but you could try changing the line in celery/app/base.py, in the __reduce_args__ that reads:

conf = (self.conf.changes if _forking._forking_is_enabled
           else self.conf._pickleable_changes())

into:

conf = (self.conf.changes if _forking._forking_is_enabled
            else {})

(these lines: https://github.com/celery/celery/blob/3.0/celery/app/base.py#L418-L419)

This should work as a workaround, and if celery was working before it shouldn't have any bad effects.
I need to find a better solution to fix it properly though.

[mpaf]

It works for me, thanks!

[sebasmagri]

I can confirm this issue and that the change fix the error.

Regards,

[titusz]

Had the same problem here on win7 with redis backend... the suggested workaround fixed it

[mpaf]

I guess more faults should be expected when using celery on Windows since not all of the tests pass. They all pass in Ubuntu. Not sure about the severity of the failed tests though.

[orf]

I've been running Celery in production on a Windows 2003 server (py 2.7) for the last few years with no issues to be had, until now.

[ask]

@mpaf: Some of the tests fail simply because the test code is incompatible (but not the actual code)

[movEAX]

I solved this problem for myself by adding lines of code below after https://github.com/celery/celery/blob/3.0/celery/app/utils.py#L71

if isinstance(d, LazySettings):
    dd = dict()
    for k in dir(d):
        attr = getattr(d._wrapped, k)
        if not callable(attr) and k.isupper() and not k.startswith('_'):
            dd[k] = attr
    d = dd

And don't forget to import LazySettings

[marcinn]

I can confirm this issue after upgrading from Celery 3.0.11 to 3.0.15 (Linux 64-bit), Django 1.4.0.

[sq9mev]

Do not confirm that issue, after upgrade to 3.0.15 from 3.0.11 on both Django 1.4.0 and Django 1.4.3 (Linux x86_64) workers start without exception. Other packages:

$ pip freeze
Django==1.4.3
PIL==1.2a0
PyXML==0.8.4
amqp==1.0.8
anyjson==0.3.3
billiard==2.7.3.21
celery==3.0.15
iwscan==0.0.0
kombu==2.5.6
libxml2-python==2.9.0
lxml==2.3.5
mercurial==2.2.3
pysqlite==2.6.3
python-dateutil==1.5
six==1.2.0
wpactrl==0.0.0

*************** PICKED VERSIONS ****************
[versions]
Fabric = 1.5.1
Markdown = 2.2.1
Pygments = 1.5
Unidecode = 0.04.10
Whoosh = 2.4.1
chardet = 2.1.1
collective.recipe.template = 1.9
django-articles = 2.4.1
django-celery = 3.0.11
django-extensions = 1.0.2
django-pagination = 1.0.7
django-tinymce = 1.5.1b4
djangorecipe = 1.3
docutils = 0.10
flup = 1.0.3.dev-20110405
geopy = 0.94.2
gocept.recipe.env = 1.0
hexagonit.recipe.cmmi = 1.6
hexagonit.recipe.download = 1.6
httplib2 = 0.7.7
infrae.subversion = 1.4.5
ipdb = 0.7
iw.recipe.subversion = 0.3.1
meld3 = 0.6.10
numpy = 1.6.2
paramiko = 1.9.0
psycopg2 = 2.4.6
py = 1.4.12
pyPdf = 1.13
pycrypto = 2.6
pyproj = 1.9.2
python-memcached = 1.48
pytz = 2012j
qrcode = 2.4.2
reportlab = 2.6
requests = 1.0.4
setuptools = 0.6c11
simplejson = 3.0.5
sorl-thumbnail = 11.12
supervisor = 3.0b1
template-utils = 0.4p2
textile = 2.1.5
vectorformats = 0.1
xmpppy = 0.5.0rc1
z3c.rml = 2.0.0
zc.buildout = 1.6.3
zc.recipe.egg = 1.3.2
zerokspot.recipe.git = 0.6.1
zope.event = 4.0.2
zope.interface = 4.0.3
zope.schema = 4.2.2

[orf]

Still happens for me with 3.0.15 on Windows 8 64bit.

[orf]

Any updates on this, its still occurring for me. It should be a one line fix.

[ask]

@orf could you try installing the 3.0 development branch?

pip install https://github.com/celery/celery/zipball/3.0

@marcinn This is not a place for you to recruit users, you say rq is simple, I would be nice if I called it nave, I still
don't litter their issue tracker.

3.0.16 does solve this problem for non-windows users, but you may have to set CELERY_FORCE_EXECV=False if you have already set it to True. Hopefully the recent change in the 3.0 branch will fix it on Windows, but sadly I don't have access to a Windows license myself.

[marcinn]

Sorry @ask, but in my case Celery is unstable as far I remember (since 2.x). I upgraded to 3.x because of some critical bugs in 2.x. And I tried many versions, many configurations and my customer is still complaining for months ("mr. marcin, please kill the rabbit again"). I can't go to vacation, because I have to "kill rabbit" (stalled celery processes) almost every day. Installing new version isn't possible due this issue (is it?). I'm trapped.

Maybe I'm too stupid or lazy to use Celery properly, or just should wait for stable release. But I can't wait more for critical updates, probably same as @orf. This is a business problem.

Don't get me wrong and keep up good job. Maybe Celery is not yet stable as we need.

[marcinn]

Yes, I know about long-running workers so I added cron jobs for restarting workers periodically. But still have some kind of problems which I can't recognize properly. The problem is that I can't do experimens in live env and can't reproduce major problems (I/O blocking?) in dev env. The simplest and safest way is to update Celery (and wait for The Rabbit), but 20 days ago I was not able to do that. I'll give a next try with 3.0.16 in this week.

Forking processes is a good feature for some (simple) cases and it's ok for me.

[adam-snyder]

@ask The 3.0 development branch worked for me on Windows. On a side note, I couldn't get pip to install celery using the command you provided so I had to manually unzip it to my site-packages. Thanks, and keep up the good work.

[keeyai]

I also had this problem on win7 64bit. pip uninstalled celery and django-celery, use the 3.0 branch above with pip install https://github.com/celery/celery/zipball/3.0, reinstalled django-celery, and it appears to be working.

[jpellerin]

I've seen this on ubuntu 12.04/64-bit with django 1.5 and celery 3.0.16. The 3.0 branch had the same problem, downgrading to 3.0.11 fixed it for me.

[marcinn]

Well, I've just tried to upgrade Celery to 3.0.16 and I can't run workers anymore (known issue - ImportError: billiard.forking). I'm using buildout.

billiard-2.7.3.22
celery-3.0.16
django_celery-3.0.11
kombu-2.5.7
Django-1.4

I have no luck. Again. :(

[ask]

@marcinn Do you have FORCE_EXECV in your settings, or are you on Windows?

[ask]

@marcinn buildout is messing things up here, you could try adding PYTHONPATH=/path/to/buildout/lib to ensure the external programs get the right PATH, or if not on Windows use the --no-execv argument