Error when starting a celery worker

[mpaf]

I create a new python environment (Python 2.7) and install django 1.4.3 and django-celery 3.0.11 . I then cd into my django project where the first two lines of settings.py have:

import djcelery
djcelery.setup_loader()

INSTALLED_APPS contains 'djcelery'

When I run:

 python manage.py celery worker

I get the following error:

-------------- celery@MIGUEL-PC v3.0.15 (Chiastic Slide)
---- **** -----
--- * *** * -- [Configuration]
-- * - **** --- . broker: amqp://guest@localhost:5672//

• ** ---------- . app: default:0x34fd9e8 (djcelery.loaders.DjangoLoader)
• ** ---------- . concurrency: 8 (processes)
• ** ---------- . events: OFF (enable -E to monitor this worker)
• ** ----------
• *** --- * --- [Queues]
  -- ******* ---- . celery: exchange:celery(direct) binding:celery
  --- ***** -----

[2013-02-13 15:40:58,119: ERROR/MainProcess] Unrecoverable error: TypeError("'LazySettings' object is not iterable",)
Traceback (most recent call last):
File "C:\Python27\Lib\site-packages\celery\worker__init__.py", line 351, in start
component.start()
File "C:\Python27\Lib\site-packages\celery\concurrency\base.py", line 111, in start
self.on_start()
File "C:\Python27\Lib\site-packages\celery\concurrency\processes__init__.py", line 80, in on_start
**self.options)
File "C:\Python27\Lib\site-packages\billiard\pool.py", line 845, in init
self._create_worker_process(i)
File "C:\Python27\Lib\site-packages\billiard\pool.py", line 915, in _create_worker_process
w.start()
File "C:\Python27\Lib\site-packages\billiard\process.py", line 119, in start
self._popen = Popen(self)
File "C:\Python27\Lib\site-packages\billiard\forking.py", line 336, in init
dump(process_obj, to_child, HIGHEST_PROTOCOL)
File "C:\Python27\Lib\site-packages\billiard\forking.py", line 127, in dump
ForkingPickler(file, protocol).dump(obj)
File "C:\Python27\lib\pickle.py", line 224, in dump
self.save(obj)
File "C:\Python27\lib\pickle.py", line 331, in save
self.save_reduce(obj=obj, *rv)
File "C:\Python27\lib\pickle.py", line 419, in save_reduce
save(state)
File "C:\Python27\lib\pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "C:\Python27\lib\pickle.py", line 649, in save_dict
self._batch_setitems(obj.iteritems())
File "C:\Python27\lib\pickle.py", line 681, in _batch_setitems
save(v)
File "C:\Python27\lib\pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "C:\Python27\lib\pickle.py", line 562, in save_tuple
save(element)
File "C:\Python27\lib\pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "C:\Python27\lib\pickle.py", line 548, in save_tuple
save(element)
File "C:\Python27\lib\pickle.py", line 306, in save
rv = reduce(self.proto)
File "C:\Python27\Lib\site-packages\celery\app\base.py", line 409, in reduce
(self.class, self.Pickler) + self.reduce_args(),
File "C:\Python27\Lib\site-packages\celery\app\base.py", line 419, in reduce_args
else self.conf._pickleable_changes())
File "C:\Python27\Lib\site-packages\celery\app\utils.py", line 77, in _pickleable_changes
R.update(d)
TypeError: 'LazySettings' object is not iterable

c:\Users\MIGUEL\Projects\django_atf>Traceback (most recent call last):
File "", line 1, in
File "C:\Python27\Lib\site-packages\billiard\forking.py", line 530, in main
self = load(from_parent)
File "C:\Python27\lib\pickle.py", line 1378, in load
return Unpickler(file).load()
File "C:\Python27\lib\pickle.py", line 858, in load
dispatchkey
File "C:\Python27\lib\pickle.py", line 880, in load_eof
raise EOFError
EOFError"

[orf]

I'm also getting this since I updated celery and billard. I run Windows, if that helps.

[ask]

@mpaf, are you also using Windows?

[mpaf]

Yes, sorry I forgot to mention. Windows 7 64 bit

[strayer]

Happens for me too. Win7 64bit.

[ask]

I don't have access to Windows at the moment,
but you could try changing the line in celery/app/base.py, in the __reduce_args__ that reads:

conf = (self.conf.changes if _forking._forking_is_enabled
           else self.conf._pickleable_changes())

into:

conf = (self.conf.changes if _forking._forking_is_enabled
            else {})

(these lines: https://github.com/celery/celery/blob/3.0/celery/app/base.py#L418-L419)

This should work as a workaround, and if celery was working before it shouldn't have any bad effects.
I need to find a better solution to fix it properly though.

[mpaf]

It works for me, thanks!

[sebasmagri]

I can confirm this issue and that the change fix the error.

Regards,

[titusz]

Had the same problem here on win7 with redis backend... the suggested workaround fixed it

[mpaf]

I guess more faults should be expected when using celery on Windows since not all of the tests pass. They all pass in Ubuntu. Not sure about the severity of the failed tests though.

[orf]

I've been running Celery in production on a Windows 2003 server (py 2.7) for the last few years with no issues to be had, until now.

[ask]

@mpaf: Some of the tests fail simply because the test code is incompatible (but not the actual code)

[movEAX]

I solved this problem for myself by adding lines of code below after https://github.com/celery/celery/blob/3.0/celery/app/utils.py#L71

if isinstance(d, LazySettings):
    dd = dict()
    for k in dir(d):
        attr = getattr(d._wrapped, k)
        if not callable(attr) and k.isupper() and not k.startswith('_'):
            dd[k] = attr
    d = dd

And don't forget to import LazySettings

[marcinn]

I can confirm this issue after upgrading from Celery 3.0.11 to 3.0.15 (Linux 64-bit), Django 1.4.0.

[sq9mev]

Do not confirm that issue, after upgrade to 3.0.15 from 3.0.11 on both Django 1.4.0 and Django 1.4.3 (Linux x86_64) workers start without exception. Other packages:

$ pip freeze
Django==1.4.3
PIL==1.2a0
PyXML==0.8.4
amqp==1.0.8
anyjson==0.3.3
billiard==2.7.3.21
celery==3.0.15
iwscan==0.0.0
kombu==2.5.6
libxml2-python==2.9.0
lxml==2.3.5
mercurial==2.2.3
pysqlite==2.6.3
python-dateutil==1.5
six==1.2.0
wpactrl==0.0.0

*************** PICKED VERSIONS ****************
[versions]
Fabric = 1.5.1
Markdown = 2.2.1
Pygments = 1.5
Unidecode = 0.04.10
Whoosh = 2.4.1
chardet = 2.1.1
collective.recipe.template = 1.9
django-articles = 2.4.1
django-celery = 3.0.11
django-extensions = 1.0.2
django-pagination = 1.0.7
django-tinymce = 1.5.1b4
djangorecipe = 1.3
docutils = 0.10
flup = 1.0.3.dev-20110405
geopy = 0.94.2
gocept.recipe.env = 1.0
hexagonit.recipe.cmmi = 1.6
hexagonit.recipe.download = 1.6
httplib2 = 0.7.7
infrae.subversion = 1.4.5
ipdb = 0.7
iw.recipe.subversion = 0.3.1
meld3 = 0.6.10
numpy = 1.6.2
paramiko = 1.9.0
psycopg2 = 2.4.6
py = 1.4.12
pyPdf = 1.13
pycrypto = 2.6
pyproj = 1.9.2
python-memcached = 1.48
pytz = 2012j
qrcode = 2.4.2
reportlab = 2.6
requests = 1.0.4
setuptools = 0.6c11
simplejson = 3.0.5
sorl-thumbnail = 11.12
supervisor = 3.0b1
template-utils = 0.4p2
textile = 2.1.5
vectorformats = 0.1
xmpppy = 0.5.0rc1
z3c.rml = 2.0.0
zc.buildout = 1.6.3
zc.recipe.egg = 1.3.2
zerokspot.recipe.git = 0.6.1
zope.event = 4.0.2
zope.interface = 4.0.3
zope.schema = 4.2.2

[orf]

Still happens for me with 3.0.15 on Windows 8 64bit.