"DatabaseWrapper objects created in a thread can only be used in that same thread" using prefork and the celery command

[hheimbuerger]

celery 3.1.24, django-celery (3.1.17), using the 'django' broker transport.

I'm launching my app with the celery command, like this: celery -A myapp worker --pool=prefork

And I get the following exception when running a task that accesses the database (single database, standard Django ORM, latest Postgres with psycopg2):

[2016-10-20 15:22:22,967] ERROR [-/-/-/celery.worker.job] Task myapp.mytask[a05031c7-3e16-4edb-b484-c51f6976d244] raised unexpected: DatabaseError("DatabaseWrapper objects created in a thread can only be used in that same thread. The object with alias 'default' was created in thread id 4628 and this is thread id 55128752.",)
Traceback (most recent call last):
  File "c:\python\2.7\lib\site-packages\celery\app\trace.py", line 240, in trace_task
    R = retval = fun(*args, **kwargs)
  File "c:\python\2.7\lib\site-packages\celery\app\trace.py", line 438, in __protected_call__
    return self.run(*args, **kwargs)
  File "E:\code\src\myapp\tasks.py", line 96, in mytask
    request_model_here()   # two stack frames of internal code removed here
  File "c:\python\2.7\lib\site-packages\django\db\models\manager.py", line 122, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "c:\python\2.7\lib\site-packages\django\db\models\query.py", line 381, in get
    num = len(clone)
  File "c:\python\2.7\lib\site-packages\django\db\models\query.py", line 240, in __len__
    self._fetch_all()
  File "c:\python\2.7\lib\site-packages\django\db\models\query.py", line 1074, in _fetch_all
    self._result_cache = list(self.iterator())
  File "c:\python\2.7\lib\site-packages\django\db\models\query.py", line 52, in __iter__
    results = compiler.execute_sql()
  File "c:\python\2.7\lib\site-packages\django\db\models\sql\compiler.py", line 846, in execute_sql
    cursor = self.connection.cursor()
  File "c:\python\2.7\lib\site-packages\django\db\backends\base\base.py", line 229, in cursor
    self.validate_thread_sharing()
  File "c:\python\2.7\lib\site-packages\django\db\backends\base\base.py", line 524, in validate_thread_sharing
    % (self.alias, self._thread_ident, thread.get_ident()))
DatabaseError: DatabaseWrapper objects created in a thread can only be used in that same thread. The object with alias 'default' was created in thread id 4628 and this is thread id 55128752.

Any idea what might cause this? I've debugged all my way through, but I'm struggling to understand how this is supposed to work when celery preforks the workers. I assume the problem is that the workers inherit the DatabaseWrapper during the fork?

This started to appear around the time I was upgrading Django from 1.8 to 1.9, but there's a lot of related changes and I cannot narrow it down to that specifically.

Related issues:

• fix django.db.backends import in patch_thread_ident django-celery#476: a patch method for django-celery that doesn't fix it for me
• Cannot access Django ORM in worker when using python3+eventlet #2829: same issue, but the official solution was to switch from manage.py celery to the celery command, which I'm already using
• How to implement terminate_job for eventlet concurrency backend? #3429: mentions the same issue with prefork, but doesn't go into detail because prefork was never the intended solution, just a test on the side

[ask]

Can you try installing celery 4.0rc5 (pip install 'celery>dev')? I wonder if the issue is in that version.

Usually we've seen this when the gevent/eventlet monkey patches are applied too late, not with the prefork pool.

I'm guessing somehow some state for the databases is initialized in the parent process, that's inherited in the child, but Celery will close the connection after fork.

You mention that you don't use manage.py celery, do you use djcelery.setup_loader() anywhere?

[hheimbuerger]

@ask

Can you try installing celery 4.0rc5 (pip install 'celery>dev')? I wonder if the issue is in that version.

Not easily, because last I checked a lot of my project is incompatible with the changes announced for Celery 4 (I'm using a lot of the parts that have been deprecated). I'll give it a try on the weekend, though.

Usually we've seen this when the gevent/eventlet monkey patches are applied too late, not with the prefork pool.

That's what I've learned, so I'm really trying to understand what I'm doing differently to cause this in prefork right now.

I'm guessing somehow some state for the databases is initialized in the parent process, that's inherited in the child, but Celery will close the connection after fork.

Sorry, I don't quite understand. Are you saying initializing the database in the parent process and then closing it after the fork is what's supposed to happen, or are you saying that's what should not happen, but if it were to happen, it might cause my issue?

Because that's what I believe I'm seeing in debugging. However, note that I haven't found a way to debug the actual celery command (because that's an executable on Windows and PyCharm won't accept it as a Python run target). Any advice on how to debug this? Can I just run the celery/bin/celery.py file from the Celery sources? Therefore so far, I've only been debugging via manage.py celery. (But the above error is still from a non-debugging run via the celery command!)

The problem is that I haven't found any documentation about how the patch is supposed to work: Is the BaseDatabaseWrapper supposed to be created in the pre-forking or post-forking? Is the patch supposed to run in the prefork configuration? If so, at what stage exactly?
Not knowing that, it's hard to debug because I don't have anything to compare what I'm seeing with.

You mention that you don't use manage.py celery, do you use djcelery.setup_loader() anywhere?

I wasn't. I've added it now to the very end of my settings.py. I didn't notice any effect, the exception still occurs in the exact same way.

[hheimbuerger]

I have resolved this issue. Another library was the culprit, it wasn't caused by celery or django-celery. I'll add details later.

[hheimbuerger]

I could eventually narrow this down to the following lines in the devourer library:

import gevent
from gevent import monkey
monkey.patch_all()

Commenting these out (remember, I'm not using gevent in celery at all, just prefork) fixed all my issues.

So the debugging advice I have for everyone reading this due to a similar issue is: even if the stack trace looks like celery, the issue might not originate in celery (thanks to monkey patching hell…)! Check your other libraries and see what kind of patching they might perform that could cause this.

Thanks for your response, @ask, and sorry for the distraction.

---

bonnierpolska/devourer#1