Yet another threading issue with m2m (v2.2.1) #131
Comments
|
quick update: this behaviour is only appears on production servers where django project is running under uwsgi. no problems appears when local dev server is used. |
|
update 2: disabling cache for Problematic environment: |
|
update 3: switching from threads to processes did not make any difference to described problem. still curious why this issue does not happen on a local development server. (runserver) :( |
|
@Suor can you please take a look at this? |
|
I will take a look tomorrow |
|
@Suor thanks! I'm still digging into a problem and using oldschool "print" debugging I've found out that m2m_changed is not called while running inside uwsgi, but is called while workin with devserver. which does not solve my problem, but well can help futher invetigation |
@once
def install_cacheops():
"""
Installs cacheops by numerous monkey patches
"""
print "INSTALL CACHEOPS 1"
monkey_mix(Manager, ManagerMixin)
monkey_mix(QuerySet, QuerySetMixin)
QuerySet._cacheprofile = QuerySetMixin._cacheprofile
QuerySet._cloning = QuerySetMixin._cloning
monkey_mix(ValuesQuerySet, QuerySetMixin, ['iterator'])
monkey_mix(ValuesListQuerySet, QuerySetMixin, ['iterator'])
monkey_mix(DateQuerySet, QuerySetMixin, ['iterator'])
print "INSTALL CACHEOPS 2"
# Install profile and signal handlers for any earlier created models
from django.db.models import get_models
for model in get_models(include_auto_created=True):
model._default_manager._install_cacheops(model)
print "INSTALL CACHEOPS 3"
# Turn off caching in admin
try:
# Use app registry in Django 1.7
from django.apps import apps
admin_used = apps.is_installed('django.contrib.admin')
except ImportError:
# Introspect INSTALLED_APPS in older djangos
from django.conf import settings
admin_used = 'django.contrib.admin' in settings.INSTALLED_APPS
print "INSTALL CACHEOPS 4"
if admin_used:
from django.contrib.admin.options import ModelAdmin
# Renamed queryset to get_queryset in Django 1.6
method_name = 'get_queryset' if hasattr(ModelAdmin, 'get_queryset') else 'queryset'
@monkey(ModelAdmin, name=method_name)
def get_queryset(self, request):
return get_queryset.original(self, request).nocache()
# Bind m2m changed handler
print "INSTALL CACHEOPS 5"
m2m_changed.connect(invalidate_m2m)
# Make buffers/memoryviews pickleable to serialize binary field data
if six.PY2:
import copy_reg
copy_reg.pickle(buffer, lambda b: (buffer, (bytes(b),)))
if six.PY3:
import copyreg
copyreg.pickle(memoryview, lambda b: (memoryview, (bytes(b),)))while running with uwsgi (1 process / 0 threads) I've noticed that code is not executing after this line settings.INSTALLED_APPS while execution install_cacheops is the following: |
|
Still no luck :(
so as a result – m2m_changed signal was not attached (and probably cacheops were not installed correctly) and all my problems were caused by this. |
|
what is working: |
|
Since I don't have full failing project I can only guess. The best guess is you messed up with import path in uwsgi configuration, so some model can't load. And error is swallowed by Django. Exception is the only way installation procedure could be broken in between as you describe. You was on the right track with your print debugging though. You can continue by checking for exceptions in failing code section. E.g. like: # Add this before install_cacheops() procedure
import traceback
from contextlib import contextmanager
@contextmanager
def print_errors(label):
try:
yield
except Exception as e:
print('%s raised in %s' % (traceback.format_exc(), label))
raise
# Change a sections between 2 and 3 to
with print_errors('import'):
from django.db.models import get_models
for model in get_models(include_auto_created=True):
print('Loading models for', model)
with print_errors('cacheops for %s' % model):
model._default_manager._install_cacheops(model)Run it and see what import causes failure. If this doesn't lead to solution then step into. Thanks open source, Django code is available :) P.S. If this leads to nothing then please say what Django version you are using and your |
|
@Suor thx for update! I've made a local modifitication for cacheops/query.py like this: import traceback
from contextlib import contextmanager
@contextmanager
def print_errors(label):
try:
yield
except Exception as e:
print('%s raised in %s' % (traceback.format_exc(), label))
raise
@once
def install_cacheops():
"""
Installs cacheops by numerous monkey patches
"""
monkey_mix(Manager, ManagerMixin)
monkey_mix(QuerySet, QuerySetMixin)
QuerySet._cacheprofile = QuerySetMixin._cacheprofile
QuerySet._cloning = QuerySetMixin._cloning
monkey_mix(ValuesQuerySet, QuerySetMixin, ['iterator'])
monkey_mix(ValuesListQuerySet, QuerySetMixin, ['iterator'])
monkey_mix(DateQuerySet, QuerySetMixin, ['iterator'])
print "BEGIN INSTALL"
# Change a sections between 2 and 3 to
with print_errors('import'):
from django.db.models import get_models
for model in get_models(include_auto_created=True):
print('Loading models for', model)
with print_errors('cacheops for %s' % model):
model._default_manager._install_cacheops(model)
print "END INSTALL"
# rest is ommite as not interesting and never reachedRestarted UWSGI, made a request to a service and here is the output: So clearly END INSTALL was not reached. I've tried to put more prints inside # inside django.db.models.loading.AppCache#get_models
import inspect
caller = inspect.stack()[1][3]
if caller == 'install_cacheops':
print "OK, I've been called from install_cacheops"get_models stops at line self._populate() which has interesting docs, which I'm not sure I understand correctly: # django.db.models.loading.AppCache#_populate
def _populate(self):
"""
Fill in all the cache information. This method is threadsafe, in the
sense that every caller will see the same state upon return, and if the
cache is already initialised, it does no work.
"""
if self.loaded:
return
# Note that we want to use the import lock here - the app loading is
# in many cases initiated implicitly by importing, and thus it is
# possible to end up in deadlock when one thread initiates loading
# without holding the importer lock and another thread then tries to
# import something which also launches the app loading. For details of
# this situation see #18251.
imp.acquire_lock()
try:
if self.loaded:
return
for app_name in settings.INSTALLED_APPS:
if app_name in self.handled:
continue
self.load_app(app_name, True)
if not self.nesting_level:
for app_name in self.postponed:
self.load_app(app_name)
self.loaded = True
finally:
imp.release_lock()So I'm still nowhere (well fixed my issue with an early import in settings). Django ==1.6.8, sys.path (from print inside install_cacheops): |
|
UPD: Tried to apply this Traceback (most recent call last):
File "/Users/Silver/.virtualenvs/myproject-server/lib/python2.7/site-packages/django/db/models/loading.py", line 24, in print_errors
yield
File "/Users/Silver/.virtualenvs/myproject-server/lib/python2.7/site-packages/django/db/models/loading.py", line 246, in get_models
self._populate()
File "/Users/Silver/.virtualenvs/myproject-server/lib/python2.7/site-packages/django/db/models/loading.py", line 90, in _populate
self.load_app(app_name)
File "/Users/Silver/.virtualenvs/myproject-server/lib/python2.7/site-packages/django/db/models/loading.py", line 111, in load_app
models = import_module('%s.models' % app_name)
File "/Users/Silver/.virtualenvs/myproject-server/lib/python2.7/site-packages/django/utils/importlib.py", line 40, in import_module
__import__(name)
File "/Users/Silver/Projects/myproject-server/myproject/apps/billing/models.py", line 14, in <module>
from .managers import TransactionManager, AggregationManager
File "/Users/Silver/Projects/myproject-server/myproject/apps/billing/managers.py", line 12, in <module>
from accounts.models import User
ImportError: cannot import name User
raised in _populate |
|
So you got your error now. Investigate why you can't import User there. |
|
And check your import sys
print sys.path |
|
@Suor thats kinda strange. It always complains about importing custom user model (accounts.User in my case). If I move import statment in a problematic location into a function, it just starts complaining about Interesting why there is no problems when:
My assumption:
|
|
OK. Solved. For other people that might find this issue – read this article carefully: http://www.caktusgroup.com/blog/2013/08/07/migrating-custom-user-model-django/
|
Hello,
I'm experiencing a serious problems with m2m fields when cacheops is enabled. I belive the problem is somewhere in threading support because it does not appear with development server (and as a result tests passes but features gets broken on production).
See the code:
model (simplified):
settings.py:
Here is how problem appears to the end user: User fills the website creation form by providing url, description and selecting few categories and hits "SAVE" button.
Websiteis now in cache and in db.User tries to update created object and there is a failure – he can update everything but not a
categoriesm2m field.So I've tried to do a couple of experiments.
I run
./manage.py shell_plusto get into console, loaded aWebsiteobject there and changed categories. Refreshed page on a website – categories were updated. NEXT PART IS STRANGE. Then I've immediately tried to update categories using website –categoriesfield were updated on db lvl, but reading still returns old m2m set on a web interface (console returns fresh results). Subsequent attempts to update object using web interface does not hit db any more. At this point I've tried to invalite_obj, invalite_model from console, but web interface still output old categories set. I belive this has nothing to do with a browser cache, nor with UI.I'm mostly positive it has something to do with threading support and has nothing do to with UI (In my case it's DRF API, so I've tried to update object using same serializer inside a console and it works fine, same as if I update an object directly without serializer layer).
Once I set
CACHEOPS_FAKEtoTrueall problems with m2m gone away. So it's for sure something related to cacheops.@Suor can you please give any directions how may I find out the source of this problem? Thank you very much for your work!
The text was updated successfully, but these errors were encountered: