Sporadic IntegrityError when calling get_thumbnail

[awidgery]

sorl-thumbnail works great, using S3Boto backend storage.

However, I occasionally get an IntegrityError (see traceback below) that makes a response fail. Any ideas?

Traceback:
File "/usr/local/lib/python2.7/dist-packages/sorl/thumbnail/shortcuts.py", line 8, in get_thumbnail
return default.backend.get_thumbnail(file_, geometry_string, **options)

File "/usr/local/lib/python2.7/dist-packages/sorl/thumbnail/base.py", line 66, in get_thumbnail
default.kvstore.set(thumbnail, source)

File "/usr/local/lib/python2.7/dist-packages/sorl/thumbnail/kvstores/base.py", line 33, in set
self._set(image_file.key, image_file)

File "/usr/local/lib/python2.7/dist-packages/sorl/thumbnail/kvstores/base.py", line 138, in _set
self._set_raw(add_prefix(key, identity), s)

File "/usr/local/lib/python2.7/dist-packages/sorl/thumbnail/kvstores/cached_db_kvstore.py", line 36, in _set_raw
kv = KVStoreModel.objects.get_or_create(key=key)[0]

File "/usr/local/lib/python2.7/dist-packages/django/db/models/manager.py", line 146, in get_or_create
return self.get_query_set().get_or_create(**kwargs)

File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 487, in get_or_create
six.reraise(*exc_info)

File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 477, in get_or_create
obj.save(force_insert=True, using=self.db)

File "/usr/local/lib/python2.7/dist-packages/django/db/models/base.py", line 546, in save
force_update=force_update, update_fields=update_fields)

File "/usr/local/lib/python2.7/dist-packages/django/db/models/base.py", line 650, in save_base
result = manager._insert([self], fields=fields, return_id=update_pk, using=using, raw=raw)

File "/usr/local/lib/python2.7/dist-packages/django/db/models/manager.py", line 215, in _insert
return insert_query(self.model, objs, fields, **kwargs)

File "/usr/local/lib/python2.7/dist-packages/django/db/models/query.py", line 1661, in insert_query
return query.get_compiler(using=using).execute_sql(return_id)

File "/usr/local/lib/python2.7/dist-packages/django/db/models/sql/compiler.py", line 937, in execute_sql
cursor.execute(sql, params)

File "/usr/local/lib/python2.7/dist-packages/django/db/backends/mysql/base.py", line 122, in execute
six.reraise(utils.IntegrityError, utils.IntegrityError(*tuple(e.args)), sys.exc_info()[2])

File "/usr/local/lib/python2.7/dist-packages/django/db/backends/mysql/base.py", line 120, in execute
return self.cursor.execute(query, args)

File "/usr/local/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 202, in execute
self.errorhandler(self, exc, value)

File "/usr/local/lib/python2.7/dist-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
raise errorclass, errorvalue

IntegrityError: (1062, "Duplicate entry 'sorl-thumbnail||image||ea147948138d781a2ef2c0caa2fcbd74' for key 'PRIMARY'")

[mnstefan]

I have the same issue.

[awidgery]

Any movement on this? It seems to occur when attempting to access or create a thumbnail very soon after creating one.

[dellis23]

I'm getting the same issue (also seen here and here). I'm using MySQL as a storage backend.

The Air Mozilla team's solution was to create a wrapper over the thumbnail method that would simply wait and try again. I will probably do the same for my project, since there are a number of outstanding pull requests and I'm not sure the work on the core would be worth it.

[dellis23]

I think this is ultimately a concurrency issue with Django's get_or_create method. I think it manifests in sorl thumbnail moreso than other applications by virtue of the frequency of access of thumbnails.

Here are a few discussions on the issue:

http://stackoverflow.com/questions/2235318/how-do-i-deal-with-this-race-condition-in-django
http://python.6.x6.nabble.com/get-or-create-can-still-cause-IntegrityError-td455416.html
https://code.djangoproject.com/ticket/13906

[dellis23]

Alrighty then. Here's a subclass I'm using now that seems to work well:

from django.core.cache import cache
from django.db import transaction, IntegrityError

from sorl.thumbnail.conf import settings
from sorl.thumbnail.kvstores.cached_db_kvstore import KVStore
from sorl.thumbnail.models import KVStore as KVStoreModel


class ConcurrentKVStore(KVStore):

    @transaction.commit_on_success
    def _set_raw(self, key, value):
        try:
            kv = KVStoreModel.objects.create(key=key)
        except IntegrityError:
            transaction.commit()
            kv = KVStoreModel.objects.get(key=key)
        kv.value = value
        kv.save()
        cache.set(key, value, settings.THUMBNAIL_CACHE_TIMEOUT)

This can be chosen using the THUMBNAIL_KVSTORE setting for sorl. It seems to work well: my integrity errors have gone away. I will report back if anything changes.

[awidgery]

I was just writing a message - could you elaborate where you've put this subclass? Have you modified the existing sorl-thumbnail code or tacked this on?

[dellis23]

In my project, I put it in sorl_kvstore.py in my application's base directory, and added THUMBNAIL_KVSTORE = 'myapp.sorl_kvstore.ConcurrentKVStore' to my settings file.

[awidgery]

Amazing! Looks like you've done it! I'll keep an eye on server logs as well and post back.
Thanks for your help :)

[dellis23]

Great, glad to help :-)

On Tue, Sep 24, 2013 at 1:08 PM, awidgery notifications@github.com wrote:

Amazing! Looks like you've done it! I'll keep an eye on server logs as
well and post back.
Thanks for your help :)

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/162#issuecomment-25023504
.

[dkrnl]

yes, have the same issue.

[mariocesar]

@codeinthehole I see that your fix is working well, I'm was about to ask you if you can test it , but I know that is difficult to test race conditions, so If you can prepare a pull request I will merge your code as it's, or I will later add a fix with a mention to your fix.

[mariocesar]

@codeinthehole I just found your pull request #167, is merged