3 queries per thumbnail #212

Closed
nebstrebor opened this Issue Mar 20, 2013 · 2 comments

3 participants

@nebstrebor

I have a monthly cafeteria menu with 300 foods, each with an image & single thumbnail. Images & thumbnails are stored on s3.

I started profiling using NewRelic and django-debug-toolbar because things are slow on my site and the time for optimization is no longer premature. New Relic informed me that for each menu load, there are 300 queries to the easy_thumbnails_thumbnail table and 600 queries the easy_thumbnails_source table. So, three per thumbnail.

The way I'm getting the thumbnail is:

from easy_thumbnails.templatetags.thumbnail import thumbnail_url

class Food(models.Model):
     image = ThumbnailerImageField(upload_to=food_image_upload_to, blank=True,         
        null=True, resize_source=dict(size=(640,0)), storage=S3BotoStorage_AllPublic(), 
        thumbnail_storage=FoodImagesS3Storage_AllPublic())

    @property
    def hoverpic_url(self):
        return thumbnail_url(self.image, 'hoverpic')

(I have the pre-generation signal turned on as well, and that's working fine).

Here's the ddt trace for the three queries:

# 1
/Users/ben/code/nutripublisher/publisher/menus/models.py in hoverpic_url(609)
*return thumbnail_url(self.image, 'hoverpic')*
/Users/ben/Envs/nutripublisher/src/easy-thumbnails/easy_thumbnails/templatetags/thumbnail.py in thumbnail_url(259)
*thumb = get_thumbnailer(source)[alias]*
/Users/ben/Envs/nutripublisher/src/easy-thumbnails/easy_thumbnails/files.py in __getitem__(280)
*return self.get_thumbnail(options)*
/Users/ben/Envs/nutripublisher/src/easy-thumbnails/easy_thumbnails/files.py in get_thumbnail(391)
*if self.thumbnail_exists(filename):*
/Users/ben/Envs/nutripublisher/src/easy-thumbnails/easy_thumbnails/files.py in thumbnail_exists(434)
*source = self.get_source_cache()*
/Users/ben/Envs/nutripublisher/src/easy-thumbnails/easy_thumbnails/files.py in get_source_cache(450)
*check_cache_miss=self.thumbnail_check_cache_miss)*
/Users/ben/Envs/nutripublisher/src/easy-thumbnails/easy_thumbnails/models.py in get_file(21)
*object = self.get(**kwargs)*

# 2
/Users/ben/code/nutripublisher/publisher/menus/models.py in hoverpic_url(609)
return thumbnail_url(self.image, 'hoverpic')
/Users/ben/Envs/nutripublisher/src/easy-thumbnails/easy_thumbnails/templatetags/thumbnail.py in thumbnail_url(259)
thumb = get_thumbnailer(source)[alias]
/Users/ben/Envs/nutripublisher/src/easy-thumbnails/easy_thumbnails/files.py in __getitem__(280)
return self.get_thumbnail(options)
/Users/ben/Envs/nutripublisher/src/easy-thumbnails/easy_thumbnails/files.py in get_thumbnail(391)
if self.thumbnail_exists(filename):
/Users/ben/Envs/nutripublisher/src/easy-thumbnails/easy_thumbnails/files.py in thumbnail_exists(437)
thumbnail = self.get_thumbnail_cache(thumbnail_name)
/Users/ben/Envs/nutripublisher/src/easy-thumbnails/easy_thumbnails/files.py in get_thumbnail_cache(459)
source = self.get_source_cache(create=True)
/Users/ben/Envs/nutripublisher/src/easy-thumbnails/easy_thumbnails/files.py in get_source_cache(450)
check_cache_miss=self.thumbnail_check_cache_miss)
/Users/ben/Envs/nutripublisher/src/easy-thumbnails/easy_thumbnails/models.py in get_file(16)
object, created = self.get_or_create(**kwargs)

# 3
/Users/ben/code/nutripublisher/publisher/menus/models.py in hoverpic_url(609)
return thumbnail_url(self.image, 'hoverpic')
/Users/ben/Envs/nutripublisher/src/easy-thumbnails/easy_thumbnails/templatetags/thumbnail.py in thumbnail_url(259)
thumb = get_thumbnailer(source)[alias]
/Users/ben/Envs/nutripublisher/src/easy-thumbnails/easy_thumbnails/files.py in __getitem__(280)
return self.get_thumbnail(options)
/Users/ben/Envs/nutripublisher/src/easy-thumbnails/easy_thumbnails/files.py in get_thumbnail(391)
if self.thumbnail_exists(filename):
/Users/ben/Envs/nutripublisher/src/easy-thumbnails/easy_thumbnails/files.py in thumbnail_exists(437)
thumbnail = self.get_thumbnail_cache(thumbnail_name)
/Users/ben/Envs/nutripublisher/src/easy-thumbnails/easy_thumbnails/files.py in get_thumbnail_cache(463)
check_cache_miss=self.thumbnail_check_cache_miss)
/Users/ben/Envs/nutripublisher/src/easy-thumbnails/easy_thumbnails/models.py in get_file(21)
object = self.get(**kwargs)

I'm not sure if this is an inefficiency that can be fixed, or if I'm doing something wrong, or if that's just the way it works. Obviously its probably not a big deal for a few thumbnails on a page, but with 300 thumbnails, it can take as much as 4 seconds, just for the queries.

I'd be happy to help fix this issue if someone points me toward a workable solution.

In the mean time, I'm looking at adding another charfield or imagefield to my model and storing the thumbnail url on the model so that i can get rid of any O(n) queries altogether. If anyone has suggestions on how to acheive this, feel free to suggest. :)

@nebstrebor nebstrebor closed this Mar 20, 2013
@nebstrebor nebstrebor reopened this Mar 20, 2013
@epicserve

I'm having this same experience. @SmileyChris, any thoughts? Could redis be used as a data store?

@SmileyChris
Owner

I'm going to mark this as as a dupe of #245 too.

Epicserve's suggestion should be a separate issue (and one I'm not adverse to).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment