Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Add caching to CompilerFilter to not re-compile unchanged files #346

Open
wants to merge 4 commits into from

5 participants

@pindia

Currently django_compressor seems to re-run precompilers on each compression, even if files are completely unchanged. I haven't been able to stop this through any combination of existing caching options. In my large project, this has gotten to the point where it takes 10-15 seconds to load every page.

This patch adds caching directly to the CompilerFilter class. Before running the compiler, it computes a hash of the input contents and checks the compressor cache for a cached output. If not found, the compiler runs as normal and the output is saved to the cache for next time. Thus, only files that have actually changed are recompiled. When used in combination with a file-based caching backend to persist across server restarts, this greatly speeds up development.

@diox
Owner

Hi,

The main problem with that approach, which has stopped us from implementing this before, is that compressor doesn't know if the files being precompiled depend on some others it's not tracking. For instance, if you are trying to precompile a.css, and inside this file there is a statement causing the precompiler to import b.css. If only b.css is changed, with your patch compressor won't know and will serve an out-of-date, cached version.

For this reason, I've always advocated for caching in the precompilers themselves. Sadly, not all of them implement it :( If we can find a satisfactory solution for this problem, I'll be happy to merge this.

This problem aside, there are a couple issues in your code (easy to fix though) :

  • It would be better to have caching happen in a separate method to allow classes inheriting from CompilerFilter to avoid it (or customize it) if they want to. Look at the templatetag implementation for some inspiration on this.
  • The cache key function should live in cache.py like the rest of cache related stuff
@pindia

Thanks for the feedback. To address the import issue, I've added a new setting COMPRESS_CACHEABLE_PRECOMPILERS, empty by default, that defines the mimetypes that are allowed to be cached. This allows the caching to be enabled only for compilers like CoffeeScript that don't have an import mechanism.

I moved the caching code into a new CachedCompilerFilter class, so classes can inherit from CompilerFilter to avoid the caching behavior or inherit from CachedCompilerFilter and override get_cache_key to customize it.

@pindia

Hmm, it says this build failed, but I can't find any actual error in the logs: https://travis-ci.org/jezdez/django_compressor/jobs/3672604

@diox
Owner

Sorry for not getting back at you sooner on this - It looks OK to me, but this is not my main area of expertise in compressor, so I'd prefer if @jezdez would look at it and merge it and not myself.

(The build thing seems to be just a travis problem, but I'm not 100% sure)

@rasca

Hi,

I've run the test suit in Py2.7 and Django 1.4 (the travis failure) and it runs successfully.

I'm also using this branch in production now with a SCSS precompiler (which supports imports) so I created the following: https://gist.github.com/4665106 so that it recompiles the .scss on every deploy (actually on every restart).

@ejo

If you're running django-compressor >= 1.3, it's possible to do this without altering django_compressor in any way. We wanted to follow pindia's example but avoid forking django-compressor. Here's how we got this running in my shop. (side note: we did not address the issue of imports in the compiled files, since we don't use any, but it would be easy enough to address that as pindia did). Many thanks to pindia for motivation and examples.

Just to be clear on the goals of this, for those who may be searching for matching issues/discussions, we wanted in development to precompile but not compress, and recent versions of django_compressor made that possible... but without precompile caching our performance was terribly slow. pindia's patches or this abstracted-out version take care of that problem.

In settings_dev.py:

COMPRESS_PRECOMPILERS = [
    ('text/coffeescript', 'our_project.utils.precompilers.CachedCompilerFilter'),
    ('text/less', 'our_project.utils.precompilers.CachedCompilerFilter'), 
]
COMPRESS_PRECOMPILER_COMMANDS = {
    'text/coffeescript': 'coffee --compile --stdio',
    'text/less': 'lessc {infile} {outfile}',    
}

Then in a new file, our_project.utils.precompilers.py:

import hashlib
from compressor.filters.base import CompilerFilter
from django.conf import settings
from django.core.cache import get_cache

cache = get_cache(settings.COMPRESS_CACHE_BACKEND)

class CachedCompilerFilter(CompilerFilter):

    commands = settings.COMPRESS_PRECOMPILER_COMMANDS

    def __init__(self, content, attrs, command=None, *args, **kwargs):
        command = self.commands.get(attrs.get('type'))
        super(CachedCompilerFilter, self).__init__(content, command=command, *args, **kwargs)

    def input(self, **kwargs):
        content_hash = hashlib.sha1(self.content.encode('utf8')).hexdigest()
        data = cache.get(content_hash)
        if data:
            return data
        data = super(CachedCompilerFilter, self).input(**kwargs)
        cache.set(content_hash, data, settings.COMPRESS_REBUILD_TIMEOUT)
        return data
@staab

:+1: on that solution, ejo. Does exactly what I want.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Dec 14, 2012
  1. @pindia
  2. @pindia
  3. @pindia

    Only cache precompilers specified in COMPRESS_CACHEABLE_PRECOMPILERS …

    pindia authored
    …setting, and refactor caching logic into new CachedCompilerFilter class
Commits on Dec 15, 2012
  1. @pindia
This page is out of date. Refresh to see the latest.
View
6 compressor/base.py
@@ -15,7 +15,7 @@
from compressor.conf import settings
from compressor.exceptions import (CompressorError, UncompressableFileError,
FilterDoesNotExist)
-from compressor.filters import CompilerFilter
+from compressor.filters import CachedCompilerFilter
from compressor.storage import default_storage, compressor_file_storage
from compressor.signals import post_compress
from compressor.utils import get_class, get_mod_func, staticfiles
@@ -212,8 +212,8 @@ def precompile(self, content, kind=None, elem=None, filename=None, **kwargs):
try:
mod = import_module(mod_name)
except ImportError:
- return True, CompilerFilter(content, filter_type=self.type,
- command=filter_or_command, filename=filename).input(
+ return True, CachedCompilerFilter(content=content, filter_type=self.type,
+ command=filter_or_command, filename=filename, mimetype=mimetype).input(
**kwargs)
try:
precompiler_class = getattr(mod, cls_name)
View
4 compressor/cache.py
@@ -126,6 +126,10 @@ def get_hashed_content(filename, length=12):
return get_hexdigest(content, length)
+def get_precompiler_cachekey(command, contents):
+ return hashlib.sha1('precompiler.%s.%s' % (command, contents)).hexdigest()
+
+
def cache_get(key):
packed_val = cache.get(key)
if packed_val is None:
View
1  compressor/conf.py
@@ -34,6 +34,7 @@ class CompressorConf(AppConf):
# ('text/stylus', 'stylus < {infile} > {outfile}'),
# ('text/x-scss', 'sass --scss {infile} {outfile}'),
)
+ CACHEABLE_PRECOMPILERS = ()
CLOSURE_COMPILER_BINARY = 'java -jar compiler.jar'
CLOSURE_COMPILER_ARGUMENTS = ''
CSSTIDY_BINARY = 'csstidy'
View
2  compressor/filters/__init__.py
@@ -1,3 +1,3 @@
# flake8: noqa
from compressor.filters.base import (FilterBase, CallbackOutputFilter,
- CompilerFilter, FilterError)
+ CompilerFilter, CachedCompilerFilter, FilterError)
View
23 compressor/filters/base.py
@@ -7,6 +7,7 @@
from django.core.files.temp import NamedTemporaryFile
from django.utils.importlib import import_module
+from compressor.cache import cache, get_precompiler_cachekey
from compressor.conf import settings
from compressor.exceptions import FilterError
from compressor.utils import get_mod_func
@@ -143,3 +144,25 @@ def input(self, **kwargs):
filtered = self.outfile.read()
self.outfile.close()
return filtered
+
+
+class CachedCompilerFilter(CompilerFilter):
+
+ def __init__(self, mimetype, **kwargs):
+ self.mimetype = mimetype
+ super(CachedCompilerFilter, self).__init__(**kwargs)
+
+ def input(self, **kwargs):
+ if self.mimetype in settings.COMPRESS_CACHEABLE_PRECOMPILERS:
+ key = self.get_cache_key()
+ data = cache.get(key)
+ if data:
+ return data
+ filtered = super(CachedCompilerFilter, self).input(**kwargs)
+ cache.set(key, filtered, settings.COMPRESS_REBUILD_TIMEOUT)
+ return filtered
+ else:
+ return super(CachedCompilerFilter, self).input(**kwargs)
+
+ def get_cache_key(self):
+ return get_precompiler_cachekey(self.command, self.content.encode('utf8'))
View
30 compressor/tests/test_filters.py
@@ -9,7 +9,7 @@
from compressor.conf import settings
from compressor.css import CssCompressor
from compressor.utils import find_command
-from compressor.filters.base import CompilerFilter
+from compressor.filters.base import CompilerFilter, CachedCompilerFilter
from compressor.filters.cssmin import CSSMinFilter
from compressor.filters.css_default import CssAbsoluteFilter
from compressor.filters.template import TemplateFilter
@@ -41,6 +41,7 @@ def setUp(self):
with open(self.filename) as f:
self.content = f.read()
self.test_precompiler = os.path.join(test_dir, 'precompiler.py')
+ settings.COMPRESS_CACHEABLE_PRECOMPILERS = ('text/css',)
def test_precompiler_infile_outfile(self):
command = '%s %s -f {infile} -o {outfile}' % (sys.executable, self.test_precompiler)
@@ -67,6 +68,33 @@ def test_precompiler_stdin_stdout_filename(self):
compiler = CompilerFilter(content=self.content, filename=self.filename, command=command)
self.assertEqual(u"body { color:#990; }%s" % os.linesep, compiler.input())
+ def test_precompiler_cache(self):
+ command = '%s %s -f {infile} -o {outfile}' % (sys.executable, self.test_precompiler)
+ compiler = CachedCompilerFilter(content=self.content, filename=self.filename, command=command, mimetype='text/css')
+ self.assertEqual(u"body { color:#990; }", compiler.input())
+ # We tell whether the precompiler actually ran by inspecting compiler.infile. If not None, the compiler had to
+ # write the input out to the file for the external command. If None, it was in the cache and thus skipped.
+ self.assertIsNotNone(compiler.infile) # Not cached
+
+ compiler = CachedCompilerFilter(content=self.content, filename=self.filename, command=command, mimetype='text/css')
+ self.assertEqual(u"body { color:#990; }", compiler.input())
+ self.assertIsNone(compiler.infile) # Cached
+
+ self.content += ' ' # Invalidate cache by slightly changing content
+ compiler = CachedCompilerFilter(content=self.content, filename=self.filename, command=command, mimetype='text/css')
+ self.assertEqual(u"body { color:#990; }", compiler.input())
+ self.assertIsNotNone(compiler.infile) # Not cached
+
+ def test_precompiler_not_cacheable(self):
+ command = '%s %s -f {infile} -o {outfile}' % (sys.executable, self.test_precompiler)
+ compiler = CachedCompilerFilter(content=self.content, filename=self.filename, command=command, mimetype='text/different')
+ self.assertEqual(u"body { color:#990; }", compiler.input())
+ self.assertIsNotNone(compiler.infile) # Not cached
+
+ compiler = CachedCompilerFilter(content=self.content, filename=self.filename, command=command, mimetype='text/different')
+ self.assertEqual(u"body { color:#990; }", compiler.input())
+ self.assertIsNotNone(compiler.infile) # Not cached
+
class CssMinTestCase(TestCase):
def test_cssmin_filter(self):
View
13 docs/settings.txt
@@ -371,6 +371,19 @@ Caching settings
:attr:`~django.conf.settings.COMPRESS_REBUILD_TIMEOUT` and
:attr:`~django.conf.settings.COMPRESS_MINT_DELAY`.
+.. attribute:: COMPRESS_CACHEABLE_PRECOMPILERS
+
+ :Default: ``()``
+
+ An iterable of precompiler mimetypes as defined in :attr:`~django.conf.settings.COMPRESS_PRECOMPILERS`
+ for which the compiler output can be cached based solely on the contents
+ of the input file. This lets Django Compressor avoid recompiling unchanged
+ files. Caching is appropriate for compilers such as CoffeeScript where files
+ are compiled one-to-one, but not for compilers such as SASS that have an
+ ``import`` mechanism for including one file from another. If caching is enabled
+ for such a compiler, Django Compressor will not know to recompile files when a file
+ they import is modified.
+
.. attribute:: COMPRESS_DEBUG_TOGGLE
:Default: None
Something went wrong with that request. Please try again.