DjangoCache Out of Disk Space Scenario

[silentjay]

I had to do an emergency delete of the disk cache /var/tmp/django_disk_cache as my server had run out of disk space.

Ever since I receive Django errors

Exception Type: ValueError at x Exception Value: Key ':1xxx' not found

Disabling DjangoCache is the ony current fix. I've checked the Django db and there's no corresponding table for DjangoCache. There's obviously some reference to these keys somewhere but I can't find them. The docs make mention to a sqlite db, but I've searched my installation and can't find it.

The help for DjangoCache also mention a clear command, which I assume I'm meant to run in a python shell, but I can't figure out how to run it and there's no examples.

[grantjenks]

After you "rm -r /var/tmp/django_disk_cache" just restart your web server. The directory and supporting SQLite database will be created for you.

[grantjenks]

Here's the relevant docs:

https://docs.djangoproject.com/en/1.11/topics/cache/#basic-usage

http://www.grantjenks.com/docs/diskcache/api.html#djangocache

To use the API, open a Django shell with "python manage.py shell" then import the cache as you do in your views and call "cache.clear()"

[grantjenks]

Is there a diskcache bug that caused the out-of-space scenario?

[silentjay]

Wow that was quick! No it wasn't a diskcache bug that caused the out-of-space scenario just my stupid fault for setting the cache too large.

I've restarted the and deleted /var/tmp/django_disk_cache multiple times and seen it's been recreated but the error remains, the only way I've been able to fix it is by changing the django cache type.

I'll try cache.clear() thanks, also where is the sqlite db located as I'd like to try deleting that as well

[grantjenks]

The SQLite database is in that directory. I think it's "/var/tmp/django_disk_cache/cache.db"

[grantjenks]

Actually it's a little more complicated. There's 8 or so subdirectories and it's in one of those.

[grantjenks]

Do you have a situation where you assume the cache is warmed? Meaning, are there keys you assume will always be there? It's hard to tell without more stack trace in your exception.

[silentjay]

Still no joy, have deleted /var/tmp/django_disk_cache/, run 'from django.core.cache import cache -> clear.cache()' in a shell, restarted nginx and gunicorn.

I'm sure not what you mean by assuming the keys are always there, I'm using it in conjunction with fancy-cache https://github.com/peterbe/django-fancy-cache, if there's a miss on the keys then I believe it executes the code as normal then caches it. Been working well for a week or two until I deleted that directory.

Here's the stack trace:

Internal Server Error: /x

ValueError at /x
Key ':1:87449ae7b6412baad697135c2326efe3' not found

Traceback:

File "/home/x/venv/local/lib/python2.7/site-packages/django/core/handlers/exception.py" in inner
  41.             response = get_response(request)

File "/home/x/venv/local/lib/python2.7/site-packages/django/core/handlers/base.py" in _get_response
  187.                 response = self.process_exception_by_middleware(e, request)

File "/home/x/venv/local/lib/python2.7/site-packages/django/core/handlers/base.py" in _get_response
  185.                 response = wrapped_callback(request, *callback_args, **callback_kwargs)

File "/home/x/venv/local/lib/python2.7/site-packages/django/utils/decorators.py" in _wrapped_view
  141.                     result = middleware.process_request(request)

File "/home/x/venv/local/lib/python2.7/site-packages/fancy_cache/middleware.py" in process_request
  176.             self.cache.incr(cache_key)

File "/home/x/venv/local/lib/python2.7/site-packages/diskcache/djangocache.py" in incr
  203.             raise ValueError("Key '%s' not found" % key)

Exception Type: ValueError at x
Exception Value: Key ':1:87449ae7b6412baad697135c2326efe3' not found

[grantjenks]

That behavior looks expected if the key is not in the cache. See the Django source for reference: https://github.com/django/django/blob/1.11.8/django/core/cache/backends/base.py#L177

So the question then is why does fancy-cache assume the key is in the cache?

[silentjay]

Well I assumed the keys were in this sqlite db I can't find which still has the old keys from before I deleted /var/tmp/django_disk_cache, but this isn't the case?

/var/tmp/django_disk_cache has been recreated but there's no sqlite db there, so I definitely must be using the original

[grantjenks]

Assuming your caches setting looks something like:

CACHES = {
    'default': {
        'BACKEND': 'diskcache.DjangoCache',
        'LOCATION': '/var/tmp/django_disk_cache/',
        'OPTIONS': {
            'size_limit': int(10e9),
        },
    },
}

You can also test the caching layer with:

$ python manage.py shell
Python 2.7.14 (default, Sep 22 2017, 00:05:22) 
>>> from django.core.cache import caches
>>> cache = caches['default']
>>> cache.set(b'test', 0)
True
>>> cache.get(b'test')
0
>>> exit()
$ python manage.py shell
Python 2.7.14 (default, Sep 22 2017, 00:05:22) 
>>> from django.core.cache import caches
>>> cache = caches['default']
>>> cache.get(b'test')
0

Does that work?

[silentjay]

Yes that works, except instead of '0' nothing is returned, which I assume is the same

[grantjenks]

Not the same actually. The default return value is None which is not displayed by the Python interpreter. Sounds like your cache is not getting created. Usually that's a file system permissions issue.

[silentjay]

Ah you're right, it had:

drwxr-xr-x silentjay www-data

changed it to:

drwxr-xr-x silentjay silentjay

and it's now working! either that or it's because I got mad and ran the clear cache command about 20 times in a row. I also changed size_limit back from 2 ** 16 to 2 ** 32

[grantjenks]

Glad it's working now!

I wonder why it didn't error earlier? I tried to support this scenario with an informative exception:

In [1]: import diskcache as dc

In [2]: cache = dc.FanoutCache('/tmp/test/cache')
---------------------------------------------------------------------------
EnvironmentError                          Traceback (most recent call last)
<ipython-input-2-b3b6bb09a133> in <module>()
----> 1 cache = dc.FanoutCache('/tmp/test/cache')

/home/grantj/env27/local/lib/python2.7/site-packages/diskcache/fanout.pyc in __init__(self, directory, shards, timeout, disk, **settings)
     36                 **settings
     37             )
---> 38             for num in range(shards)
     39         )
     40         self._hash = self._shards[0].disk.hash

/home/grantj/env27/local/lib/python2.7/site-packages/diskcache/fanout.pyc in <genexpr>((num,))
     36                 **settings
     37             )
---> 38             for num in range(shards)
     39         )
     40         self._hash = self._shards[0].disk.hash

/home/grantj/env27/local/lib/python2.7/site-packages/diskcache/core.pyc in __init__(self, directory, timeout, disk, **settings)
    371                         error.errno,
    372                         'Cache directory "%s" does not exist'
--> 373                         ' and could not be created' % self._directory
    374                     )
    375 

EnvironmentError: [Errno 13] Cache directory "/tmp/test/cache/000" does not exist and could not be created

In [3]: !ls -al /tmp/test
total 8
drwxr-xr-x  2 www-data www-data 4096 Dec 13 17:04 .
drwxrwxrwt 10 root     root     4096 Dec 13 17:07 ..

Also, if I setup a dummy Django project, I get that error again:

$ python manage.py shell
Python 2.7.12 (default, Jul  1 2016, 15:12:24) 
Type "copyright", "credits" or "license" for more information.

IPython 5.5.0 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: from django.core.cache import caches

In [2]: cache = caches['default']
---------------------------------------------------------------------------
EnvironmentError                          Traceback (most recent call last)
<ipython-input-2-c8c8d8058ea1> in <module>()
----> 1 cache = caches['default']

/home/grantj/env27/local/lib/python2.7/site-packages/django/core/cache/__init__.pyc in __getitem__(self, alias)
     78             )
     79 
---> 80         cache = _create_cache(alias)
     81         self._caches.caches[alias] = cache
     82         return cache

/home/grantj/env27/local/lib/python2.7/site-packages/django/core/cache/__init__.pyc in _create_cache(backend, **kwargs)
     53         raise InvalidCacheBackendError(
     54             "Could not find backend '%s': %s" % (backend, e))
---> 55     return backend_cls(location, params)
     56 
     57 

/home/grantj/env27/local/lib/python2.7/site-packages/diskcache/djangocache.pyc in __init__(self, directory, params)
     26         options = params.get('OPTIONS', {})
     27         self._directory = directory
---> 28         self._cache = FanoutCache(directory, shards, timeout, **options)
     29         self.memoize = self._cache.memoize
     30 

/home/grantj/env27/local/lib/python2.7/site-packages/diskcache/fanout.pyc in __init__(self, directory, shards, timeout, disk, **settings)
     36                 **settings
     37             )
---> 38             for num in range(shards)
     39         )
     40         self._hash = self._shards[0].disk.hash

/home/grantj/env27/local/lib/python2.7/site-packages/diskcache/fanout.pyc in <genexpr>((num,))
     36                 **settings
     37             )
---> 38             for num in range(shards)
     39         )
     40         self._hash = self._shards[0].disk.hash

/home/grantj/env27/local/lib/python2.7/site-packages/diskcache/core.pyc in __init__(self, directory, timeout, disk, **settings)
    371                         error.errno,
    372                         'Cache directory "%s" does not exist'
--> 373                         ' and could not be created' % self._directory
    374                     )
    375 

EnvironmentError: [Errno 13] Cache directory "/tmp/test/cache/000" does not exist and could not be created

[grantjenks]

Oh wait! I just noticed your comment about the size limit. "2 ** 16" is 65,536 which is really much too low to be practical. Your cache was likely evicting keys to meet the size limit just after they were written. It's happened to me before too.

[silentjay]

That would be it then. Might be worth a warning in the docs. Thanks again for the help.