Compression for cached values

[mkhorton]

Is it possible to compress values before storing them in the cache (and decompressing them on retrieval)? If not, would there be an appropriate place to add a hook for this?

I'm trying Flask-Caching with a redis deployment, and it looks like a significant amount of memory could be saved by compressing the cached values, and uncompressing them when retrieved from the cache.

Performing the compression in the body of the function is not appropriate, since other parts of the web app would require un-compressed values to be returned by the relevant functions.

Thanks for a great library!

[sh4nks]

You could implement a wrapper around the cache object that handles that for you and then use the wrapped cache object to retrieve and set the values.

[mkhorton]

Thanks for the response. You mean wrap @cache and @memoize using functools, to create new decorators? That sounds a lot easier than I was expecting -- is there anything special about Flask-Caching to know about before implementing?

[sh4nks]

I haven't thought about wrapping the decorators but I guess that would work as well.

I thought about wrapping the cache.set, etc functions (https://github.com/sh4nks/flask-caching/blob/master/flask_caching/__init__.py#L246-L282). Another possibility would be to create a new redis backend and implement it directly in the backend.

[filak]

"Is it possible to compress values before storing them in the cache (and decompressing them on retrieval)? If not, would there be an appropriate place to add a hook for this?"

I would like to have that option for the FileSystemCache backend. I need to cache big dictionaries with a lot of text returned by Elasticsearch backend, sometimes the cached files have few MB size. I thought something simple like replacing open(filename... with gzip.open(filename... in the FS cache class might do the trick but with no success so far...

I use both memoize decorator and not so often set/get methods.

Would you be so kind and provide some hints how to achieve that ?

Or maybe this is not going to help to do faster responses at all...

[sh4nks]

You could create a new caching backend (i.e. based on the FileSystemCache) which implements this feature. The decorators are using the set, get, etc methods from backend and as such they will inherit the functionality as well.

[filak]

OK, I have created the backends (simple mods of filesystemcache.py):

• https://github.com/filak/flask-caching/blob/master/flask_caching/contrib/filesystemcachegz.py
• https://github.com/filak/flask-caching/blob/master/flask_caching/contrib/filesystemcachegzjson.py
• https://github.com/filak/flask-caching/blob/master/flask_caching/contrib/filesystemcachejson.py

Plus the Windows deleting bug #238 should be fixed as well.