Redisign the cahce API

[hexagonrecursion]

The current cache API is:

class Cache:
    def get(self, key):
        pass

    def set(self, key, value, expires=None):
        pass

    def delete(self, key):
        pass

    def close(self):
        pass

This makes fixing #145, #180 and #238 difficult because you have to store the entire response body somewhere before you can save it with Cache.set().

Currently CacheControl buffers the response body in memory which is just wrong - thou shalt not assume that there are no files large enough to not fit in RAM. This is the cause of #145 and #180. Additionally this makes CacheControl completely unusable with response bodies larder than (2^32)-1 bytes no matter how much RAM you have due to a design limitation of the msgpack format #238.

I have considered a less invasive approach of buffering the response body in a temporary file, but there are at least two problems with this:

1. On Linux /tmp is often a tmpfs - a filesystem backed by virtual memory. It can swap out, but swap is not always enabled. tmpfs has a configurable maximum size limit. swap also often has a size limit. Basically: this would defeat the point - we would not be able to reliably cache a 100GB response even if there is enough storage for the cache.
2. Unoptimal IO: to cache a 10GB response we would write 10GB, read 10GB and write 10GB again instead of writing 10GB once.

[hexagonrecursion]

Fixing #238 would also require a new serialization format because cc=4 stores both the body and the header in a single msgpack object.
Edit: fixing #145 and #180 will also require a new serialization format.

[ionrock]

Hey! Thanks for looking at this. I know this limitation has been considered before, but the discrepancies between different operating systems and potential use cases made it challenging to find a valid fix. The result was we did the simplest thing! One idea I had when looking through your commentary was that you might be able to extend the heuristic functionality to allow someone to implement a large response handler. For example, if the Content-Length is larger than some value (1GB maybe?) then you might trigger a custom buffer that avoids the extra reads / writes.

I haven't looked at the code for a while, but I think heuristics only are for adjusting the response object before it is sent to the actual caching logic. You'd likely have to pass some extra info to trigger the buffering or potentially swap the response's body with your custom buffer. There are probably other colors to paint the bikeshed too!

I mention it as I suspect changing the core API might be challenging at this point whereas extending the heuristics might provide a more reasonable upgrade path over time.

[hexagonrecursion]

discrepancies between different operating systems

What kind of discrepancies?

potential use cases

What kind of use cases?

One idea I had when looking through your commentary was that you might be able to extend the heuristic functionality to allow someone to implement a large response handler. For example, if the Content-Length is larger than some value (1GB maybe?) then you might trigger a custom buffer that avoids the extra reads / writes.

Exactly how would this work? Where would the body be stored while it's being downloaded?