Add max_age parameter to .get()

[ppwfx]

Hey,

would you accept a pull request that adds a max_age parameter to .get(). So that it returns a default value if the cached value is too old.

I guess something like this is not possible yet, or?

[grantjenks]

An expired key is functionally the same as a missing key. The "get" method already supports a "default" keyword parameter for this purpose. Does that not work for you?

[ppwfx]

Let's say I want to get pricing data. In my test's its totally fine if they are 5 days old. In production their age should be max 60s. Currently the expire time is defined when a key is set.

The idea would be to set the "expire" on a read and not on a write. In the end a creation timestamp needs to be added when setting a value and read would check if that timestamp is younger than max age.

I hope that clarified it.

I think that's not possible right now or?

[grantjenks]

You can just "set" the value again.

In [1]: import diskcache

In [2]: cache = diskcache.Cache('/tmp')

In [3]: cache.set(b'key', b'value', expire=3600)
Out[3]: True

In [4]: cache.get(b'key', expire_time=True)
Out[4]: ('value', 1511756942.215775)

In [5]: cache.set(b'key', b'value', expire=60)
Out[5]: True

In [6]: cache.get(b'key', expire_time=True)
Out[6]: ('value', 1511753418.529129)

The second expire time is 60 seconds rather than the original 3600 seconds.

As a function, it looks like:

In [7]: def set_expire_time(cache, key, expire):
   ...:     value = cache[key]
   ...:     cache.set(key, value, expire)

[ppwfx]

I think that does not work for the use case.

How can I guarantee that the data is only max 60s old?
How can I make that work across more than one process?

Out[6]: ('value', 1511753418.529129) could be up to 3600s + 60s seconds old data

That's how I quickly implemented it.

def get(key: str, max_age=0):
    v = disk_cache.get(key, default=None)

    if v is None:
        return v

    max_age_timestamp = int(datetime.datetime.now(tz=datetime.timezone.utc).timestamp()) - max_age

    if v["timestamp"] < max_age_timestamp:
        return None

    return v["value"]


def set(key: str, value: object) -> bool:
    value = {
        "timestamp": int(datetime.datetime.now(tz=datetime.timezone.utc).timestamp()),
        "value": value,
    }
    return disk_cache.set(key, value)```

[grantjenks]

You can store the current time in the "tag" for the item. Something like:

In [1]: import diskcache

In [2]: cache = diskcache.Cache('/tmp/diskcache')

In [3]: import time

In [22]: cache.clear()
Out[22]: 1

In [23]: cache.set(b'key', b'value', expire=3600, tag=time.time())
Out[23]: True

In [24]: cache.get(b'key', expire_time=True, tag=True)
Out[24]: ('value', 1511822834.686602, 1511819234.6866)  # <-- expire time is tag + 3600

In [25]: def set_expire_time(cache, key, expire):
    ...:     now = time.time()
    ...:     value, store_time = cache.get(key, tag=True)
    ...:     expire -= now - store_time
    ...:     if expire < 0:
    ...:         return
    ...:     cache.set(key, value, expire, tag=store_time)
    ...:     
    ...:      

In [26]: set_expire_time(cache, b'key', 60)

In [27]: cache.get(b'key', expire_time=True, tag=True)  # <-- expire time is tag + 60
Out[27]: ('value', 1511819294.6867409, 1511819234.6866)

Using the tag is just one option. You could also store the current time in the value as you've done.