Clean cached data quickly

[wenhaocs]

Hi,

We are using cachelib in a storage system with RAFT consensus. We are thinking of a way to invalidate all cache items in the cache quickly on partition leader change, to avoid cache incoherence. Since it's leader change, the callback must finish quickly, maybe 1s.

I am wondering whether cachelib can (1) support removing all keys in the cache with one function call (2) do this quickly. Our goal is just to make sure those keys are not found when calling cache.find(). It may be achieved by clearing all meta data or data in cachelib.

Thanks

[sathyaphoenix]

The easiest way to do this would be to have an epoch for all the cache items that is bumped up on version change and let your application code do epoch checks after find() while having some background thread purge expired epochs or let them fall out of cache as the reads taper off. This does not accomplish your goal as you outline it above on how it can be done, but wonder if this meet the application level requirement to wipe out the cache. Few more questions

1. what's the intent with clearing the cache on leader change ? Is it purely for coeherence to avoid stale reads from being served while routing change propagates ?

2. do you have additional requirement to reclaim the cache space within a certain timeframe to make space for new data ?

[wenhaocs]

Hi Sathya,

Thanks for the answer. I am a little bit concerned about the additional overhead for epoch check in OLTP database reads. Regarding the questions:

(1) Currently we are invalidating cache in RAFT commit logs, so the nodes storing replicas also have a chance to purge cache items on updates. But we are implementing another store which does not need replicas. To unify the interface, we are considering only invaliding the cache items in leaders when serving writes for the store with replicas. Once a leader becomes a follower, its cached items will become stale. We then purge them once it becomes leader again. This process must be done quickly. So we cannot scan items in the storage and purge them in cache one by one. That's why we want to invalidate cache items as a whole, which could be faster.

(2) I think we do need cache space to be reclaimed if there is a way to drop all cache items quickly. But I think we can still endure it if this process is not fast, comparing with reading stale data. Our assumption of observation is leader change is not frequent, thus dropping cache is not frequent. There is little chance of OOM caused by frequently dropping cache.

[sathyaphoenix]

if the intent is to drop all the cache at once on leader role change, have you tried doing cache reset and reinstantiating a new cache ? Can you stall reads briefly while the cache is being reset ? (assuming you will have to synchronize the cache instance in the call sites to pick up the right one with the concurrent purge)

[sathyaphoenix]

What do you mean by reset? I don't see such interface in CacheAllocator.h

I was referring to destroying the Cache instance and recreating another one and using it. But that will need co-ordination with the threads accessing the cache instance.

Naively if cachelib were to do this, it would involve an approach similar to either of these ideas. ie. Have an epoch in each item, an api to invalidate the entire cache and validate epoch on apis. Or, have a reset like API that resets each of the internal component.

[wenhaocs]

If I can release the constraint of deadline from 1s to some larger value, say 1min, can Cachelib expose an API to do some internal cleaning like map.clear()?

[therealgymmy]

@wenhaocs: Can you explicitly reset the cache instance and create a new one? Something like:

// std::unique_ptr<LruAllocator> myCache;
void resetCache() {
// Block other threads from accessing cache when this is happening
myCache.reset();
myCache = /* create a new instance */;
}

[sathyaphoenix]

@therealgymmy I think the problem with that approach in their case is that the myCache pointer is passed to many places and we need to centralize a way to obtain the cache reference that is synchronized with this reset.

@wenhaocs There are two possible ways like outlined above that come to my mind and I am open to any other approaches you might have. Let's figure out if we want to pursue any of this.

1. store an epoch in the item and handle the invalidation through epochs.
2. implement cache reset functionality by reseting the state of all the internal components. This is easier said than done, but it might need a lot more thought on how to accomplish this with concurrent API calls to CacheLib and also think about hybrid cache semantics.

What Jimmy suggested is a variant of #2, but handled entirely through existing shutdown, destroy, recreate flow. That is also a better alternative than #2 IMO.

[wenhaocs]

Hi @sathyaphoenix @therealgymmy , I think we will go with choice 1.
Rather than store an epoch with value, we will construct the key with epoch included. In this way, we can quickly tell if a key belongs to the previous epoch (due to leader change), rather than retrieving value first. We will also set up a background thread to clean all stale items on leader change. Thanks for all the help.