Confused about sync vs async refresh

[NathanHazout]

Let's say I have a LoadingCache with a refreshAfterWrite of 1 minute.

• LoadingCache::refresh says "Loads a new value for the key, asynchronously.", also "will call CacheLoader.reload(K, V)".
• Caffeine:: refreshAfterWrite says "performed by calling AsyncCacheLoader.asyncLoad(K, Executor)"
• The default implementation of reload simply calls load()

Based on the above, I have a few questions:

• Why is there a CacheLoader and an AsyncCacheLoad? LoadingCache vs AsyncLoadingCache? If the documentation states the data will be loaded asynchronously regardless? Which are are we meant to use?
• Similarly, why CacheLoader has both reload and asyncReload? Which one are we meant to implement?

If I'm not doing anything with the old data, is is safe to ONLY implement the load method?

What happens when thousands of requests try to get() the same key, and the refresh time has been reached?
Will Caffeine understand to call reload only once?

I asked these because in our testing we saw the load method being called a lot more than once per minute.

[ben-manes]

The LoadingCache.refresh documentation states "Caches loaded by a {@link CacheLoader} will call {@link CacheLoader#reload}..." which is the common case, but it doesn't preclude caches built from an AsyncCacheLoader. Internally the implementations only call asyncReload and rely on the delegation chain to resolve to the implementation logic.

AsyncLoadingCache stores a future into the cache and has a synchronous() view for a LoadingCache. It uses a AsyncCacheLoader, but can be given a CacheLoader (a subtype) for user convenience if their logic is best expressed synchronously and redefines the delegation chain from asyncReload => asyncLoad to asyncReload => reload => load. You can implement only the load method and the provided current value will be ignored and has no cost for the cache to retrieve as a necessity anyway. Since the loaders are a @FunctionalInterface you can use a lambda expression for simple cases and it will work as desired.

If I'm not doing anything with the old data, is is safe to ONLY implement the load method?

Yes. The old value is provided in the rarer cases when you can perform a cheap version check to validate it is still up to date instead of a more costly full load. For example if fetching the value uses a provider's API quota but they offers a free HEAD ETag for the unique version identifier, then you can use that optimization trick to avoid the financial penalty.

What happens when thousands of requests try to get() the same key, and the refresh time has been reached?
Will Caffeine understand to call reload only once? I asked these because in our testing we saw the load method being called a lot more than once per minute.

Yes, the cache will memoize this, perform the call once, and ignore redundant attempts to refresh if no other writes occurred for that key. If the mapping was updated or removed while a refresh is in-flight then that reload will be abandoned. This is to try to provide per-key linearizable reload semantics to avoid the reload inserting a stale version of the data into the cache (e.g. Th1: start reload of v1 => Th2: update to v2 => Th1: drop (v1-current => v1-loaded) result and forward the reloaded value to the removal listener). If the test is overwriting the mapping while also performing explicit refreshes then you would see multiple in-flight reloads, but only the last valid one will replace the value (caveat wrt some unreleased fixes). This assumes v3, whereas prior versions did not make as strong a promise and was documented as more naive.

If you are performing many concurrent reloads then I'd recommend considering coalescing reloads over a time/space period so that you can batch them more efficiently.

Let me know if you have any further questions or clarifications.

[ben-manes]

Caffeine.refreshAfterWrite does have a typo wrt asyncLoad should be asyncReload.

[NathanHazout]

Thanks!

Just to make sure however, when you write LoadingCache<Key, Graph> graphs = Caffeine.newBuilder()..., are you getting a LoadingCache or an AsyncLoadingCache ?

If you get a LoadingCache, does it work asynchronously for refreshes?

Also, if you pass a CacheLoader (and not an AsyncCacheLoader), will refreshes still be async?

In other words, is there even a scenario where refresh is NOT async?

[ben-manes]

• Caffeine.build(loader) will give you a LoadingCache which is a bounded ConcurrentMap<K, V>.
• Caffeine.buildAsyn(loader) will give you an AsyncLoadingCache which is a bounded ConcurrentMap<K, CompletableFuture<V>>.

All refreshes are asynchronous and are performed outside of any atomic action within the underlying ConcurrentMap. All usages call asyncReload on a reload, while LoadingCache.refresh can call asyncLoad if the entry is absent.

If the async loading function returns a completed future, e.g. due to a caller-runs executor, this is still asynchronous wrt the ConcurrentMap that holds the cached entries. If triggered by a refreshAfterWrite then the calling get will return the new reloaded value, if successfully replacing the mapping, instead of the previous one it had found.

Some users want per-entry linearizability of the cached mapping. A strict version of that is possible with a Caffeine.build(loader) synchronous cache for regular loads due to using ConcurrentMap.compute to atomically load the value. In that case, a refresh is an optimistic non-atomic load that uses a compare-and-swap approach, abandoning itself if another write to that mapping could have caused a conflict. A Caffeine.buildAsync(loader) asynchronous cache cannot provide linearizability of the V value, since it stores a CompletableFuture<V> mapping, so it similarly linearizable in respect to the future mapping (less useful). The caveat is that ensuring refresh linearizable is complex and error prone, so it is a best effort goal that might be accidentally violated (e.g. unreleased fixes on master when reviewing it recently).

You can think of a refresh as an optimistic transaction. It will never be blocking within the cache's data map during its load and is therefore asynchronous to it. It may be that the reload's work is performed on another thread, but it could be on the caller if that is how the user computed it.