-
Notifications
You must be signed in to change notification settings - Fork 442
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Determine if scan server tablet metadata cache blocks on refresh #4544
Comments
Maybe the refreshAfterWrite could be used? https://github.com/ben-manes/caffeine/wiki/Refresh |
I will look into this to see how the Cache is being used but Caffeine does support a refresh option which allows an updated value to be loaded in the background and the old value is returned until complete. We probably need to change the configuration here to use the refresh |
@EdColeman - I see you just beat me to the same comment :) |
I did not know about the refresh capability, that seems like it could really smooth out scan server performance for frequently used tablets. Also, that seems to indicate that the current code is probably blocking on refreshing frequently used tablets. |
It seems like we may be able to target the changes here for 2.1 or at least for 3.1 |
It could be a nice change for 2.1, it definitely seems like a performance bug if its periodically blocking. |
We may be able to use this refresh capability to further smooth out performance of frequently used tablets. Looking at the caffeine refresh docs, it seems custom code can be called on refresh. We could add code there to write out any new scan server refs if needed. That could be follow on work. It would further reduce delays around frequently used tablets. That could look like the following.
|
From afar, it does sound like refreshAfterWrite could work very well for you. It avoids the latency spike when popular items expire by reloading early, while allowing seldomly used ones to fade away by expiring. It is intended to hide latency. You can override As reloads are triggered by individual lookups, it can be beneficial to batch these requests into a single load. A small buffering delay can be used to accumulate reloads over a time/space window, perform the bulk load, and complete the individual operations. This works nicely because that buffering delay is not application-facing, as it continues to receive the current value while the reload is in progress. The example uses Reactor for simplicity as it then only requires about a dozen lines of code. Hope that helps! |
@ben-manes - Thanks for the input, that helps. One question I was wondering is that currently this uses a I think the main benefit of the |
This adds a property to configure the scan server tablet metadata Caffeine cache to refresh cached tablet metadata in the background on cache hits after the refresh time has passed. The refresh time is expressed as a percentage of the expiration time. This allows the cached entries to refresh before expiration if they are frequently used so that scans will not be blocked waiting on a refresh on expiration. Entries still expire if no cache hits come after the refresh time and expiration time passes. See: https://github.com/ben-manes/caffeine/wiki/Refresh This closes apache#4544
This adds a property to configure the scan server tablet metadata Caffeine cache to refresh cached tablet metadata in the background on cache hits after the refresh time has passed. The refresh time is expressed as a percentage of the expiration time. This allows the cached entries to refresh before expiration if they are frequently used so that scans will not be blocked waiting on a refresh on expiration. Entries still expire if no cache hits come after the refresh time and expiration time passes. See: https://github.com/ben-manes/caffeine/wiki/Refresh This closes apache#4544
@cshannon, I think you're right on all points. I'll give more depth just for fun. In regards to automatic refresh, the sync/async caches are the same. The vast majority of the code is shared (BoundedLocalCache.refreshIfNeeded) with thin adapters to the corresponding interfaces (LocalLoadingCache, LocalAsyncLoadingCache). When a refresh is triggered, it calls AsyncCacheLoader.asyncReload(k, v, executor), which CacheLoader overrides to wrap and call the synchronous reload(k, v) for convenience. There is a little juggling if the cache is storing a future value, but that's all internal details.
A negative aspect of all async calls is the need for an executor. The cache doesn't manage its own threads so it defaults to the shared JVM-wide |
@ben-manes - Thanks for the explanation, that helps a lot. Sounds like the Virtual thread support will be quite nice when it's ready in the future. |
The ScanServer has a cache of tablet metadata with an expire after write time. Whan a tablet is constantly being read from this cache, what happens when the expiration time arrives? Does it drop it cause a block read for the next cache use? Ideally items in the cache that were recently used could be reloaded in such a way that the expiration does not cause blocking for the reload, but not sure if this is possible w/ caffine.
The first use of tablet should cause a blocking read on the cache, but if the tablet continues to be used frequently it would be nice if its refreshed in the background somehow w/o causing further blocking reads form the cache.
The text was updated successfully, but these errors were encountered: