Determine if scan server tablet metadata cache blocks on refresh

[keith-turner]

The ScanServer has a cache of tablet metadata with an expire after write time. Whan a tablet is constantly being read from this cache, what happens when the expiration time arrives? Does it drop it cause a block read for the next cache use? Ideally items in the cache that were recently used could be reloaded in such a way that the expiration does not cause blocking for the reload, but not sure if this is possible w/ caffine.

The first use of tablet should cause a blocking read on the cache, but if the tablet continues to be used frequently it would be nice if its refreshed in the background somehow w/o causing further blocking reads form the cache.

[EdColeman]

Maybe the refreshAfterWrite could be used? https://github.com/ben-manes/caffeine/wiki/Refresh

[cshannon]

I will look into this to see how the Cache is being used but Caffeine does support a refresh option which allows an updated value to be loaded in the background and the old value is returned until complete. We probably need to change the configuration here to use the refresh

[cshannon]

@EdColeman - I see you just beat me to the same comment :)

[keith-turner]

I did not know about the refresh capability, that seems like it could really smooth out scan server performance for frequently used tablets. Also, that seems to indicate that the current code is probably blocking on refreshing frequently used tablets.

[cshannon]

It seems like we may be able to target the changes here for 2.1 or at least for 3.1

[keith-turner]

It seems like we may be able to target the changes here for 2.1 or at least for 3.1

It could be a nice change for 2.1, it definitely seems like a performance bug if its periodically blocking.

[keith-turner]

We may be able to use this refresh capability to further smooth out performance of frequently used tablets. Looking at the caffeine refresh docs, it seems custom code can be called on refresh. We could add code there to write out any new scan server refs if needed. That could be follow on work. It would further reduce delays around frequently used tablets. That could look like the following.

1. Load tablet metadata for table A
2. Tablet A uses files F1,F2,F3 so write out scan server refs for those
3. Later refresh tablet A and find it uses files F1,F2,F3,F4.. as part of refresh write out a new scans server ref for F4

[ben-manes]

From afar, it does sound like refreshAfterWrite could work very well for you. It avoids the latency spike when popular items expire by reloading early, while allowing seldomly used ones to fade away by expiring. It is intended to hide latency.

You can override CacheLoader.reload(key, value) if you can use the value as part of your logic. This is useful when you can cheaply validate if the current value is up to date, e.g. a version / timestamp, or if you can apply a differential update rather than a full reload.

As reloads are triggered by individual lookups, it can be beneficial to batch these requests into a single load. A small buffering delay can be used to accumulate reloads over a time/space window, perform the bulk load, and complete the individual operations. This works nicely because that buffering delay is not application-facing, as it continues to receive the current value while the reload is in progress. The example uses Reactor for simplicity as it then only requires about a dozen lines of code.

Hope that helps!

[cshannon]

@ben-manes - Thanks for the input, that helps. One question I was wondering is that currently this uses a LoadingCache vs AsyncLoadingCache and I was wondering if there would be any reason to switch and any difference with how the refresh behavior works. The refresh operation looks like it will be done in the background even for a LoadingCache and the old value returned, so I think the current LoadingCache is ok in this case.

I think the main benefit of the AsyncLoadingCache is when the computation is slow for the load so you reduce contention on the cache when other threads are hitting it since you put the loads onto an executor. So for the sync cache I assume the performance is based on how fine grained the locking is when running computeIfAbsent(). In this case I'm not sure how much of a benefit that would be, I would need to look into it more.