Does cachelib support persist cache by call specific API?

[yiguolei]

Currently, cachelib could persist cache by call shutdown api. But if the process is killed by 'kill -9', then no signal could be catch by the process so that could not call shutdown api.

Does cachelib have an persistence API and the application could call it periodically? And could load the cache even if the process is killed by 'kill -9'? The cache is not up to date is accepted.

[leozzx]

no, cachelib doesn't support online backup. Could you explain why do you need persistence in the case of "kill -9"?

[sathyaphoenix]

Unlike storage engines, cachelib does not offer crash consistent storage of items. for caches, this is not a deal breaker since durability is not a prime concern for the cache. Besides, caches operate with background evictions, so the application already expects the cache to be empty or missing items. In steady state, cachelib has DRAM data structures that need to be persisted for recovery on restart. If a cache process is receiving a kill -9, we can not guarantee the state of these DRAM data structures to be consistent for shutdown. So I would not recommend catching the kill signal and doing a shutdown unless you can guarantee that the application threads have stopped accessing the cache and shutdown api can be safely called.

There is an internal experimental feature, but it is not the "kill -9" feature you are probably looking for. With that you can snapshot the cache by shutting it down (after ensuring application thread stop accessing cache) and copying the state to another machine to bootstrap another cache from it.

[yiguolei]

Our service is an OLAP service, we want to store the data to S3 and use cachelib to cache data blocks(about 1MB every block) to local disk(the total cache size is about 1TB - 5TB) to accelerate query performance. But the process does not have an elegant shutdown method, we usually call 'kill -9' to stop the process for example during upgrading program version. If the cache is not backuped, then all cache is lost after start up and we will see a performance decline because the cache is too large, it need a lot of time to fetch data from S3 to local disk.

We also meet this problem during machine restarting process.

I do not know if I am using cachelib the right way. Hope to get advice from you.....

[sathyaphoenix]

But the process does not have an elegant shutdown method, we usually call 'kill -9' to stop the process for example during upgrading program version.

You can shutdown cachelib and restore using the shutDown() API documented here. However, to use this API, you must shut down cachelib from one thread and ensure no other threads can be calling other APIs. It is undefined behavior to call shutdown while cache is being concurrently accessed. Hope that helps. I suggest installing a SIGSTOP instead of kill -9 and co-ordinating your application shutdown in that signal handler. As an example, cachebench does this by invoking the runner abort on a SIGSTOP and runner's abort in turn co-ordinates with the threads to stop accessing the cache and then shutdown the cache.

[sathyaphoenix]

Our service is an OLAP service, we want to store the data to S3 and use cachelib to cache data blocks(about 1MB every block) to local disk(the total cache size is about 1TB - 5TB) to accelerate query performance

Can you share more details on the OLAP service :) I am interested in learning about cachelib use cases in the OSS forum.

[yiguolei]

It is apache doris (https://github.com/apache/incubator-doris) a mpp based OLAP engine inspired by Google Mesa. All data files is immutable. We want to save the data file to S3 and use cachelib to accelerate the query performance.

BTW does cachelib only support reading cache files by DirectIO? I have tested it and find the pagecache doesn't increasing...

Does it have any config to allow using buffered read?

[sathyaphoenix]

BTW does cachelib only support reading cache files by DirectIO? I have tested it and find the pagecache doesn't increasing...

CacheLib IO to SSD are direct IO since CacheLib will cache the items in DRAM upon a find() API call until the access to the item becoming cold for it to be evicted from it's dram cache. You can control the dram cache footprint in hybrid cache to get similar performance as buffered IO.

[amosbird]

I think this is a quite valid feature request. The Cachelib research paper highlights the goodness of supporting "Frequent Restart".

Warm restarts: To handle code updates seamlessly, CacheLib can perform warm restarts that retain the state of the cache. This overcomes the need to warm up caches everytime they are restarted.

It's not reliable to enforce some cache shutdown operation in order to retain a clean state.