Join GitHub today
GitHub is home to over 31 million developers working together to host and review code, manage projects, and build software together.Sign up
cache: problems with non-file-based storage plugins #2454
@markus2330 as discussed in the last meeting, caching works with cascading keys too. Because of your comment about "/", I tested it and found it to be an exception.
The problem is that non-storage based backends like
I see two approaches to solve this:
Thank you for reporting this problem!
Isn't this problem unrelated to cascading keys and simply a problem of that we need some way to properly implement and use "noresolver" (for the mentioned mountpoints but also for
The noresolver could work like: It gets all keys, and compares with the current keys. If the keys are the same, it returns 0 (cache hit). Otherwise, it returns 1 (update needed).
But actually, your idea (2.) to simply always append all keys again is maybe even better: then we do not need so complicated resolvers. And we wanted to use this behavior for command-line anyway. What is the performance penalty for doing so?
You're right. In case of the cache, cascading keys just trigger the problem, but are not the cause.
Well, the optimal case for the mmap cache is of course not appending anything. In that case we only update pointers, which was very fast as shown with mmapstorage.
Comparing the keysets sounds questionable to me. In case of a cache hit, we still have to compare two keysets. In case of a cache miss we have to correctly cut out the old keys and then append the new ones. It already does not sound very efficient.
Solution 2 more detailed:
In kdbGet() we would omit the problematic keys from the cache. The same is basically already done for the default split (the last split with user supplied keys, not from the backends). I propose to simply add a backendflag_t to mark such "problematic" backends, for easy detection.
After a cache hit we would then fetch only the problematic keys in
When noresolver decides that there is no change, no ksAppend would be needed. The disadvantage is that you need to detect if there is no change. It is not clear for me if the detection or the appending is more expensive. For example, in a naive implementation of the ulimit plugin you would need to execute
This would invalidate the OPMPHM. Maybe the ksAppend of existing keys is not so expensive after all: it only needs O(1) to locate where to insert and then O(1) to swap the key.
In any case: This problem does not need to be solved with your current (soon-to-come) PR.