Join GitHub today
GitHub is home to over 50 million developers working together to host and review code, manage projects, and build software together.Sign up
Feature request: ability to get the list of keys, or a subset of them. #163
I needed the ability to remove all keys matching particular patterns, where one key might match more than one pattern, so regions would not suffice.
The work around is to have OnAdd/OnRemove hooks that keep track of all the keys, but this is inefficient as many of the backends have clear ways to get the keys (in memory and redis are the ones I'm using for the project).
I totally get that this is kinda a great feature.
What I would suggest is, how about building that feature for in-memory cache only? For in-memory, this is actually relatively easy to control and do and fast.
So, how about make that feature available client side only and in case you have a distributed layer, you'll have to either not use the feature or build your own solution.
Maybe there will be solution for Redis some day with all the new features coming in 4.0, but right now, I really don't want to have any method calling
What do you think?
The library for redis (StackExchange.Redis) does use SCAN with a pattern match rather than KEYS if the redis server is new enough to support it (>= 2.8.0), so it's somewhat less intense.
We have two use cases:
Your suggestion would work for case 1, and maybe that's more useful to others and thus worth implementing at the library level.
The "hashes of keys" solution is very similar to the way regions support is implemented (at least for the memory and redis cases), except on multiple dimensions rather than just the one.
and we want to be able to either remove all where "Foo=a" or, at a different time, "Bar=y" (basically deleting on named property)
Maybe there's a better way to solve this particular case.
I think the Redis use-case would always be very dependent on how you setup Redis and what you actually need.
Storing another lookup in Redis for all keys is just not something I'd like to have. That's not performing very well as you would have to keep it somewhat in sync. There could be millions of keys eventually...
Yes this would be similar to the regions implementation with Redis. But I know that the list of keys, i keep for a region, might not be 100% accurate and it doesn't really matter in that case. Again, if you would store really tons of keys in one region, a clear region would also be pretty slow. But it is a trade off for that feature which I think is fine.
In addition to all that, only Redis has good support for searching at all. Memcached and other distributed caches don't support that at all. That's why I never added that feature in the first place ;)
But again, I think it would be fine to have that implemented for all the in-memory caches.
From quiet observation, I think there's two features here:
While i understand 1. being quite expensive in some distributed caches such as Redis, and non existent in others, it is largely seperate from 2.
Currently, there might be situations where i need to get ~100 items from the cache, and if these are in Redis, this could mean 100 requests. If there was a
Worst case scenario, the distributed cache doesn't have a facility for getting multiple keys in one command, and cache manager has to do what the user is otherwise forced to do - iterate and get each key individually.
@twgraham You are absolutely right that those are two different operations.
That's why this feature doesn't exist, I'd implement it as an iterator on the top level anyways. Then, a MGet might have draw backs, you have to handle what should happen if one or more given keys do not exist. I could return either nulls to keep positions or only return found key/values. Also, I'd have to return both, key and value. There might be more logic involved, so, I think it makes more sense for anyone who needs it, to implement it with a simple loop+ custom logic...
Regarding the "search" like feature, I'll look more into it again when I find some time. Still interested in getting that feature done. I'm just not very happy with the current approach in the related PR and might have to change it a lot ...
Thanks for the explanation.
Regarding a) do you have any further reference material (code aside) about how cache manager uses Redis to store and retrieve values?
In regards to b) my concern is for cold starts, and expiration (although I suppose that's up to me to tune).
I suppose we can agree on c).
I think that storing cache keys would be very useful for tracing purposes. I have also read the related PR request.
It is obvious that due to memory limitations you cannot store O(n) keys in a hashtable/hashset in the BaseCacheManager. I made some calculations, complications would be catastrophic.
But I have suggestion about this:
The main problem is the key insertion time (log(n) I think on average) of the TRIE of course. Which can be done in a seperate thread right after sending the key to the handlers.
What do you think?