Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improvement: Add filters to SCAN #3323

Closed
allanwax opened this issue Jun 16, 2016 · 2 comments
Closed

Improvement: Add filters to SCAN #3323

allanwax opened this issue Jun 16, 2016 · 2 comments

Comments

@allanwax
Copy link

I've written a small Java program to copy and transform keys and their values from one redis cluster to another. I have to process tens of millions of keys. To do the job properly I need to send a round-trip request to get the type of the keys returned from SCAN which causes time and network traffic to add up quickly with millions of keys.

I'd like to suggest a modification to the SCAN command to allow for a filter on type or other attributes so I can batch the copy by type and save the round trip. Something like:

scan 0 match * count 100 type hash

or

scan 0 match * count 100 filter type=hash

which will filter the scan results to only return keys that represent a hash type. This avoids the request for the type. I can then parallelize the copy process to do hash, set, zset, etc. as separate tasks and avoid the overhead of the type call.

I can imagine many kinds of filters, such as keys that will expire in 1 hour, sets that contain more than nnn elements, lists with exactly one element, etc.

Perhaps filters can be combined : filter type=list AND ttl<3600

@itamarhaber
Copy link
Member

This has been partially addressed (support for TYPE) by #6116

@itamarhaber
Copy link
Member

Closing - feel free to reopen or create a new issue (for extended filtering)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants