New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

KEYS * in production #959

Closed
FGRibreau opened this Issue Feb 20, 2013 · 3 comments

Comments

Projects
None yet
2 participants
@FGRibreau

FGRibreau commented Feb 20, 2013

Hello everyone,

So yes the this may seems to have been written by someone who hasn't read Redis documentation but please bear with me, this is not what you think.

As a Redis GUI developer, founder of Redsmin, I'm looking for workarounds to enable Redis users to query the keyspace without doing any KEYS command. Currently Redsmin finder uses KEYS underneath like any other Redis GUI client.

The goal here is to find techniques that could empower an efficient key finder without:

  • requiring the user to do/configure anything on its side
  • slow down large Redis instances

Here are some ideas I've come up with so far (please consider them from a Redis GUI point of view):

When a user connects for the first time its Redis instance, spawn a new local redis instance as slave, and then run KEYS against the slave.

For self-hosted Redis client that could be a solution but as a SaaS provider it rises a lot of memory consumption and security issues.
Even if memory consumption and security was not an issue on our side (a.k.a Redsmin servers side) KEYS * on highly active databases could still slow them down.
Issue:

  • Well this solution involves KEYS so...
(variant) Let users set their own slaves and configure the GUI to use the slave

Fully-hosted solution won't have to deal with memory consumption and can concentrate on feature and security.
Issues:

  • It requires the user to setup a slave
  • Well this solution involves KEYS so...
When a user connects for the first time its Redis instance, listen for changes thanks to the newly available keyspace notifications

Issues:

  • The GUI will first need to request all keys (with KEYS *)
  • Redis notifications won't be reliable (because of its fire & forget design) and may create inconsistencies.
  • It'll only works with 2.8+ redis server
  • The user (or the GUI) must run CONFIG SET notify-keyspace-events yes
  • Well this solution involves KEYS * so...
When a user requests a pattern (like *a*b*:0*) from the keyspace the first call will make a KEYS * and will then be cached, the pattern will be applied on the GUI side. Further user requests will be made directly on the cache.

Issues:

  • How do the GUI knows when to get request fresh data with KEYS * (without requiring the user to deal with this issue)
  • Well this solution involves KEYS * so...

I may be wrong, but I don't think that a solution exists right know. I know that the main reason behind these limitations is to keep Redis codebase simple.

So what are our options?

@antirez

This comment has been minimized.

Show comment
Hide comment
@antirez

antirez Feb 20, 2013

Owner

Hello,

I think that a simpler solution is this, from @pietern

#579

The only reason I did not yet merged it is that the code involves an algorithm that was not well documented as of today, otherwise I would put SCAN into 2.8 without even thinking about it since it is useful, and the code was written by Pieter that is a guarantee of quality for me.

Soon or later I should find the time to understand how it works and document it if Pieter will not do it before me, so that I can merge that.

Owner

antirez commented Feb 20, 2013

Hello,

I think that a simpler solution is this, from @pietern

#579

The only reason I did not yet merged it is that the code involves an algorithm that was not well documented as of today, otherwise I would put SCAN into 2.8 without even thinking about it since it is useful, and the code was written by Pieter that is a guarantee of quality for me.

Soon or later I should find the time to understand how it works and document it if Pieter will not do it before me, so that I can merge that.

@FGRibreau

This comment has been minimized.

Show comment
Hide comment
@FGRibreau

FGRibreau Feb 20, 2013

Awesome! I've never heard about this one before, it still planned to port it back to 2.6?

We then (Redis GUI Clients) could use SCAN if available and fallback to KEYS otherwise, nice!

FGRibreau commented Feb 20, 2013

Awesome! I've never heard about this one before, it still planned to port it back to 2.6?

We then (Redis GUI Clients) could use SCAN if available and fallback to KEYS otherwise, nice!

@FGRibreau FGRibreau closed this Feb 20, 2013

@antirez

This comment has been minimized.

Show comment
Hide comment
@antirez

antirez Feb 20, 2013

Owner

More likely a 2.8 thing, but if the code looks very safe I may backport. Cheers!

Owner

antirez commented Feb 20, 2013

More likely a 2.8 thing, but if the code looks very safe I may backport. Cheers!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment