Extended SET command

[antirez]

SET and variants

SET is one of the most used Redis commands, and has many variants:

• SETNX -- Set if not exists
• SETEX -- Set with expire
• PSETEX -- Set with milliseconds expire

There is demand for additional variants such as SETEXNX, becuase it is a pretty common pattern in caching and sessions handling to set an object only if not already inside the cache, and setting it with an expire. This is also useful for poor's man locking, and so forth.

There is the argument that this need can be addressed with scripting, however probably this is enough common that a very high performance solution is expected, and also we can exploit the need in order to make our SET story more clear for the users.

Trivia: the reason currently we have this mess of commands is because of an early limitation of the Redis protocol in the early stages. The binary-safe argument was only possible as last argument, so it was not easy to extend already existing commands, and I kept introducing new ones. Finally the protocol was fixed and now our hands are free.

Proposal

This proposal extends the SET command to cover all the already existing variants in form of options, so that later all the SET variants will be deprecated.
At the same time the proposal adds a new functionality in form of an option that only executes the SET command if the key already exists.

The new form of the command will be:

SET key value option option option ...

The available options will be:

• nx -- Execute the SET only if the key does not already exists
• xx -- Execute the SET only if the key already exists
• ex -- Set an expire if the SET will be executed.
• px -- Set a milliseconds expire if the SET will be executed.

Examples:

SET foo bar xx # Set foo to bar only if key foo already exists.
SET foo value ex 1000 nx # Set foo to value with an expire of 1000 seconds, if not exists.
SET foo bar ex 10 # Same as SETEX foo 10 bar

The new command will allow more variants in the future if needed and is simple to remember. The option names were chosen to be short to write and easy to remember just remembering the first character:

• Non existing
• eXisting
• Eexpire
• Pprecision expire

Feedbacks welcomed.

[mgravell]

I think it would be a positive change, and would make taking locks a breeze - just:

set {lockname} {lockvalue} nx ex {timeout}

My concern, however, would be the return value and the impact on older clients; set has historically returned +ok; however now it would presumably be taking on the return signature of setnx, i.e. :0 or :1. This could negatively impact older clients that are checking for a +ok response, with false-negative failures when using regular set.

Unless, of course, the intent is to return +ok if nx / xx are not used, and :0 / :1 if nx / xx are used.

Thoughts?

[antirez]

Good point Marc, my reasoning was to always return just :0 / :1 actually because in the vanilla form SET can't fail, so probably nobody checks the return value, but yep if there is an explicit check, this will break existing clients. There is to evaluate if this is the case or not.

The backward compatible way of returning OK with no options works for sure, but given that we are going for fixing it, maybe it's better to try hard to return a consistent return value. However there is also an alternative of returning "+OK" if the command works as expected and "+NOOP" or alike in case nx or xx prevented the command from being executed. In that case we have a single return value, it is backward compatible, but it introduces something new to know for the users.

Ok there is to think a bit about this :-) thanks for the hint!

[melo]

@mgravell I don't see the return value as an issue.

If you plan to use the new command, you will probably check the client library to see if already supports it or not, and update it if not.

As long as the old SET version in the new code returns the same old +OK return code, old code (code that uses an old client lib) will only use the current syntax, and will keep returning the current response code.

@antirez Do we need EX and PX? Why not always precision expire? Also, why the short codes (EX instead of EXPIRE)?

[mgravell]

@melo it is the same command; the proposal (see also antirez's comment) would potentially change the result of the existing set command; there is no "new command" here. Old clients, continuing to use the command that they always have, but against a new server version, may suddenly get a different response.

[antirez]

@melo unfortunately the new version would NOT return OK even in the old form... But :1

About ex/px, given that we have options, it is probably not too bad to also allow people to use seconds that is what 99% of people do with expires, so my reason is to add two options solely based on the frequency of usage.

About the short options, two reasons:

• 1. fast to type.
• 2. fast to check in the code, no string comparison but inline character checks.

About point 2, because the initial character is different for every option so far, checking this will be very very fast just with a switch or alike for the first char, and only later we'll check the rest of the string, that must just be 'x' + nullterm.

[mgravell]

@antirez "set can't fail" is incorrect; it might be true in the "happy path", but it can fail in many other ways:

• when reading the db file / synchronizing ("not ready" iirc)
• with "READONLY You can't write against a read only slave."
• and possibly a few others

What I have not done is look at all the available client-library implementations to see which disregard the result completely; which just check for -message (for error conditions), and which positively check for +ok.

I've checked my own library, and I have an ExpectOk() there; obviously that is totally trivial for me to fix; my point there is : it is not beyond reason that other clients have their equivalent.

[antirez]

True but in that case a -ERR exception will be raised exactly like in the new SET case, that's why I don't think people check for the 'OK' string itself but for exceptions (errors).

[mgravell]

I chose another library at random and looked at the "set", and indeed in that library it just checked for -message failures. I guess it comes down to the confidence level of libraries being overly conservative in what they accept.

[melo]

I may be missing something obvious, but I see no reason that the new code must return a different result from the one we have today if the simple SET k v format is used.

Only if options are used does a new return code makes sense, IMVHO.

[antirez]

@mgravell yep I think we'll break almost nothing at all in the practice, and anyway this is for 2.8 so we'll make sure to put a big warning in the release notes :-)

@melo I want to have a single return value because often people do something like that:

cmd = ... code to dynamically generate a set ...
result = redis.set(*cmd)
if result == 0 # We were unable to set the key
   .... some other code ...
end

Basically in complex code, often inside library clients implementation, you don't know the form it is used, and to check explicitly if the form is a "pure" one or not will make the check more complex.

[dragonsinth]

This seems awesome, love the proposal. Don't have any brilliant suggestions on the return value tho.

[antirez]

Dudes, quick poll on Twitter revealed with this change we'll break a lot of code.

Currently the best option we have as an exit strategy is +OK on success, +NOOP on no op performed.
But +NOOP kind of sucks, a better more clear to understand string would be much better.

[antirez]

I reconsidered the return value issue, basically OK / NOOP will not help us further, and will create an asymmetry as all the other commands use the :1 / :0 pattern.

The reason why it does not help is that in the initial time after 2.8 introduction clients will not be always updated, so people will check for explicit NOOP returned by the command.
Later probably most clients will update to return true/false to allow for the if (redis.set(...)) pattern, and this will break the code checking for NOOP.

So probably to return :0 and :1 is our best bet, with information about the change.

Similarly I would like to rename HMSET to HSET (variadic version by default) and deprecate HMSET.

So practically speaking, what this will break?

• Application code that explicitly tests for OK returned by set.
• Client libraries that try to ensure SET returns OK
• Client libraries that process the reply from SET in a way that a status reply is expected and an integer reply can't be handled properly.

In some way or the other we should try to move forward and fix the API by providing changes and deprecating the old way of doing things. There isn't a lot to fix in the Redis API, but this time I don't think there are better options.

[e-oz]

In languages with dynamic typing it doesn't matter, will this function return +OK or 1. if (redis.set(... will be possible and correct anyway. But not with +NOOP. Dynamic typed languages are widely used in web-dev, so I guess it will not be a big problem.

For compatibility it will be easy to create wrappers for old variants of SET in clients. Because client library can't afford to declare support only one (last) version of Redis :)

[antirez]

@Jamm For both Ruby and Lua for instance numerical 0 is a true value... so I'm not sure if it's really that different compared to NOOP.

[e-oz]

Sorry, but +NOOP looks ugly.

[e-oz]

If 1/0 is a generic pattern, then Ruby and Lua clients should just reflect it and take as true/false... I think it will be more predictable behavior, than one more special value as NOOP.

[robreiss]

I don't have much experience with Redis yet, just brainstorming:

Can we put the options at the front?
SET -e 60 key1 val1
SET -ye 60 key1 val1 key2 val2 ...
SET -n key1 val1

y = yes exists, n = not exists, e = expire, p = millisecond expire

I could see this being nice when generating a variadic SET via a loop or something, ie. don't have to tag on the options after the loop.

[aleksraiden]

robreiss - i think, your example hasnt backward compartability and protocol/command style of other command at Redis, IMHO

[arnodirlam]

Great proposal, I like the idea of making the Redis API more consistent and flexible at the same time.

How about adding two more options:

• eq -- Execute the SET only if the existing value equals given parameter (after eq)
• ne -- Execute the SET only if the existing value does not equal given parameter (after ne)

Of course this would require comparing both characters, not just the first one, but there are might be sensible use cases for these options, e.g. setting a value only if it hasn't been overwritten in the meantime with one atomic operation (kind of an optimistic lock). What do you guys think?

[charsyam]

How about adding new command just follow above rule
for example "setx", as you know, setnx and set have different response and, it can cause many problem.

so I think, supporting new command, as time goes by, we can deprecated setnx and other thing.

[ptaoussanis]

My vote:

• I really like this proposal. Very happy to see there's still thought going into improving the basics when possible.
• I like the idea of returning :0/:1. I think a documented, sensible break would be better than the +NOOP (which will still leave unnecessary difficulties in the future). It's one clean break, for a good benefit.
• I also like @arnodirlam's suggestion of also adding something like eq, ne since these are themselves such common/useful cases.

@antirez What about also offering a config option (disabled by default) that retains the old SET semantics for those few clients that do break and haven't yet been updated?

[grosser]

👍 please add this or add setexnx, we need a 100% reliable way of locking.

[antirez]

Dudes after much considering all this issue, my guess is that we are best served by the following return value:

• If the set was performed, return +OK, no breakage with the past.
• If the set was not performed, return a null bulk reply like EXEC does with WATCH when the transaction is aborted.

So this is perfectly compatible with the past, and also should work without issues using the new format even with clients not explicitly accounting for the new format because the null bulk reply is often returned as "nil" that most languages will evaluate as a false value.

And here the bottom line is, with this return values making it backward-compatible 100% with the old SET semantics, we can marge in current 2.6 branch and have it ASAP. I think it's a good idea after all. Comments welcomed.

[mgravell]

That's a fair solution - zero impact on clients that don't use the new extended api, so no surprises. And some nice single-trip conditional operations. Very cute.

[antirez]

Thanks Marc, the implementation is now available in the unstable branch with tests. Everything seems fine, so I'm planning to backport this into 2.6 ASAP (before 2.6.12).

[antirez]

Implemented, back ported into 2.6, 2.6.12 released, and documented at redis.io. Closing :-)

Thank you for all your feedbacks!

[mgravell]

The locking guidance on http://redis.io/commands/setnx should probably be updated with this in mind. The most obvious lock now is simply:

set {key} {token} [ex|px] {timeout} nx

with the check: if you get +OK, you got the lock; if you get *0, you didn't.

Edit: I see this is mentioned in "set", but IMO the "setnx" page should strongly suggest using the "set" version, only using the "setnx" implementation for down-level servers.

[antirez]

@mgravell agreed, doing it right now. I'm also writing the small Lua script required to safely unlock for reference so that I can add it into the SET page. Thanks.