Adds new pop-push commands (LMOVE, BLMOVE)

[felipou]

Adds lpoprpush, rpoprpush and lpoplpush, and the equivalent blocking commands: blpoprpush, brpoprpush and blpoplpush.

[felipou]

I started implementing this almost 5 years ago, so I decided to create a new PR. The old one is here: #2664

[felipou]

I was in doubt about mainly 3 things:

• Tests: I just replicated all the RPOPLPUSH and BRPOPLPUSH tests. They add quite some testing time (only the blocking ones), but I think they add very little value, since the code is practically the same. But anyway, I felt that I should replicate then all, it's easier to just erase some now if that's the case.
• Comments: I didn't add any new comments, just modified the existing ones. Didn't really feel the need to, since the lists code is quite clean and easy to understand, and my changes don't make it more complicated.
• Coding style: I did my best to comply to overall coding style, although I haven't seen any guideline, just tried to look at the surrounding code to grasp it. But I'm a strong advocate of strict and consistent coding style, so I was confused because there are some bits of code that don't seem to comply with the overall style, like for example max line length. I tried to keep my code under 80 characters, but there were some long lines already, so when I modified these I just kept it as one-line. If anyone wants to point something that is not conformant with the project coding style, I'd be very grateful and eager to fix it.

[itamarhaber]

I started implementing this almost 5 years ago

+1 for being persistent :)

[felipou]

Did a rebase with unstable solving a minor conflict (wasn't an actual conflict, just a change very close to my changes flagged as a possible conflict). As a plus it solved the CI checks that were failing before (and wasn't my fault as expected).

[antirez]

Looks like a good coincise implementation @felipou, thanks. And since this is a plain extension of the original implementation, I guess the replication part should work as expected (didn't check the details). So indeed it is worth a careful review and a potential merge.

[felipou]

Great, let me know if there is anything else I can do to help or improve the code!

[itamarhaber]

@yossigo @oranagra @madolson @soloestoy please review and consider for 6.0.6

@felipou would it be too much to ask you to help with the respective docs once approved? ;)

[oranagra]

@felipou thank you for this PR.
i've added a few minor comments about the code.
some suggestions about improving some tests, and trimming others.

[felipou]

@felipou would it be too much to ask you to help with the respective docs once approved? ;)

Not at all, please count me in for anything you need, it's a pleasure contributing to Redis!

[felipou]

@oranagra Thanks for your review, I'll go over everything right now. I may struggle a bit with the tests, since I've never worked with tcl, I just copied the existing tests for RPOPLPUSH, but I think I'll manage.

Just a quick question: what is the usual process here, do I overwrite my branch (push --force), or should I push multiple commits and afterwards we squash them (or maybe don't even squash, just merge as it is)? For now I'll just push new commits, but I'm asking because I like to keep my git history very clean, so I usually do a lot of overwrites on feature branches to keep them as tidy as possible before merging.

[oranagra]

@felipou i like my git history clean too, we'll either squash it at the end, or if for some reason we want to keep some changes in separate commit, we'll do some cleanup.
for now you can either add more commits, or amend and push-f.

if you need help with the tests let me know, i think you'll just be able to use your clipboard and common sense..

[felipou]

@oranagra I just pushed some new commits addressing everything you commented. I just replied every thread instead of resolving the conversations, if you want I can resolve all the trivial ones (removing duplicate tests for example).

[oranagra]

@felipou thanks you. everything looks great.
hope to merge it soon (although not sure it can go into 6.0).

p.s. out of curiosity i run the list test.
original version took me 11 seconds
the one before my review comments took 26 seconds
and the last revision took just 7 seconds.
so adding tests and still runs faster.

[felipou]

@oranagra Great, thanks!

Makes total sense that it runs faster too, the idea you gave of replacing the after calls with wait_for_condition is a very good optimization, no sense in waiting a full 1s for something that usually happens a lot faster.

[oranagra]

@redis/core-team since this PR adds commands (which we can't later remove or rename), it requires more than lazy consensus.
There's no need to review the code again, but please approve the addition of these 6 new commands.

[madolson]

I think the new commands are sensible.

I want to bring up a separate point that I think we should rewrite the blocking command infrastructure. Today it works by marking the client as blocked on a key with a bunch of special variables, when that key is written to we used those stored special variables to complete the request. This has several issues:

1. We maintain a lot of duplicate code to process the requests.
2. The code as written is sort of spaghetti code.
3. We may still have to re-block after the command is executed, which requires special handling in that case which is unnecessary.
4. Explicit rewriting for replication.

Instead, I think we should merge the blocking framework with the mechanism that was outlined for the background threads. If a client is block from a non-existent key, it should just remove itself from the event handler and register that it is blocked on a key. When that key is touched, the blocked client will be unblocked and will attempt to re-execute the command that blocked it in the first place. This shouldn't introduce any major compatibility issues and should throw the same exceptions we do today. It can then inline the command it has within the replication system. It should hopefully remove a lot of the weird checks that we have to do that are included in the CR.

I think this refactor is worth it either for this change or so that it is easier to make these sort of changes in the future. (If we agree it's worth it for this change, I'm okay with this specific PR getting merged and a separate one for the refactor)

[felipou]

Just pushed 3 commits with:

• The joined tests (I changed them a bit from the commit I had sent before, just to make it a little simpler by avoiding nested ifs)
• The left and right shared strings
• Replacement of rewriteClientCommandVector with more generic function that both commands can use
• Compatibility for rpoplpush/brpoplpush when propagating or rewriting

I'll now experiment with the wherefrom/whereto parameters of the blockForKeys function, and update the redis-doc PR.

[felipou]

Option 1 - using LIST_NONE: felipou@073bb77
Option 2 - using a struct: felipou@0cd3feb

Not sure if it was just me choosing terrible names, but option 2 doesn't look very good.

[oranagra]

all looks good. the only thing that may be missing is maybe add a test to cover the propagation of BRPOPLPUSH and BLMOVE to AOF or Replicas.
i see replication.tcl has these two tests:

        test {BRPOPLPUSH replication, when blocking against empty list} {
        test {BRPOPLPUSH replication, list exists} {

i guess these can be improved to look at the command stats to check which command was executed on the replica.
and add similar tests for BLMOVE.

maybe move proc cmdstat from integration/introspection-2.tcl to support/util.tcl and use it in replication.tcl

[felipou]

Just pushed the commit to create the internal listPos struct as you suggested @oranagra, looks much better now, thanks!

[felipou]

I'll take a look at the tests you mentioned.

[felipou]

Ok, I think I managed. TCL is really new to me, so I didn't know much about what was possible. I created a new proc to avoid changing the file tests/unit/introspection-2.tcl too much.

[oranagra]

hope my changes actually work (not a TCL expert either)

[felipou]

Ok, great, thanks for the orientation! This actually helped me understand something I had suspected but wasn't really sure: the propagated command is actually the non-blocking version (I had to fix that for the tests to pass).

That's how it worked in the rpoplpush command before my changes, so I just replicated the same idea (BLMOVE propagates LMOVE). Should we change that?

[oranagra]

@felipou change what?
you're right, the propagated command is the non-blocking one.
and with the changes we did, when the original command was [B]RPOPLPUSH we propagate an RPOPLPUSH, and when it's [B]LMOVE we propagate an LMOVE.
is that not what we did? i think the tests prove it.
so what did you suggest to change?

[felipou]

I thought maybe this wasn't the intended behaviour, and that perhaps we should propagate the same command (the blocking one) instead.

[oranagra]

@redis/core-team please approve the addition of new LMOVE and BLMOVE commands (deprecating [B]RPOPLPUSH).

Note that when receiving a BRPOPLPUSH we'll still propagate an RPOPLPUSH (but on BLMOVE RIGHT LEFT we'll propagate an LMOVE)

[oranagra]

merged.. @felipou thanks a lot for sticking that long and implementing all the feedback.

[felipou]

Hey, thank you all very much for all the feedback and for taking this in, especially you @oranagra! It was my first significant contribution to an important open-source project, and I have to say, the experience was great. Looking forward to contribute more!