Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix the bug that CLIENT REPLY OFF|SKIP cannot receive push notifications #11875

Merged
merged 12 commits into from Mar 12, 2023

Conversation

enjoy-binbin
Copy link
Collaborator

@enjoy-binbin enjoy-binbin commented Mar 2, 2023

This bug seems to be there forever, CLIENT REPLY OFF|SKIP will
mark the client with CLIENT_REPLY_OFF or CLIENT_REPLY_SKIP flags.
With these flags, prepareClientToWrite called by addReply* will
return C_ERR directly. So the client can't receive the Pub/Sub
messages and any other push notifications, e.g client side tracking.

In this PR, we adding a CLIENT_PUSHING flag, disables the reply
silencing flags. When adding push replies, set the flag, after the reply,
clear the flag. Then add the flag check in prepareClientToWrite.

Fixes #11874

Note, the SUBSCRIBE command response is a bit awkward,
see redis/redis-doc#2327

This bug seems to be there forever, CLIENT REPLY OFF|SKIP will
mark the client with CLIENT_REPLY_OFF or CLIENT_REPLY_SKIP flags.
With these flags, prepareClientToWrite called by addReply* will
return C_ERR directly. So the client can't receive the Pub/Sub
messages.

In this PR, we clear these flags when sending Pub/Sub messages,
and restore them after sending if any.

Fixes redis#11874
@oranagra
Copy link
Member

oranagra commented Mar 3, 2023

What about other push notifications? E.g client side tracking.
I think we better handle it in networking.c since these functions that add a push message to the output buffer are not really a reply, and should not be skipped (no matter who calls them)

@enjoy-binbin
Copy link
Collaborator Author

I thought about modifying it in something like addReplyPubsubMessage or networking.c. But I can't think of any better way.

changing addReply* will cause a huge diff.

and I don't want to add extra flags to the client, some flags like, add a flag (PUSHING?) before push, clear the flag after push
then in networking.c, by checking the flag

so in the first place, i chooice this way, and then ignore the other push notifications first. maybe the client flag is the right way? do you have any other ideas?

@oranagra
Copy link
Member

oranagra commented Mar 4, 2023

but why not put the exact same solution you implemented (backup and restore a flag) in addReplyPubsubMessage itself?

looking at the code now (i was AFK earlier), i see it's insufficient as well.
sendTrackingMessage calls addReplyPubsubMessage, but it also calls other networking.c primitives (like addReplyProto, addReplyBulkCBuffer).

also addReplyPushLen just adds the push header, and unlike addReplyPubsubMessage doesn't wrap the entire thing.

what i think we wanna do is:
add these flags addReplyPubsubMessage, but also in all places that call addReplyPushLen.
in addition we need some assertion that will help us catch bugs.
we can't just assert in addReplyPushLen that the flag is off, since the tests are unlikely to catch it, so maybe indeed a flag that disables the reply silencing flags (like PUSHING) is better.
maybe we can also make sure that any call to prepareClientToWrite when that client is not the executing_client, is done with the PUSHING flag set?

@enjoy-binbin
Copy link
Collaborator Author

so maybe indeed a flag that disables the reply silencing flags (like PUSHING) is better.

seems to be yes. i added the PUSHING flag.

maybe we can also make sure that any call to prepareClientToWrite when that client is not the executing_client, is done with the PUSHING flag set?

Is this the way to do it? (the one i commented)
the executing_client is only set in call, there are some addReply* (called by rejectCommandSds) in processCommand, in these cases, executing_client is not set, and it will crash

src/debug.c Outdated Show resolved Hide resolved
src/networking.c Outdated Show resolved Hide resolved
src/pubsub.c Outdated Show resolved Hide resolved
src/tracking.c Outdated Show resolved Hide resolved
src/networking.c Outdated Show resolved Hide resolved
Copy link
Member

@oranagra oranagra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. now maybe add a test or two?

@enjoy-binbin
Copy link
Collaborator Author

LGTM. now maybe add a test or two?

yean, the next step is to add tests, i will try to cover it

@oranagra oranagra added this to To Do in 6.0 Backport via automation Mar 7, 2023
@enjoy-binbin enjoy-binbin changed the title Fix the bug that CLIENT REPLY OFF|SKIP cannot receive Pub/Sub messages Fix the bug that CLIENT REPLY OFF|SKIP cannot receive push notifications Mar 7, 2023
tests/unit/pubsub.tcl Outdated Show resolved Hide resolved
tests/unit/pubsub.tcl Outdated Show resolved Hide resolved
tests/unit/pubsub.tcl Outdated Show resolved Hide resolved
tests/unit/pubsub.tcl Outdated Show resolved Hide resolved
tests/unit/pubsub.tcl Outdated Show resolved Hide resolved
tests/unit/tracking.tcl Outdated Show resolved Hide resolved
tests/unit/tracking.tcl Outdated Show resolved Hide resolved
@oranagra oranagra merged commit 416842e into redis:unstable Mar 12, 2023
@enjoy-binbin enjoy-binbin deleted the pubsub_client_reply_off_skip branch March 12, 2023 15:54
@oranagra oranagra added the release-notes indication that this issue needs to be mentioned in the release notes label Mar 14, 2023
oranagra pushed a commit to oranagra/redis that referenced this pull request Mar 20, 2023
…ons (redis#11875)

This bug seems to be there forever, CLIENT REPLY OFF|SKIP will
mark the client with CLIENT_REPLY_OFF or CLIENT_REPLY_SKIP flags.
With these flags, prepareClientToWrite called by addReply* will
return C_ERR directly. So the client can't receive the Pub/Sub
messages and any other push notifications, e.g client side tracking.

In this PR, we adding a CLIENT_PUSHING flag, disables the reply
silencing flags. When adding push replies, set the flag, after the reply,
clear the flag. Then add the flag check in prepareClientToWrite.

Fixes redis#11874

Note, the SUBSCRIBE command response is a bit awkward,
see redis/redis-doc#2327

Co-authored-by: Oran Agra <oran@redislabs.com>
(cherry picked from commit 416842e)
@oranagra oranagra mentioned this pull request Mar 20, 2023
oranagra pushed a commit that referenced this pull request Mar 20, 2023
…ons (#11875)

This bug seems to be there forever, CLIENT REPLY OFF|SKIP will
mark the client with CLIENT_REPLY_OFF or CLIENT_REPLY_SKIP flags.
With these flags, prepareClientToWrite called by addReply* will
return C_ERR directly. So the client can't receive the Pub/Sub
messages and any other push notifications, e.g client side tracking.

In this PR, we adding a CLIENT_PUSHING flag, disables the reply
silencing flags. When adding push replies, set the flag, after the reply,
clear the flag. Then add the flag check in prepareClientToWrite.

Fixes #11874

Note, the SUBSCRIBE command response is a bit awkward,
see redis/redis-doc#2327

Co-authored-by: Oran Agra <oran@redislabs.com>
(cherry picked from commit 416842e)
Mixficsol pushed a commit to Mixficsol/redis that referenced this pull request Apr 12, 2023
…ons (redis#11875)

This bug seems to be there forever, CLIENT REPLY OFF|SKIP will
mark the client with CLIENT_REPLY_OFF or CLIENT_REPLY_SKIP flags.
With these flags, prepareClientToWrite called by addReply* will
return C_ERR directly. So the client can't receive the Pub/Sub
messages and any other push notifications, e.g client side tracking.

In this PR, we adding a CLIENT_PUSHING flag, disables the reply
silencing flags. When adding push replies, set the flag, after the reply,
clear the flag. Then add the flag check in prepareClientToWrite.

Fixes redis#11874

Note, the SUBSCRIBE command response is a bit awkward,
see redis/redis-doc#2327

Co-authored-by: Oran Agra <oran@redislabs.com>
@oranagra oranagra moved this from To Do to In progress in 6.0 Backport Apr 13, 2023
oranagra pushed a commit to oranagra/redis that referenced this pull request Apr 17, 2023
…ons (redis#11875)

This bug seems to be there forever, CLIENT REPLY OFF|SKIP will
mark the client with CLIENT_REPLY_OFF or CLIENT_REPLY_SKIP flags.
With these flags, prepareClientToWrite called by addReply* will
return C_ERR directly. So the client can't receive the Pub/Sub
messages and any other push notifications, e.g client side tracking.

In this PR, we adding a CLIENT_PUSHING flag, disables the reply
silencing flags. When adding push replies, set the flag, after the reply,
clear the flag. Then add the flag check in prepareClientToWrite.

Fixes redis#11874

Note, the SUBSCRIBE command response is a bit awkward,
see redis/redis-doc#2327

Co-authored-by: Oran Agra <oran@redislabs.com>
(cherry picked from commit 416842e)
(cherry picked from commit f8ae7a4)
@oranagra oranagra mentioned this pull request Apr 17, 2023
oranagra pushed a commit to oranagra/redis that referenced this pull request Apr 17, 2023
…ons (redis#11875)

This bug seems to be there forever, CLIENT REPLY OFF|SKIP will
mark the client with CLIENT_REPLY_OFF or CLIENT_REPLY_SKIP flags.
With these flags, prepareClientToWrite called by addReply* will
return C_ERR directly. So the client can't receive the Pub/Sub
messages and any other push notifications, e.g client side tracking.

In this PR, we adding a CLIENT_PUSHING flag, disables the reply
silencing flags. When adding push replies, set the flag, after the reply,
clear the flag. Then add the flag check in prepareClientToWrite.

Fixes redis#11874

Note, the SUBSCRIBE command response is a bit awkward,
see redis/redis-doc#2327

Co-authored-by: Oran Agra <oran@redislabs.com>
(cherry picked from commit 416842e)
(cherry picked from commit f8ae7a4)
(cherry picked from commit 96814a32da61e5ed523864e00609a4aa6be065b3)
@oranagra oranagra mentioned this pull request Apr 17, 2023
oranagra pushed a commit that referenced this pull request Apr 17, 2023
…ons (#11875)

This bug seems to be there forever, CLIENT REPLY OFF|SKIP will
mark the client with CLIENT_REPLY_OFF or CLIENT_REPLY_SKIP flags.
With these flags, prepareClientToWrite called by addReply* will
return C_ERR directly. So the client can't receive the Pub/Sub
messages and any other push notifications, e.g client side tracking.

In this PR, we adding a CLIENT_PUSHING flag, disables the reply
silencing flags. When adding push replies, set the flag, after the reply,
clear the flag. Then add the flag check in prepareClientToWrite.

Fixes #11874

Note, the SUBSCRIBE command response is a bit awkward,
see redis/redis-doc#2327

Co-authored-by: Oran Agra <oran@redislabs.com>
(cherry picked from commit 416842e)
(cherry picked from commit f8ae7a4)
(cherry picked from commit 96814a32da61e5ed523864e00609a4aa6be065b3)
oranagra pushed a commit that referenced this pull request Apr 17, 2023
…ons (#11875)

This bug seems to be there forever, CLIENT REPLY OFF|SKIP will
mark the client with CLIENT_REPLY_OFF or CLIENT_REPLY_SKIP flags.
With these flags, prepareClientToWrite called by addReply* will
return C_ERR directly. So the client can't receive the Pub/Sub
messages and any other push notifications, e.g client side tracking.

In this PR, we adding a CLIENT_PUSHING flag, disables the reply
silencing flags. When adding push replies, set the flag, after the reply,
clear the flag. Then add the flag check in prepareClientToWrite.

Fixes #11874

Note, the SUBSCRIBE command response is a bit awkward,
see redis/redis-doc#2327

Co-authored-by: Oran Agra <oran@redislabs.com>
(cherry picked from commit 416842e)
(cherry picked from commit f8ae7a4)
@oranagra oranagra moved this from In progress to Done in 6.0 Backport Jul 2, 2023
enjoy-binbin added a commit to enjoy-binbin/redis that referenced this pull request Jul 31, 2023
…ons (redis#11875)

This bug seems to be there forever, CLIENT REPLY OFF|SKIP will
mark the client with CLIENT_REPLY_OFF or CLIENT_REPLY_SKIP flags.
With these flags, prepareClientToWrite called by addReply* will
return C_ERR directly. So the client can't receive the Pub/Sub
messages and any other push notifications, e.g client side tracking.

In this PR, we adding a CLIENT_PUSHING flag, disables the reply
silencing flags. When adding push replies, set the flag, after the reply,
clear the flag. Then add the flag check in prepareClientToWrite.

Fixes redis#11874

Note, the SUBSCRIBE command response is a bit awkward,
see redis/redis-doc#2327

Co-authored-by: Oran Agra <oran@redislabs.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
release-notes indication that this issue needs to be mentioned in the release notes
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

[BUG] CLIENT REPLY OFF|SKIP would disable push notifications
4 participants