Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RPUSH commands infinitely repeated with follower-proxy #13

Closed
aphyr opened this issue Mar 3, 2020 · 2 comments
Closed

RPUSH commands infinitely repeated with follower-proxy #13

aphyr opened this issue Mar 3, 2020 · 2 comments

Comments

@aphyr
Copy link
Collaborator

aphyr commented Mar 3, 2020

With Redis f88f866 and Redisraft 1b3fbf6, executing a single RPUSH command against a fresh cluster creates an infinite loop, wherein that RPUSH command is executed repeatedly, gradually ballooning the size of the value, state machine, and log. It also seems to cause some (all?) redis-cli requests against the node which executed the RPUSH to hang indefinitely.

For example, try this command with a fresh cluster:

/opt/redis/redis-cli rpush "a-list" "x"
(integer) 1

redis-cli monitor shows the Raft log applying endless RPUSH operations:

1583259973.634023 [0 ?:0] "RAFT" "rpush" "a-list" "x"
1583259973.634067 [0 ?:0] "RAFT" "rpush" "a-list" "x"
1583259973.634087 [0 ?:0] "RAFT" "rpush" "a-list" "x"
1583259973.634111 [0 ?:0] "RAFT" "rpush" "a-list" "x"
1583259973.634153 [0 ?:0] "RAFT" "rpush" "a-list" "x"
1583259973.634181 [0 ?:0] "RAFT" "rpush" "a-list" "x"
1583259973.634208 [0 ?:0] "RAFT" "rpush" "a-list" "x"

In this Jepsen run, we execute multiple RPUSH operations, each with a distinct integer value: you can see their consequences interleaved in reads:

[:r 0 [1 1 1 1 1 2 3 1 1 1 1 1 1 1 4 1 1 1 1 1 1 1 1 1 5 9 10 6 2 2 2 3 3 2 3 3 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 4 4 4 1 1 4 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 7 1 1 1 1 1 1 1 1 1 1 1 8]

This bug appears linked to follower-proxy=yes; running without follower proxies seems to resolve the issue.

yossigo added a commit that referenced this issue Mar 4, 2020
@yossigo
Copy link
Collaborator

yossigo commented Mar 4, 2020

This should be fixed in #15.

@aphyr
Copy link
Collaborator Author

aphyr commented Mar 4, 2020

Yup! Looks good as of d589127

@aphyr aphyr closed this as completed Mar 4, 2020
sjpotter added a commit to sjpotter/redisraft that referenced this issue Oct 7, 2021
…edisLabs#13)

Before, we only did "auto commit" of entries if the number of voters == 1.  As we can have a "non voting leader", this is wrong, as the voter can be another node, not the leader, therefore we add a condition to ensure that the leader is a voter too, so that we know that the leader is the single voter.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants