raft: don't promote to voter if previous config is not committed #17675
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mmaslankaprv
previously approved these changes
Apr 5, 2024
bharathv
previously approved these changes
Apr 5, 2024
|
/ci-repeat |
dotnwat
reviewed
Apr 5, 2024
Comment on lines
-69
to
+86
| consistency_level>> | ||
| consistency_level, | ||
| isolated_t>> |
|
Build seems to be failing here, @ztlpn |
ztlpn
force-pushed
the
raft-voter-add-divergence
branch
from
44f1272
to
195feac
Compare
ztlpn
force-pushed
the
raft-voter-add-divergence
branch
from
195feac
to
4e7a69a
Compare
mmaslankaprv
previously approved these changes
Apr 8, 2024
|
ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/47564#018ec35c-4118-46c6-b0bd-e5ebdb0177ca ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/47564#018ec363-6bdd-4bf8-829f-e0ac88543d61 ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/47600#018ec749-0f07-41dc-afcd-eee752811673 ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/47637#018eca7c-1e2d-4a4c-b647-0fbe5ef9bc7a |
ztlpn
force-pushed
the
raft-voter-add-divergence
branch
from
4e7a69a
to
86c526f
Compare
Otherwise we may add several new voters in quick succession, that the old voters will not know of, resulting in a possibility of non-intersecting quorums. Example scenario: 1. we start with leader id 1, committed configuration: voters:1,2,3; learners:4,5 2. 1,4,5 are partitioned from 2,3 3. 1 finishes recovery of 4 and 5 and adds them as voters 4. now 1 can remain the leader getting responses from 1,4,5 (that form a quorum in the new voter set 1,2,3,4,5) and 2,3 can elect a leader among themselves, say 2 (because 2,3 is a quorum in the old voter set 1,2,3) 5. now 1 and 2 both think that they are legitimate leaders and can commit new entries, resulting in divergent logs. To prevent this, in step 3 we don't add 5 as voter until the configuration with voter 4 is committed (or vice versa).
If we test reconfig with faulty network, nodes leaving the raft group are not guaranteed to be up to date because they might have left the group before they have been fully recovered. So we restrict the after-reconfig checks to target nodes.
ztlpn
force-pushed
the
raft-voter-add-divergence
branch
from
86c526f
to
cea0c6c
Compare
mmaslankaprv
approved these changes
Apr 11, 2024
|
/backport v23.3.x |
|
/backport v23.2.x |
|
Failed to create a backport PR to v23.2.x branch. I tried: |
|
Failed to create a backport PR to v23.3.x branch. I tried: |
This was referenced Apr 11, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Otherwise we may add several new voters in quick succession, that the old voters will not know of, resulting in a possibility of non-intersecting quorums.
Example scenario:
To prevent this, in step 3 we don't add 5 as voter until the configuration with voter 4 is committed (or vice versa).
Backports Required
Release Notes
Bug Fixes