box: disable split-brain detection until schema is upgraded #9050
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
andreyaksenov
approved these changes
Aug 28, 2023
grafin
reviewed
Sep 1, 2023
sergepetrenko
force-pushed
the
gh-8996-relax-split-brain-detection
branch
from
e005089
to
8f38270
Compare
grafin
approved these changes
Sep 7, 2023
Gerold103
approved these changes
Sep 13, 2023
sergepetrenko
added
full-ci
sergepetrenko
force-pushed
the
gh-8996-relax-split-brain-detection
branch
2 times, most recently
from
5f1d8c8
to
b1c8973
Compare
|
@Gerold103, @grafin, please take another look. I've noticed I never woke the fibers up. Fixed that and added a test case to avoid such errors. |
ochaplashkin
approved these changes
Sep 18, 2023
grafin
approved these changes
Sep 18, 2023
|
Great job on noticing the problem! |
Gerold103
approved these changes
Sep 26, 2023
test/replication-luatest/gh_8996_synchro_filter_enable_test.lua
Outdated
Show resolved
Hide resolved
Our split-brain detection machinery relies among other things on all nodes tracking the synchro queue confirmed lsn. This tracking was only added together with the split-brain detection. Only the synchro queue owner tracked the confirmed lsn before. This means that after an upgrade all the replicas remember the latest confirmed lsn as 0, and any PROMOTE/DEMOTE request from the queue owner is treated as a split brain. Let's fix this and only enable split-brain detection on the replica set once the schema version is updated. Thanks to the synchro queue freeze on restart, this can only happen after a new PROMOTE or DEMOTE entry is written by one of the nodes, and thus the correct confirmed lsn is propagated with this PROMOTE/DEMOTE to all the cluster members. Closes tarantool#8996 NO_DOC=bugfix
sergepetrenko
force-pushed
the
gh-8996-relax-split-brain-detection
branch
from
b1c8973
to
1d0e13f
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Our split-brain detection machinery relies among other things on all nodes tracking the synchro queue confirmed lsn. This tracking was only added together with the split-brain detection. Only the synchro queue owner tracked the confirmed lsn before.
This means that after an upgrade all the replicas remember the latest confirmed lsn as 0, and any PROMOTE/DEMOTE request from the queue owner is treated as a split brain.
Let's fix this and only enable split-brain detection on the replica set once the schema version is updated. Thanks to the synchro queue freeze on restart, this can only happen after a new PROMOTE or DEMOTE entry is written by one of the nodes, and thus the coorect confirmed lsn is propagated with this PROMOTE/DEMOTE to all the cluster members.
Closes #8996
NO_DOC=bugfix