r/consensus: fixed handling replicate result when term changed #3003
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mmaslankaprv
requested review from
VadimPlh,
jcsp,
rystsov and
ztlpn
as code owners
mmaslankaprv
force-pushed
the
fix-replicate-when-term-changed
branch
from
2c2b69b to
74c78dc
Compare
jcsp
reviewed
Nov 18, 2021
Fixed problem that may lead to situation in which replication might be claimed successful but replicated entries were truncated. This might lead to consistency violation. Consider an example: ``` F1 - fiber processing replicate in replicate entries STM F2 - fiber processing append entries request F1 - append to leader log F1 - dispatch requests to followers and flush on leader F1 - wait for dispatches and flush F2 - processing new append entries with term greater than current F2 - grab oplock F2 - new term detected, step down, update term F1 - check stop condition, detected term update F1 - check term of self appended entry matches stored term -> TRUE F1 - claim successful replication F2 - truncate log, new leader log doesn't match <- violation ``` The problem was caused by the fact that `raft::replicate_entries_stm` did not wait for committed index update before evaluation replication results. This way a result might have been calculated based on the incorrect assumptions. Waiting for actual update of committed index before proceeding to result calculation give us a guarantee that all the state was update to decide if replication was successful. Signed-off-by: Michal Maslanka <michal@vectorized.io>
mmaslankaprv
force-pushed
the
fix-replicate-when-term-changed
branch
from
74c78dc to
c230235
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixed problem that may lead to situation in which replication might be
claimed successful but replicated entries were truncated. This might
lead to consistency violation.
Consider an example:
The problem was caused by the fact that
raft::replicate_entries_stmdid not wait for committed index update before evaluation replication results. This way a result might have been calculated based on the incorrect assumptions. Waiting for actual update of committed index before proceeding to result calculation give us a guarantee that all the state was update to decide if replication was successful.Signed-off-by: Michal Maslanka michal@vectorized.io