raft: Set the RecentActive flag for newly added nodes #7830
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
I found that enabling the CheckQuorum flag led to spurious leader elections when new nodes joined. It looks like in the time between a new node joining the cluster, and that node first communicating with the leader, the quorum check could fail because the new node looks inactive. To solve this, set the RecentActive flag when nodes are first added. This gives a grace period for the node to communicate before it causes the quorum check to fail. Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
|
Any thoughts on this propsed change? The extra leader elections when |
|
can you write a test for this? |
|
this change looks reasonable. |
This test verifies that adding a node does not cause the leader to step down until at least one full ElectionTick cycle elapses. Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
|
Thanks. I've added a test. |
|
LGTM. |
|
@aaronlehmann Thanks. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I found that enabling the
CheckQuorumflag led to spurious leader elections when new nodes joined. It looks like in the time between a new node joining the cluster, and that node first communicating with the leader, the quorum check could fail because the new node looks inactive. To solve this, set theRecentActiveflag when nodes are first added. This gives a grace period for the node to communicate before it causes the quorum check to fail.This seems like a decent way to solve the problem, but please feel free to suggest other approaches instead.