Make Leader Election timeout configurable #3263
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
thefringeninja
requested changes
Nov 3, 2021
hayley-jean
force-pushed
the
hayley-jean/make-election-timeout-configurable
branch
from
3f1f715
to
e0c302c
Compare
thefringeninja
previously approved these changes
Nov 3, 2021
shaan1337
reviewed
Nov 8, 2021
shaan1337
previously approved these changes
Nov 8, 2021
There was a problem hiding this comment.
Noticed an issue which isn't directly related to this PR:
Setting --leader-election-timeout-ms to 1 millisecond (I've commented out the >= 1 second check) then making two nodes elect a leader crashes both nodes with the following error:
[11102Unhandled exception. , 4,11:09:32.628,FTL] Global Unhandled Exception occurred.
System.NullReferenceException: Object reference not set to an instance of an object.
at Grpc.Net.Client.Internal.GrpcCall`2.DeadlineExceededCallback(Object state)
at System.Threading.TimerQueueTimer.<>c.<.cctor>b__23_0(Object state)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
at System.Threading.TimerQueueTimer.CallCallback(Boolean isThreadPool)
at System.Threading.TimerQueueTimer.Fire(Boolean isThreadPool)
at System.Threading.TimerQueue.FireNextTimers()
at System.Threading.TimerQueue.AppDomainTimerCallback(Int32 id)
bb44607
- Adds the LeaderElectionTimeoutMs option which configures the LiveUntil and Deadlines of messages in ElectionsService - Only accept the LiveUntil timespan on messages to send over gRPC. The LiveUntil will be used for the deadline - Don't use the GossipInterval as a LiveUntil in the GossipService Co-authored-by: Shaan Nobee <sniper111@gmail.com>
hayley-jean
force-pushed
the
hayley-jean/make-election-timeout-configurable
branch
from
bb44607
to
2a69cef
Compare
shaan1337
approved these changes
Nov 8, 2021
thefringeninja
approved these changes
Nov 8, 2021
thefringeninja
deleted the
hayley-jean/make-election-timeout-configurable
branch
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Added: LeaderElectionTimeoutMs option to allow configuring the timeout for election messages.
Backports PR #3121 to 20.10.
Fixes https://github.com/EventStore/home/issues/605.
Adds the
LeaderElectionTimeoutMsoption which configures the LiveUntil and Deadlines of messages in ElectionsService