Do not run check in cets_discovery on nodedown #50
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
When a node goes down, it would be kicked from other nodes in the cluster by global. This process in not instant. This process happens even if node is alive and could reconnect. Reconnecting too fast would interfere with prevent_overlapped_partitions logic though. We still allow for this node to reconnect, but only during the regular checks. The change temporary puts the node into unavalable_nodes list though
This allows us to try to reconnect to the remote node after some period of time after the netsplit. Enters regular phase once we contact the remote node.
arcusfelis
force-pushed
the
less-checks
branch
from
ba68b94
to
89f62d1
Compare
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #50 +/- ##
==========================================
+ Coverage 98.33% 98.35% +0.02%
==========================================
Files 10 10
Lines 780 792 +12
==========================================
+ Hits 767 779 +12
Misses 13 13 ☔ View full report in Codecov by Sentry. |
arcusfelis
force-pushed
the
less-checks
branch
from
89f62d1
to
469f177
Compare
arcusfelis
changed the title
arcusfelis
force-pushed
the
less-checks
branch
from
cc8a9b7
to
eddf756
Compare
chrzaszcz
reviewed
Feb 28, 2024
Call peer to stop the processes Use proc_lib to spawn processes for better error reporting in tests
More reliable disconnect_node in tests Link disco process in tests
chrzaszcz
approved these changes
Mar 4, 2024
There was a problem hiding this comment.
Looks good in general, but cets_SUITE is becoming huge, and for me it's really hard to follow the changes there. IMO we need to extract some helpers to separate modules - especially that this suite is full of generic helpers. It should be a separate story - please don't extend this PR.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Patches for disco.
Do not call check on nodedown. This prevents immediate reconnect (which could confuse code inside global, trying to execute logic for prevent_overlapping_partitions).
If nodedown is received, wait 30 second before the next check. This allows the restarting node to initiate the connection on its own, without the whole cluster trying to connect to it.
Improve logging for when wait_for_ready fails.
Provide extra args into handle_down_fun: is_leader and remote_node