storage_service: node_ops_cmd_handler: decommission rollback fix #16508
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…the node if's already removed This is a regression after scylladb#15903. Before these changes del_leaving_endpoint took IP as a parameter and did nothing if it was called with a non-existent IP. The problem was revealed by the dtest test_remove_garbage_members_from_group0_after_abort_decommission[Announcing_that_I_have_left_the_ring-]. The test was flaky as in most cases the node died before the gossiper notification reached all the other nodes. To make it fail consistently and reproduce the problem one can move the info log 'Announcing that I have' after the sleep and add additional sleep after it in storage_service::leave_ring function. Fixes scylladb#16466
🟢 CI State: SUCCESS✅ - Build Build Details:
|
bhalevy
approved these changes
Dec 21, 2023
dgarcia360
pushed a commit
to dgarcia360/scylla
that referenced
this pull request
Apr 30, 2024
…the node if's already removed This is a regression after scylladb#15903. Before these changes del_leaving_endpoint took IP as a parameter and did nothing if it was called with a non-existent IP. The problem was revealed by the dtest test_remove_garbage_members_from_group0_after_abort_decommission[Announcing_that_I_have_left_the_ring-]. The test was flaky as in most cases the node died before the gossiper notification reached all the other nodes. To make it fail consistently and reproduce the problem one can move the info log 'Announcing that I have' after the sleep and add additional sleep after it in storage_service::leave_ring function. Fixes scylladb#16466 Closes scylladb#16508
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a regression after #15903. Before that PR
del_leaving_endpointtookIPas a parameter and did nothing if it was called with a non-existent IP. After the PRdel_leaving_endpointtakeshost_idas a parameter and we need to convert IP tohost_idbefore we can call it. The problem is that under certain conditions the node may be already removed and we can't do so.This patch restores original behaviour, we just do nothing if we can't find the node by
IP.The problem was revealed by the dtest test_remove_garbage_members_from_group0_after_abort_decommission[Announcing_that_I_have_left_the_ring-]. The test was flaky as in most cases the node died before the
gossiper notification reached all the other nodes. To make it fail consistently and reproduce the problem one can move the info log
Announcing that I haveafter the sleep and add additional sleep after it instorage_service::leave_ringfunction.Fixes #16466