quincy: rbd-mirror: clean up stale pool replayers and callouts better #57305
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The code in Mirror::update_pool_replayers() responsible for shutting down and removing stale pool replayers kicks in only in case the peer is removed, but not if the peer changes. However, the code responsible for (re)starting pool replayers in the same method _does_ create and start a new pool replayer in that case. As a result, we can end up with nearly identical pool replayers running at the same time, hogging OS resources and confusing instance_id tracking logic and mirror status reporting at the very least. The root cause is that PeerSpec is matched normally (i.e. based on all fields) when it comes to m_pool_replayers, and based only on UUID when it comes to pool_peers. This was missed in commit 5463e1a ("rbd-mirror: extract optional peer mon_host/key values from MON"). Signed-off-by: Ilya Dryomov <idryomov@gmail.com> (cherry picked from commit def93cb)
If a pool replayer is removed in an error state (e.g. after failing to connect to the remote cluster), its callout should be removed as well. Otherwise, the error would persist causing "daemon health: ERROR" status to be reported even after a new pool replayer is created and started successfully. Fixes: https://tracker.ceph.com/issues/65487 Signed-off-by: Ilya Dryomov <idryomov@gmail.com> (cherry picked from commit b7e7964)
ajarr
approved these changes
May 21, 2024
wait_for_replay_complete() doesn't wait for image status to get updated. This didn't matter previously because these tests are run on two different pools and nothing else was following. Signed-off-by: Ilya Dryomov <idryomov@gmail.com> (cherry picked from commit d1d8482) Conflicts: qa/workunits/rbd/rbd_mirror_bootstrap.sh [ commit 3fd8a03 ("qa/workunits/rbd: merge journal and snapshot test scripts") not in quincy ]
idryomov
force-pushed
the
wip-65487-quincy
branch
from
7beadb9
to
952f397
Compare
|
Forgot to adjust for the lack of |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
backport tracker: https://tracker.ceph.com/issues/65818
backport of #57082
parent tracker: https://tracker.ceph.com/issues/65487