rbd-mirror: generally skip replay/resync if remote image is not primary #46759
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Initialize local_promotion_state and remote_promotion_state to UNKNOWN instead of counterintuitive PRIMARY and NON_PRIMARY -- half the time the final values are flipped. Then is_local_primary() and is_linked() can be strengthened as a non-existent image should stay in UNKNOWN. Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
Replay and resync should generally be skipped if the remote image is not primary. If this is not done for replay, snapshot-based mirroring can run into a livelock if the primary image is demoted while a mirror snapshot is being synced. On the demote site, rbd-mirror would pick up the just demoted image, grab the exclusive lock on it and idle waiting for a new mirror snapshot to be created. On the (still) non-primary site, rbd-mirror would eventually finish syncing that mirror snapshot and attempt to unlink from it on the demote site. These attempts would fail with EROFS due to exclusive lock being held in the "refuse proxied maintenance operations" mode, blocking forward progress (syncing of the demotion snapshot so that the non-primary image can be orderly promoted to primary, etc). If this is not done for resync, data loss can ensue as the just demoted image would be immediately trashed, underneath the non-primary site that is still syncing. Currently this is done in PrepareReplayRequest only for journal-based mirroring. Note that it is conditional: if the local image is linked to the remote image, proceeding is desirable. Generalize this check, consolidate it with a related check in PrepareRemoteImageRequest and move the result to BootstrapRequest to cover both "local image does not exist" and "local image is unlinked" cases for both modes. Fixes: https://tracker.ceph.com/issues/54448 Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
Now unused (and if it was used, the entire StateBuilder is passed in anyway). Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
It was botched in commit 2bca9ee ("rbd-mirror: consolidate prepare local/remote image steps to bootstrap") and went unnoticed because currently no special handling is needed for disconnected clients -- is_disconnected() check happens to be the last step and it doesn't generate an error. Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
There is a difference: non-primary means NON_PRIMARY promotion state, while "not primary" can refer to any of NON_PRIMARY, ORPHAN or UNKNOWN promotion states. Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
|
jenkins test windows |
MrFreezeex
approved these changes
Jun 21, 2022
There was a problem hiding this comment.
This seems right! I also did some changes in this part of the code in #43359 and I think it should be a bit cleaner with this new approach for the multipeer work. It might even (I have little hope though as I also have some on the journal mode) fix some bugs I have there.
ideepika
approved these changes
Jun 21, 2022
There was a problem hiding this comment.
tested with reliable reproducer https://tracker.ceph.com/attachments/download/6024/test_many.sh it resolves the issue manifesting as read-only filesystem error, eventually leading to failure in peer unlinking.
This was referenced Jun 22, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Contribution Guidelines
To sign and title your commits, please refer to Submitting Patches to Ceph.
If you are submitting a fix for a stable branch (e.g. "pacific"), please refer to Submitting Patches to Ceph - Backports for the proper workflow.
Checklist
Show available Jenkins commands
jenkins retest this pleasejenkins test classic perfjenkins test crimson perfjenkins test signedjenkins test make checkjenkins test make check arm64jenkins test submodulesjenkins test dashboardjenkins test dashboard cephadmjenkins test apijenkins test docsjenkins render docsjenkins test ceph-volume alljenkins test ceph-volume toxjenkins test windows