[DNM]reef: rgw: sync fairness bidding #52550
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Casey Bodley <cbodley@redhat.com> (cherry picked from commit 40c0179)
Signed-off-by: Casey Bodley <cbodley@redhat.com> (cherry picked from commit 43e95e6)
multisite metadata sync fairness
The approach of this commit is to allow multiple RGWs to participate in the multisite metadata sync.
Before this commit only single RGW has caught the all the sync locks.
This feature is using bidding algorithm.
For each lock, RGW is randomizing a number from 0 to shard count and for each shard is picking randomally one number and giving it as the bid_amount.
each one of those vectors each RGW handles are being sent using watch notify (based on RADOS watch notify).
Each time the RGW tries to lock it will compare its bid for the lock and the bids of other rgws, if the current RGW has the highest bid it will try to acquire the lock.
Important configs:
rgw_sync_work_period - For how long the RGW will sync until it will send unlock (very important in the beggining, because in the beginning only single RGW holds the locks)
rgw_sync_lease_period - not new to this commit but affecting it, For how many seconds the RGW will request from the RADOS to keep the lock, mainly important in case of failure, so automatically the RGW will lose a lock if it's down
Fixes: https://tracker.ceph.com/issues/41230
Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
Signed-off-by: Or Friedmann <ofriedma@ibm.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 38fa043)
Signed-off-by: Shilpa Jagannath <smanjara@redhat.com> (cherry picked from commit 6c99d33)
If we are outbid, exit after updating sync status. Signed-off-by: Shilpa Jagannath <smanjara@redhat.com> (cherry picked from commit 8ee42ab)
Signed-off-by: Shilpa Jagannath <smanjara@redhat.com> (cherry picked from commit 31f129f)
…) control flow - check for is_highest_bidder() before even attempting to take the lock - don't block on RGWMetaSyncShardNotifyCR() - other minor fixes Signed-off-by: Shilpa Jagannath <smanjara@redhat.com> (cherry picked from commit 14c4713)
Signed-off-by: Shilpa Jagannath <smanjara@redhat.com> (cherry picked from commit 410246f)
|
thanks @smanjara! i hadn't planned to take this for the initial reef release, but i wouldn't object to a later point release. i think it would help to clean up multisite tests first though, so we're better able to catch any regressions on reef |
yeah, makes sense @cbodley should I close the pr for now or leave it be for later rebase? |
|
@smanjara you can just add the DNM tag |
smanjara
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
backport tracker: https://tracker.ceph.com/issues/61172
backport of #45958
parent tracker: https://tracker.ceph.com/issues/41230
this backport was staged using ceph-backport.sh version 16.0.0.6848
find the latest version at https://github.com/ceph/ceph/blob/main/src/script/ceph-backport.sh