Add consensus operation to restart shard transfer #3703
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
timvisee
commented
Feb 27, 2024
Comment on lines
358
to
368
| // Abort and start transfer | ||
| self.handle_transfer( | ||
| collection_id.clone(), | ||
| ShardTransferOperations::Abort { | ||
| transfer: transfer.key(), | ||
| reason: "restart transfer".into(), | ||
| }, | ||
| ) | ||
| .await?; | ||
| self.handle_transfer(collection_id, ShardTransferOperations::Start(transfer)) | ||
| .await?; |
There was a problem hiding this comment.
Here I just call abort and start right after each other, as part of a single operation. It keeps the implementation very simple.
Maybe there's more to it though.
100 tasks
ffuugoo
approved these changes
Feb 28, 2024
generall
approved these changes
Feb 29, 2024
generall
reviewed
Feb 29, 2024
Comment on lines
+376
to
+379
| let mut new_transfer = transfer.clone(); | ||
| // Preserve sync flag from the old transfer | ||
| new_transfer.sync = old_transfer.sync; | ||
|
|
There was a problem hiding this comment.
This is new stuff from my side, might require extra look
There was a problem hiding this comment.
Shall we do an OR, to sync if the old OR the new one has sync=true?
Suggested change
| let mut new_transfer = transfer.clone(); | |
| // Preserve sync flag from the old transfer | |
| new_transfer.sync = old_transfer.sync; | |
| // Preserve sync flag from the old transfer | |
| let mut new_transfer = transfer.clone(); | |
| new_transfer.sync |= old_transfer.sync; | |
There was a problem hiding this comment.
new_transfer.sync = old_transfer.sync || new_transfer.sync. Don't use bitwise ops for logic.
There was a problem hiding this comment.
I've accidentally hit the green button.
I'm handling this here: #3728
6 tasks
timvisee
added a commit
that referenced
this pull request
Mar 5, 2024
* Add consensus operation to restart shard transfer with different config * Require shard transfer restart to have a changed configuration * implement api --------- Co-authored-by: generall <andrey@vasnetsov.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Tracked in: #3477
Add a consensus operation to restart a shard transfer with a different configuration.
This will make falling back to a different shard transfer method a whole lot nicer. We make sure to properly clean up and prepare for the different shard transfer method. Without it, we're manually managing and update shard replica set states to make transfers happen which is very fragile. Restarting this way also makes sure that the actual current transfer method and progress is properly reported.
The operation ensures that:
It is similar to calling abort/start right after each other, but in a single consensus operation.
If we'll merge this in 1.8, we'll be able to make use of it in Qdrant 1.9.
All Submissions:
devbranch. Did you create your branch fromdev?New Feature Submissions:
cargo +nightly fmt --allcommand prior to submission?cargo clippy --all --all-featurescommand?Changes to Core Features: