Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add basic resharding types #4216

Merged
merged 8 commits into from
May 13, 2024
Merged

Add basic resharding types #4216

merged 8 commits into from
May 13, 2024

Conversation

ffuugoo
Copy link
Contributor

@ffuugoo ffuugoo commented May 10, 2024

Tracked in #4213.

This PR adds a few basic types/variants/fields that are required as the most fundamental building blocks for resharding.

TODO:

  • add basic structure with resharding state to Collection
  • add second hashring to ShardHolder::rings
  • add Resharding replica state

All Submissions:

  • Contributions should target the dev branch. Did you create your branch from dev?
  • Have you followed the guidelines in our Contributing document?
  • Have you checked to ensure there aren't other open Pull Requests for the same update/change?

New Feature Submissions:

  1. Does your submission pass tests?
  2. Have you formatted your code locally using cargo +nightly fmt --all command prior to submission?
  3. Have you checked your code using cargo clippy --all --all-features command?

Changes to Core Features:

  • Have you added an explanation of what your changes do and why you'd like us to include them?
  • Have you written new tests for your core changes, as applicable?
  • Have you successfully ran tests with your changes locally?

@ffuugoo ffuugoo requested a review from timvisee May 10, 2024 12:14
@ffuugoo
Copy link
Contributor Author

ffuugoo commented May 10, 2024

@timvisee Mostly done, I just need to analyze how ReplicaState is used in a few places to add proper handling for ReplicaState::Resharding. Help welcome. :D

@ffuugoo ffuugoo mentioned this pull request May 10, 2024
76 tasks
Properly (?) handle `ReplicaState::Resharding` everywhere (?)
Add missing documentation
Remove `ReplicaState::Resharding` from the OpenAPI spec
@ffuugoo ffuugoo marked this pull request as ready for review May 10, 2024 14:38
@@ -920,6 +922,9 @@ pub enum ReplicaState {
// Shard is undergoing recovery by an external node
// Normally rejects updates, accepts updates if force is true
Recovery,
// Points are being migrated to this shard as part of resharding
#[schemars(skip)]
Copy link
Member

@timvisee timvisee May 13, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 I think this is good enough for now to hide our stuff from APIs. The OpenAPI spec has not been changed in this PR.

Copy link
Member

@timvisee timvisee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It implements two things: a new shard replica state and a different hash ring configuration.

This can be done in two separate PRs, and I think we should prefer that. It allow us to discuss better and allows faster merging.

lib/collection/src/shards/shard_holder.rs Show resolved Hide resolved
lib/collection/src/collection/resharding.rs Outdated Show resolved Hide resolved
@ffuugoo ffuugoo merged commit dae311e into dev May 13, 2024
17 checks passed
@ffuugoo ffuugoo deleted the add-basic-resharding-types branch May 13, 2024 11:31
generall pushed a commit that referenced this pull request May 26, 2024
@timvisee timvisee modified the milestone: v1.11: resharding Jul 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants