Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Connect resharding finish operation #4503

Merged
merged 4 commits into from
Jun 20, 2024
Merged

Connect resharding finish operation #4503

merged 4 commits into from
Jun 20, 2024

Conversation

timvisee
Copy link
Member

@timvisee timvisee commented Jun 19, 2024

Tracked in: #4213
Depends on: #4507

When the resharding driver is done, properly end resharding and clean things up.

This is one of the necessary changes we need to complete a resharding operation front to end. I'd like this so we can start testing with it soon.

Tasks

All Submissions:

  • Contributions should target the dev branch. Did you create your branch from dev?
  • Have you followed the guidelines in our Contributing document?
  • Have you checked to ensure there aren't other open Pull Requests for the same update/change?

New Feature Submissions:

  1. Does your submission pass tests?
  2. Have you formatted your code locally using cargo +nightly fmt --all command prior to submission?
  3. Have you checked your code using cargo clippy --all --all-features command?

@timvisee timvisee requested a review from ffuugoo June 19, 2024 11:39
Comment on lines 268 to 269
// TODO(resharding): don't commit resharding here, commit through consensus in driver
self.commit_write_hashring(resharding_key.clone())?;
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is here until we propose hash ring changes through consensus in the driver. That'll be in a separate PR.

Copy link
Contributor

@ffuugoo ffuugoo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't like it, but OK. 😂

Though, I'd structure finish_resharding like this:

    pub fn finish_resharding(&mut self, resharding_key: ReshardKey) -> CollectionResult<()> {
        // TODO(resharding): don't commit resharding here, commit through consensus in driver
        self.commit_write_hashring(resharding_key.clone())?;

        self.check_resharding(&resharding_key, |_| {
            // TODO(resharding): Check resharding is in the correct state to finish resharding!
            Ok(())
        })?;

        self.resharding_state.write(|state| {
            state.take();
        })?;

        Ok(())
    }

I.e., put commit_write_hashring call before check_resharding.

@timvisee
Copy link
Member Author

timvisee commented Jun 19, 2024

put commit_write_hashring call before check_resharding.

Then it panics due to surrounding checks.

We can also implement the consensus calls for setting resharding states first, because then we don't have to do this.

@timvisee timvisee changed the base branch from dev to resharding-commit-hashrings June 19, 2024 14:32
@timvisee timvisee marked this pull request as draft June 19, 2024 14:33
@timvisee timvisee force-pushed the resharding-commit-hashrings branch from 3ea6d23 to 7c525fe Compare June 19, 2024 14:37
@timvisee
Copy link
Member Author

I moved this on top of #4507 so that we don't need the temporary 'commit write' hack anymore.

This now just connects the finish resharding consensus message.

Base automatically changed from resharding-commit-hashrings to dev June 19, 2024 15:22
timvisee and others added 2 commits June 20, 2024 09:19
...and tweak `commit_read`/`commit_write`/`finish` ops to work properly
@timvisee timvisee marked this pull request as ready for review June 20, 2024 07:19
@timvisee timvisee merged commit f107b98 into dev Jun 20, 2024
17 checks passed
@timvisee timvisee deleted the resharding-end branch June 20, 2024 08:07
generall pushed a commit that referenced this pull request Jun 21, 2024
* End resharding and clean up

* Track resharding stage in `ReshardState`...

...and tweak `commit_read`/`commit_write`/`finish` ops to work properly

* Refactor hashring check for migrating and deleting migrated points

* When propagating deletes, only wait for the last batch

---------

Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
@timvisee timvisee added this to the v1.11: resharding milestone Jul 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants