Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove clock tags when forwarding operations during reshading shard transfer #4516

Merged
merged 2 commits into from
Jun 21, 2024

Conversation

ffuugoo
Copy link
Contributor

@ffuugoo ffuugoo commented Jun 20, 2024

Tracked in #4213.

Resharding shard transfer initializes ForwardProxy that forwards incoming updates from local shard X to remote shard Y.

Our clock tag logic does not work well enough in such case (though, it still kinda works!), so we just remove clock tags during ForwardProxy::update, if wrapped_shard and remote_shard IDs are different.

TODO (for a follow-up PR):

When ForwardProxy forwards updates from one shard to another one, it should also "split" the operation with a hashring and only forward relevant part of the operation to the remote shard. (Or, alternatively, remote shard should split incoming operations and only apply relevant part.)

It seems a lot more annoying to implement, though, so I'll leave it for a separate PR.

All Submissions:

  • Contributions should target the dev branch. Did you create your branch from dev?
  • Have you followed the guidelines in our Contributing document?
  • Have you checked to ensure there aren't other open Pull Requests for the same update/change?

New Feature Submissions:

  1. Does your submission pass tests?
  2. Have you formatted your code locally using cargo +nightly fmt --all command prior to submission?
  3. Have you checked your code using cargo clippy --all --all-features command?

Changes to Core Features:

  • Have you added an explanation of what your changes do and why you'd like us to include them?
  • Have you written new tests for your core changes, as applicable?
  • Have you successfully ran tests with your changes locally?

…tes...

...to remote shard with a different shard id
@ffuugoo ffuugoo mentioned this pull request Jun 20, 2024
76 tasks
…ate`

Co-authored-by: Tim Visée <tim+github@visee.me>
@ffuugoo ffuugoo merged commit 77fb5cc into dev Jun 21, 2024
17 checks passed
@ffuugoo ffuugoo deleted the resharding-forward-proxy-clock-tag-fix branch June 21, 2024 10:26
generall pushed a commit that referenced this pull request Jun 21, 2024
…ransfer (#4516)

Co-authored-by: Tim Visée <tim+github@visee.me>
@timvisee timvisee added this to the v1.11: resharding milestone Jul 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants