-
Notifications
You must be signed in to change notification settings - Fork 2.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
raftstore: add a concurrency limiter for receiving snapshots #17015
Conversation
This commit starts to introduce a snapshot precheck mechanism to reduce unnecessary snapshot drops and generations. The mechanism functions as follows: Before a leader sends a snapshot to a follower, it first sends a precheck message. This message serves as a preliminary inquiry to the follower, seeking confirmation of its readiness to receive a snapshot. Upon receiving the message, the follower consults its concurrency limiter and returns a response to the leader. The leader will only proceed to generate the snapshot after the precheck is passed. A passed precheck means the leader has reserved a spot on the follower so the subsequent snapshot send should succeed. The reservation has a TTL and the leader is supposed to complete the snapshot generation and transmission within the TTL timeframe. Note that this commit implements the concurrency limiter without actually using it. A follow-up commit will update the snapshot sending process and make use of this concurrency limiter. Signed-off-by: Bisheng Huang <hbisheng@gmail.com>
[REVIEW NOTIFICATION] This pull request has been approved by:
To complete the pull request process, please ask the reviewers in the list to review by filling The full list of commands accepted by this bot can be found here. Reviewer can indicate their review by submitting an approval review. |
Hi @hbisheng. Thanks for your PR. I'm waiting for a tikv member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
This commit implements a snapshot precheck process that leverages the snapshot concurrency limiter introduced in tikv#17015. The leader only proceeds to generate the snapshot after it receives a precheck succeed message from the follower. The precheck request and response are sent via the raft `ExtraMessage` where two new message types are added in kvproto. TODO: 1. Handle the case where the follower is on an old version without the snapshot precheck API. 2. Throttle the snapshot precheck requests, if necessary. Signed-off-by: Bisheng Huang <hbisheng@gmail.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
/cc @overvenus could you help review this? The approach right now is similar to what we discussed last week except that the follower would not return a string token and would not ask for a token when receiving the snapshot. The follower merely maintains a counter where each counted object has a TTL. I think this is sufficient for slowing down snapshot requests from the leader. Also, the existing |
Co-authored-by: Neil Shen <overvenus@gmail.com> Signed-off-by: Bisheng Huang <hbisheng@gmail.com>
/ok-to-test |
/test |
@overvenus: The
Use In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
/test all |
…ed to 0 when limit is 0 Signed-off-by: Bisheng Huang <hbisheng@gmail.com>
Signed-off-by: Bisheng Huang <hbisheng@gmail.com>
/merge |
@overvenus: It seems you want to merge this PR, I will help you trigger all the tests: /run-all-tests You only need to trigger
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository. |
This pull request has been accepted and is ready to merge. Commit hash: 801a02d
|
@hbisheng: Your PR was out of date, I have automatically updated it for you. If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository. |
This commit implements a snapshot precheck process that leverages the snapshot concurrency limiter introduced in tikv#17015. The leader only proceeds to generate the snapshot after it receives a precheck succeed message from the follower. The precheck request and response are sent via the raft `ExtraMessage` where two new message types are added in kvproto. TODO: 1. Handle the case where the follower is on an old version without the snapshot precheck API. 2. Throttle the snapshot precheck requests, if necessary. Signed-off-by: Bisheng Huang <hbisheng@gmail.com>
This commit implements a snapshot precheck process that leverages the snapshot concurrency limiter introduced in tikv#17015. The leader only proceeds to generate the snapshot after it receives a precheck succeed message from the follower. The precheck request and response are sent via the raft `ExtraMessage` where two new message types are added in kvproto. TODO: 1. Handle the case where the follower is on an old version without the snapshot precheck API. 2. Throttle the snapshot precheck requests, if necessary. Signed-off-by: Bisheng Huang <hbisheng@gmail.com>
…17019) close #15972 This commit implements a snapshot precheck process that leverages the snapshot concurrency limiter introduced in #17015. The leader only proceeds to generate the snapshot after it receives a precheck succeed message from the follower. The precheck request and response are sent via the raft `ExtraMessage` where two new message types are added in kvproto. Signed-off-by: Bisheng Huang <hbisheng@gmail.com> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
What is changed and how it works?
Issue Number: Ref #15972
What's Changed:
Related changes
pingcap/docs
/pingcap/docs-cn
:Check List
Tests
Side effects
Release note