raftstore: add a concurrency limiter for receiving snapshots

[hbisheng]

What is changed and how it works?

Issue Number: Ref #15972

What's Changed:

This commit starts to introduce a snapshot precheck mechanism to reduce
unnecessary snapshot drops and generations. The mechanism functions as follows: 

Before a leader sends a snapshot to a follower, it first sends a precheck
message. This message serves as a preliminary inquiry to the follower, seeking
confirmation of its readiness to receive a snapshot. Upon receiving the message,
the follower consults its concurrency limiter and returns a response to the
leader. The leader will only proceed to generate the snapshot after the precheck
is passed. 

A passed precheck means the leader has reserved a spot on the follower so the
subsequent snapshot send should succeed. The reservation has a TTL and the
leader is supposed to complete the snapshot generation and transmission within
the TTL timeframe.

Note that this commit implements the concurrency limiter without actually using
it. A follow-up commit will update the snapshot sending process and make use of
this concurrency limiter.

Related changes

• PR to update pingcap/docs/pingcap/docs-cn:
• Need to cherry-pick to the release branch

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No code

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Release note

None

[ti-chi-bot]

[REVIEW NOTIFICATION]

This pull request has been approved by:

• Connor1996
• overvenus

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

[ti-chi-bot]

Hi @hbisheng. Thanks for your PR.

I'm waiting for a tikv member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[Connor1996]

LGTM

[hbisheng]

/cc @overvenus could you help review this?

The approach right now is similar to what we discussed last week except that the follower would not return a string token and would not ask for a token when receiving the snapshot. The follower merely maintains a counter where each counted object has a TTL. I think this is sufficient for slowing down snapshot requests from the leader. Also, the existing concurrent_recv_snap_limit is still in place to protect the receiver from receiving too many snapshots.

[overvenus]

/ok-to-test

[overvenus]

/test

[ti-chi-bot]

@overvenus: The /test command needs one or more targets.
The following commands are available to trigger required jobs:

• /test pull-unit-test

Use /test all to run all jobs.

In response to this:

/test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[overvenus]

/test all

[overvenus]

/merge

[ti-chi-bot]

@overvenus: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

[ti-chi-bot]

This pull request has been accepted and is ready to merge.

Commit hash: 801a02d

[ti-chi-bot]

@hbisheng: Your PR was out of date, I have automatically updated it for you.

If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.