storage: precheck whether the peer is leader when acquiring latches failed (#13254)

[ti-srebot]

cherry-pick #13254 to release-6.1
You can switch your code base to this Pull Request by using git-extras:

# In tikv repo:
git pr https://github.com/tikv/tikv/pull/13318

After apply modifications, you can push your change to this PR via:

git push git@github.com:ti-srebot/tikv.git pr/13318:release-6.1-58fa80e0de0d

---

What is changed and how it works?

Issue Number: Close #12966

What's Changed:

When a tikv is isolated from other tikv instances, some requests will be 
blocked in raftstore and the corresponding latches are not released. 
Following requests which require the latches will receive ServerIsBusy error
and keep retrying. However, In such case, peers on the tikv are not leader
anymore. The client is supposed to receive NotLeader error immediately.

This commit introduces fail fast mode to scheduler. When a request 
fails to acquire any latch, scheduler checks if the peer is still leader.
If it still the leader, schedule the request as usual, fail fast otherwise.

Related changes

• PR to update pingcap/docs/pingcap/docs-cn:
• Need to cherry-pick to the release branch

Check List

Tests

• Unit test
  • latch
  • storage
• Integration test
  • raftkv
• Manual test (add detailed scripts or steps below)
  • Check issue one az is isolated from other az networks, and the service is unavailable for 4 minutes #12966

Now

Release note

Fix the problem that QPS may drop to zero for several mintues when a tikv is partitioned.

[ti-chi-bot]

[REVIEW NOTIFICATION]

This pull request has been approved by:

• sticnarf
• tonyxuqqi

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

[ti-srebot]

/run-all-tests

[ti-srebot]

@cosven please accept the invitation then you can push to the cherry-pick pull requests. Comment with /cherry-pick-invite if the invitation is expired.
https://github.com/ti-srebot/tikv/invitations

[cosven]

Conflicts are resolved and this PR is ready for review.

/cc @sticnarf @tonyxuqqi @BusyJay

Compared to the original PR, this PR has little change. You can see all changes in this commit f1a52c0 .

[sticnarf]

LGTM

[BusyJay]

/merge

[ti-chi-bot]

@BusyJay: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

[ti-chi-bot]

This pull request has been accepted and is ready to merge.

Commit hash: bd001fa