raftstore: fix the issue deadlock between apply snapshot and create peer (#10910)

[ti-srebot]

cherry-pick #10910 to release-5.2
You can switch your code base to this Pull Request by using git-extras:

# In tikv repo:
git pr https://github.com/tikv/tikv/pull/10918

After apply modifications, you can push your change to this PR via:

git push git@github.com:ti-srebot/tikv.git pr/10918:release-5.2-1e453231b7dd

---

Signed-off-by: nolouch nolouch@gmail.com

What problem does this PR solve?

Issue Number: close #10909

Problem Summary:

What is changed and how it works?

Introduced by: #10802
The deadlock happened between the apply snapshot and create new peer, we can confirm that the deadlock was caused by the locking sequence of store_meta and global_replication_state. as below stacks information shown, one thread call on_ready_apply_snapshot, another one call maybe_create_peer_internal.

#0  0x00007f2536da14ed in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f2536d9cdcb in _L_lock_883 () from /lib64/libpthread.so.0
#2  0x00007f2536d9cc98 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x000055a6ffc1505e in raftstore::store::fsm::peer::PeerFsmDelegate$LT$EK$C$ER$C$T$GT$::on_ready_apply_snapshot::hb8ba6dd7648d255a ()
#4  0x000055a6ffc104b6 in raftstore::store::fsm::peer::PeerFsmDelegate$LT$EK$C$ER$C$T$GT$::post_raft_ready_append::h1fcc32c1524e2da0 ()
#5  0x000055a6ffc278b2 in raftstore::store::fsm::store::RaftPoller$LT$EK$C$ER$C$T$GT$::handle_raft_ready_write::h01da8c1bdafe94bc ()
#6  0x000055a6ff94e036 in _$LT$raftstore..store..fsm..store..RaftPoller$LT$EK$C$ER$C$T$GT$$u20$as$u20$batch_system..batch..PollHandler$LT$raftstore..store..fsm..peer..PeerFsm$LT$EK$C$ER$GT$$C$raftstore..store..fsm..store..StoreFsm$LT$EK$GT$$GT$$GT$::end::ha140758f8328ea0d ()
#7  0x000055a6ff903e8e in batch_system::batch::Poller$LT$N$C$C$C$Handler$GT$::poll::h5c7b49aadf7c2a97 ()
#8  0x000055a6ffce20ae in std::sys_common::backtrace::__rust_begin_short_backtrace::h900083fbcc827393 ()
#9  0x000055a6ffcf8afd in std::panicking::try::do_call::hfbcd62ff2215291e ()
#10 0x000055a6ffe967ad in core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hb9b7ea1369649798 ()
#11 0x000055a7003b6897 in call_once<(), dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global> () at /rustc/2faabf579323f5252329264cc53ba9ff803429a3/library/alloc/src/boxed.rs:1572
#12 call_once<(), alloc::boxed::Box<dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global>, alloc::alloc::Global> () at /rustc/2faabf579323f5252329264cc53ba9ff803429a3/library/alloc/src/boxed.rs:1572
#13 std::sys::unix::thread::Thread::new::thread_start::hb71b17934c5f5e68 () at library/std/src/sys/unix/thread.rs:91
#14 0x00007f2536d9add5 in start_thread () from /lib64/libpthread.so.0
#15 0x00007f25363a3ead in clone () from /lib64/libc.so.6

#0  0x00007f2536da14ed in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f2536d9cdcb in _L_lock_883 () from /lib64/libpthread.so.0
#2  0x00007f2536d9cc98 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x000055a6ffc455c2 in raftstore::store::fsm::store::StoreFsmDelegate$LT$EK$C$ER$C$T$GT$::maybe_create_peer_internal::hb4a38ea8f75478f8 ()
#4  0x000055a6ffc385ac in raftstore::store::fsm::store::StoreFsmDelegate$LT$EK$C$ER$C$T$GT$::maybe_create_peer::h8f442248e8d62dfc ()
#5  0x000055a6ffc37521 in raftstore::store::fsm::store::StoreFsmDelegate$LT$EK$C$ER$C$T$GT$::on_raft_message::h4d43b5dfc6a12f45 ()
#6  0x000055a6ffc3453a in raftstore::store::fsm::store::StoreFsmDelegate$LT$EK$C$ER$C$T$GT$::handle_msgs::h6fa8c55f9c0a2df8 ()

What's Changed:

Related changes

• PR to update pingcap/docs/pingcap/docs-cn:
• PR to update pingcap/tidb-ansible:
• Need to cherry-pick to the release branch

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No code

Side effects

• Performance regression
  • Consumes more CPU
  • Consumes more MEM
• Breaking backward compatibility

Release note

raftstore: fix the deadlock issue in rafstore

[ti-chi-bot]

[REVIEW NOTIFICATION]

This pull request has been approved by:

• BusyJay
• zhouqiang-cl

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

[ti-srebot]

/run-all-tests

[ti-srebot]

@nolouch please accept the invitation then you can push to the cherry-pick pull requests. Comment with /cherry-pick-invite if the invitation is expired.
https://github.com/ti-srebot/tikv/invitations

[nolouch]

/test

[zhouqiang-cl]

/test

[zhouqiang-cl]

/merge

[ti-chi-bot]

@zhouqiang-cl: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

[ti-chi-bot]

@zhouqiang-cl: /merge in this pull request requires 2 approval(s).

In response to this:

/merge

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

[nolouch]

/test

[zhouqiang-cl]

/test

[zhouqiang-cl]

LGTM

[zhouqiang-cl]

/merge

[ti-chi-bot]

@zhouqiang-cl: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

[ti-chi-bot]

This pull request has been accepted and is ready to merge.

Commit hash: c3a583c

[NingLin-P]

LGTM

[nolouch]

/test