[Dynamic Regions] Multi-RocksDB's tikv panic with resource leak detected: callback of index read

[lilinghai]

Bug Report

What version of TiKV are you using?

master

What operating system and CPU are you using?

Steps to reproduce

[FATAL] [[lib.rs:497](http://lib.rs:497/)] ["resource leak detected: callback of index read."] [backtrace="   0: tikv_util::set_panic_hook::{{closure}}\n             at /home/jenkins/agent/workspace/build-common/go/src/[github.com/pingcap/tikv/components/tikv_util/src/lib.rs:496:18\n](http://github.com/pingcap/tikv/components/tikv_util/src/lib.rs:496:18/n)   1: <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call\n             at /rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/[boxed.rs:2032](http://boxed.rs:2032/):9\n      std::panicking::rust_panic_with_hook\n             at /rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/[panicking.rs:692](http://panicking.rs:692/):13\n   2: std::panicking::begin_panic_handler::{{closure}}\n             at /rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/[panicking.rs:579](http://panicking.rs:579/):13\n   3: std::sys_common::backtrace::__rust_end_short_backtrace\n             at /rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/[backtrace.rs:137](http://backtrace.rs:137/):18\n   4: rust_begin_unwind\n             at /rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/[panicking.rs:575](http://panicking.rs:575/):5\n   5: core::panicking::panic_fmt\n             at /rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/[panicking.rs:65](http://panicking.rs:65/):14\n   6: <tikv_util::MustConsumeVec<T> as core::ops::drop::Drop>::drop\n             at /home/jenkins/agent/workspace/build-common/go/src/[github.com/pingcap/tikv/components/tikv_util/src/lib.rs:456:13\n](http://github.com/pingcap/tikv/components/tikv_util/src/lib.rs:456:13/n)      core::ptr::drop_in_place<tikv_util::MustConsumeVec<(kvproto::protos::raft_cmdpb::RaftCmdRequest,raftstore_v2::router::response_channel::BaseChannel<raftstore_v2::router::response_channel::QueryResult>,core::option::Option<u64>)>>\n             at /rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ptr/[mod.rs:490](http://mod.rs:490/):1\n   7: core::ptr::drop_in_place<raftstore::store::read_queue::ReadIndexRequest<raftstore_v2::router::response_channel::BaseChannel<raftstore_v2::router::response_channel::QueryResult>>>\n             at /rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ptr/[mod.rs:490](http://mod.rs:490/):1\n   8: core::ptr::drop_in_place<[raftstore::store::read_queue::ReadIndexRequest<raftstore_v2::router::response_channel::BaseChannel<raftstore_v2::router::response_channel::QueryResult>>]>\n             at /rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ptr/[mod.rs:490](http://mod.rs:490/):1\n      <alloc::collections::vec_deque::VecDeque<T,A> as core::ops::drop::Drop>::drop\n             at /rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/collections/vec_deque/[mod.rs:161](http://mod.rs:161/):13\n      core::ptr::drop_in_place<alloc::collections::vec_deque::VecDeque<raftstore::store::read_queue::ReadIndexRequest<raftstore_v2::router::response_channel::BaseChannel<raftstore_v2::router::response_channel::QueryResult>>>>\n             at /rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ptr/[mod.rs:490](http://mod.rs:490/):1\n      core::ptr::drop_in_place<raftstore::store::read_queue::ReadIndexQueue<raftstore_v2::router::response_channel::BaseChannel<raftstore_v2::router::response_channel::QueryResult>>>\n             at /rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ptr/[mod.rs:490](http://mod.rs:490/):1\n   9: core::ptr::drop_in_place<raftstore_v2::raft::peer::Peer<engine_rocks::engine::RocksEngine,raft_log_engine::engine::RaftLogEngine>>\n             at /rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ptr/[mod.rs:490](http://mod.rs:490/):1\n  10: core::ptr::drop_in_place<raftstore_v2::fsm::peer::PeerFsm<engine_rocks::engine::RocksEngine,raft_log_engine::engine::RaftLogEngine>>\n             at /rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ptr/[mod.rs:490](http://mod.rs:490/):1\n  11: core::ptr::drop_in_place<alloc::boxed::Box<raftstore_v2::fsm::peer::PeerFsm<engine_rocks::engine::RocksEngine,raft_log_engine::engine::RaftLogEngine>>>\n             at /rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ptr/[mod.rs:490](http://mod.rs:490/):1\n      batch_system::fsm::FsmState<N>::release\n             at /home/jenkins/agent/workspace/build-common/go/src/[github.com/pingcap/tikv/components/batch-system/src/fsm.rs:152:50\n](http://github.com/pingcap/tikv/components/batch-system/src/fsm.rs:152:50/n)      batch_system::mailbox::BasicMailbox<Owner>::release\n             at /home/jenkins/agent/workspace/build-common/go/src/[github.com/pingcap/tikv/components/batch-system/src/mailbox.rs:54:9\n](http://github.com/pingcap/tikv/components/batch-system/src/mailbox.rs:54:9/n)      batch_system::batch::Batch<N,C>::release\n             at /home/jenkins/agent/workspace/build-common/go/src/[github.com/pingcap/tikv/components/batch-system/src/batch.rs:124:9\n](http://github.com/pingcap/tikv/components/batch-system/src/batch.rs:124:9/n)      batch_system::batch::Batch<N,C>::schedule\n             at /home/jenkins/agent/workspace/build-common/go/src/[github.com/pingcap/tikv/components/batch-system/src/batch.rs:167:51\n](http://github.com/pingcap/tikv/components/batch-system/src/batch.rs:167:51/n)  12: batch_system::batch::Poller<N,C,Handler>::poll\n             at /home/jenkins/agent/workspace/build-common/go/src/[github.com/pingcap/tikv/components/batch-system/src/batch.rs:445:17\n](http://github.com/pingcap/tikv/components/batch-system/src/batch.rs:445:17/n)  13: batch_system::batch::BatchSystem<N,C>::start_poller::{{closure}}\n             at /home/jenkins/agent/workspace/build-common/go/src/[github.com/pingcap/tikv/components/batch-system/src/batch.rs:550:17\n](http://github.com/pingcap/tikv/components/batch-system/src/batch.rs:550:17/n)      <std::thread::Builder as tikv_util::sys::thread::StdThreadBuildWrapper>::spawn_wrapper::{{closure}}\n             at /home/jenkins/agent/workspace/build-common/go/src/[github.com/pingcap/tikv/components/tikv_util/src/sys/thread.rs:415:23\n](http://github.com/pingcap/tikv/components/tikv_util/src/sys/thread.rs:415:23/n)      std::sys_common::backtrace::__rust_begin_short_backtrace\n             at /rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/[backtrace.rs:121](http://backtrace.rs:121/):18\n  14: std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}\n             at /rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/[mod.rs:551](http://mod.rs:551/):17\n      <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once\n             at /rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/panic/[unwind_safe.rs:271](http://unwind_safe.rs:271/):9\n      std::panicking::try::do_call\n             at /rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/[panicking.rs:483](http://panicking.rs:483/):40\n      std::panicking::try\n             at /rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/[panicking.rs:447](http://panicking.rs:447/):19\n      std::panic::catch_unwind\n             at /rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/[panic.rs:137](http://panic.rs:137/):14\n      std::thread::Builder::spawn_unchecked_::{{closure}}\n             at /rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/[mod.rs:550](http://mod.rs:550/):30\n      core::ops::function::FnOnce::call_once{{vtable.shim}}\n             at /rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/[function.rs:513](http://function.rs:513/):5\n  15: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once\n             at /rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/[boxed.rs:2000](http://boxed.rs:2000/):9\n      <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once\n             at /rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/[boxed.rs:2000](http://boxed.rs:2000/):9\n      std::sys::unix::thread::Thread::new::thread_start\n             at /rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/[thread.rs:108](http://thread.rs:108/):17\n  16: start_thread\n  17: clone\n"] [location=/home/jenkins/agent/workspace/build-common/go/src/[github.com/pingcap/tikv/components/tikv_util/src/lib.rs:456](http://github.com/pingcap/tikv/components/tikv_util/src/lib.rs:456)] [thread_name=rs-3-1]
store/TiDB master append auto-retry auto-retry-limit :default txn-mode optimistic isolation :read-committed nemesis partition-half,partition-pd-leader,partition-ring,schedules/20230228T123945.000Z/node-3.node-peer.jepsen-tps-1652555-1-201/kv.log:[2023/02/28 12:41:53.207 +00:00] [FATAL] [[lib.rs:497](http://lib.rs:497/)] ["resource leak detected: callback of index read."] [backtrace="   0: tikv_util::set_panic_hook::{{closure}}\n             at /home/jenkins/agent/workspace/build-common/go/src/[github.com/pingcap/tikv/components/tikv_util/src/lib.rs:496:18\n](http://github.com/pingcap/tikv/components/tikv_util/src/lib.rs:496:18/n)   1: <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call\n             at /rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/[boxed.rs:2032](http://boxed.rs:2032/):9\n      std::panicking::rust_panic_with_hook\n             at /rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/[panicking.rs:692](http://panicking.rs:692/):13\n   2: std::panicking::begin_panic_handler::{{closure}}\n             at /rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/[panicking.rs:579](http://panicking.rs:579/):13\n   3: std::sys_common::backtrace::__rust_end_short_backtrace\n             at /rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/[backtrace.rs:137](http://backtrace.rs:137/):18\n   4: rust_begin_unwind\n             at /rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/[panicking.rs:575](http://panicking.rs:575/):5\n   5: core::panicking::panic_fmt\n             at /rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/[panicking.rs:65](http://panicking.rs:65/):14\n   6: <tikv_util::MustConsumeVec<T> as core::ops::drop::Drop>::drop\n             at /home/jenkins/agent/workspace/build-common/go/src/[github.com/pingcap/tikv/components/tikv_util/src/lib.rs:456:13\n](http://github.com/pingcap/tikv/components/tikv_util/src/lib.rs:456:13/n)      core::ptr::drop_in_place<tikv_util::MustConsumeVec<(kvproto::protos::raft_cmdpb::RaftCmdRequest,raftstore_v2::router::response_channel::BaseChannel<raftstore_v2::router::response_channel::QueryResult>,core::option::Option<u64>)>>\n             at /rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ptr/[mod.rs:490](http://mod.rs:490/):1\n   7: core::ptr::drop_in_place<raftstore::store::read_queue::ReadIndexRequest<raftstore_v2::router::response_channel::BaseChannel<raftstore_v2::router::response_channel::QueryResult>>>\n             at /rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ptr/[mod.rs:490](http://mod.rs:490/):1\n   8: core::ptr::drop_in_place<[raftstore::store::read_queue::ReadIndexRequest<raftstore_v2::router::response_channel::BaseChannel<raftstore_v2::router::response_channel::QueryResult>>]>\n             at /rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ptr/[mod.rs:490](http://mod.rs:490/):1\n      <alloc::collections::vec_deque::VecDeque<T,A> as core::ops::drop::Drop>::drop\n             at /rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/collections/vec_deque/[mod.rs:161](http://mod.rs:161/):13\n      core::ptr::drop_in_place<alloc::collections::vec_deque::VecDeque<raftstore::store::read_queue::ReadIndexRequest<raftstore_v2::router::response_channel::BaseChannel<raftstore_v2::router::response_channel::QueryResult>>>>\n             at /rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ptr/[mod.rs:490](http://mod.rs:490/):1\n      core::ptr::drop_in_place<raftstore::store::read_queue::ReadIndexQueue<raftstore_v2::router::response_channel::BaseChannel<raftstore_v2::router::response_channel::QueryResult>>>\n             at /rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ptr/[mod.rs:490](http://mod.rs:490/):1\n   9: core::ptr::drop_in_place<raftstore_v2::raft::peer::Peer<engine_rocks::engine::RocksEngine,raft_log_engine::engine::RaftLogEngine>>\n             at /rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ptr/[mod.rs:490](http://mod.rs:490/):1\n  10: core::ptr::drop_in_place<raftstore_v2::fsm::peer::PeerFsm<engine_rocks::engine::RocksEngine,raft_log_engine::engine::RaftLogEngine>>\n             at /rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ptr/[mod.rs:490](http://mod.rs:490/):1\n  11: core::ptr::drop_in_place<alloc::boxed::Box<raftstore_v2::fsm::peer::PeerFsm<engine_rocks::engine::RocksEngine,raft_log_engine::engine::RaftLogEngine>>>\n             at /rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ptr/[mod.rs:490](http://mod.rs:490/):1\n      batch_system::fsm::FsmState<N>::release\n             at /home/jenkins/agent/workspace/build-common/go/src/[github.com/pingcap/tikv/components/batch-system/src/fsm.rs:152:50\n](http://github.com/pingcap/tikv/components/batch-system/src/fsm.rs:152:50/n)      batch_system::mailbox::BasicMailbox<Owner>::release\n             at /home/jenkins/agent/workspace/build-common/go/src/[github.com/pingcap/tikv/components/batch-system/src/mailbox.rs:54:9\n](http://github.com/pingcap/tikv/components/batch-system/src/mailbox.rs:54:9/n)      batch_system::batch::Batch<N,C>::release\n             at /home/jenkins/agent/workspace/build-common/go/src/[github.com/pingcap/tikv/components/batch-system/src/batch.rs:124:9\n](http://github.com/pingcap/tikv/components/batch-system/src/batch.rs:124:9/n)      batch_system::batch::Batch<N,C>::schedule\n             at /home/jenkins/agent/workspace/build-common/go/src/[github.com/pingcap/tikv/components/batch-system/src/batch.rs:167:51\n](http://github.com/pingcap/tikv/components/batch-system/src/batch.rs:167:51/n)  12: batch_system::batch::Poller<N,C,Handler>::poll\n             at /home/jenkins/agent/workspace/build-common/go/src/[github.com/pingcap/tikv/components/batch-system/src/batch.rs:445:17\n](http://github.com/pingcap/tikv/components/batch-system/src/batch.rs:445:17/n)  13: batch_system::batch::BatchSystem<N,C>::start_poller::{{closure}}\n             at /home/jenkins/agent/workspace/build-common/go/src/[github.com/pingcap/tikv/components/batch-system/src/batch.rs:550:17\n](http://github.com/pingcap/tikv/components/batch-system/src/batch.rs:550:17/n)      <std::thread::Builder as tikv_util::sys::thread::StdThreadBuildWrapper>::spawn_wrapper::{{closure}}\n             at /home/jenkins/agent/workspace/build-common/go/src/[github.com/pingcap/tikv/components/tikv_util/src/sys/thread.rs:415:23\n](http://github.com/pingcap/tikv/components/tikv_util/src/sys/thread.rs:415:23/n)      std::sys_common::backtrace::__rust_begin_short_backtrace\n             at /rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/[backtrace.rs:121](http://backtrace.rs:121/):18\n  14: std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}\n             at /rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/[mod.rs:551](http://mod.rs:551/):17\n      <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once\n             at /rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/panic/[unwind_safe.rs:271](http://unwind_safe.rs:271/):9\n      std::panicking::try::do_call\n             at /rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/[panicking.rs:483](http://panicking.rs:483/):40\n      std::panicking::try\n             at /rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/[panicking.rs:447](http://panicking.rs:447/):19\n      std::panic::catch_unwind\n             at /rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/[panic.rs:137](http://panic.rs:137/):14\n      std::thread::Builder::spawn_unchecked_::{{closure}}\n             at /rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/[mod.rs:550](http://mod.rs:550/):30\n      core::ops::function::FnOnce::call_once{{vtable.shim}}\n             at /rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/[function.rs:513](http://function.rs:513/):5\n  15: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once\n             at /rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/[boxed.rs:2000](http://boxed.rs:2000/):9\n      <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once\n             at /rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/[boxed.rs:2000](http://boxed.rs:2000/):9\n      std::sys::unix::thread::Thread::new::thread_start\n             at /rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/[thread.rs:108](http://thread.rs:108/):17\n  16: start_thread\n  17: clone\n"] [location=/home/jenkins/agent/workspace/build-common/go/src/[github.com/pingcap/tikv/components/tikv_util/src/lib.rs:456](http://github.com/pingcap/tikv/components/tikv_util/src/lib.rs:456)

What did you expect?

What did happened?