tikv panic with fatal error when run ha-tidb-random-to-all-network-partition-last-for-10m

[Lily2025]

Bug Report

What version of TiKV are you using?

[2023/03/19 13:13:59.601 +08:00] [INFO] [lib.rs:91] ["Release Version: 6.7.0-alpha"]
[2023/03/19 13:13:59.601 +08:00] [INFO] [lib.rs:91] ["Edition: Community"]
[2023/03/19 13:13:59.601 +08:00] [INFO] [lib.rs:91] ["Git Commit Hash: 4baf9e7"]
[2023/03/19 13:13:59.601 +08:00] [INFO] [lib.rs:91] ["Git Commit Branch: heads/refs/tags/v6.7.0-alpha"]
[2023/03/19 13:13:59.601 +08:00] [INFO] [lib.rs:91] ["UTC Build Time: Unknown (env var does not exist when building)"]
[2023/03/19 13:13:59.601 +08:00] [INFO] [lib.rs:91] ["Rust Version: rustc 1.67.0-nightly (96ddd32c4 2022-11-14)"]
[2023/03/19 13:13:59.601 +08:00] [INFO] [lib.rs:91] ["Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure"]
[2023/03/19 13:13:59.601 +08:00] [INFO] [lib.rs:91] ["Profile: dist_release"]

What operating system and CPU are you using?

8c/32g

Steps to reproduce

https://tcms.pingcap.net/dashboard/executions/plan/1682471
ha-tidb-random-to-all-network-partition-last-for-10m

What did you expect?

tikv shoud not panic

What did happened?

[2023/03/19 13:13:54.604 +08:00] [FATAL] [lib.rs:497] ["Resolved future is not supposed to be polled again."] [backtrace=" 0: tikv_util::set_panic_hook::{{closure}}\n at home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/components/tikv_util/src/lib.rs:496:18\n 1: <alloc::boxed::Box<F,A> as core::ops::function::Fn>::call\n at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:2032:9\n std::panicking::rust_panic_with_hook\n at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:692:13\n 2: std::panicking::begin_panic::{{closure}}\n at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:608:9\n 3: std::sys_common::backtrace::__rust_end_short_backtrace\n at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:137:18\n 4: std::panicking::begin_panic\n at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:607:12\n 5: <grpcio::task::CqFuture as core::future::future::Future>::poll\n at rust/registry/src/github.com-1ecc6299db9ec823/grpcio-0.10.4/src/task/mod.rs:90:13\n 6: grpcio::call::SinkBase::poll_ready\n at rust/registry/src/github.com-1ecc6299db9ec823/grpcio-0.10.4/src/call/mod.rs:796:24\n 7: <grpcio::call::server::ServerStreamingSink as futures_sink::Sink<(T,grpcio::call::WriteFlags)>>::poll_close\n at rust/registry/src/github.com-1ecc6299db9ec823/grpcio-0.10.4/src/call/server.rs:562:28\n <&mut S as futures_sink::Sink>::poll_close\n at rust/registry/src/github.com-1ecc6299db9ec823/futures-sink-0.3.15/src/lib.rs:130:9\n <futures_util::sink::close::Close<Si,Item> as core::future::future::Future>::poll\n at rust/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.15/src/sink/close.rs:30:9\n backup_stream::checkpoint_manager::SubscriptionManager::remove_subscription::{{closure}}\n at home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/components/backup-stream/src/checkpoint_manager.rs:113:21\n <core::future::from_generator::GenFuture as core::future::future::Future>::poll\n at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/future/mod.rs:91:19\n backup_stream::checkpoint_manager::SubscriptionManager::emit_events::{{closure}}\n at home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/components/backup-stream/src/checkpoint_manager.rs:104:41\n <core::future::from_generator::GenFuture as core::future::future::Future>::poll\n at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/future/mod.rs:91:19\n backup_stream::checkpoint_manager::SubscriptionManager::main_loop::{{closure}}\n at home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/components/backup-stream/src/checkpoint_manager.rs:73:45\n <core::future::from_generator::GenFuture as core::future::future::Future>::poll\n at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/future/mod.rs:91:19\n tokio::runtime::task::core::CoreStage::poll::{{closure}}\n at rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/runtime/task/core.rs:184:17\n tokio::loom::std::unsafe_cell::UnsafeCell::with_mut\n at rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/loom/std/unsafe_cell.rs:14:9\n tokio::runtime::task::core::CoreStage::poll\n at rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/runtime/task/core.rs:174:13\n 8: tokio::runtime::task::harness::poll_future::{{closure}}\n at rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/runtime/task/harness.rs:480:19\n <core::panic::unwind_safe::AssertUnwindSafe as core::ops::function::FnOnce<()>>::call_once\n at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/panic/unwind_safe.rs:271:9\n std::panicking::try::do_call\n at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:483:40\n std::panicking::try\n at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:447:19\n std::panic::catch_unwind\n at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:137:14\n tokio::runtime::task::harness::poll_future\n at rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/runtime/task/harness.rs:468:18\n tokio::runtime::task::harness::Harness<T,S>::poll_inner\n at rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/runtime/task/harness.rs:104:27\n tokio::runtime::task::harness::Harness<T,S>::poll\n at rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/runtime/task/harness.rs:57:15\n tokio::runtime::task::raw::poll\n at rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/runtime/task/raw.rs:194:5\n 9: tokio::runtime::task::raw::RawTask::poll\n at rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/runtime/task/raw.rs:134:18\n tokio::runtime::task::LocalNotified::run\n at rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/runtime/task/mod.rs:385:9\n tokio::runtime::scheduler::multi_thread::worker::Context::run_task::{{closure}}\n at rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/runtime/scheduler/multi_thread/worker.rs:421:13\n tokio::coop::with_budget::{{closure}}\n at rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/coop.rs:102:9\n std::thread::local::LocalKey::try_with\n at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/local.rs:446:16\n std::thread::local::LocalKey::with\n at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/local.rs:422:9\n tokio::coop::with_budget\n at rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/coop.rs:95:5\n tokio::coop::budget\n at rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/coop.rs:72:5\n tokio::runtime::scheduler::multi_thread::worker::Context::run_task\n at rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/runtime/scheduler/multi_thread/worker.rs:420:9\n 10: tokio::runtime::scheduler::multi_thread::worker::Context::run\n at rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/runtime/scheduler/multi_thread/worker.rs:387:24\n tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}\n at rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/runtime/scheduler/multi_thread/worker.rs:372:17\n tokio::macros::scoped_tls::ScopedKey::set\n at rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/macros/scoped_tls.rs:61:9\n tokio::runtime::scheduler::multi_thread::worker::run\n at rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/runtime/scheduler/multi_thread/worker.rs:369:5\n tokio::runtime::scheduler::multi_thread::worker::Launch::launch::{{closure}}\n at rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/runtime/scheduler/multi_thread/worker.rs:348:45\n <tokio::runtime::blocking::task::BlockingTask as core::future::future::Future>::poll\n at rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/runtime/blocking/task.rs:42:21\n tokio::runtime::task::core::CoreStage::poll::{{closure}}\n at rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/runtime/task/core.rs:184:17\n tokio::loom::std::unsafe_cell::UnsafeCell::with_mut\n at rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/loom/std/unsafe_cell.rs:14:9\n tokio::runtime::task::core::CoreStage::poll\n at rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/runtime/task/core.rs:174:13\n tokio::runtime::task::harness::poll_future::{{closure}}\n at rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/runtime/task/harness.rs:480:19\n <core::panic::unwind_safe::AssertUnwindSafe as core::ops::function::FnOnce<()>>::call_once\n at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/panic/unwind_safe.rs:271:9\n std::panicking::try::do_call\n at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:483:40\n std::panicking::try\n at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:447:19\n std::panic::catch_unwind\n at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:137:14\n tokio::runtime::task::harness::poll_future\n at rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/runtime/task/harness.rs:468:18\n tokio::runtime::task::harness::Harness<T,S>::poll_inner\n at rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/runtime/task/harness.rs:104:27\n tokio::runtime::task::harness::Harness<T,S>::poll\n at rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/runtime/task/harness.rs:57:15\n tokio::runtime::task::raw::poll\n at rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/runtime/task/raw.rs:194:5\n 11: tokio::runtime::task::raw::RawTask::poll\n at rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/runtime/task/raw.rs:134:18\n tokio::runtime::task::UnownedTask::run\n at rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/runtime/task/mod.rs:422:9\n tokio::runtime::blocking::pool::Task::run\n at rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/runtime/blocking/pool.rs:111:9\n tokio::runtime::blocking::pool::Inner::run\n at rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/runtime/blocking/pool.rs:346:17\n tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}\n at rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.21.2/src/runtime/blocking/pool.rs:321:13\n std::sys_common::backtrace::rust_begin_short_backtrace\n at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:121:18\n 12: std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}}\n at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:551:17\n <core::panic::unwind_safe::AssertUnwindSafe as core::ops::function::FnOnce<()>>::call_once\n at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/panic/unwind_safe.rs:271:9\n std::panicking::try::do_call\n at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:483:40\n std::panicking::try\n at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:447:19\n std::panic::catch_unwind\n at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:137:14\n std::thread::Builder::spawn_unchecked::{{closure}}\n at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/mod.rs:550:30\n core::ops::function::FnOnce::call_once{{vtable.shim}}\n at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:513:5\n 13: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce>::call_once\n at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:2000:9\n <alloc::boxed::Box<F,A> as core::ops::function::FnOnce>::call_once\n at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/boxed.rs:2000:9\n std::sys::unix::thread::Thread::new::thread_start\n at rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/thread.rs:108:17\n 14: start_thread\n 15: __GI___clone\n"] [location=/rust/registry/src/github.com-1ecc6299db9ec823/grpcio-0.10.4/src/task/mod.rs:90] [thread_name=backup-stream]

[Lily2025]

/type bug
/severity critical

[Lily2025]

/assign joccau

[ti-chi-bot]

@Lily2025: GitHub didn't allow me to assign the following users: joccau.

Note that only tikv members with read permissions, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

In response to this:

/assign joccau

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[joccau]

/assign @YuJuncen

[joccau]

/remove-label may-affects-4.0
/remove-label may-affects-5.0
/remove-label may-affects-5.1
/remove-label may-affects-5.2
/remove-label may-affects-5.3
/remove-label may-affects-5.4
/remove-label may-affects-6.1
/remove-label may-affects-6.5

[joccau]

/remove-label may-affects-4.0

[joccau]

/remove-label may-affects-4.0
/remove-label may-affects-5.0
/remove-label may-affects-5.1
/remove-label may-affects-5.2
/remove-label may-affects-5.3
/remove-label may-affects-5.4
/remove-label may-affects-6.1
/remove-label may-affects-6.5

[ti-chi-bot]

@joccau: These labels are not set on the issue: may-affects-4.0.

In response to this:

/remove-label may-affects-4.0
/remove-label may-affects-5.0
/remove-label may-affects-5.1
/remove-label may-affects-5.2
/remove-label may-affects-5.3
/remove-label may-affects-5.4
/remove-label may-affects-6.1
/remove-label may-affects-6.5

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.