Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

resolved_ts: unexpected panic when build_scan_task #12231

Closed
zyguan opened this issue Mar 22, 2022 · 2 comments · Fixed by #12244
Closed

resolved_ts: unexpected panic when build_scan_task #12231

zyguan opened this issue Mar 22, 2022 · 2 comments · Fixed by #12244
Labels
affects-5.3 This bug affects 5.3.x versions. affects-5.4 affects-6.0 severity/major type/bug Type: Issue - Confirmed a bug

Comments

@zyguan
Copy link
Contributor

zyguan commented Mar 22, 2022

Bug Report

What version of TiKV are you using?

master @ 9a43faf

What operating system and CPU are you using?

kubernetes

Steps to reproduce

run jepsen sequential workload with the following options:

 {:auto-retry-limit :default,
 :ssh-private-key "jepsen.pem",
 :isolation :repeatable-read,
 :concurrency 10,
 :predicate-read false,
 :tarball-url
 "http://minio.pingcap.net:9000/tp-team/tests/jepsen/tidb-master.tar.gz",
 :txn-mode "pessimistic",
 :force-reinstall true,
 :update-in-place false,
 :ssh
 {:username "root",
  :password "root",
  :strict-host-key-checking false,
  :private-key-path "jepsen.pem"},
 :nemesis-long-recovery false,
 :follower-read true,
 :nemesis
 {:interval 10,
  :schedules true,
  :kill true,
  :stop true,
  :partition-one true},
 :nodes
 ["node-0.node-peer.jepsen-tps-662537-1-191"
  "node-1.node-peer.jepsen-tps-662537-1-191"
  "node-2.node-peer.jepsen-tps-662537-1-191"
  "node-3.node-peer.jepsen-tps-662537-1-191"
  "node-4.node-peer.jepsen-tps-662537-1-191"],
 :test-count 1,
 :read-lock "FOR UPDATE",
 :use-index false,
 :os #object[tidb.core.Image 0x19fec3d6 "tidb.core.Image@19fec3d6"],
 :time-limit 300,
 :recovery-time 10,
 :version "master",
 :auto-retry :default,
 :workload :sequential,
 :init-txn-sql
 ("set @@tidb_enable_async_commit = 0, @@tidb_enable_1pc = 0"
  "set @@tidb_enable_async_commit = 1, @@tidb_enable_1pc = 0"
  "set @@tidb_enable_async_commit = 1, @@tidb_enable_1pc = 1"),
 :init-sql
 ("set @@tidb_enable_mutation_checker=1, @@tidb_txn_assertion_level=strict")}

What did you expect?

Test passed.

What did happened?

TiKV panicked

[2022/03/21 10:00:44.118 +00:00] [FATAL] [lib.rs:468] ["called `Result::unwrap()` on an `Err` value: channel has been closed"] [backtrace="   0: tikv_util::set_panic_hook::{{closure}}\n             at /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/components/tikv_util/src/lib.rs:467:18\n   1: std::panicking::rust_panic_with_hook\n             at /rustc/1e12aef3fab243407f9d71ba9956cb2a1bf105d5/library/std/src/panicking.rs:702:17\n   2: std::panicking::begin_panic_handler::{{closure}}\n             at /rustc/1e12aef3fab243407f9d71ba9956cb2a1bf105d5/library/std/src/panicking.rs:588:13\n   3: std::sys_common::backtrace::__rust_end_short_backtrace\n             at /rustc/1e12aef3fab243407f9d71ba9956cb2a1bf105d5/library/std/src/sys_common/backtrace.rs:138:18\n   4: rust_begin_unwind\n             at /rustc/1e12aef3fab243407f9d71ba9956cb2a1bf105d5/library/std/src/panicking.rs:584:5\n   5: core::panicking::panic_fmt\n             at /rustc/1e12aef3fab243407f9d71ba9956cb2a1bf105d5/library/core/src/panicking.rs:143:14\n   6: core::result::unwrap_failed\n             at /rustc/1e12aef3fab243407f9d71ba9956cb2a1bf105d5/library/core/src/result.rs:1749:5\n   7: core::result::Result<T,E>::unwrap\n             at /rustc/1e12aef3fab243407f9d71ba9956cb2a1bf105d5/library/core/src/result.rs:1065:23\n      resolved_ts::endpoint::Endpoint<T,E,C>::build_scan_task::{{closure}}\n             at /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/components/resolved_ts/src/endpoint.rs:368:17\n   8: <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call\n             at /rustc/1e12aef3fab243407f9d71ba9956cb2a1bf105d5/library/alloc/src/boxed.rs:1868:9\n      resolved_ts::scanner::ScannerPool<T,E>::spawn_task::{{closure}}\n             at /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/components/resolved_ts/src/scanner.rs:96:25\n   9: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll\n             at /rustc/1e12aef3fab243407f9d71ba9956cb2a1bf105d5/library/core/src/future/mod.rs:84:19\n      tokio::runtime::task::core::CoreStage<T>::poll::{{closure}}\n             at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/core.rs:161:17\n      tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut\n             at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/loom/std/unsafe_cell.rs:14:9\n      tokio::runtime::task::core::CoreStage<T>::poll\n             at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/core.rs:151:13\n      tokio::runtime::task::harness::poll_future::{{closure}}\n             at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/harness.rs:467:19\n      <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once\n             at /rustc/1e12aef3fab243407f9d71ba9956cb2a1bf105d5/library/core/src/panic/unwind_safe.rs:271:9\n      std::panicking::try::do_call\n             at /rustc/1e12aef3fab243407f9d71ba9956cb2a1bf105d5/library/std/src/panicking.rs:492:40\n      std::panicking::try\n             at /rustc/1e12aef3fab243407f9d71ba9956cb2a1bf105d5/library/std/src/panicking.rs:456:19\n      std::panic::catch_unwind\n             at /rustc/1e12aef3fab243407f9d71ba9956cb2a1bf105d5/library/std/src/panic.rs:137:14\n      tokio::runtime::task::harness::poll_future\n             at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/harness.rs:455:18\n  10: tokio::runtime::task::harness::Harness<T,S>::poll_inner\n             at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/harness.rs:103:27\n      tokio::runtime::task::harness::Harness<T,S>::poll\n             at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/harness.rs:57:15\n      tokio::runtime::task::raw::poll\n             at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/raw.rs:128:5\n  11: tokio::runtime::task::raw::RawTask::poll\n             at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/raw.rs:80:18\n      tokio::runtime::task::LocalNotified<S>::run\n             at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/mod.rs:347:9\n      tokio::runtime::thread_pool::worker::Context::run_task::{{closure}}\n             at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/thread_pool/worker.rs:425:13\n      tokio::coop::with_budget::{{closure}}\n             at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/coop.rs:102:9\n      std::thread::local::LocalKey<T>::try_with\n             at /rustc/1e12aef3fab243407f9d71ba9956cb2a1bf105d5/library/std/src/thread/local.rs:413:16\n      std::thread::local::LocalKey<T>::with\n             at /rustc/1e12aef3fab243407f9d71ba9956cb2a1bf105d5/library/std/src/thread/local.rs:389:9\n      tokio::coop::with_budget\n             at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/coop.rs:95:5\n      tokio::coop::budget\n             at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/coop.rs:72:5\n      tokio::runtime::thread_pool::worker::Context::run_task\n             at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/thread_pool/worker.rs:424:9\n  12: tokio::runtime::thread_pool::worker::Context::run\n             at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/thread_pool/worker.rs:391:24\n      tokio::runtime::thread_pool::worker::run::{{closure}}\n             at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/thread_pool/worker.rs:376:17\n      tokio::macros::scoped_tls::ScopedKey<T>::set\n             at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/macros/scoped_tls.rs:61:9\n      tokio::runtime::thread_pool::worker::run\n             at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/thread_pool/worker.rs:373:5\n      tokio::runtime::thread_pool::worker::Launch::launch::{{closure}}\n             at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/thread_pool/worker.rs:352:45\n      <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll\n             at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/blocking/task.rs:42:21\n      tokio::runtime::task::core::CoreStage<T>::poll::{{closure}}\n             at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/core.rs:161:17\n      tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut\n             at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/loom/std/unsafe_cell.rs:14:9\n      tokio::runtime::task::core::CoreStage<T>::poll\n             at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/core.rs:151:13\n      tokio::runtime::task::harness::poll_future::{{closure}}\n             at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/harness.rs:467:19\n      <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once\n             at /rustc/1e12aef3fab243407f9d71ba9956cb2a1bf105d5/library/core/src/panic/unwind_safe.rs:271:9\n      std::panicking::try::do_call\n             at /rustc/1e12aef3fab243407f9d71ba9956cb2a1bf105d5/library/std/src/panicking.rs:492:40\n      std::panicking::try\n             at /rustc/1e12aef3fab243407f9d71ba9956cb2a1bf105d5/library/std/src/panicking.rs:456:19\n      std::panic::catch_unwind\n             at /rustc/1e12aef3fab243407f9d71ba9956cb2a1bf105d5/library/std/src/panic.rs:137:14\n      tokio::runtime::task::harness::poll_future\n             at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/harness.rs:455:18\n      tokio::runtime::task::harness::Harness<T,S>::poll_inner\n             at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/harness.rs:103:27\n      tokio::runtime::task::harness::Harness<T,S>::poll\n             at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/harness.rs:57:15\n      tokio::runtime::task::raw::poll\n             at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/raw.rs:128:5\n  13: tokio::runtime::task::raw::RawTask::poll\n             at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/raw.rs:80:18\n      tokio::runtime::task::UnownedTask<S>::run\n             at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/mod.rs:384:9\n      tokio::runtime::blocking::pool::Task::run\n             at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/blocking/pool.rs:91:9\n      tokio::runtime::blocking::pool::Inner::run\n             at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/blocking/pool.rs:308:17\n      tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}\n             at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/blocking/pool.rs:288:17\n      std::sys_common::backtrace::__rust_begin_short_backtrace\n             at /rustc/1e12aef3fab243407f9d71ba9956cb2a1bf105d5/library/std/src/sys_common/backtrace.rs:122:18\n  14: std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}\n             at /rustc/1e12aef3fab243407f9d71ba9956cb2a1bf105d5/library/std/src/thread/mod.rs:498:17\n      <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once\n             at /rustc/1e12aef3fab243407f9d71ba9956cb2a1bf105d5/library/core/src/panic/unwind_safe.rs:271:9\n      std::panicking::try::do_call\n             at /rustc/1e12aef3fab243407f9d71ba9956cb2a1bf105d5/library/std/src/panicking.rs:492:40\n      std::panicking::try\n             at /rustc/1e12aef3fab243407f9d71ba9956cb2a1bf105d5/library/std/src/panicking.rs:456:19\n      std::panic::catch_unwind\n             at /rustc/1e12aef3fab243407f9d71ba9956cb2a1bf105d5/library/std/src/panic.rs:137:14\n      std::thread::Builder::spawn_unchecked_::{{closure}}\n             at /rustc/1e12aef3fab243407f9d71ba9956cb2a1bf105d5/library/std/src/thread/mod.rs:497:30\n      core::ops::function::FnOnce::call_once{{vtable.shim}}\n             at /rustc/1e12aef3fab243407f9d71ba9956cb2a1bf105d5/library/core/src/ops/function.rs:227:5\n  15: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once\n             at /rustc/1e12aef3fab243407f9d71ba9956cb2a1bf105d5/library/alloc/src/boxed.rs:1854:9\n      <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once\n             at /rustc/1e12aef3fab243407f9d71ba9956cb2a1bf105d5/library/alloc/src/boxed.rs:1854:9\n      std::sys::unix::thread::Thread::new::thread_start\n             at /rustc/1e12aef3fab243407f9d71ba9956cb2a1bf105d5/library/std/src/sys/unix/thread.rs:108:17\n  16: start_thread\n  17: clone\n"] [location=/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/components/resolved_ts/src/endpoint.rs:374] [thread_name=inc-scan]
   0: tikv_util::set_panic_hook::{{closure}}
             at /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/components/tikv_util/src/lib.rs:467:18
   1: std::panicking::rust_panic_with_hook
             at /rustc/1e12aef3fab243407f9d71ba9956cb2a1bf105d5/library/std/src/panicking.rs:702:17
   2: std::panicking::begin_panic_handler::{{closure}}
             at /rustc/1e12aef3fab243407f9d71ba9956cb2a1bf105d5/library/std/src/panicking.rs:588:13
   3: std::sys_common::backtrace::__rust_end_short_backtrace
             at /rustc/1e12aef3fab243407f9d71ba9956cb2a1bf105d5/library/std/src/sys_common/backtrace.rs:138:18
   4: rust_begin_unwind
             at /rustc/1e12aef3fab243407f9d71ba9956cb2a1bf105d5/library/std/src/panicking.rs:584:5
   5: core::panicking::panic_fmt
             at /rustc/1e12aef3fab243407f9d71ba9956cb2a1bf105d5/library/core/src/panicking.rs:143:14
   6: core::result::unwrap_failed
             at /rustc/1e12aef3fab243407f9d71ba9956cb2a1bf105d5/library/core/src/result.rs:1749:5
   7: core::result::Result<T,E>::unwrap
             at /rustc/1e12aef3fab243407f9d71ba9956cb2a1bf105d5/library/core/src/result.rs:1065:23
      resolved_ts::endpoint::Endpoint<T,E,C>::build_scan_task::{{closure}}
             at /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/components/resolved_ts/src/endpoint.rs:368:17
   8: <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call
             at /rustc/1e12aef3fab243407f9d71ba9956cb2a1bf105d5/library/alloc/src/boxed.rs:1868:9
      resolved_ts::scanner::ScannerPool<T,E>::spawn_task::{{closure}}
             at /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/components/resolved_ts/src/scanner.rs:96:25
   9: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/1e12aef3fab243407f9d71ba9956cb2a1bf105d5/library/core/src/future/mod.rs:84:19
      tokio::runtime::task::core::CoreStage<T>::poll::{{closure}}
             at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/core.rs:161:17
      tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
             at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/loom/std/unsafe_cell.rs:14:9
      tokio::runtime::task::core::CoreStage<T>::poll
             at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/core.rs:151:13
      tokio::runtime::task::harness::poll_future::{{closure}}
             at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/harness.rs:467:19
      <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/1e12aef3fab243407f9d71ba9956cb2a1bf105d5/library/core/src/panic/unwind_safe.rs:271:9
      std::panicking::try::do_call
             at /rustc/1e12aef3fab243407f9d71ba9956cb2a1bf105d5/library/std/src/panicking.rs:492:40
      std::panicking::try
             at /rustc/1e12aef3fab243407f9d71ba9956cb2a1bf105d5/library/std/src/panicking.rs:456:19
      std::panic::catch_unwind
             at /rustc/1e12aef3fab243407f9d71ba9956cb2a1bf105d5/library/std/src/panic.rs:137:14
      tokio::runtime::task::harness::poll_future
             at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/harness.rs:455:18
  10: tokio::runtime::task::harness::Harness<T,S>::poll_inner
             at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/harness.rs:103:27
      tokio::runtime::task::harness::Harness<T,S>::poll
             at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/harness.rs:57:15
      tokio::runtime::task::raw::poll
             at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/raw.rs:128:5
  11: tokio::runtime::task::raw::RawTask::poll
             at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/raw.rs:80:18
      tokio::runtime::task::LocalNotified<S>::run
             at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/mod.rs:347:9
      tokio::runtime::thread_pool::worker::Context::run_task::{{closure}}
             at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/thread_pool/worker.rs:425:13
      tokio::coop::with_budget::{{closure}}
             at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/coop.rs:102:9
      std::thread::local::LocalKey<T>::try_with
             at /rustc/1e12aef3fab243407f9d71ba9956cb2a1bf105d5/library/std/src/thread/local.rs:413:16
      std::thread::local::LocalKey<T>::with
             at /rustc/1e12aef3fab243407f9d71ba9956cb2a1bf105d5/library/std/src/thread/local.rs:389:9
      tokio::coop::with_budget
             at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/coop.rs:95:5
      tokio::coop::budget
             at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/coop.rs:72:5
      tokio::runtime::thread_pool::worker::Context::run_task
             at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/thread_pool/worker.rs:424:9
  12: tokio::runtime::thread_pool::worker::Context::run
             at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/thread_pool/worker.rs:391:24
      tokio::runtime::thread_pool::worker::run::{{closure}}
             at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/thread_pool/worker.rs:376:17
      tokio::macros::scoped_tls::ScopedKey<T>::set
             at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/macros/scoped_tls.rs:61:9
      tokio::runtime::thread_pool::worker::run
             at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/thread_pool/worker.rs:373:5
      tokio::runtime::thread_pool::worker::Launch::launch::{{closure}}
             at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/thread_pool/worker.rs:352:45
      <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll
             at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/blocking/task.rs:42:21
      tokio::runtime::task::core::CoreStage<T>::poll::{{closure}}
             at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/core.rs:161:17
      tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
             at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/loom/std/unsafe_cell.rs:14:9
      tokio::runtime::task::core::CoreStage<T>::poll
             at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/core.rs:151:13
      tokio::runtime::task::harness::poll_future::{{closure}}
             at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/harness.rs:467:19
      <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/1e12aef3fab243407f9d71ba9956cb2a1bf105d5/library/core/src/panic/unwind_safe.rs:271:9
      std::panicking::try::do_call
             at /rustc/1e12aef3fab243407f9d71ba9956cb2a1bf105d5/library/std/src/panicking.rs:492:40
      std::panicking::try
             at /rustc/1e12aef3fab243407f9d71ba9956cb2a1bf105d5/library/std/src/panicking.rs:456:19
      std::panic::catch_unwind
             at /rustc/1e12aef3fab243407f9d71ba9956cb2a1bf105d5/library/std/src/panic.rs:137:14
      tokio::runtime::task::harness::poll_future
             at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/harness.rs:455:18
      tokio::runtime::task::harness::Harness<T,S>::poll_inner
             at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/harness.rs:103:27
      tokio::runtime::task::harness::Harness<T,S>::poll
             at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/harness.rs:57:15
      tokio::runtime::task::raw::poll
             at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/raw.rs:128:5
  13: tokio::runtime::task::raw::RawTask::poll
             at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/raw.rs:80:18
      tokio::runtime::task::UnownedTask<S>::run
             at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/task/mod.rs:384:9
      tokio::runtime::blocking::pool::Task::run
             at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/blocking/pool.rs:91:9
      tokio::runtime::blocking::pool::Inner::run
             at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/blocking/pool.rs:308:17
      tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}
             at /rust/registry/src/github.com-1ecc6299db9ec823/tokio-1.17.0/src/runtime/blocking/pool.rs:288:17
      std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/1e12aef3fab243407f9d71ba9956cb2a1bf105d5/library/std/src/sys_common/backtrace.rs:122:18
  14: std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}
             at /rustc/1e12aef3fab243407f9d71ba9956cb2a1bf105d5/library/std/src/thread/mod.rs:498:17
      <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/1e12aef3fab243407f9d71ba9956cb2a1bf105d5/library/core/src/panic/unwind_safe.rs:271:9
      std::panicking::try::do_call
             at /rustc/1e12aef3fab243407f9d71ba9956cb2a1bf105d5/library/std/src/panicking.rs:492:40
      std::panicking::try
             at /rustc/1e12aef3fab243407f9d71ba9956cb2a1bf105d5/library/std/src/panicking.rs:456:19
      std::panic::catch_unwind
             at /rustc/1e12aef3fab243407f9d71ba9956cb2a1bf105d5/library/std/src/panic.rs:137:14
      std::thread::Builder::spawn_unchecked_::{{closure}}
             at /rustc/1e12aef3fab243407f9d71ba9956cb2a1bf105d5/library/std/src/thread/mod.rs:497:30
      core::ops::function::FnOnce::call_once{{vtable.shim}}
             at /rustc/1e12aef3fab243407f9d71ba9956cb2a1bf105d5/library/core/src/ops/function.rs:227:5
  15: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/1e12aef3fab243407f9d71ba9956cb2a1bf105d5/library/alloc/src/boxed.rs:1854:9
      <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/1e12aef3fab243407f9d71ba9956cb2a1bf105d5/library/alloc/src/boxed.rs:1854:9
      std::sys::unix::thread::Thread::new::thread_start
             at /rustc/1e12aef3fab243407f9d71ba9956cb2a1bf105d5/library/std/src/sys/unix/thread.rs:108:17
  16: start_thread
  17: clone

full logs can be accessed here
@zyguan zyguan added type/bug Type: Issue - Confirmed a bug severity/major labels Mar 22, 2022
@you06
Copy link
Contributor

you06 commented Mar 22, 2022

The panic is caused by call unwrap on Err(...), @NingLin-P PTAL.

tikv/components/resolved_ts/src/endpoint.rs

Lines 367 to 375 in 9a43faf

on_error: Some(Box::new(move |observe_id, _region, e| {
scheduler_error
.schedule(Task::ReRegisterRegion {
region_id,
observe_id,
cause: format!("met error while handle scan task {:?}", e),
})
.unwrap();
RTS_SCAN_TASKS.with_label_values(&["abort"]).inc();

@NingLin-P
Copy link
Member

NingLin-P commented Mar 22, 2022 •
edited
Loading

The scheduler failed to schedule Task::ReRegisterRegion because the store is stopping, and likely the resolved-ts worker is stopped thus close the channel. From the logs:

[2022/03/21 10:00:43.801 +00:00] [INFO] [signal_handler.rs:19] ["receive signal 15, stopping server..."]
...
[2022/03/21 10:00:44.100 +00:00] [FATAL] [lib.rs:468] ["called `Result::unwrap()` on an `Err` value: channel has been closed"]

Perhaps we should print a warning log instead of unwrap when fail to schedule task /cc @5kbpers

5kbpers added a commit to 5kbpers/tikv that referenced this issue Mar 23, 2022
@5kbpers
close tikv#12231
402705d 
Signed-off-by: 5kbpers <tangminghua@pingcap.com>
@5kbpers 5kbpers mentioned this issue Mar 23, 2022
Closed
@NingLin-P NingLin-P mentioned this issue Mar 23, 2022
Merged
@ti-srebot ti-srebot mentioned this issue Mar 23, 2022
Merged
ti-chi-bot pushed a commit that referenced this issue Mar 23, 2022
@ti-srebot @NingLin-P
resolved_ts: fix shutdown panic (#12244) (#12245)
c30c2af 
close #12231, ref #12244

resolved_ts: fix shutdown panic

Signed-off-by: ti-srebot <ti-srebot@pingcap.com>

Co-authored-by: NingLin-P <linningde25@gmail.com>
@jebter jebter added affects-5.3 This bug affects 5.3.x versions. affects-5.4 affects-6.0 affects-6.1 and removed affects-6.1 labels Jun 22, 2022
This was referenced Jun 22, 2022
Merged
Closed
ti-chi-bot added a commit that referenced this issue Jun 23, 2022
@ti-srebot @NingLin-P @ti-chi-bot
resolved_ts: fix shutdown panic (#12244) (#12881)
37e8e2d 
close #12231, ref #12244

resolved_ts: fix shutdown panic

Signed-off-by: ti-srebot <ti-srebot@pingcap.com>

Co-authored-by: NingLin-P <linningde25@gmail.com>
Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
affects-5.3 This bug affects 5.3.x versions. affects-5.4 affects-6.0 severity/major type/bug Type: Issue - Confirmed a bug
Projects
None yet
Milestone
No milestone
4 participants
@jebter @zyguan @you06 @NingLin-P