v5.2.4 upgrade to v6.1.0, tikv report panic for " [FATAL] [lib.rs:491] ["rocksdb background error. db: raft, reason: compaction, error: IO ..."

[seiya-annie]

Bug Report

What version of TiKV are you using?

Release Version: 6.1.0-alpha
Edition: Community
Git Commit Hash: f20a51d
Git Commit Branch: heads/refs/tags/v6.1.0
UTC Build Time: 2022-05-27 10:09:25
Rust Version: rustc 1.60.0-nightly (1e12aef3f 2022-02-13)
Enable Features: jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure
Profile: dist_release

What operating system and CPU are you using?

Steps to reproduce

1. install tidb cluster with config:
   tikv config:
   server.labels: {zone: "z1", host: "h5"}
   security.encryption.data-encryption-method: aes256-ctr
   security.encryption.master-key.path: /tiup/deploy/master_key
   security.encryption.master-key.type: file
2. upgrade to v6.1.0
3. check node status

What did you expect?

upgrade successfully.
all component works normally

What did happened?

[2022/05/30 14:35:31.898 +08:00] [FATAL] [lib.rs:491] ["rocksdb background error. db: raft, reason: compaction, error: IO error: No such file or directoryWhile open a file for random read: /tiup/data/tikv-20260/raft/000026.sst: No such file or directory"] [backtrace=" 0: tikv_util::set_panic_hook::{{closure}}\n at /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/components/tikv_util/src/lib.rs:490:18\n 1: std::panicking::rust_panic_with_hook\n at /rust/toolchains/nightly-2022-02-14-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:702:17\n 2: std::panicking::begin_panic_handler::{{closure}}\n at /rust/toolchains/nightly-2022-02-14-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:588:13\n 3: std::sys_common::backtrace::__rust_end_short_backtrace\n at /rust/toolchains/nightly-2022-02-14-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:138:18\n 4: rust_begin_unwind\n at /rust/toolchains/nightly-2022-02-14-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:584:5\n 5: core::panicking::panic_fmt\n at /rust/toolchains/nightly-2022-02-14-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/panicking.rs:143:14\n 6: <engine_rocks::event_listener::RocksEventListener as rocksdb::event_listener::EventListener>::on_background_error\n at /home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tikv/components/engine_rocks/src/event_listener.rs:150:13\n 7: rocksdb::event_listener::on_background_error\n at /rust/git/checkouts/rust-rocksdb-a9a28e74c6ead8ef/de8310c/src/event_listener.rs:334:5\n 8: _ZN24crocksdb_eventlistener_t17OnBackgroundErrorEN7rocksdb21BackgroundErrorReasonEPNS0_6StatusE\n at /rust/git/checkouts/rust-rocksdb-a9a28e74c6ead8ef/de8310c/librocksdb_sys/crocksdb/c.cc:2351:24\n 9: _ZN7rocksdb12EventHelpers23NotifyOnBackgroundErrorERKSt6vectorISt10shared_ptrINS_13EventListenerEESaIS4_EENS_21BackgroundErrorReasonEPNS_6StatusEPNS_17InstrumentedMutexEPb\n at /rust/git/checkouts/rust-rocksdb-a9a28e74c6ead8ef/de8310c/librocksdb_sys/rocksdb/db/event_helpers.cc:53:32\n 10: _ZN7rocksdb12ErrorHandler10SetBGErrorERKNS_6StatusENS_21BackgroundErrorReasonE\n at /rust/git/checkouts/rust-rocksdb-a9a28e74c6ead8ef/de8310c/librocksdb_sys/rocksdb/db/error_handler.cc:219:42\n 11: _ZN7rocksdb6DBImpl20BackgroundCompactionEPbPNS_10JobContextEPNS_9LogBufferEPNS0_19PrepickedCompactionENS_3Env8PriorityE\n at /rust/git/checkouts/rust-rocksdb-a9a28e74c6ead8ef/de8310c/librocksdb_sys/rocksdb/db/db_impl/db_impl_compaction_flush.cc:2799:30\n 12: _ZN7rocksdb6DBImpl24BackgroundCallCompactionEPNS0_19PrepickedCompactionENS_3Env8PriorityE\n at /rust/git/checkouts/rust-rocksdb-a9a28e74c6ead8ef/de8310c/librocksdb_sys/rocksdb/db/db_impl/db_impl_compaction_flush.cc:2319:72\n 13: _ZN7rocksdb6DBImpl16BGWorkCompactionEPv\n at /rust/git/checkouts/rust-rocksdb-a9a28e74c6ead8ef/de8310c/librocksdb_sys/rocksdb/db/db_impl/db_impl_compaction_flush.cc:2093:61\n 14: _ZNKSt8functionIFvvEEclEv\n at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/std_function.h:687:14\n _ZN7rocksdb14ThreadPoolImpl4Impl8BGThreadEm\n at /rust/git/checkouts/rust-rocksdb-a9a28e74c6ead8ef/de8310c/librocksdb_sys/rocksdb/util/threadpool_imp.cc:266:9\n 15: _ZN7rocksdb14ThreadPoolImpl4Impl15BGThreadWrapperEPv\n at /rust/git/checkouts/rust-rocksdb-a9a28e74c6ead8ef/de8310c/librocksdb_sys/rocksdb/util/threadpool_imp.cc:307:15\n 16: execute_native_thread_routine\n 17: start_thread\n at /usr/src/debug/glibc-2.17-c758a686/nptl/pthread_create.c:307\n 18: \n at /usr/src/debug////////glibc-2.17-c758a686/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:111\n"] [location=components/engine_rocks/src/event_listener.rs:150] [thread_name=]
[2022/05/30 14:35:47.136 +08:00] [INFO] [lib.rs:79] ["Welcome to TiKV"]

[seiya-annie]

tikv.log.tar.gz

[seiya-annie]

clinic url: https://clinic.pingcap.com.cn/portal/#/orgs/33/clusters/7103411108942772330

[Lily2025]

/type bug
/severity Critical
/assign tabokie

[tabokie]

Automatically recovers after one panic, changing the severity to major.