v6.2.0-alpha: change "data-encryption-method" from "aes192-ctr" to "plaintext" got failed

[jiayang-zheng]

Bug Report

What version of TiKV are you using?

Release Version: 6.2.0-alpha
Edition: Community

What operating system and CPU are you using?

X86_64

Steps to reproduce

1. set data-encryption-method = "aes192-ctr" and start tidb cluster using tiup;
2. write some IO using sysbench;
3. set data-encryption-method = "plaintext" then restart tikv node;
4. write IO using sysbench;

What did you expect?

disable encryption successfully

What did happened?

assert failed at rust/tikv/components/encryption/src/manager/mod.rs:620 OR rust/tikv/components/encryption/src/manager/mod.rs:650:17

[2022/07/20 22:53:51.140 +08:00] [FATAL] [lib.rs:494] ["assertion failed: file.iv.is_empty()"] [backtrace=" 0: tikv_util::set_panic_hook::{{closure}}\n at rust/tikv/components/tikv_util/src/lib.rs:493:18\n 1: std::panicking::rust_panic_with_hook\n at /rustc/7c4b47696907d64eff5621a64eb3c6e795a9ec77/library/std/src/panicking.rs:702:17\n 2: std::panicking::begin_panic_handler::{{closure}}\n at /rustc/7c4b47696907d64eff5621a64eb3c6e795a9ec77/library/std/src/panicking.rs:586:13\n 3: std::sys_common::backtrace::__rust_end_short_backtrace\n at /rustc/7c4b47696907d64eff5621a64eb3c6e795a9ec77/library/std/src/sys_common/backtrace.rs:138:18\n 4: rust_begin_unwind\n at /rustc/7c4b47696907d64eff5621a64eb3c6e795a9ec77/library/std/src/panicking.rs:584:5\n 5: core::panicking::panic_fmt\n at /rustc/7c4b47696907d64eff5621a64eb3c6e795a9ec77/library/core/src/panicking.rs:142:14\n 6: core::panicking::panic\n at /rustc/7c4b47696907d64eff5621a64eb3c6e795a9ec77/library/core/src/panicking.rs:48:5\n 7: encryption::manager::DataKeyManager::open_file_with_writer\n at rust/tikv/components/encryption/src/manager/mod.rs:620:17\n 8: <raft_log_engine::engine::ManagedFileSystem as raft_engine::env::FileSystem>::new_writer\n at rust/tikv/components/raft_log_engine/src/engine.rs:236:38\n 9: raft_engine::file_pipe_log::log_file::build_file_writer\n at .cargo/git/checkouts/raft-engine-35ec7b0b2c07ddd2/7a436ea/src/file_pipe_log/log_file.rs:36:18\n 10: raft_engine::file_pipe_log::pipe::SinglePipe::rotate_imp\n at .cargo/git/checkouts/raft-engine-35ec7b0b2c07ddd2/7a436ea/src/file_pipe_log/pipe.rs:174:21\n 11: raft_engine::file_pipe_log::pipe::SinglePipe::maybe_sync\n at .cargo/git/checkouts/raft-engine-35ec7b0b2c07ddd2/7a436ea/src/file_pipe_log/pipe.rs:260:29\n 12: <raft_engine::file_pipe_log::pipe::DualPipes as raft_engine::pipe_log::PipeLog>::maybe_sync\n at .cargo/git/checkouts/raft-engine-35ec7b0b2c07ddd2/7a436ea/src/file_pipe_log/pipe.rs:365:9\n 13: raft_engine::engine::Engine<F,P>::write\n at .cargo/git/checkouts/raft-engine-35ec7b0b2c07ddd2/7a436ea/src/engine.rs:169:33\n 14: <raft_log_engine::engine::RaftLogEngine as engine_traits::raft_engine::RaftEngine>::consume_and_shrink\n at rust/tikv/components/raft_log_engine/src/engine.rs:474:9\n 15: raftstore::store::async_io::write::Worker<EK,ER,N,T>::write_to_db\n at rust/tikv/components/raftstore/src/store/async_io/write.rs:524:13\n 16: <raftstore::store::fsm::store::RaftPoller<EK,ER,T> as batch_system::batch::PollHandler<raftstore::store::fsm::peer::PeerFsm<EK,ER>,raftstore::store::fsm::store::StoreFsm>>::end\n at rust/tikv/components/raftstore/src/store/fsm/store.rs:951:17\n 17: batch_system::batch::Poller<N,C,Handler>::poll\n at rust/tikv/components/batch-system/src/batch.rs:482:13\n 18: batch_system::batch::BatchSystem<N,C>::start_poller::{{closure}}\n at rust/tikv/components/batch-system/src/batch.rs:590:17\n 19: <std::thread::Builder as tikv_util::sys::thread::StdThreadBuildWrapper>::spawn_wrapper::{{closure}}\n at rust/tikv/components/tikv_util/src/sys/thread.rs:414:23\n 20: std::sys_common::backtrace::_rust_begin_short_backtrace\n at /rustc/7c4b47696907d64eff5621a64eb3c6e795a9ec77/library/std/src/sys_common/backtrace.rs:122:18\n 21: std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}}\n at /rustc/7c4b47696907d64eff5621a64eb3c6e795a9ec77/library/std/src/thread/mod.rs:500:17\n 22: <core::panic::unwind_safe::AssertUnwindSafe as core::ops::function::FnOnce<()>>::call_once\n at /rustc/7c4b47696907d64eff5621a64eb3c6e795a9ec77/library/core/src/panic/unwind_safe.rs:271:9\n 23: std::panicking::try::do_call\n at /rustc/7c4b47696907d64eff5621a64eb3c6e795a9ec77/library/std/src/panicking.rs:492:40\n 24: _rust_try\n 25: std::panicking::try\n at /rustc/7c4b47696907d64eff5621a64eb3c6e795a9ec77/library/std/src/panicking.rs:456:19\n 26: std::panic::catch_unwind\n at /rustc/7c4b47696907d64eff5621a64eb3c6e795a9ec77/library/std/src/panic.rs:137:14\n 27: std::thread::Builder::spawn_unchecked::{{closure}}\n at /rustc/7c4b47696907d64eff5621a64eb3c6e795a9ec77/library/std/src/thread/mod.rs:499:30\n 28: core::ops::function::FnOnce::call_once{{vtable.shim}}\n at /rustc/7c4b47696907d64eff5621a64eb3c6e795a9ec77/library/core/src/ops/function.rs:248:5\n 29: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce>::call_once\n at /rustc/7c4b47696907d64eff5621a64eb3c6e795a9ec77/library/alloc/src/boxed.rs:1866:9\n <alloc::boxed::Box<F,A> as core::ops::function::FnOnce>::call_once\n at /rustc/7c4b47696907d64eff5621a64eb3c6e795a9ec77/library/alloc/src/boxed.rs:1866:9\n std::sys::unix::thread::Thread::new::thread_start\n at /rustc/7c4b47696907d64eff5621a64eb3c6e795a9ec77/library/std/src/sys/unix/thread.rs:108:17\n 30: start_thread\n at /usr/src/debug/glibc-2.17-c758a686/nptl/pthread_create.c:307\n 31: gnu_dev_makedev\n at /usr/src/debug/glibc-2.17-c758a686/misc/../sysdeps/unix/sysv/linux/makedev.c:37\n"] [location=/home/jiayang/rust/tikv/components/encryption/src/manager/mod.rs:620] [thread_name=raftstore-1-0]

---

[2022/07/20 23:01:51.102 +08:00] [FATAL] [lib.rs:494] ["assertion failed: file.iv.is_empty()"] [backtrace=" 0: tikv_util::set_panic_hook::{{closure}}\n at rust/tikv/components/tikv_util/src/lib.rs:493:18\n 1: std::panicking::rust_panic_with_hook\n at /rustc/7c4b47696907d64eff5621a64eb3c6e795a9ec77/library/std/src/panicking.rs:702:17\n 2: std::panicking::begin_panic_handler::{{closure}}\n at /rustc/7c4b47696907d64eff5621a64eb3c6e795a9ec77/library/std/src/panicking.rs:586:13\n 3: std::sys_common::backtrace::__rust_end_short_backtrace\n at /rustc/7c4b47696907d64eff5621a64eb3c6e795a9ec77/library/std/src/sys_common/backtrace.rs:138:18\n 4: rust_begin_unwind\n at /rustc/7c4b47696907d64eff5621a64eb3c6e795a9ec77/library/std/src/panicking.rs:584:5\n 5: core::panicking::panic_fmt\n at /rustc/7c4b47696907d64eff5621a64eb3c6e795a9ec77/library/core/src/panicking.rs:142:14\n 6: core::panicking::panic\n at /rustc/7c4b47696907d64eff5621a64eb3c6e795a9ec77/library/core/src/panicking.rs:48:5\n 7: encryption::manager::DataKeyManager::open_file_with_reader\n at rust/tikv/components/encryption/src/manager/mod.rs:650:17\n 8: <raft_log_engine::engine::ManagedFileSystem as raft_engine::env::FileSystem>::new_reader\n at rust/tikv/components/raft_log_engine/src/engine.rs:220:38\n 9: raft_engine::file_pipe_log::log_file::build_file_reader\n at .cargo/git/checkouts/raft-engine-35ec7b0b2c07ddd2/7a436ea/src/file_pipe_log/log_file.rs:145:18\n 10: raft_engine::file_pipe_log::pipe_builder::DualPipesBuilder::recover_queue_imp::{{closure}}\n at .cargo/git/checkouts/raft-engine-35ec7b0b2c07ddd2/7a436ea/src/file_pipe_log/pipe_builder.rs:301:27\n 11: core::ops::function::impls::<impl core::ops::function::FnMut for &F>::call_mut\n at /rustc/7c4b47696907d64eff5621a64eb3c6e795a9ec77/library/core/src/ops/function.rs:268:13\n 12: core::ops::function::impls::<impl core::ops::function::FnOnce for &mut F>::call_once\n at /rustc/7c4b47696907d64eff5621a64eb3c6e795a9ec77/library/core/src/ops/function.rs:301:13\n 13: core::option::Option::map\n at /rustc/7c4b47696907d64eff5621a64eb3c6e795a9ec77/library/core/src/option.rs:909:29\n 14: <core::iter::adapters::map::Map<I,F> as core::iter::traits::iterator::Iterator>::next\n at /rustc/7c4b47696907d64eff5621a64eb3c6e795a9ec77/library/core/src/iter/adapters/map.rs:103:9\n 15: rayon::iter::plumbing::Folder::consume_iter\n at .cargo/registry/src/github.com-1ecc6299db9ec823/rayon-1.5.0/src/iter/plumbing/mod.rs:178:21\n 16: <rayon::iter::map::MapFolder<C,F> as rayon::iter::plumbing::Folder>::consume_iter\n at .cargo/registry/src/github.com-1ecc6299db9ec823/rayon-1.5.0/src/iter/map.rs:248:21\n 17: rayon::iter::plumbing::Producer::fold_with\n at .cargo/registry/src/github.com-1ecc6299db9ec823/rayon-1.5.0/src/iter/plumbing/mod.rs:110:9\n 18: rayon::iter::plumbing::bridge_producer_consumer::helper\n at .cargo/registry/src/github.com-1ecc6299db9ec823/rayon-1.5.0/src/iter/plumbing/mod.rs:438:13\n 19: rayon::iter::plumbing::bridge_producer_consumer::helper::{{closure}}\n at .cargo/registry/src/github.com-1ecc6299db9ec823/rayon-1.5.0/src/iter/plumbing/mod.rs:427:21\n 20: rayon_core::join::join_context::call_b::{{closure}}\n at .cargo/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.9.0/src/join/mod.rs:129:25\n 21: <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute::call::{{closure}}\n at .cargo/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.9.0/src/job.rs:113:21\n 22: <core::panic::unwind_safe::AssertUnwindSafe as core::ops::function::FnOnce<()>>::call_once\n at /rustc/7c4b47696907d64eff5621a64eb3c6e795a9ec77/library/core/src/panic/unwind_safe.rs:271:9\n 23: std::panicking::try::do_call\n at /rustc/7c4b47696907d64eff5621a64eb3c6e795a9ec77/library/std/src/panicking.rs:492:40\n 24: __rust_try\n 25: std::panicking::try\n at /rustc/7c4b47696907d64eff5621a64eb3c6e795a9ec77/library/std/src/panicking.rs:456:19\n 26: std::panic::catch_unwind\n at /rustc/7c4b47696907d64eff5621a64eb3c6e795a9ec77/library/std/src/panic.rs:137:14\n 27: rayon_core::unwind::halt_unwinding\n at .cargo/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.9.0/src/unwind.rs:17:5\n 28: <rayon_core::job::StackJob<L,F,R> as rayon_core::job::Job>::execute\n at .cargo/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.9.0/src/job.rs:119:38\n 29: rayon_core::job::JobRef::execute\n at .cargo/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.9.0/src/job.rs:59:9\n 30: rayon_core::registry::WorkerThread::execute\n at .cargo/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.9.0/src/registry.rs:753:9\n 31: rayon_core::registry::WorkerThread::wait_until_cold\n at .cargo/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.9.0/src/registry.rs:730:17\n 32: rayon_core::registry::WorkerThread::wait_until\n at .cargo/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.9.0/src/registry.rs:704:13\n 33: rayon_core::registry::main_loop\n at .cargo/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.9.0/src/registry.rs:837:5\n 34: rayon_core::registry::ThreadBuilder::run\n at .cargo/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.9.0/src/registry.rs:56:18\n 35: <rayon_core::registry::DefaultSpawn as rayon_core::registry::ThreadSpawn>::spawn::{{closure}}\n at .cargo/registry/src/github.com-1ecc6299db9ec823/rayon-core-1.9.0/src/registry.rs:101:20\n 36: std::sys_common::backtrace::_rust_begin_short_backtrace\n at /rustc/7c4b47696907d64eff5621a64eb3c6e795a9ec77/library/std/src/sys_common/backtrace.rs:122:18\n 37: std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}}\n at /rustc/7c4b47696907d64eff5621a64eb3c6e795a9ec77/library/std/src/thread/mod.rs:500:17\n 38: <core::panic::unwind_safe::AssertUnwindSafe as core::ops::function::FnOnce<()>>::call_once\n at /rustc/7c4b47696907d64eff5621a64eb3c6e795a9ec77/library/core/src/panic/unwind_safe.rs:271:9\n 39: std::panicking::try::do_call\n at /rustc/7c4b47696907d64eff5621a64eb3c6e795a9ec77/library/std/src/panicking.rs:492:40\n 40: _rust_try\n 41: std::panicking::try\n at /rustc/7c4b47696907d64eff5621a64eb3c6e795a9ec77/library/std/src/panicking.rs:456:19\n 42: std::panic::catch_unwind\n at /rustc/7c4b47696907d64eff5621a64eb3c6e795a9ec77/library/std/src/panic.rs:137:14\n 43: std::thread::Builder::spawn_unchecked::{{closure}}\n at /rustc/7c4b47696907d64eff5621a64eb3c6e795a9ec77/library/std/src/thread/mod.rs:499:30\n 44: core::ops::function::FnOnce::call_once{{vtable.shim}}\n at /rustc/7c4b47696907d64eff5621a64eb3c6e795a9ec77/library/core/src/ops/function.rs:248:5\n 45: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce>::call_once\n at /rustc/7c4b47696907d64eff5621a64eb3c6e795a9ec77/library/alloc/src/boxed.rs:1866:9\n <alloc::boxed::Box<F,A> as core::ops::function::FnOnce>::call_once\n at /rustc/7c4b47696907d64eff5621a64eb3c6e795a9ec77/library/alloc/src/boxed.rs:1866:9\n std::sys::unix::thread::Thread::new::thread_start\n at /rustc/7c4b47696907d64eff5621a64eb3c6e795a9ec77/library/std/src/sys/unix/thread.rs:108:17\n 46: start_thread\n at /usr/src/debug/glibc-2.17-c758a686/nptl/pthread_create.c:307\n 47: gnu_dev_makedev\n at /usr/src/debug/glibc-2.17-c758a686/misc/../sysdeps/unix/sysv/linux/makedev.c:37\n"] [location=/home/jiayang/rust/tikv/components/encryption/src/manager/mod.rs:650] [thread_name=]

[Lily2025]

/type bug
/severity major