Some test cases failed when run make docker_test

[han-ian]

Bug Report

What version of TiKV are you using?

v6.5.9

What operating system and CPU are you using?

x86/64 centos 7

Steps to reproduce

run make docker_test

What did you expect?

all test case succeed.

What did happened?

test cases::test_snap::test_shutdown_when_snap_gc ... ok
test cases::test_snap::test_snapshot_clean_up_logs_with_unfinished_log_gc ... ok
test cases::test_split_region::test_split_not_to_split_existing_different_uninitialied_peer has been running for over 60 seconds
test cases::test_snap::test_snapshot_gc_after_failed ... ok
test cases::test_split_region::test_split_not_to_split_existing_region has been running for over 60 seconds
thread 'cases::test_snap::test_snapshot_recover_from_raft_write_failure' panicked at 'assertion failed: ents.is_empty()', tests/failpoints/cases/test_snap.rs:789:5
stack backtrace:
0: 0x560857a2b4a0 - std::backtrace_rs::backtrace::libunwind::trace::he534b130db6360ed
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/std/src/../../backtrace/src/backtrace/libunwind.rs:93:5
1: 0x560857a2b4a0 - std::backtrace_rs::backtrace::trace_unsynchronized::hc3486e10c7777b38
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
2: 0x560857a2b4a0 - std::sys_common::backtrace::_print_fmt::ha24198a22cf70db0
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/std/src/sys_common/backtrace.rs:65:5
3: 0x560857a2b4a0 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::he928616f28db1c44
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/std/src/sys_common/backtrace.rs:44:22
4: 0x560857a54c6e - core::fmt::write::hfc4e1f18015528cf
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/core/src/fmt/mod.rs:1209:17
5: 0x560857a24925 - std::io::Write::write_fmt::h1fe0ff478ba9434a
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/std/src/io/mod.rs:1682:15
6: 0x560857a2b265 - std::sys_common::backtrace::_print::hc5210df762c2ee4d
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/std/src/sys_common/backtrace.rs:47:5
7: 0x560857a2b265 - std::sys_common::backtrace::print::h01d5ef97fb586ab1
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/std/src/sys_common/backtrace.rs:34:9
8: 0x560857a2cf5f - std::panicking::default_hook::{{closure}}::hfaf4dc37dc2875b0
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/std/src/panicking.rs:267:22
9: 0x560857a2cc9a - std::panicking::default_hook::h22df008dfc46e0e0
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/std/src/panicking.rs:286:9
10: 0x560857a2d65c - std::panicking::rust_panic_with_hook::h8bb52976cbf68a37
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/std/src/panicking.rs:688:13
11: 0x560857a2d3b1 - std::panicking::begin_panic_handler::{{closure}}::hb6582e9b11a93445
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/std/src/panicking.rs:577:13
12: 0x560857a2b94c - std::sys_common::backtrace::__rust_end_short_backtrace::hd69a588bdd28408e
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/std/src/sys_common/backtrace.rs:137:18
13: 0x560857a2d112 - rust_begin_unwind
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/std/src/panicking.rs:575:5
14: 0x560851c61e43 - core::panicking::panic_fmt::hb1806527c2beebe3
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/core/src/panicking.rs:65:14
15: 0x560851c61f1d - core::panicking::panic::h667f61e6a5fa2c88
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/core/src/panicking.rs:114:5
16: 0x560851df76bc - failpoints::cases::test_snap::test_snapshot_recover_from_raft_write_failure::hc2498ea17dafb8bd
at /tikv/tests/failpoints/cases/test_snap.rs:789:5
17: 0x560851db92ee - failpoints::cases::test_snap::test_snapshot_recover_from_raft_write_failure::{{closure}}::h73f4a4aebfb80726
at /tikv/tests/failpoints/cases/test_snap.rs:743:52
18: 0x560851db92ee - core::ops::function::FnOnce::call_once::haa4bb74e0d9f8669
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/core/src/ops/function.rs:513:5
19: 0x560852e89755 - test_util::runner::run_test_with_hook::{{closure}}::{{closure}}::h56aa978f76671ce6
20: 0x560852e7e0e4 - core::ops::function::FnOnce::call_once{{vtable.shim}}::h49b40c9f92a7590e
21: 0x56085511abaa - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce>::call_once::h312fea4ff6db7ecf
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/alloc/src/boxed.rs:2000:9
22: 0x56085511abaa - test::__rust_begin_short_backtrace::hb48c52ec89163dba
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/test/src/lib.rs:637:18
23: 0x5608550ed7f3 - test::run_test::{{closure}}::hb6f66070f49159ed
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/test/src/lib.rs:620:34
24: 0x5608550ed7f3 - core::ops::function::FnOnce::call_once{{vtable.shim}}::hedad2e997023f271
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/core/src/ops/function.rs:513:5
25: 0x560855119b56 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce>::call_once::h312fea4ff6db7ecf
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/alloc/src/boxed.rs:2000:9
26: 0x560855119b56 - <core::panic::unwind_safe::AssertUnwindSafe as core::ops::function::FnOnce<()>>::call_once::hdb969e3902e87449
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/core/src/panic/unwind_safe.rs:271:9
27: 0x560855119b56 - std::panicking::try::do_call::h6da79ecfef558898
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/std/src/panicking.rs:483:40
28: 0x560855119b56 - std::panicking::try::h7f668fbaf77473b2
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/std/src/panicking.rs:447:19
29: 0x560855119b56 - std::panic::catch_unwind::ha6924da3c5c9fc14
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/std/src/panic.rs:137:14
30: 0x560855119b56 - test::run_test_in_process::hb6aceb0c3bf0084e
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/test/src/lib.rs:660:27
31: 0x560855119b56 - test::run_test::run_test_inner::{{closure}}::h1f9d76cb3b5cab45
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/test/src/lib.rs:554:39
32: 0x5608550e7ff4 - test::run_test::run_test_inner::{{closure}}::hbb1c64585fff0899
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/test/src/lib.rs:581:37
33: 0x5608550e7ff4 - std::sys_common::backtrace::rust_begin_short_backtrace::hc55a7d34ba091187
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/std/src/sys_common/backtrace.rs:121:18
34: 0x5608550ed6b5 - std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}}::h5f0678941ebc6010
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/std/src/thread/mod.rs:551:17
35: 0x5608550ed6b5 - <core::panic::unwind_safe::AssertUnwindSafe as core::ops::function::FnOnce<()>>::call_once::h7b5b1b1b480c0d0b
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/core/src/panic/unwind_safe.rs:271:9
36: 0x5608550ed6b5 - std::panicking::try::do_call::h75b027545d7dd24a
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/std/src/panicking.rs:483:40
37: 0x5608550ed6b5 - std::panicking::try::h3231c5548f9f3483
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/std/src/panicking.rs:447:19
38: 0x5608550ed6b5 - std::panic::catch_unwind::h4b3465df06d90c4b
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/std/src/panic.rs:137:14
39: 0x5608550ed6b5 - std::thread::Builder::spawn_unchecked::{{closure}}::h44dacc2db01da8f5
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/std/src/thread/mod.rs:550:30
40: 0x5608550ed6b5 - core::ops::function::FnOnce::call_once{{vtable.shim}}::h8e859a243f1af552
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/core/src/ops/function.rs:513:5
41: 0x560857a33343 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce>::call_once::h613d5e218c101168
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/alloc/src/boxed.rs:2000:9
42: 0x560857a33343 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce>::call_once::hecab0373f4228b53
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/alloc/src/boxed.rs:2000:9
43: 0x560857a33343 - std::sys::unix::thread::Thread::new::thread_start::hd2791a9cabec1fda
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/std/src/sys/unix/thread.rs:108:17
44: 0x7f00f65dec02 - start_thread
45: 0x7f00f6662ed4 - __GI___clone
46: 0x0 -
test cases::test_snap::test_snapshot_recover_from_raft_write_failure ... FAILED
test cases::test_split_region::test_split_not_to_split_existing_tombstone_region has been running for over 60 seconds

test cases::test_merge::test_merge_with_concurrent_pessimistic_locking has been running for over 60 seconds
test cases::test_merge::test_node_failed_merge_before_succeed_merge has been running for over 60 seconds
test cases::test_merge::test_node_merge_cascade_merge_with_apply_yield has been running for over 60 seconds
test cases::test_merge::test_node_merge_catch_up_logs_leader_election has been running for over 60 seconds
test cases::test_merge::test_node_merge_catch_up_logs_no_need has been running for over 60 seconds
thread 'cases::test_hibernate::test_store_disconnect_with_hibernate' panicked at 'can't get value Some("v2") for key 6B32', components/test_raftstore/src/util.rs:75:5
stack backtrace:
0: 0x560857a2b4a0 - std::backtrace_rs::backtrace::libunwind::trace::he534b130db6360ed
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/std/src/../../backtrace/src/backtrace/libunwind.rs:93:5
1: 0x560857a2b4a0 - std::backtrace_rs::backtrace::trace_unsynchronized::hc3486e10c7777b38
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
2: 0x560857a2b4a0 - std::sys_common::backtrace::_print_fmt::ha24198a22cf70db0
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/std/src/sys_common/backtrace.rs:65:5
3: 0x560857a2b4a0 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::he928616f28db1c44
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/std/src/sys_common/backtrace.rs:44:22
4: 0x560857a54c6e - core::fmt::write::hfc4e1f18015528cf
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/core/src/fmt/mod.rs:1209:17
5: 0x560857a24925 - std::io::Write::write_fmt::h1fe0ff478ba9434a
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/std/src/io/mod.rs:1682:15
6: 0x560857a2b265 - std::sys_common::backtrace::_print::hc5210df762c2ee4d
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/std/src/sys_common/backtrace.rs:47:5
7: 0x560857a2b265 - std::sys_common::backtrace::print::h01d5ef97fb586ab1
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/std/src/sys_common/backtrace.rs:34:9
8: 0x560857a2cf5f - std::panicking::default_hook::{{closure}}::hfaf4dc37dc2875b0
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/std/src/panicking.rs:267:22
9: 0x560857a2cc9a - std::panicking::default_hook::h22df008dfc46e0e0
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/std/src/panicking.rs:286:9
10: 0x560857a2d65c - std::panicking::rust_panic_with_hook::h8bb52976cbf68a37
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/std/src/panicking.rs:688:13
11: 0x560857a2d3f7 - std::panicking::begin_panic_handler::{{closure}}::hb6582e9b11a93445
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/std/src/panicking.rs:579:13
12: 0x560857a2b94c - std::sys_common::backtrace::__rust_end_short_backtrace::hd69a588bdd28408e
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/std/src/sys_common/backtrace.rs:137:18
13: 0x560857a2d112 - rust_begin_unwind
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/std/src/panicking.rs:575:5
14: 0x560851c61e43 - core::panicking::panic_fmt::hb1806527c2beebe3
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/core/src/panicking.rs:65:14
15: 0x560852195a74 - test_raftstore::util::must_get::h3eeb538acd7351a4
16: 0x560852195adf - test_raftstore::util::must_get_equal::h4a08e0904f78b6fd
17: 0x560851cf9de2 - failpoints::cases::test_hibernate::test_store_disconnect_with_hibernate::h9baedad0dbe30eaa
at /tikv/tests/failpoints/cases/test_hibernate.rs:132:5
18: 0x560851db6e7e - failpoints::cases::test_hibernate::test_store_disconnect_with_hibernate::{{closure}}::hc810dae9a98e2b30
at /tikv/tests/failpoints/cases/test_hibernate.rs:96:43
19: 0x560851db6e7e - core::ops::function::FnOnce::call_once::h90853f1795f2c984
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/core/src/ops/function.rs:513:5
20: 0x560852e89755 - test_util::runner::run_test_with_hook::{{closure}}::{{closure}}::h56aa978f76671ce6
21: 0x560852e7e0e4 - core::ops::function::FnOnce::call_once{{vtable.shim}}::h49b40c9f92a7590e
22: 0x56085511abaa - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce>::call_once::h312fea4ff6db7ecf
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/alloc/src/boxed.rs:2000:9
23: 0x56085511abaa - test::__rust_begin_short_backtrace::hb48c52ec89163dba
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/test/src/lib.rs:637:18
24: 0x5608550ed7f3 - test::run_test::{{closure}}::hb6f66070f49159ed
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/test/src/lib.rs:620:34
25: 0x5608550ed7f3 - core::ops::function::FnOnce::call_once{{vtable.shim}}::hedad2e997023f271
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/core/src/ops/function.rs:513:5
26: 0x560855119b56 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce>::call_once::h312fea4ff6db7ecf
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/alloc/src/boxed.rs:2000:9
27: 0x560855119b56 - <core::panic::unwind_safe::AssertUnwindSafe as core::ops::function::FnOnce<()>>::call_once::hdb969e3902e87449
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/core/src/panic/unwind_safe.rs:271:9
28: 0x560855119b56 - std::panicking::try::do_call::h6da79ecfef558898
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/std/src/panicking.rs:483:40
29: 0x560855119b56 - std::panicking::try::h7f668fbaf77473b2
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/std/src/panicking.rs:447:19
30: 0x560855119b56 - std::panic::catch_unwind::ha6924da3c5c9fc14
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/std/src/panic.rs:137:14
31: 0x560855119b56 - test::run_test_in_process::hb6aceb0c3bf0084e
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/test/src/lib.rs:660:27
32: 0x560855119b56 - test::run_test::run_test_inner::{{closure}}::h1f9d76cb3b5cab45
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/test/src/lib.rs:554:39
33: 0x5608550e7ff4 - test::run_test::run_test_inner::{{closure}}::hbb1c64585fff0899
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/test/src/lib.rs:581:37
34: 0x5608550e7ff4 - std::sys_common::backtrace::rust_begin_short_backtrace::hc55a7d34ba091187
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/std/src/sys_common/backtrace.rs:121:18
35: 0x5608550ed6b5 - std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}}::h5f0678941ebc6010
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/std/src/thread/mod.rs:551:17
36: 0x5608550ed6b5 - <core::panic::unwind_safe::AssertUnwindSafe as core::ops::function::FnOnce<()>>::call_once::h7b5b1b1b480c0d0b
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/core/src/panic/unwind_safe.rs:271:9
37: 0x5608550ed6b5 - std::panicking::try::do_call::h75b027545d7dd24a
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/std/src/panicking.rs:483:40
38: 0x5608550ed6b5 - std::panicking::try::h3231c5548f9f3483
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/std/src/panicking.rs:447:19
39: 0x5608550ed6b5 - std::panic::catch_unwind::h4b3465df06d90c4b
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/std/src/panic.rs:137:14
40: 0x5608550ed6b5 - std::thread::Builder::spawn_unchecked::{{closure}}::h44dacc2db01da8f5
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/std/src/thread/mod.rs:550:30
41: 0x5608550ed6b5 - core::ops::function::FnOnce::call_once{{vtable.shim}}::h8e859a243f1af552
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/core/src/ops/function.rs:513:5
42: 0x560857a33343 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce>::call_once::h613d5e218c101168
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/alloc/src/boxed.rs:2000:9
43: 0x560857a33343 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce>::call_once::hecab0373f4228b53
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/alloc/src/boxed.rs:2000:9
44: 0x560857a33343 - std::sys::unix::thread::Thread::new::thread_start::hd2791a9cabec1fda
at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/std/src/sys/unix/thread.rs:108:17
45: 0x7f00f65dec02 - start_thread
46: 0x7f00f6662ed4 - __GI___clone
47: 0x0 -
test cases::test_hibernate::test_store_disconnect_with_hibernate ... FAILED
test cases::test_merge::test_node_merge_catch_up_logs_restart has been running for over 60 seconds
test cases::test_merge::test_node_merge_crash_before_snapshot_then_catch_up_logs has been running for over 60 seconds

failures:
cases::test_disk_full::test_disk_full_for_region_follower
cases::test_disk_full::test_disk_full_for_region_leader
cases::test_hibernate::test_store_disconnect_with_hibernate
cases::test_snap::test_snapshot_recover_from_raft_write_failure

test result: FAILED. 248 passed; 4 failed; 1 ignored; 0 measured; 0 filtered out; finished in 497.61s

[han-ian]

Another case, test cases blocked about half an hour.
But I can run case each case individually.
I think it is the reason of failpoint concurrent execution.

test event_loader::tests::test_disk_read ... ok
test metadata::checkpoint_cache::test::test_basic ... ok
walking /tmp/068b0150-dd32-413b-9039-2cb44e6e39ee
walking /tmp/068b0150-dd32-413b-9039-2cb44e6e39ee/v1
walking /tmp/068b0150-dd32-413b-9039-2cb44e6e39ee/v1/backupmeta
walking /tmp/068b0150-dd32-413b-9039-2cb44e6e39ee/v1/backupmeta/1721114033187-c1a9ee87-40fc-4701-ab96-d1296ae94259.meta
walking /tmp/068b0150-dd32-413b-9039-2cb44e6e39ee/v1/20240716
walking /tmp/068b0150-dd32-413b-9039-2cb44e6e39ee/v1/20240716/07
walking /tmp/068b0150-dd32-413b-9039-2cb44e6e39ee/v1/20240716/07/1
walking /tmp/068b0150-dd32-413b-9039-2cb44e6e39ee/v1/20240716/07/1/451179717115772928-efd87ca1-0500-45b5-bd09-2a3b0206b037.log
walking /tmp/068b0150-dd32-413b-9039-2cb44e6e39ee/v1/20240716/07/1/451179717115772928-8cfab479-ef6c-4760-9c34-80c84ef757bc.log
test router::tests::test_basic_file ... ok
test endpoint::test::test_start ... ok
test utils::test::test_wait_group ... ok
test router::tests::test_do_flush has been running for over 60 seconds
test router::tests::test_flush_on_events_race has been running for over 60 seconds
test router::tests::test_flush_with_error has been running for over 60 seconds
test router::tests::test_flush_with_pausing_self has been running for over 60 seconds

[glorv]

I think the root cause is the same with #17268.

[han-ian]

I would like to know how the tikv dev team performs CI.
From my tests, make test has failed test cases (usually different test cases) both on linux physical machines and in docker containers.
I suspect that the test cases were executed concurrently (sharing the same temporary directory or triggering the same failpoint).

[glorv]

Currently, we use nextest in the CI environment. Seems nextest provides better isolation than the default test framework, so some issue will be covered when running with nextest.

You can run tikv's test with nextest by make test_with_nextest, this should provide stabler test output.

[han-ian]

Need to mention that if you run the test cases as a root user, some cases will fail because the read-only property of the file is not valid for the root user.

Just as #14594 (comment)

[glorv]

/cc @BornChanger