Flaky test cases::test_rawkv::test_leader_transfer

[wuhuizuo]

Bug Report

What version of TiKV are you using?

What operating system and CPU are you using?

Steps to reproduce

What did you expect?

What did happened?

Ref:

• PR: ci: add codecov.yaml file #14994
• CI build url: https://ci.pingcap.net/blue/organizations/jenkins/tikv_ghpr_test/detail/tikv_ghpr_test/35871/pipeline

Log:

[2023-06-26T03:38:58.018Z] ------------

[2023-06-26T03:38:58.018Z]      Summary [ 215.814s] 1749 tests run: 1748 passed (1 flaky), 1 failed, 1804 skipped

[2023-06-26T03:38:58.018Z]    FLAKY 2/3 [   1.702s] tests::failpoints cases::test_rawkv::test_region_merge

[2023-06-26T03:38:58.018Z]   TRY 3 FAIL [   1.198s] tests::failpoints cases::test_rawkv::test_leader_transfer

[2023-06-26T03:38:58.018Z] 

[2023-06-26T03:38:58.018Z] --- TRY 3 STDOUT:        tests::failpoints cases::test_rawkv::test_leader_transfer ---

[2023-06-26T03:38:58.018Z] 

[2023-06-26T03:38:58.018Z] running 1 test

[2023-06-26T03:38:58.018Z] test cases::test_rawkv::test_leader_transfer ... FAILED

[2023-06-26T03:38:58.018Z] 

[2023-06-26T03:38:58.018Z] failures:

[2023-06-26T03:38:58.018Z] 

[2023-06-26T03:38:58.018Z] failures:

[2023-06-26T03:38:58.018Z]     cases::test_rawkv::test_leader_transfer

[2023-06-26T03:38:58.018Z] 

[2023-06-26T03:38:58.018Z] test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 290 filtered out; finished in 1.11s

[2023-06-26T03:38:58.018Z] 

[2023-06-26T03:38:58.018Z] 

[2023-06-26T03:38:58.018Z] --- TRY 3 STDERR:        tests::failpoints cases::test_rawkv::test_leader_transfer ---

[2023-06-26T03:38:58.018Z] thread 'cases::test_rawkv::test_leader_transfer' panicked at '', components/test_raftstore/src/util.rs:1314:5

[2023-06-26T03:38:58.018Z] stack backtrace:

[2023-06-26T03:38:58.018Z]    0: rust_begin_unwind

[2023-06-26T03:38:58.018Z]              at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/std/src/panicking.rs:575:5

[2023-06-26T03:38:58.018Z]    1: core::panicking::panic_fmt

[2023-06-26T03:38:58.018Z]              at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/core/src/panicking.rs:65:14

[2023-06-26T03:38:58.018Z]    2: test_raftstore::util::must_raw_put

[2023-06-26T03:38:58.018Z]    3: failpoints::cases::test_rawkv::TestSuite::must_raw_put

[2023-06-26T03:38:58.018Z]              at /home/jenkins/tikv-src/tests/failpoints/cases/test_rawkv.rs:65:9

[2023-06-26T03:38:58.018Z]    4: failpoints::cases::test_rawkv::test_leader_transfer

[2023-06-26T03:38:58.018Z]              at /home/jenkins/tikv-src/tests/failpoints/cases/test_rawkv.rs:196:9

[2023-06-26T03:38:58.018Z]    5: failpoints::cases::test_rawkv::test_leader_transfer::{{closure}}

[2023-06-26T03:38:58.018Z]              at /home/jenkins/tikv-src/tests/failpoints/cases/test_rawkv.rs:153:27

[2023-06-26T03:38:58.018Z]    6: core::ops::function::FnOnce::call_once

[2023-06-26T03:38:58.018Z]              at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/core/src/ops/function.rs:513:5

[2023-06-26T03:38:58.018Z]    7: test_util::runner::run_test_with_hook::{{closure}}::{{closure}}

[2023-06-26T03:38:58.018Z]    8: core::ops::function::FnOnce::call_once{{vtable.shim}}

[2023-06-26T03:38:58.018Z]    9: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once

[2023-06-26T03:38:58.018Z]              at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/alloc/src/boxed.rs:2000:9

[2023-06-26T03:38:58.018Z] note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

[2023-06-26T03:38:58.018Z] 

[2023-06-26T03:38:58.018Z] error: test run failed

[2023-06-26T03:38:58.018Z] + gdb -c 'core.*' -batch -ex 'info threads' -ex 'thread apply all bt'

[2023-06-26T03:38:58.018Z] /home/jenkins/agent/tikv-master/build/core.*: No such file or directory.

[2023-06-26T03:38:58.018Z] No threads.

[wuhuizuo]

other failed case: raftstore::test_flashback::test_flashback_with_in_memory_pessimistic_locks::test_raftstore_v2_new_server_cluster

[2023-06-26T04:28:06.447Z] ------------

[2023-06-26T04:28:06.447Z]      Summary [ 205.152s] 1790 tests run: 1789 passed, 1 failed, 1763 skipped

[2023-06-26T04:28:06.447Z]   TRY 3 FAIL [   0.879s] tests::integrations raftstore::test_flashback::test_flashback_with_in_memory_pessimistic_locks::test_raftstore_v2_new_server_cluster

[2023-06-26T04:28:06.447Z] 

[2023-06-26T04:28:06.447Z] --- TRY 3 STDOUT:        tests::integrations raftstore::test_flashback::test_flashback_with_in_memory_pessimistic_locks::test_raftstore_v2_new_server_cluster ---

[2023-06-26T04:28:06.447Z] 

[2023-06-26T04:28:06.447Z] running 1 test

[2023-06-26T04:28:06.447Z] test raftstore::test_flashback::test_flashback_with_in_memory_pessimistic_locks::test_raftstore_v2_new_server_cluster ... FAILED

[2023-06-26T04:28:06.447Z] 

[2023-06-26T04:28:06.447Z] failures:

[2023-06-26T04:28:06.447Z] 

[2023-06-26T04:28:06.447Z] failures:

[2023-06-26T04:28:06.447Z]     raftstore::test_flashback::test_flashback_with_in_memory_pessimistic_locks::test_raftstore_v2_new_server_cluster

[2023-06-26T04:28:06.447Z] 

[2023-06-26T04:28:06.447Z] test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 708 filtered out; finished in 0.84s

[2023-06-26T04:28:06.447Z] 

[2023-06-26T04:28:06.447Z] 

[2023-06-26T04:28:06.447Z] --- TRY 3 STDERR:        tests::integrations raftstore::test_flashback::test_flashback_with_in_memory_pessimistic_locks::test_raftstore_v2_new_server_cluster ---

[2023-06-26T04:28:06.447Z] thread 'raftstore::test_flashback::test_flashback_with_in_memory_pessimistic_locks::test_raftstore_v2_new_server_cluster' panicked at 'assertion failed: !pessimistic_locks.is_writable()', tests/integrations/raftstore/test_flashback.rs:69:9

[2023-06-26T04:28:06.447Z] stack backtrace:

[2023-06-26T04:28:06.447Z]    0: rust_begin_unwind

[2023-06-26T04:28:06.447Z]              at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/std/src/panicking.rs:575:5

[2023-06-26T04:28:06.447Z]    1: core::panicking::panic_fmt

[2023-06-26T04:28:06.447Z]              at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/core/src/panicking.rs:65:14

[2023-06-26T04:28:06.447Z]    2: core::panicking::panic

[2023-06-26T04:28:06.447Z]              at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/core/src/panicking.rs:114:5

[2023-06-26T04:28:06.447Z]    3: integrations::raftstore::test_flashback::test_flashback_with_in_memory_pessimistic_locks::test_raftstore_v2_new_server_cluster

[2023-06-26T04:28:06.447Z]              at /home/jenkins/tikv-src/tests/integrations/raftstore/test_flashback.rs:69:9

[2023-06-26T04:28:06.447Z]    4: integrations::raftstore::test_flashback::test_flashback_with_in_memory_pessimistic_locks::test_raftstore_v2_new_server_cluster::{{closure}}

[2023-06-26T04:28:06.447Z]              at /home/jenkins/tikv-src/tests/integrations/raftstore/test_flashback.rs:27:54

[2023-06-26T04:28:06.447Z]    5: core::ops::function::FnOnce::call_once

[2023-06-26T04:28:06.447Z]              at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/core/src/ops/function.rs:513:5

[2023-06-26T04:28:06.447Z]    6: test_util::runner::run_test_with_hook::{{closure}}::{{closure}}

[2023-06-26T04:28:06.447Z]    7: core::ops::function::FnOnce::call_once{{vtable.shim}}

[2023-06-26T04:28:06.447Z]    8: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once

[2023-06-26T04:28:06.447Z]              at /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/alloc/src/boxed.rs:2000:9

[2023-06-26T04:28:06.447Z] note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

[2023-06-26T04:28:06.447Z] 

[2023-06-26T04:28:06.447Z] error: test run failed

build url: https://ci.pingcap.net/blue/organizations/jenkins/tikv_ghpr_test/detail/tikv_ghpr_test/35873/pipeline/69