bug: CI test stepdown failed

[Veeupup]

here is the log and CI link https://github.com/datafuselabs/openraft/runs/4700908123?check_suite_focus=true#step:4:417

running 1 test
test stepdown ... FAILED

failures:

---- stepdown stdout ----
thread 'stepdown' panicked at 'assertion failed: `(left == right)`: expected node 1 to have state machine last_applied_log 2-4, got 3-4

Diff < left / right > :
 LogId {
<    term: 3,
>    term: 2,
     index: 4,
 }

', openraft/tests/fixtures/mod.rs:701:13
stack backtrace:
   0:     0x55cb16b53f4c - std::backtrace_rs::backtrace::libunwind::trace::h65ef482bb9b15649
                               at /rustc/8f3238f898163f09726c3d2b2cc9bafb09da26f3/library/std/src/../../backtrace/src/backtrace/libunwind.rs:93:5
   1:     0x55cb16b53f4c - std::backtrace_rs::backtrace::trace_unsynchronized::hf1ee7630128bf9a9
                               at /rustc/8f3238f898163f09726c3d2b2cc9bafb09da26f3/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:     0x55cb16b53f4c - std::sys_common::backtrace::_print_fmt::haddc20e8865333bd
                               at /rustc/8f3238f898163f09726c3d2b2cc9bafb09da26f3/library/std/src/sys_common/backtrace.rs:67:5
   3:     0x55cb16b53f4c - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::hc06b166f304d5a13
                               at /rustc/8f3238f898163f09726c3d2b2cc9bafb09da26f3/library/std/src/sys_common/backtrace.rs:46:22
   4:     0x55cb16b7b96c - core::fmt::write::h212f7b7266b9a26a
                               at /rustc/8f3238f898163f09726c3d2b2cc9bafb09da26f3/library/core/src/fmt/mod.rs:1149:17
   5:     0x55cb16b4e913 - std::io::Write::write_fmt::hbfd04b0aa1968e83
                               at /rustc/8f3238f898163f09726c3d2b2cc9bafb09da26f3/library/std/src/io/mod.rs:1660:15
   6:     0x55cb16b56702 - std::sys_common::backtrace::_print::h9d519f4e309ac3d8
                               at /rustc/8f3238f898163f09726c3d2b2cc9bafb09da26f3/library/std/src/sys_common/backtrace.rs:49:5
   7:     0x55cb16b56702 - std::sys_common::backtrace::print::h8362bac372870d46
                               at /rustc/8f3238f898163f09726c3d2b2cc9bafb09da26f3/library/std/src/sys_common/backtrace.rs:36:9
   8:     0x55cb16b56702 - std::panicking::default_hook::{{closure}}::hdf453592ec91e76e
                               at /rustc/8f3238f898163f09726c3d2b2cc9bafb09da26f3/library/std/src/panicking.rs:211:50
   9:     0x55cb16b56351 - std::panicking::default_hook::h9b626066b2c6b270
                               at /rustc/8f3238f898163f09726c3d2b2cc9bafb09da26f3/library/std/src/panicking.rs:225:9
  10:     0x55cb16b56d53 - std::panicking::rust_panic_with_hook::h90c32e4f1cc54562
                               at /rustc/8f3238f898163f09726c3d2b2cc9bafb09da26f3/library/std/src/panicking.rs:606:17
  11:     0x55cb16b56a70 - std::panicking::begin_panic_handler::{{closure}}::h6751672be4522935
                               at /rustc/8f3238f898163f09726c3d2b2cc9bafb09da26f3/library/std/src/panicking.rs:502:13
  12:     0x55cb16b543f4 - std::sys_common::backtrace::__rust_end_short_backtrace::hdecab70784de07d3
                               at /rustc/8f3238f898163f09726c3d2b2cc9bafb09da26f3/library/std/src/sys_common/backtrace.rs:139:18
  13:     0x55cb16b567a9 - rust_begin_unwind
                               at /rustc/8f3238f898163f09726c3d2b2cc9bafb09da26f3/library/std/src/panicking.rs:498:5
  14:     0x55cb163ba401 - core::panicking::panic_fmt::h3a6bf5e754065d1c
                               at /rustc/8f3238f898163f09726c3d2b2cc9bafb09da26f3/library/core/src/panicking.rs:107:14
  15:     0x55cb164f3c04 - stepdown::fixtures::RaftRouter::assert_storage_state::{{closure}}::h436e5f88acb8f336
                               at /home/runner/work/openraft/openraft/openraft/tests/fixtures/mod.rs:701:13
  16:     0x55cb16550429 - <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll::hd7c940c832f59945
                               at /rustc/8f3238f898163f09726c3d2b2cc9bafb09da26f3/library/core/src/future/mod.rs:84:19
  17:     0x55cb1661945a - stepdown::stepdown::{{closure}}::h7dff7e0fc7262991
                               at /home/runner/work/openraft/openraft/openraft/tests/stepdown.rs:131:99
  18:     0x55cb1654e1e9 - <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll::hb0fac9f34dea2935
                               at /rustc/8f3238f898163f09726c3d2b2cc9bafb09da26f3/library/core/src/future/mod.rs:84:19
  19:     0x55cb165ec630 - tokio::park::thread::CachedParkThread::block_on::{{closure}}::hc967d3ff0ac1d916
                               at /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.15.0/src/park/thread.rs:263:54
  20:     0x55cb1662e199 - tokio::coop::with_budget::{{closure}}::hece6e7fff23fbe9b
                               at /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.15.0/src/coop.rs:102:9
  21:     0x55cb164d6aea - std::thread::local::LocalKey<T>::try_with::hfd9320b0b9fec881
                               at /rustc/8f3238f898163f09726c3d2b2cc9bafb09da26f3/library/std/src/thread/local.rs:412:16
  22:     0x55cb164d52fd - std::thread::local::LocalKey<T>::with::h72ed484b2dfa81fb
                               at /rustc/8f3238f898163f09726c3d2b2cc9bafb09da26f3/library/std/src/thread/local.rs:388:9
  23:     0x55cb165ec3ed - tokio::coop::with_budget::h0ed9a56ecce39f22
                               at /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.15.0/src/coop.rs:95:5
  24:     0x55cb165ec3ed - tokio::coop::budget::h64415ea0ab85f3d6
                               at /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.15.0/src/coop.rs:72:5
  25:     0x55cb165ec3ed - tokio::park::thread::CachedParkThread::block_on::h4e7f5e9c04ec7ab7
                               at /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.15.0/src/park/thread.rs:263:31
  26:     0x55cb1655782e - tokio::runtime::enter::Enter::block_on::h2de1d90cc0290968
                               at /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.15.0/src/runtime/enter.rs:151:13
  27:     0x55cb163c3923 - tokio::runtime::thread_pool::ThreadPool::block_on::h1c0b9c616e7bcdbc
                               at /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.15.0/src/runtime/thread_pool/mod.rs:77:9
  28:     0x55cb164e7b46 - tokio::runtime::Runtime::block_on::hc0f1407cae225896
                               at /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.15.0/src/runtime/mod.rs:463:43
  29:     0x55cb164855a1 - stepdown::stepdown::hafcf32fcebecedb3
                               at /home/runner/work/openraft/openraft/openraft/tests/stepdown.rs:136:5
  30:     0x55cb16613c4e - stepdown::stepdown::{{closure}}::h7550cf795b8ff982
                               at /home/runner/work/openraft/openraft/openraft/tests/stepdown.rs:24:7
  31:     0x55cb16442efe - core::ops::function::FnOnce::call_once::h7271fc4f3780d659
                               at /rustc/8f3238f898163f09726c3d2b2cc9bafb09da26f3/library/core/src/ops/function.rs:227:5
  32:     0x55cb16676b03 - core::ops::function::FnOnce::call_once::h2f9ae408a2054227
                               at /rustc/8f3238f898163f09726c3d2b2cc9bafb09da26f3/library/core/src/ops/function.rs:227:5
  33:     0x55cb16676b03 - test::__rust_begin_short_backtrace::h946635b2b37a3f0d
                               at /rustc/8f3238f898163f09726c3d2b2cc9bafb09da26f3/library/test/src/lib.rs:585:5
  34:     0x55cb166757af - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h327757468c02cf0a
                               at /rustc/8f3238f898163f09726c3d2b2cc9bafb09da26f3/library/alloc/src/boxed.rs:1811:9
  35:     0x55cb166757af - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h51682cfbb1c3ae3c
                               at /rustc/8f3238f898163f09726c3d2b2cc9bafb09da26f3/library/core/src/panic/unwind_safe.rs:271:9
  36:     0x55cb166757af - std::panicking::try::do_call::hacea7f0da16ddb2e
                               at /rustc/8f3238f898163f09726c3d2b2cc9bafb09da26f3/library/std/src/panicking.rs:406:40
  37:     0x55cb166757af - std::panicking::try::hb51105ac5e97d475
                               at /rustc/8f3238f898163f09726c3d2b2cc9bafb09da26f3/library/std/src/panicking.rs:370:19
  38:     0x55cb166757af - std::panic::catch_unwind::h3e6525b13276184d
                               at /rustc/8f3238f898163f09726c3d2b2cc9bafb09da26f3/library/std/src/panic.rs:133:14
  39:     0x55cb166757af - test::run_test_in_process::hc169dee883f59795
                               at /rustc/8f3238f898163f09726c3d2b2cc9bafb09da26f3/library/test/src/lib.rs:608:18
  40:     0x55cb166757af - test::run_test::run_test_inner::{{closure}}::hf2b675a51a8a715b
                               at /rustc/8f3238f898163f09726c3d2b2cc9bafb09da26f3/library/test/src/lib.rs:500:39
  41:     0x55cb1667eb11 - test::run_test::run_test_inner::{{closure}}::h24c5be3fc3c26378
                               at /rustc/8f3238f898163f09726c3d2b2cc9bafb09da26f3/library/test/src/lib.rs:527:37
  42:     0x55cb1667eb11 - std::sys_common::backtrace::__rust_begin_short_backtrace::h02f81884a4850f72
                               at /rustc/8f3238f898163f09726c3d2b2cc9bafb09da26f3/library/std/src/sys_common/backtrace.rs:123:18
  43:     0x55cb1664b5df - std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}}::h4b4d68bc1bf14409
                               at /rustc/8f3238f898163f09726c3d2b2cc9bafb09da26f3/library/std/src/thread/mod.rs:477:17
  44:     0x55cb1664b5df - <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h41705b2de488f292
                               at /rustc/8f3238f898163f09726c3d2b2cc9bafb09da26f3/library/core/src/panic/unwind_safe.rs:271:9
  45:     0x55cb1664b5df - std::panicking::try::do_call::hb0c49e6a8b1782a9
                               at /rustc/8f3238f898163f09726c3d2b2cc9bafb09da26f3/library/std/src/panicking.rs:406:40
  46:     0x55cb1664b5df - std::panicking::try::hc9baa64f4cf6e2d3
                               at /rustc/8f3238f898163f09726c3d2b2cc9bafb09da26f3/library/std/src/panicking.rs:370:19
  47:     0x55cb1664b5df - std::panic::catch_unwind::h38c4576076b1268c
                               at /rustc/8f3238f898163f09726c3d2b2cc9bafb09da26f3/library/std/src/panic.rs:133:14
  48:     0x55cb1664b5df - std::thread::Builder::spawn_unchecked::{{closure}}::h12b59a33f3b9f983
                               at /rustc/8f3238f898163f09726c3d2b2cc9bafb09da26f3/library/std/src/thread/mod.rs:476:30
  49:     0x55cb1664b5df - core::ops::function::FnOnce::call_once{{vtable.shim}}::h6d952349615f77cc
                               at /rustc/8f3238f898163f09726c3d2b2cc9bafb09da26f3/library/core/src/ops/function.rs:227:5
  50:     0x55cb16b5cce3 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::hc32e0330a23c98fc
                               at /rustc/8f3238f898163f09726c3d2b2cc9bafb09da26f3/library/alloc/src/boxed.rs:1811:9
  51:     0x55cb16b5cce3 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::hc425e01eeb6518b7
                               at /rustc/8f3238f898163f09726c3d2b2cc9bafb09da26f3/library/alloc/src/boxed.rs:1811:9
  52:     0x55cb16b5cce3 - std::sys::unix::thread::Thread::new::thread_start::hda5c72740ad584b7
                               at /rustc/8f3238f898163f09726c3d2b2cc9bafb09da26f3/library/std/src/sys/unix/thread.rs:108:17
  53:     0x7fa030cb7609 - start_thread
  54:     0x7fa030a89293 - clone
  55:                0x0 - <unknown>


failures:
    stepdown

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 5.63s

error: test failed, to rerun pass '-p openraft --test stepdown'

cc @drmingdrmer

[github-actions]

👋 Thanks for opening this issue!

Get help or engage by:

• /help : to print help messages.
• /assignme : to assign this issue to you.

[drmingdrmer]

Thank you man.

Because within CI if some task is delayed a new round of election will be fired.

[lichuang]

use a bigger election timeout to avoid this problem : #100