Reactor stall in redpanda

[mmaslankaprv]

Redpanda version: redpanda-21.6.3-1.x86_64

2021-06-29 20:43:15	
Reactor stalled for 3708 ms on shard 0. Backtrace: 0x284aa49 0x284cb49 0x28cf9fd341df 0x11a1f01 0x11a9051 0x2870144 0x2872db7 0x279c0c9 0x279a120 0xd652a5 0x2d910cc 0x281a1 0xd6330d
2021-06-29 20:43:13	
Reactor stalled for 1967 ms on shard 0. Backtrace: 0x284aa49 0x284cb49 0x28cf9fd341df 0x11a1eff 0x11a9051 0x2870144 0x2872db7 0x279c0c9 0x279a120 0xd652a5 0x2d910cc 0x281a1 0xd6330d
2021-06-29 20:43:12	
Reactor stalled for 1051 ms on shard 0. Backtrace: 0x284aa49 0x284cb49 0x28cf9fd341df 0x11a1ee0 0x11a9051 0x2870144 0x2872db7 0x279c0c9 0x279a120 0xd652a5 0x2d910cc 0x281a1 0xd6330d
2021-06-29 20:43:12	
Reactor stalled for 572 ms on shard 0. Backtrace: 0x284aa49 0x284cb49 0x28cf9fd341df 0x11a1f0a 0x11a9051 0x2870144 0x2872db7 0x279c0c9 0x279a120 0xd652a5 0x2d910cc 0x281a1 0xd6330d

JIRA Link: CORE-666

[mmaslankaprv]

void seastar::backtrace<seastar::backtrace_buffer::append_backtrace_oneline()::{lambda(seastar::frame)#1}>(seastar::backtrace_buffer::append_backtrace_oneline()::{lambda(seastar::frame)#1}&&) at /v/build/v_deps_build/seastar-prefix/src/seastar/include/seastar/util/backtrace.hh:59
 (inlined by) seastar::backtrace_buffer::append_backtrace_oneline() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:772
 (inlined by) seastar::print_with_backtrace(seastar::backtrace_buffer&, bool) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:791
 (inlined by) seastar::internal::cpu_stall_detector::generate_trace() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:1258
seastar::internal::cpu_stall_detector::maybe_report() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:1104
 (inlined by) seastar::internal::cpu_stall_detector::on_signal() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:1118
 (inlined by) seastar::reactor::block_notifier(int) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:1241
 ?? ??:0detail::base_named_type<long, raft::raft_group_id_type, std::__1::integral_constant<bool, true> >::operator==(detail::base_named_type<long, raft::raft_group_id_type, std::__1::integral_constant<bool, true> > const&) const at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0285c386480c1c2d3-1/vectorized/redpanda/vbuild/release/clang/../../../src/v/utils/named_type.h:40
 (inlined by) std::__1::equal_to<detail::base_named_type<long, raft::raft_group_id_type, std::__1::integral_constant<bool, true> > >::operator()(detail::base_named_type<long, raft::raft_group_id_type, std::__1::integral_constant<bool, true> > const&, detail::base_named_type<long, raft::raft_group_id_type, std::__1::integral_constant<bool, true> > const&) const at /vectorized/llvm/bin/../include/c++/v1/functional:692
 (inlined by) bool absl::lts_20210324::container_internal::raw_hash_set<absl::lts_20210324::container_internal::FlatHashMapPolicy<detail::base_named_type<long, raft::raft_group_id_type, std::__1::integral_constant<bool, true> >, raft::heartbeat_manager::follower_request_meta>, absl::lts_20210324::hash_internal::Hash<detail::base_named_type<long, raft::raft_group_id_type, std::__1::integral_constant<bool, true> > >, std::__1::equal_to<detail::base_named_type<long, raft::raft_group_id_type, std::__1::integral_constant<bool, true> > >, std::__1::allocator<std::__1::pair<detail::base_named_type<long, raft::raft_group_id_type, std::__1::integral_constant<bool, true> > const, raft::heartbeat_manager::follower_request_meta> > >::EqualElement<detail::base_named_type<long, raft::raft_group_id_type, std::__1::integral_constant<bool, true> > >::operator()<detail::base_named_type<long, raft::raft_group_id_type, std::__1::integral_constant<bool, true> >, std::__1::piecewise_construct_t const&, std::__1::tuple<detail::base_named_type<long, raft::raft_group_id_type, std::__1::integral_constant<bool, true> > const&>, std::__1::tuple<raft::heartbeat_manager::follower_request_meta const&> >(detail::base_named_type<long, raft::raft_group_id_type, std::__1::integral_constant<bool, true> > const&, std::__1::piecewise_construct_t const&, std::__1::tuple<detail::base_named_type<long, raft::raft_group_id_type, std::__1::integral_constant<bool, true> > const&>&&, std::__1::tuple<raft::heartbeat_manager::follower_request_meta const&>&&) const at /vectorized/include/absl/container/internal/raw_hash_set.h:1474
 (inlined by) _ZN4absl12lts_2021032418container_internal15memory_internal17DecomposePairImplINS1_12raw_hash_setINS1_17FlatHashMapPolicyIN6detail15base_named_typeIlN4raft18raft_group_id_typeENSt3__117integral_constantIbLb1EEEEENS8_17heartbeat_manager21follower_request_metaEEENS0_13hash_internal4HashISD_EENSA_8equal_toISD_EENSA_9allocatorINSA_4pairIKSD_SF_EEEEE12EqualElementISD_EERSO_NSA_5tupleIJRKSF_EEEEEDTclclsr3stdE7declvalIT_EEclsr3stdE7declvalIRKT0_EEL_ZNSA_L19piecewise_constructEEclsr3stdE7declvalINSV_IJS10_EEEEEclsr3stdE7declvalIT1_EEEEOSZ_NSN_IS13_S14_EE at /vectorized/include/absl/container/internal/container_memory.h:139
 (inlined by) _ZN4absl12lts_2021032418container_internal13DecomposePairINS1_12raw_hash_setINS1_17FlatHashMapPolicyIN6detail15base_named_typeIlN4raft18raft_group_id_typeENSt3__117integral_constantIbLb1EEEEENS7_17heartbeat_manager21follower_request_metaEEENS0_13hash_internal4HashISC_EENS9_8equal_toISC_EENS9_9allocatorINS9_4pairIKSC_SE_EEEEE12EqualElementISC_EEJRSO_EEEDTclsr15memory_internalE17DecomposePairImplclsr3stdE7forwardIT_Efp_Ecl8PairArgsspclsr3stdE7forwardIT0_Efp0_EEEEOSU_DpOSV_ at /vectorized/include/absl/container/internal/container_memory.h:206
 (inlined by) _ZN4absl12lts_2021032418container_internal17FlatHashMapPolicyIN6detail15base_named_typeIlN4raft18raft_group_id_typeENSt3__117integral_constantIbLb1EEEEENS5_17heartbeat_manager21follower_request_metaEE5applyINS1_12raw_hash_setISD_NS0_13hash_internal4HashISA_EENS7_8equal_toISA_EENS7_9allocatorINS7_4pairIKSA_SC_EEEEE12EqualElementISA_EEJRSO_EEEDTclsr4absl18container_internalE13DecomposePairclsr3stdE7declvalIT_EEspclsr3stdE7declvalIT0_EEEEOSU_DpOSV_ at /vectorized/include/absl/container/flat_hash_map.h:580
 (inlined by) _ZN4absl12lts_2021032418container_internal18hash_policy_traitsINS1_17FlatHashMapPolicyIN6detail15base_named_typeIlN4raft18raft_group_id_typeENSt3__117integral_constantIbLb1EEEEENS6_17heartbeat_manager21follower_request_metaEEEvE5applyINS1_12raw_hash_setISE_NS0_13hash_internal4HashISB_EENS8_8equal_toISB_EENS8_9allocatorINS8_4pairIKSB_SD_EEEEE12EqualElementISB_EEJRSQ_ESE_EEDTclsrT1_5applyclsr3stdE7forwardIT_Efp_Espclsr3stdE7forwardIT0_Efp0_EEEOSX_DpOSY_ at /vectorized/include/absl/container/internal/hash_policy_traits.h:170
 (inlined by) absl::lts_20210324::container_internal::raw_hash_set<absl::lts_20210324::container_internal::FlatHashMapPolicy<detail::base_named_type<long, raft::raft_group_id_type, std::__1::integral_constant<bool, true> >, raft::heartbeat_manager::follower_request_meta>, absl::lts_20210324::hash_internal::Hash<detail::base_named_type<long, raft::raft_group_id_type, std::__1::integral_constant<bool, true> > >, std::__1::equal_to<detail::base_named_type<long, raft::raft_group_id_type, std::__1::integral_constant<bool, true> > >, std::__1::allocator<std::__1::pair<detail::base_named_type<long, raft::raft_group_id_type, std::__1::integral_constant<bool, true> > const, raft::heartbeat_manager::follower_request_meta> > >::iterator absl::lts_20210324::container_internal::raw_hash_set<absl::lts_20210324::container_internal::FlatHashMapPolicy<detail::base_named_type<long, raft::raft_group_id_type, std::__1::integral_constant<bool, true> >, raft::heartbeat_manager::follower_request_meta>, absl::lts_20210324::hash_internal::Hash<detail::base_named_type<long,raft::raft_group_id_type, std::__1::integral_constant<bool, true> > >, std::__1::equal_to<detail::base_named_type<long, raft::raft_group_id_type, std::__1::integral_constant<bool, true> > >, std::__1::allocator<std::__1::pair<detail::base_named_type<long, raft::raft_group_id_type, std::__1::integral_constant<bool, true> > const, raft::heartbeat_manager::follower_request_meta> > >::find<detail::base_named_type<long, raft::raft_group_id_type, std::__1::integral_constant<bool, true> > >(detail::base_named_type<long, raft::raft_group_id_type, std::__1::integral_constant<bool, true> > const&, unsigned long) at /vectorized/include/absl/container/internal/raw_hash_set.h:1374
 (inlined by) raft::heartbeat_manager::process_reply(detail::base_named_type<int, model::node_id_model_type, std::__1::integral_constant<bool, true> >, absl::lts_20210324::flat_hash_map<detail::base_named_type<long, raft::raft_group_id_type, std::__1::integral_constant<bool, true> >, raft::heartbeat_manager::follower_request_meta, absl::lts_20210324::hash_internal::Hash<detail::base_named_type<long, raft::raft_group_id_type, std::__1::integral_constant<bool, true> > >, std::__1::equal_to<detail::base_named_type<long, raft::raft_group_id_type, std::__1::integral_constant<bool, true> > >, std::__1::allocator<std::__1::pair<detail::base_named_type<long, raft::raft_group_id_type, std::__1::integral_constant<bool, true> > const, raft::heartbeat_manager::follower_request_meta> > >, boost::outcome_v2::basic_result<raft::heartbeat_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::heartbeat_reply, std::__1::error_code, void> >) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0285c386480c1c2d3-1/vectorized/redpanda/vbuild/release/clang/../../../src/v/raft/heartbeat_manager.cc:235
operator() at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0285c386480c1c2d3-1/vectorized/redpanda/vbuild/release/clang/../../../src/v/raft/heartbeat_manager.cc:187
 (inlined by) _ZNSt3__18__invokeIRZN4raft17heartbeat_manager12do_heartbeatEONS2_14node_heartbeatEE3$_6JN5boost10outcome_v212basic_resultINS1_15heartbeat_replyENS_10error_codeENS8_6policy32error_code_throw_as_system_errorISA_SB_vEEEEEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOSG_DpOSH_ at /vectorized/llvm/bin/../include/c++/v1/type_traits:3694
 (inlined by) std::__1::invoke_result<raft::heartbeat_manager::do_heartbeat(raft::heartbeat_manager::node_heartbeat&&)::$_6&, boost::outcome_v2::basic_result<raft::heartbeat_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::heartbeat_reply, std::__1::error_code, void> > >::type std::__1::invoke<raft::heartbeat_manager::do_heartbeat(raft::heartbeat_manager::node_heartbeat&&)::$_6&, boost::outcome_v2::basic_result<raft::heartbeat_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::heartbeat_reply, std::__1::error_code, void> > >(raft::heartbeat_manager::do_heartbeat(raft::heartbeat_manager::node_heartbeat&&)::$_6&, boost::outcome_v2::basic_result<raft::heartbeat_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::heartbeat_reply, std::__1::error_code, void> >&&) at /vectorized/llvm/bin/../include/c++/v1/functional:2989
 (inlined by) auto seastar::internal::future_invoke<raft::heartbeat_manager::do_heartbeat(raft::heartbeat_manager::node_heartbeat&&)::$_6&, boost::outcome_v2::basic_result<raft::heartbeat_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::heartbeat_reply, std::__1::error_code, void> > >(raft::heartbeat_manager::do_heartbeat(raft::heartbeat_manager::node_heartbeat&&)::$_6&, boost::outcome_v2::basic_result<raft::heartbeat_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::heartbeat_reply, std::__1::error_code, void> >&&) at /vectorized/include/seastar/core/future.hh:1211
 (inlined by) operator() at /vectorized/include/seastar/core/future.hh:1582
 (inlined by) void seastar::futurize<void>::satisfy_with_result_of<seastar::future<boost::outcome_v2::basic_result<raft::heartbeat_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::heartbeat_reply, std::__1::error_code, void> > >::then_impl_nrvo<raft::heartbeat_manager::do_heartbeat(raft::heartbeat_manager::node_heartbeat&&)::$_6, seastar::future<void> >(raft::heartbeat_manager::do_heartbeat(raft::heartbeat_manager::node_heartbeat&&)::$_6&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, raft::heartbeat_manager::do_heartbeat(raft::heartbeat_manager::node_heartbeat&&)::$_6&,seastar::future_state<boost::outcome_v2::basic_result<raft::heartbeat_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::heartbeat_reply, std::__1::error_code, void> > >&&)#1}::operator()(seastar::internal::promise_base_with_type<void>&&, raft::heartbeat_manager::do_heartbeat(raft::heartbeat_manager::node_heartbeat&&)::$_6&, seastar::future_state<boost::outcome_v2::basic_result<raft::heartbeat_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::heartbeat_reply, std::__1::error_code, void> > >&&) const::{lambda()#1}>(seastar::internal::promise_base_with_type<void>&&, raft::heartbeat_manager::do_heartbeat(raft::heartbeat_manager::node_heartbeat&&)::$_6&&) at /vectorized/include/seastar/core/future.hh:2117
 (inlined by) operator() at /vectorized/include/seastar/core/future.hh:1575
 (inlined by) seastar::continuation<seastar::internal::promise_base_with_type<void>, raft::heartbeat_manager::do_heartbeat(raft::heartbeat_manager::node_heartbeat&&)::$_6, seastar::future<boost::outcome_v2::basic_result<raft::heartbeat_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::heartbeat_reply, std::__1::error_code, void> > >::then_impl_nrvo<raft::heartbeat_manager::do_heartbeat(raft::heartbeat_manager::node_heartbeat&&)::$_6, seastar::future<void> >(raft::heartbeat_manager::do_heartbeat(raft::heartbeat_manager::node_heartbeat&&)::$_6&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, raft::heartbeat_manager::do_heartbeat(raft::heartbeat_manager::node_heartbeat&&)::$_6&, seastar::future_state<boost::outcome_v2::basic_result<raft::heartbeat_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::heartbeat_reply, std::__1::error_code, void> > >&&)#1}, boost::outcome_v2::basic_result<raft::heartbeat_reply, std::__1::error_code, boost::outcome_v2::policy::error_code_throw_as_system_error<raft::heartbeat_reply, std::__1::error_code, void> > >::run_and_dispose() at /vectorized/include/seastar/core/future.hh:767
 seastar::reactor::run_tasks(seastar::reactor::task_queue&) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2263
 (inlined by) seastar::reactor::run_some_tasks() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2672
seastar::reactor::run() at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2831
seastar::app_template::run_deprecated(int, char**, std::__1::function<void ()>&&) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/app-template.cc:207
seastar::app_template::run(int, char**, std::__1::function<seastar::future<int> ()>&&) at /v/build/v_deps_build/seastar-prefix/src/seastar/src/core/app-template.cc:115
application::run(int, char**) at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0285c386480c1c2d3-1/vectorized/redpanda/vbuild/release/clang/../../../src/v/redpanda/application.cc:130
main at /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-0285c386480c1c2d3-1/vectorized/redpanda/vbuild/release/clang/../../../src/v/redpanda/main.cc:38

[andrewhsu]

this seems like the closest issue i can find for what i see...

build 30025 from PR #10521, ducktape-build-debug-clang error in:

test_id:    rptest.tests.rpk_start_test.RpkRedpandaStartTest.test_rpc_tls_enable
status:     FAIL
run time:   1 minute 17.886 seconds


    TimeoutError('Redpanda service docker-rp-11 failed to start within 60 sec using rpk')
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 135, in run
    data = self.run_test()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 227, in run_test
    return self.test_context.function(self.test)
  File "/root/tests/rptest/services/cluster.py", line 49, in wrapped
    r = f(self, *args, **kwargs)
  File "/root/tests/rptest/tests/rpk_start_test.py", line 388, in test_rpc_tls_enable
    self.redpanda._for_nodes(self.redpanda.nodes,
  File "/root/tests/rptest/services/redpanda.py", line 999, in _for_nodes
    return list(executor.map(cb, nodes))
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 621, in result_iterator
    yield _result_or_cancel(fs.pop())
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 319, in _result_or_cancel
    return fut.result(timeout)
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 458, in result
    return self.__get_result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/root/tests/rptest/tests/rpk_start_test.py", line 384, in start_cluster
    self.redpanda.start_node_with_rpk(node, args, clean_node=False)
  File "/root/tests/rptest/services/redpanda.py", line 2186, in start_node_with_rpk
    self.start_service(node, start_rp)
  File "/root/tests/rptest/services/redpanda.py", line 2216, in start_service
    start()
  File "/root/tests/rptest/services/redpanda.py", line 2177, in start_rp
    wait_until(
  File "/usr/local/lib/python3.10/dist-packages/ducktape/utils/util.py", line 57, in wait_until
    raise TimeoutError(err_msg() if callable(err_msg) else err_msg) from last_exception
ducktape.errors.TimeoutError: Redpanda service docker-rp-11 failed to start within 60 sec using rpk

led me to look in the docker logs and i found:

INFO  2023-05-26 22:19:24,440 [shard  0] main - application.cc:1177 - Partition manager started
Reactor stalled for 54 ms on shard 0. Backtrace: 0xeb526 0x46a06c5 0x469fb3a 0x4461bd3 0x445cd05 0x445c7b7 0x445cf84 0x44631aa 0x42abf 0xf8f4f 0x3eaf4bf 0x410fff3 0x410fc65 0x672d33d 0x672b13a 0x6729d9a 0x786ffc7 0x742a664 0x7429701 0x7428f6f 0x7427794 0x7426ee2 0x742444d 0x74234d2 0x742288f 0x74226a3 0x7421fdf 0x7421cb9 0x74219e0 0x742160d 0x74213ca 0x742122f 0x7421183 0x741d16f 0x4a03800 0x4a035f8 0x4a033ea 0x4a032ff 0x4a02eaa 0x4a02a19 0x4a0392f 0x4a02b6b 0x4a0269a 0x4a023df 0x684e9e0 0x66343f9 0x635f011 0x6347ec3 0x637a1e6 0x6399da0 0x63997b0 0x6399754 0x63996ea 0x6399547 0x63992c2 0x63990d8 0x40d82bd 0x4ad1d14
kernel callstack:
Reactor stalled for 1174 ms on shard 15. Backtrace: 0xeb526 0x46a06c5 0x469fb3a 0x4461bd3 0x445cd05 0x445c7b7 0x445cf84 0x44631aa 0x42abf 0x1826ac5f 0x1b91676d 0x3c58dcb 0x414c98b 0x414c848 0x414c2a4 0x414b75d 0x41327b0 0x4112916 0x4115a6b 0x4106df6 0x672b8fc 0x6729d9a 0x786ffc7 0x742a664 0x7429701 0x7428f6f 0x7427794 0x7426ee2 0x742444d 0x74234d2 0x742288f 0x74226a3 0x743c70f 0x743c1fd 0x4485864 0x4492981 0x4498a3c 0x4612a1d 0x4611620 0x46114e0 0x4611484 0x460ca60 0x57b0301 0x57b0118 0x426cfdc 0x91016 0x1166cf

however, the same test passed for ducktape-build-release-clang

[abhijat]

https://buildkite.com/redpanda/redpanda/builds/30108#018868e9-8cf3-4d68-a263-ae0e21754d82

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 135, in run
    data = self.run_test()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 227, in run_test
    return self.test_context.function(self.test)
  File "/root/tests/rptest/services/cluster.py", line 49, in wrapped
    r = f(self, *args, **kwargs)
  File "/root/tests/rptest/tests/rpk_start_test.py", line 388, in test_rpc_tls_enable
    self.redpanda._for_nodes(self.redpanda.nodes,
  File "/root/tests/rptest/services/redpanda.py", line 999, in _for_nodes
    return list(executor.map(cb, nodes))
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 621, in result_iterator
    yield _result_or_cancel(fs.pop())
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 319, in _result_or_cancel
    return fut.result(timeout)
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 458, in result
    return self.__get_result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/root/tests/rptest/tests/rpk_start_test.py", line 384, in start_cluster
    self.redpanda.start_node_with_rpk(node, args, clean_node=False)
  File "/root/tests/rptest/services/redpanda.py", line 2186, in start_node_with_rpk
    self.start_service(node, start_rp)
  File "/root/tests/rptest/services/redpanda.py", line 2216, in start_service
    start()
  File "/root/tests/rptest/services/redpanda.py", line 2177, in start_rp
    wait_until(
  File "/usr/local/lib/python3.10/dist-packages/ducktape/utils/util.py", line 57, in wait_until
    raise TimeoutError(err_msg() if callable(err_msg) else err_msg) from last_exception
ducktape.errors.TimeoutError: Redpanda service docker-rp-19 failed to start within 60 sec using rpk

In broker log there are several reactor stalls:

Reactor stalled for 222 ms on shard 35. Backtrace: 0xeb526 0x46a06c5 0x469fb3a 0x4461bd3 0x445cd05 0x445c7b7 0x445cf84 0x44631aa 0x42abf 0xa8d16 0xa8943 0x14030e 0x1a1afb 0x4f44fa5 0x4116287 0x4116e91 0x4111c4d 0x41119fa 0x4111da9 0x400df44 0x400de74 0x412c24c 0x4107478 0x742b136 0x742af8f 0x742aea8 0x742af08 0x6840e9e 0x691175a 0x6911528 0x6911274 0x6910123 0x691466d 0x4485864 0x4492981 0x4498a3c 0x4612a1d 0x4611620 0x46114e0 0x4611484 0x460ca60 0x57b0301 0x57b0118 0x426cfdc 0x91016 0x1166cf
kernel callstack: 0xffffffffffffff80 0xffffffffae06f9b0 0xffffffffae976d72 0xffffffffaea00b62
Reactor stalled for 222 ms on shard 9. Backtrace: 0xeb526 0x46a06c5 0x469fb3a 0x4461bd3 0x445cd05 0x445c7b7 0x445cf84 0x44631aa 0x42abf 0xb49a7 0x60763f6 0x607628f 0x607611e 0x6076033 0x415d184 0x415c1ab 0x415753f 0x41166cb 0x4116245 0x4116e91 0x4111c4d 0x41119fa 0x4111da9 0x400df44 0x400de74 0x412c24c 0x4107478 0x742b136 0x742af8f 0x742aea8 0x742af08 0x6840e9e 0x691175a 0x6911528 0x6911274 0x6910123 0x691466d 0x4485864 0x4492981 0x4498a3c 0x4612a1d 0x4611620 0x46114e0 0x4611484 0x460ca60 0x57b0301 0x57b0118 0x426cfdc 0x91016 0x1166cf
kernel callstack:
Reactor stalled for 224 ms on shard 31. Backtrace: 0xeb526 0x46a06c5 0x469fb3a 0x4461bd3 0x445cd05 0x445c7b7 0x445cf84 0x44631aa 0x42abf 0x6074057 0x6074002 0x6073e00 0x939c11 0x9cb2be 0x9ca32f 0x4157270 0x4116691 0x4116245 0x41178e1 0x4116df2 0x4111c4d 0x41119fa 0x4111da9 0x400df44 0x400de74 0x412c24c 0x4107478 0x742b136 0x742af8f 0x742aea8 0x742af08 0x6840e9e 0x691175a 0x6911528 0x6911274 0x6910123 0x691466d 0x4485864 0x4492981 0x4498a3c 0x4612a1d 0x4611620 0x46114e0 0x4611484 0x460ca60 0x57b0301 0x57b0118 0x426cfdc 0x91016 0x1166cf
kernel callstack:
Reactor stalled for 228 ms on shard 5. Backtrace: 0xeb526 0x46a06c5 0x469fb3a 0x4461bd3 0x445cd05 0x445c7b7 0x445cf84 0x44631aa 0x42abf 0x16a2c3 0x14adcf 0x1402ff 0x19c057 0x9cb016 0x9ca4c3 0x4157270 0x4116679 0x4116245 0x4116e91 0x4111c4d 0x41119fa 0x4111da9 0x400df44 0x400de74 0x412c24c 0x4107478 0x742b136 0x742af8f 0x742aea8 0x742af08 0x6840e9e 0x691175a 0x6911528 0x6911274 0x6910123 0x691466d 0x4485864 0x4492981 0x4498a3c 0x4612a1d 0x4611620 0x46114e0 0x4611484 0x460ca60 0x57b0301 0x57b0118 0x426cfdc 0x91016 0x1166cf
kernel callstack:
Reactor stalled for 224 ms on shard 7. Backtrace: 0xeb526 0x46a06c5 0x469fb3a 0x4461bd3 0x445cd05 0x445c7b7 0x445cf84 0x44631aa 0x42abf 0x6076afd 0x6076534 0x607628f 0x607611e 0x6076033 0x415d184 0x415c1ab 0x415753f 0x41166cb 0x4116245 0x4116e91 0x4111c4d 0x41119fa 0x4111da9 0x400df44 0x400de74 0x412c24c 0x4107478 0x742b136 0x742af8f 0x742aea8 0x742af08 0x6840e9e 0x691175a 0x6911528 0x6911274 0x6910123 0x691466d 0x4485864 0x4492981 0x4498a3c 0x4612a1d 0x4611620 0x46114e0 0x4611484 0x460ca60 0x57b0301 0x57b0118 0x426cfdc 0x91016 0x1166cf
kernel callstack:
Reactor stalled for 228 ms on shard 13. Backtrace: 0xeb526 0x46a06c5 0x469fb3a 0x4461bd3 0x445cd05 0x445c7b7 0x445cf84 0x44631aa 0x42abf 0x4f40ff1 0x4f407aa 0x4f400fe 0x4f3fcd4 0x4f3fab4 0x4f3f85d 0x607366e 0x19596d9 0x41166b9 0x4116245 0x4116e91 0x4111c4d 0x41119fa 0x4111da9 0x400df44 0x400de74 0x412c24c 0x4107478 0x742b136 0x742af8f 0x742aea8 0x742af08 0x6840e9e 0x691175a 0x6911528 0x6911274 0x6910123 0x691466d 0x4485864 0x4492981 0x4498a3c 0x4612a1d 0x4611620 0x46114e0 0x4611484 0x460ca60 0x57b0301 0x57b0118 0x426cfdc 0x91016 0x1166cf
kernel callstack:
Reactor stalled for 228 ms on shard 1. Backtrace: 0xeb526 0x46a06c5 0x469fb3a 0x4461bd3 0x445cd05 0x445c7b7 0x445cf84 0x44631aa 0x42abf 0x5ef26af 0x5ef91fd 0x5ef9138 0x5ef908c 0x5ef901c 0x5ef8a28 0xbe2bce 0x742aff9 0x742aea8 0x742af08 0x6840e9e 0x691175a 0x6911528 0x6911274 0x6910123 0x691466d 0x4485864 0x4492981 0x4498a3c 0x4612a1d 0x4611620 0x46114e0 0x4611484 0x460ca60 0x57b0301 0x57b0118 0x426cfdc 0x91016 0x1166cf
Reactor stalled for 227 ms on shard 43. Backtrace: 0xeb526 0x46a06c5 0x469fb3a 0x4461bd3 0x445cd05 0x445c7b7 0x445cf84 0x44631aa 0x42abf 0xb4932 0x60763f6 0x607628f 0x607611e 0x6076033 0x415d184 0x415c1ab 0x415753f 0x41166cb 0x4116245 0x4116e91 0x4111c4d 0x41119fa 0x4111da9 0x400df44 0x400de74 0x412c24c 0x4107478 0x742b136 0x742af8f 0x742aea8 0x742af08 0x6840e9e 0x691175a 0x6911528 0x6911274 0x6910123 0x691466d 0x4485864 0x4492981 0x4498a3c 0x4612a1d 0x4611620 0x46114e0 0x4611484 0x460ca60 0x57b0301 0x57b0118 0x426cfdc 0x91016 0x1166cf
kernel callstack:
kernel callstack:
Reactor stalled for 232 ms on shard 12. Backtrace: 0xeb526 0x46a06c5 0x469fb3a 0x4461bd3 0x445cd05 0x445c7b7 0x445cf84 0x44631aa 0x42abf 0x2d23c51 0x9cb4c6 0x9ca32f 0x4157270 0x4116691 0x4116245 0x4116e91 0x4111c4d 0x41119fa 0x4111da9 0x400df44 0x400de74 0x412c24c 0x4107478 0x742b136 0x742af8f 0x742aea8 0x742af08 0x6840e9e 0x691175a 0x6911528 0x6911274 0x6910123 0x691466d 0x4485864 0x4492981 0x4498a3c 0x4612a1d 0x4611620 0x46114e0 0x4611484 0x460ca60 0x57b0301 0x57b0118 0x426cfdc 0x91016 0x1166cf
kernel callstack:
Reactor stalled for 226 ms on shard 46. Backtrace: 0xeb526 0x46a06c5 0x469fb3a 0x4461bd3 0x445cd05 0x445c7b7 0x445cf84 0x44631aa 0x42abf 0xb48e6 0x1a2246 0x1a20ad 0x1a1884 0x17f2d6 0x41161cc 0x41178e1 0x4116df2 0x4111c4d 0x41119fa 0x4111da9 0x400df44 0x400de74 0x412c24c 0x4107478 0x742b136 0x742af8f 0x742aea8 0x742af08 0x6840e9e 0x691175a 0x6911528 0x6911274 0x6910123 0x691466d 0x4485864 0x4492981 0x4498a3c 0x4612a1d 0x4611620 0x46114e0 0x4611484 0x460ca60 0x57b0301 0x57b0118 0x426cfdc 0x91016 0x1166cf
kernel callstack:
Reactor stalled for 230 ms on shard 39. Backtrace: 0xeb526 0x46a06c5 0x469fb3a 0x4461bd3 0x445cd05 0x445c7b7 0x445cf84 0x44631aa 0x42abf 0x6073ee1 0x6073da0 0x939bd0 0x9cb2be 0x9ca32f 0x4157270 0x4116679 0x4116245 0x41178e1 0x4116df2 0x4111c4d 0x41119fa 0x4111da9 0x400df44 0x400de74 0x412c24c 0x4107478 0x742b420 0x742afef 0x742aea8 0x742af08 0x6840e9e 0x691175a 0x6911528 0x6911274 0x6910123 0x691466d 0x4485864 0x4492981 0x4498a3c 0x4612a1d 0x4611620 0x46114e0 0x4611484 0x460ca60 0x57b0301 0x57b0118 0x426cfdc 0x91016 0x1166cf
kernel callstack:
Reactor stalled for 228 ms on shard 47. Backtrace: 0xeb526 0x46a06c5 0x469fb3a 0x4461bd3 0x445cd05 0x445c7b7 0x445cf84 0x44631aa 0x42abf 0x4117420 0x4111c4d 0x41119fa 0x4111da9 0x400df44 0x400de74 0x412c24c 0x4107478 0x742b136 0x742af8f 0x742aea8 0x742af08 0x6840e9e 0x691175a 0x6911528 0x6911274 0x6910123 0x691466d 0x4485864 0x4492981 0x4498a3c 0x4612a1d 0x4611620 0x46114e0 0x4611484 0x460ca60 0x57b0301 0x57b0118 0x426cfdc 0x91016 0x1166cf
kernel callstack:
Reactor stalled for 231 ms on shard 8. Backtrace: 0xeb526 0x46a06c5 0x469fb3a 0x4461bd3 0x445cd05 0x445c7b7 0x445cf84 0x44631aa 0x42abf 0x1a20e5 0x1a1e16 0x4f44fa5 0x4116287 0x41178e1 0x4116df2 0x4111c4d 0x41119fa 0x4111da9 0x400df44 0x400de74 0x412c24c 0x4107478 0x742b136 0x742af8f 0x742aea8 0x742af08 0x6840e9e 0x691175a 0x6911528 0x6911274 0x6910123 0x691466d 0x4485864 0x4492981 0x4498a3c 0x4612a1d 0x4611620 0x46114e0 0x4611484 0x460ca60 0x57b0301 0x57b0118 0x426cfdc 0x91016 0x1166cf
kernel callstack:
Reactor stalled for 234 ms on shard 4. Backtrace: 0xeb526 0x46a06c5 0x469fb3a 0x4461bd3 0x445cd05 0x445c7b7 0x445cf84 0x44631aa 0x42abf 0x607654b 0x607628f 0x607611e 0x6076033 0x415d184 0x415c1ab 0x415753f 0x41166cb 0x4116245 0x41178e1 0x4116df2 0x4111c4d 0x41119fa 0x4111da9 0x400df44 0x400de74 0x412c24c 0x4107478 0x742b136 0x742af8f 0x742aea8 0x742af08 0x6840e9e 0x691175a 0x6911528 0x6911274 0x6910123 0x691466d 0x4485864 0x4492981 0x4498a3c 0x4612a1d 0x4611620 0x46114e0 0x4611484 0x460ca60 0x57b0301 0x57b0118 0x426cfdc 0x91016 0x1166cf
kernel callstack:
Reactor stalled for 231 ms on shard 14. Backtrace: 0xeb526 0x46a06c5 0x469fb3a 0x4461bd3 0x445cd05 0x445c7b7 0x445cf84 0x44631aa 0x42abf 0x13f237 0x60764bb 0x607628f 0x607611e 0x6076033 0x415d184 0x415c1ab 0x415753f 0x41166cb 0x4116245 0x4116e91 0x4111c4d 0x41119fa 0x4111da9 0x400df44 0x400de74 0x412c24c 0x4107478 0x742b136 0x742af8f 0x742aea8 0x742af08 0x6840e9e 0x691175a 0x6911528 0x6911274 0x6910123 0x691466d 0x4485864 0x4492981 0x4498a3c 0x4612a1d 0x4611620 0x46114e0 0x4611484 0x460ca60 0x57b0301 0x57b0118 0x426cfdc 0x91016 0x1166cf
kernel callstack:
Reactor stalled for 236 ms on shard 10. Backtrace: 0xeb526 0x46a06c5 0x469fb3a 0x4461bd3 0x445cd05 0x445c7b7 0x445cf84 0x44631aa 0x42abf 0x6075e32 0x415d184 0x415c1ab 0x415753f 0x41166cb 0x4116245 0x4116e91 0x4111c4d 0x41119fa 0x4111da9 0x400df44 0x400de74 0x412c24c 0x4107478 0x742b136 0x742af8f 0x742aea8 0x742af08 0x6840e9e 0x691175a 0x6911528 0x6911274 0x6910123 0x691466d 0x4485864 0x4492981 0x4498a3c 0x4612a1d 0x4611620 0x46114e0 0x4611484 0x460ca60 0x57b0301 0x57b0118 0x426cfdc 0x91016 0x1166cf
kernel callstack:
Reactor stalled for 233 ms on shard 17. Backtrace: 0xeb526 0x46a06c5 0x469fb3a 0x4461bd3 0x445cd05 0x445c7b7 0x445cf84 0x44631aa 0x42abf 0x4f49ef7 0x4f7784a 0x4f77442 0x414292c 0x4155ed3 0x4155c0d 0x41558ce 0x415543d 0x4154fe2 0x4154b8f 0x416241a 0x41620da 0x416125f 0x4160e27 0x41349ac 0x41170c5 0x4111c4d 0x41119fa 0x4111da9 0x400df44 0x400de74 0x412c24c 0x4107478 0x742b136 0x742af8f 0x742aea8 0x742af08 0x6840e9e 0x691175a 0x6911528 0x6911274 0x6910123 0x691466d 0x4485864 0x4492981 0x4498a3c 0x4612a1d 0x4611620 0x46114e0 0x4611484 0x460ca60 0x57b0301 0x57b0118 0x426cfdc 0x91016 0x1166cf
kernel callstack:
Reactor stalled for 228 ms on shard 34. Backtrace: 0xeb526 0x46a06c5 0x469fb3a 0x4461bd3 0x445cd05 0x445c7b7 0x445cf84 0x44631aa 0x42abf 0x4f4a0ab 0x4f49ee8 0x4f776b5 0x4f77442 0x414292c 0x4156013 0x4155c0d 0x4155807 0x415543d 0x4154fe2 0x4154b8f 0x416241a 0x41620da 0x416125f 0x4160e27 0x41349ac 0x41170c5 0x4111c4d 0x41119fa 0x4111da9 0x400df44 0x400de74 0x412c24c 0x4107478 0x742b136 0x742af8f 0x742aea8 0x742af08 0x6840e9e 0x691175a 0x6911528 0x6911274 0x6910123 0x691466d 0x4485864 0x4492981 0x4498a3c 0x4612a1d 0x4611620 0x46114e0 0x4611484 0x460ca60 0x57b0301 0x57b0118 0x426cfdc 0x91016 0x1166cf