latency degradation during add nodes on mixed workload - >180% degradation

[fgelcer]

Installation details
Scylla version (or git commit hash): 5.4.0~dev.20230629.f6f974cdeb11 with build-id 7afc85749bdc68e7ee32eead35d51badd480c79f
Cluster size: 3
OS (RHEL/CentOS/Ubuntu/AWS AMI): ami-0b7891d4fe168d4b1 (eu-west-1)
test_id: 810279ff-2820-4ca9-8549-f221b56c231f

based on previous runs of this operation, we have a degradation of 185% in the latency (this run had 32.03 ms, where this is the AVG of the 3 add nodes, 71.3, 14.54 and 10.24, while previous one was 11.22 ms).

we have 1817 stalls during the 3 add nodes operations

decoded:

Processed 1817 stalls lasting a total of 15867 milliseconds.
min=6 avg=8.7 median=6 p95=13 p99=18 p999=None max=18

This graph is printed in bottom-up order, where callees are printed first.
Use --direction=top-down to print callers first.

[level#index/out_of pct%] below denotes:
  level  - nesting level in the graph
  index  - index of node among to its siblings
  out_of - number of siblings
  pct    - percentage of total stall time of this call relative to its siblings

Command line options: --address-threshold=0x100000000 --executable=scylla/libexec/scylla --direction=bottom-up --branch-threshold=0.05 --file=mixed/add_nodes_stalls.log

++[0#1/1 100%] addr=0x56aa573 total=15867 count=1817 avg=9:
|              seastar::backtrace<seastar::backtrace_buffer::append_backtrace_oneline()::{lambda(seastar::frame)#1}> at ./build/release/seastar/./seastar/include/seastar/util/backtrace.hh:64
|              (inlined by) ?? at ./build/release/seastar/./seastar/src/core/reactor.cc:825
|              (inlined by) seastar::print_with_backtrace at ./build/release/seastar/./seastar/src/core/reactor.cc:844
++           - addr=0x56a9920:
|              seastar::internal::cpu_stall_detector::generate_trace at ./build/release/seastar/./seastar/src/core/reactor.cc:1414
++           - addr=0x56aad4f:
|              seastar::reactor::block_notifier at ./build/release/seastar/./seastar/src/core/reactor.cc:1151
|              (inlined by) ?? at ./build/release/seastar/./seastar/src/core/reactor.cc:1171
|              (inlined by) seastar::reactor::block_notifier at ./build/release/seastar/./seastar/src/core/reactor.cc:1451
++           - addr=0x3cb1f: ?? ??:0
  ++[1#1/401 10%] addr=0x1ed4ae5 total=1509 count=185 avg=8:
  |               utils::uleb64_decode_forwards<void (&)(char const*, unsigned long), void (&)(char const*, unsigned long)> at ././utils/vle.hh:105
  |               (inlined by) ?? at ./utils/logalloc.cc:1619
  |               (inlined by) ?? at ./utils/logalloc.cc:1717
  |               (inlined by) logalloc::region_impl::compact_segment_locked at ./utils/logalloc.cc:1873
  | ++[2#1/2 54%] addr=0x1ec973b total=821 count=124 avg=7:
  | |             logalloc::tracker::impl::compact_and_evict_locked at ./utils/logalloc.cc:2213
  | |             (inlined by) logalloc::tracker::impl::compact_and_evict_locked at ./utils/logalloc.cc:2694
  | | ++[3#1/1 100%] addr=0x1ed1330 total=5957 count=883 avg=7:
  | | |              logalloc::allocating_section::reserve at ./utils/logalloc.cc:2618
  | | |              (inlined by) ?? at ./utils/logalloc.cc:1279
  | | |              (inlined by) ?? at ./utils/logalloc.cc:1292
  | | |              (inlined by) ?? at ./utils/logalloc.cc:2871
  | |   ++[4#1/13 38%] addr=0x1b749e3 total=5637 count=548 avg=10:
  | |   |              replica::memtable::apply at ././utils/logalloc.hh:469
  | |   |              (inlined by) ?? at ././utils/logalloc.hh:520
  | |   |              (inlined by) replica::memtable::apply(frozen_mutation const&, seastar::lw_shared_ptr<schema const> const&, db::rp_handle&&)::$_9::operator() at ./replica/memtable.cc:789
  | |   |              (inlined by) with_allocator<replica::memtable::apply(frozen_mutation const&, seastar::lw_shared_ptr<schema const> const&, db::rp_handle&&)::$_9> at ././utils/allocation_strategy.hh:313
  | |   |              (inlined by) replica::memtable::apply at ./replica/memtable.cc:788
  | |   ++           - addr=0x1a9850a:
  | |   |              seastar::futurize<void>::invoke<replica::table::apply(frozen_mutation const&, seastar::lw_shared_ptr<schema const>, db::rp_handle&&, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >)::$_45&> at ./replica/table.cc:2257
  | |   |              (inlined by) replica::table::apply(frozen_mutation const&, seastar::lw_shared_ptr<schema const>, db::rp_handle&&, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >)::$_45::operator() at ./replica/table.cc:2280
  | |   |              (inlined by) seastar::futurize<void>::invoke<replica::table::apply(frozen_mutation const&, seastar::lw_shared_ptr<schema const>, db::rp_handle&&, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >)::$_45&> at ././seastar/include/seastar/core/future.hh:2003
  | |   | ++[5#1/1 100%] addr=0x1a11ec9 total=5674 count=553 avg=10:
  | |   | |              seastar::futurize_invoke<replica::table::apply(frozen_mutation const&, seastar::lw_shared_ptr<schema const>, db::rp_handle&&, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >)::$_45&> at ././seastar/include/seastar/core/future.hh:2037
  | |   | |              (inlined by) ?? at ./replica/dirty_memory_manager.hh:572
  | |   | |              (inlined by) replica::table::apply at ./replica/table.cc:2279
  | |   |   ++[6#1/2 99%] addr=0x189c2cd total=5604 count=546 avg=10:
  | |   |   |             replica::database::apply_in_memory(frozen_mutation const&, seastar::lw_shared_ptr<schema const>, db::rp_handle&&, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >)::$_25::operator() at ./replica/database.cc:1817
  | |   |   |             (inlined by) ?? at ././seastar/include/seastar/core/future.hh:2006
  | |   |   |             (inlined by) seastar::futurize_invoke<replica::database::apply_in_memory(frozen_mutation const&, seastar::lw_shared_ptr<schema const>, db::rp_handle&&, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >)::$_25> at ././seastar/include/seastar/core/future.hh:2037
  | |   |   |             (inlined by) seastar::internal::invoke_func_with_gate<replica::database::apply_in_memory(frozen_mutation const&, seastar::lw_shared_ptr<schema const>, db::rp_handle&&, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >)::$_25> at ././seastar/include/seastar/core/gate.hh:278
  | |   |   |             (inlined by) seastar::with_gate<replica::database::apply_in_memory(frozen_mutation const&, seastar::lw_shared_ptr<schema const>, db::rp_handle&&, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >)::$_25> at ././seastar/include/seastar/core/gate.hh:297
  | |   |   |             (inlined by) ?? at ./replica/database.cc:1816
  | |   |   |             (inlined by) replica::database::do_apply at ./replica/database.cc:2040
  | |   |   ++          - addr=0x193c8ef:
  | |   |   |             std::__invoke_impl<seastar::future<void>, seastar::future<void> (replica::database::* const&)(seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>), replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce> > at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:74
  | |   |   |             (inlined by) std::__invoke<seastar::future<void> (replica::database::* const&)(seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>), replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce> > at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:96
  | |   |   |             (inlined by) ?? at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/functional:167
  | |   |   |             (inlined by) seastar::noncopyable_function<seastar::future<void> (replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>)>::direct_vtable_for<std::_Mem_fn<seastar::future<void> (replica::database::*)(seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>)> >::call at ././seastar/include/seastar/util/noncopyable_function.hh:129
  | |   |   ++          - addr=0x195ca09:
  | |   |   |             seastar::noncopyable_function<seastar::future<void> (replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>)>::direct_vtable_for<seastar::inheriting_concrete_execution_stage<seastar::future<void>, replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce> >::make_stage_for_group(seastar::scheduling_group)::{lambda(replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>)#1}>::call at ././seastar/include/seastar/util/noncopyable_function.hh:215
  | |   |   |             (inlined by) seastar::inheriting_concrete_execution_stage<seastar::future<void>, replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce> >::make_stage_for_group(seastar::scheduling_group)::{lambda(replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>)#1}::operator() at ././seastar/include/seastar/core/execution_stage.hh:342
  | |   |   |             (inlined by) seastar::noncopyable_function<seastar::future<void> (replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>)>::direct_vtable_for<seastar::inheriting_concrete_execution_stage<seastar::future<void>, replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce> >::make_stage_for_group(seastar::scheduling_group)::{lambda(replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>)#1}>::call at ././seastar/include/seastar/util/noncopyable_function.hh:129
  | |   |   ++          - addr=0x195c5dd:
  | |   |   |             seastar::concrete_execution_stage<seastar::future<void>, replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce> >::do_flush at ././seastar/include/seastar/util/noncopyable_function.hh:215
  | |   |   |             (inlined by) std::__invoke_impl<seastar::future<void>, seastar::noncopyable_function<seastar::future<void> (replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>)>&, replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce> > at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:61
  | |   |   |             (inlined by) std::__invoke<seastar::noncopyable_function<seastar::future<void> (replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>)>&, replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce> > at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:96
  | |   |   |             (inlined by) std::__apply_impl<seastar::noncopyable_function<seastar::future<void> (replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>)>&, std::tuple<replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce> >, 0ul, 1ul, 2ul, 3ul, 4ul, 5ul, 6ul> at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/tuple:1852
  | |   |   |             (inlined by) std::apply<seastar::noncopyable_function<seastar::future<void> (replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>)>&, std::tuple<replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce> > > at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/tuple:1863
  | |   |   |             (inlined by) ?? at ././seastar/include/seastar/core/future.hh:1973
  | |   |   |             (inlined by) ?? at ././seastar/include/seastar/core/execution_stage.hh:251
  | |   |   ++          - addr=0x5667ac3:
  | |   |   |             seastar::execution_stage::flush()::$_1::operator() at ./build/release/seastar/./seastar/src/core/execution_stage.cc:149
  | |   |   |             (inlined by) ?? at ./build/release/seastar/./seastar/include/seastar/core/future.hh:2003
  | |   |   |             (inlined by) seastar::lambda_task<seastar::execution_stage::flush()::$_1>::run_and_dispose at ./build/release/seastar/./seastar/include/seastar/core/make_task.hh:45
  | |   |   ++          - addr=0x56bb654:
  | |   |   |             seastar::reactor::run_some_tasks at ./build/release/seastar/./seastar/src/core/reactor.cc:2590
  | |   |   |             (inlined by) seastar::reactor::run_some_tasks at ./build/release/seastar/./seastar/src/core/reactor.cc:3053
  | |   |   | ++[7#1/1 100%] addr=0x56bc8d7 total=15745 count=1798 avg=9:
  | |   |   | |              seastar::reactor::do_run at ./build/release/seastar/./seastar/src/core/reactor.cc:3222
  | |   |   |   ++[8#1/2 85%] addr=0x56df0a1 total=13351 count=1531 avg=9:
  | |   |   |   |             seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_96::operator() at ./build/release/seastar/./seastar/src/core/reactor.cc:4426
  | |   |   |   |             (inlined by) std::__invoke_impl<void, seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_96&> at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:61
  | |   |   |   |             (inlined by) std::__invoke_r<void, seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_96&> at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:111
  | |   |   |   |             (inlined by) std::_Function_handler<void (), seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_96>::_M_invoke at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/std_function.h:290
  | |   |   |   ++          - addr=0x568be6a:
  | |   |   |   |             seastar::posix_thread::start_routine at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/std_function.h:591
  | |   |   |   |             (inlined by) seastar::posix_thread::start_routine at ./build/release/seastar/./seastar/src/core/posix.cc:90
  | |   |   |   ++          - addr=0x8b12c: ?? ??:0
  | |   |   |   ++          - addr=0x10cbbf: ?? ??:0
  | |   |   |   ++[8#2/2 15%] addr=0x56bbc19 total=2394 count=267 avg=9:
  | |   |   |   |             seastar::reactor::run at ./build/release/seastar/./seastar/src/core/reactor.cc:3105
  | |   |   |   ++          - addr=0x565fb35:
  | |   |   |   |             seastar::app_template::run_deprecated at ./build/release/seastar/./seastar/src/core/app-template.cc:276
  | |   |   |   ++          - addr=0x565eca8:
  | |   |   |   |             seastar::app_template::run at ./build/release/seastar/./seastar/src/core/app-template.cc:167
  | |   |   |   ++          - addr=0x122de13:
  | |   |   |   |             scylla_main at ./main.cc:627
  | |   |   |   ++          - addr=0x122f9a0:
  | |   |   |   |             std::function<int (int, char**)>::operator() at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/std_function.h:591
  | |   |   |   ++          - addr=0x122c237:
  | |   |   |   |             main at ./main.cc:1995
  | |   |   |   ++          - addr=0x2750f: ?? ??:0
  | |   |   |   ++          - addr=0x275c8: ?? ??:0
  | |   |   |   ++          - addr=0x1229de4:
  | |   |   |   |             _start at ??:?
  | |   |   ++[6#2/2 1%] 1 more branches total=70 count=7 avg=10
  | |   |+[4#2/13 15%] addr=0x214040f total=2210 count=170 avg=13:
  | |   |              sstables::index_consume_entry_context<sstables::index_consumer>::process_state at ././utils/logalloc.hh:469
  | |   |              (inlined by) ?? at ././utils/logalloc.hh:520
  | |   |              (inlined by) ?? at ./sstables/index_reader.hh:71
  | |   |              (inlined by) sstables::index_consume_entry_context<sstables::index_consumer>::process_state at ./sstables/index_reader.hh:271
  | |   |+           - addr=0x213de7a:
  | |   |              data_consumer::continuous_data_consumer<sstables::index_consume_entry_context<sstables::index_consumer> >::operator() at ././sstables/consumer.hh:542
  | |   |              (inlined by) data_consumer::continuous_data_consumer<sstables::index_consume_entry_context<sstables::index_consumer> >::operator() at ././sstables/consumer.hh:559
  | |   | ++[5#1/1 100%] addr=0x213ce96 total=2223 count=171 avg=13:
  | |   | |              std::__invoke_impl<seastar::future<seastar::consumption_result<char> >, sstables::index_consume_entry_context<sstables::index_consumer>&, seastar::temporary_buffer<char> > at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:61
  | |   | |              (inlined by) std::__invoke<sstables::index_consume_entry_context<sstables::index_consumer>&, seastar::temporary_buffer<char> > at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:96
  | |   | |              (inlined by) ?? at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/refwrap.h:358
  | |   | |              (inlined by) operator() at ././seastar/include/seastar/core/iostream-impl.hh:222
  | |   | ++           - addr=0x214778a:
  | |   | |              seastar::internal::repeater<seastar::input_stream<char>::consume<std::reference_wrapper<sstables::index_consume_entry_context<sstables::index_consumer> > >(std::reference_wrapper<sstables::index_consume_entry_context<sstables::index_consumer> >&&)::{lambda()#1}>::run_and_dispose at ././seastar/include/seastar/core/future.hh:2006
  | |   | |              (inlined by) seastar::futurize_invoke<seastar::input_stream<char>::consume<std::reference_wrapper<sstables::index_consume_entry_context<sstables::index_consumer> > >(std::reference_wrapper<sstables::index_consume_entry_context<sstables::index_consumer> >&&)::{lambda()#1}&> at ././seastar/include/seastar/core/future.hh:2037
  | |   | |              (inlined by) seastar::internal::repeater<seastar::input_stream<char>::consume<std::reference_wrapper<sstables::index_consume_entry_context<sstables::index_consumer> > >(std::reference_wrapper<sstables::index_consume_entry_context<sstables::index_consumer> >&&)::{lambda()#1}>::run_and_dispose at ././seastar/include/seastar/core/loop.hh:79
  | |   | ++           - addr=0x56bb654:
  | |   | |              seastar::reactor::run_some_tasks at ./build/release/seastar/./seastar/src/core/reactor.cc:2590
  | |   | |              (inlined by) seastar::reactor::run_some_tasks at ./build/release/seastar/./seastar/src/core/reactor.cc:3053
  | |   | -> continued at addr=0x56bb654 above
  | |   |+[4#3/13 13%] addr=0x2151ea4 total=1952 count=319 avg=6:
  | |   |              seastar::future<void>::then_impl<sstables::index_reader::advance_lower_and_check_if_present(dht::ring_position_view, std::optional<position_in_partition_view>)::{lambda()#1}::operator()() const::{lambda()#1}, seastar::future<bool> > at ././utils/logalloc.hh:469
  | |   |              (inlined by) ?? at ././utils/logalloc.hh:520
  | |   |              (inlined by) sstables::index_reader::advance_lower_and_check_if_present(dht::ring_position_view, std::optional<position_in_partition_view>)::{lambda()#1}::operator()() const::{lambda()#1}::operator() at ./sstables/index_reader.hh:932
  | |   |              (inlined by) ?? at ././seastar/include/seastar/core/future.hh:2006
  | |   |              (inlined by) ?? at ././seastar/include/seastar/core/future.hh:1848
  | |   |              (inlined by) seastar::future<void>::then_impl<sstables::index_reader::advance_lower_and_check_if_present(dht::ring_position_view, std::optional<position_in_partition_view>)::{lambda()#1}::operator()() const::{lambda()#1}, seastar::future<bool> > at ././seastar/include/seastar/core/future.hh:1473
  | |   |+           - addr=0x217b077:
  | |   |              seastar::continuation<seastar::internal::promise_base_with_type<bool>, sstables::index_reader::advance_lower_and_check_if_present(dht::ring_position_view, std::optional<position_in_partition_view>)::{lambda()#1}, seastar::future<void>::then_impl_nrvo<{lambda()#1}, sstables::index_reader::advance_lower_and_check_if_present(dht::ring_position_view, std::optional<position_in_partition_view>)::{lambda()#1}<bool> >({lambda()#1}&&)::{lambda(seastar::internal::promise_base_with_type<bool>&&, {lambda()#1}&, seastar::future_state<seastar::internal::monostate>&&)#1}, void>::run_and_dispose at ././seastar/include/seastar/core/future.hh:1399
  | |   |              (inlined by) sstables::index_reader::advance_lower_and_check_if_present(dht::ring_position_view, std::optional<position_in_partition_view>)::{lambda()#1}::operator() at ./sstables/index_reader.hh:930
  | |   |              (inlined by) std::__invoke_impl<seastar::future<bool>, sstables::index_reader::advance_lower_and_check_if_present(dht::ring_position_view, std::optional<position_in_partition_view>)::{lambda()#1}&> at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:61
  | |   |              (inlined by) std::__invoke<sstables::index_reader::advance_lower_and_check_if_present(dht::ring_position_view, std::optional<position_in_partition_view>)::{lambda()#1}&> at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:96
  | |   |              (inlined by) std::invoke<sstables::index_reader::advance_lower_and_check_if_present(dht::ring_position_view, std::optional<position_in_partition_view>)::{lambda()#1}&> at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/functional:110
  | |   |              (inlined by) seastar::internal::future_invoke<sstables::index_reader::advance_lower_and_check_if_present(dht::ring_position_view, std::optional<position_in_partition_view>)::{lambda()#1}&, seastar::internal::monostate> at ././seastar/include/seastar/core/future.hh:1155
  | |   |              (inlined by) seastar::future<void>::then_impl_nrvo<sstables::index_reader::advance_lower_and_check_if_present(dht::ring_position_view, std::optional<position_in_partition_view>)::{lambda()#1}, seastar::future<bool> >(sstables::index_reader::advance_lower_and_check_if_present(dht::ring_position_view, std::optional<position_in_partition_view>)::{lambda()#1}&&)::{lambda(seastar::internal::promise_base_with_type<bool>&&, {lambda()#1}&, seastar::future_state<auto:1&&::monostate>&&)#1}::operator()(seastar::internal::promise_base_with_type<bool>, seastar::internal::promise_base_with_type<bool>&&, seastar::future_state<sstables::index_reader::advance_lower_and_check_if_present(dht::ring_position_view, std::optional<position_in_partition_view>)::{lambda()#1}&&::monostate>) const::{lambda()#1}::operator() at ././seastar/include/seastar/core/future.hh:1458
  | |   |              (inlined by) ?? at ././seastar/include/seastar/core/future.hh:1991
  | |   |              (inlined by) seastar::future<void>::then_impl_nrvo<sstables::index_reader::advance_lower_and_check_if_present(dht::ring_position_view, std::optional<position_in_partition_view>)::{lambda()#1}, seastar::future<bool> >(sstables::index_reader::advance_lower_and_check_if_present(dht::ring_position_view, std::optional<position_in_partition_view>)::{lambda()#1}&&)::{lambda(seastar::internal::promise_base_with_type<bool>&&, {lambda()#1}&, seastar::future_state<auto:1&&::monostate>&&)#1}::operator() at ././seastar/include/seastar/core/future.hh:1454
  | |   |              (inlined by) seastar::continuation<seastar::internal::promise_base_with_type<bool>, sstables::index_reader::advance_lower_and_check_if_present(dht::ring_position_view, std::optional<position_in_partition_view>)::{lambda()#1}, seastar::future<void>::then_impl_nrvo<{lambda()#1}, sstables::index_reader::advance_lower_and_check_if_present(dht::ring_position_view, std::optional<position_in_partition_view>)::{lambda()#1}<bool> >({lambda()#1}&&)::{lambda(seastar::internal::promise_base_with_type<bool>&&, {lambda()#1}&, seastar::future_state<seastar::internal::monostate>&&)#1}, void>::run_and_dispose at ././seastar/include/seastar/core/future.hh:741
  | |   |+           - addr=0x56bb654:
  | |   |              seastar::reactor::run_some_tasks at ./build/release/seastar/./seastar/src/core/reactor.cc:2590
  | |   |              (inlined by) seastar::reactor::run_some_tasks at ./build/release/seastar/./seastar/src/core/reactor.cc:3053
  | |   -> continued at addr=0x56bb654 above
  | |   |+[4#4/13 13%] addr=0x1b71559 total=1941 count=230 avg=8:
  | |   |              replica::memtable::make_flat_reader_opt at ././utils/logalloc.hh:469
  | |   |              (inlined by) ?? at ././utils/logalloc.hh:520
  | |   |              (inlined by) ?? at ./replica/memtable.cc:709
  | |   |+           - addr=0x19bdec8:
  | |   |              replica::table::add_memtables_to_reader_list(std::vector<flat_mutation_reader_v2, std::allocator<flat_mutation_reader_v2> >&, seastar::lw_shared_ptr<schema const> const&, reader_permit const&, nonwrapping_interval<dht::ring_position> const&, query::partition_slice const&, tracing::trace_state_ptr const&, seastar::bool_class<streamed_mutation::forwarding_tag>, seastar::bool_class<mutation_reader::partition_range_forwarding_tag>, std::function<void (unsigned long)>) const::$_56::operator() at ./replica/table.cc:194
  | |   |              (inlined by) replica::table::add_memtables_to_reader_list at ./replica/table.cc:205
  | |   |+           - addr=0x19be571:
  | |   |              replica::table::make_reader_v2 at ./replica/table.cc:259
  | |   |+           - addr=0x1a98b92:
  | |   |              replica::table::as_mutation_source() const::$_48::operator() at ./replica/table.cc:2484
  | |   |              (inlined by) std::__invoke_impl<flat_mutation_reader_v2, replica::table::as_mutation_source() const::$_48&, seastar::lw_shared_ptr<schema const>, reader_permit, nonwrapping_interval<dht::ring_position> const&, query::partition_slice const&, tracing::trace_state_ptr, seastar::bool_class<streamed_mutation::forwarding_tag>, seastar::bool_class<mutation_reader::partition_range_forwarding_tag> > at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:61
  | |   |              (inlined by) std::__invoke_r<flat_mutation_reader_v2, replica::table::as_mutation_source() const::$_48&, seastar::lw_shared_ptr<schema const>, reader_permit, nonwrapping_interval<dht::ring_position> const&, query::partition_slice const&, tracing::trace_state_ptr, seastar::bool_class<streamed_mutation::forwarding_tag>, seastar::bool_class<mutation_reader::partition_range_forwarding_tag> > at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:114
  | |   |              (inlined by) std::_Function_handler<flat_mutation_reader_v2 (seastar::lw_shared_ptr<schema const>, reader_permit, nonwrapping_interval<dht::ring_position> const&, query::partition_slice const&, tracing::trace_state_ptr, seastar::bool_class<streamed_mutation::forwarding_tag>, seastar::bool_class<mutation_reader::partition_range_forwarding_tag>), replica::table::as_mutation_source() const::$_48>::_M_invoke at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/std_function.h:290
  | |   |+           - addr=0x1a1ae63:
  | |   |              query::querier::querier at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/std_function.h:591
  | |   |              (inlined by) ?? at ././readers/mutation_source.hh:143
  | |   |              (inlined by) query::querier_base::querier_base at ././querier.hh:91
  | |   |              (inlined by) querier at ././querier.hh:158
  | |   |+           - addr=0x1a1665e:
  | |   |              replica::table::query at ./replica/table.cc:2393
  | |   |+           - addr=0x1953667:
  | |   |              replica::database::query(seastar::lw_shared_ptr<schema const>, query::read_command const&, query::result_options, std::vector<nonwrapping_interval<dht::ring_position>, std::allocator<nonwrapping_interval<dht::ring_position> > > const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>)::$_21::operator() at ./replica/database.cc:1568
  | |   |              (inlined by) seastar::noncopyable_function<seastar::future<void> (reader_permit)>::indirect_vtable_for<replica::database::query(seastar::lw_shared_ptr<schema const>, query::read_command const&, query::result_options, std::vector<nonwrapping_interval<dht::ring_position>, std::allocator<nonwrapping_interval<dht::ring_position> > > const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>)::$_21>::call at ././seastar/include/seastar/util/noncopyable_function.hh:158
  | |   | ++[5#1/1 100%] addr=0x429f493 total=1954 count=231 avg=8:
  | |   | |              reader_concurrency_semaphore::execution_loop at ././seastar/include/seastar/util/noncopyable_function.hh:215
  | |   | |              (inlined by) reader_concurrency_semaphore::execution_loop at ./reader_concurrency_semaphore.cc:921
  | |   |   ++[6#1/2 87%] addr=0x12a67da total=1709 count=205 avg=8:
  | |   |   |             seastar::internal::coroutine_traits_base<void>::promise_type::run_and_dispose at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/coroutine:244
  | |   |   |             (inlined by) seastar::internal::coroutine_traits_base<void>::promise_type::run_and_dispose at ././seastar/include/seastar/core/coroutine.hh:125
  | |   |   | ++[7#1/1 100%] addr=0x56bb654 total=1779 count=212 avg=8:
  | |   |   | |              seastar::reactor::run_some_tasks at ./build/release/seastar/./seastar/src/core/reactor.cc:2590
  | |   |   | |              (inlined by) seastar::reactor::run_some_tasks at ./build/release/seastar/./seastar/src/core/reactor.cc:3053
  | |   |   | -> continued at addr=0x56bb654 above
  | |   |   ++[6#2/2 13%] addr=0x1a41d66 total=245 count=26 avg=9:
  | |   |   |             seastar::coroutine::internal::maybe_yield_awaiter::run_and_dispose at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/coroutine:135
  | |   |   |             (inlined by) seastar::coroutine::internal::maybe_yield_awaiter::run_and_dispose at ././seastar/include/seastar/coroutine/maybe_yield.hh:53
  | |   |     ++[7#1/1 100%] addr=0x56bb654 total=1145 count=176 avg=7:
  | |   |     |              seastar::reactor::run_some_tasks at ./build/release/seastar/./seastar/src/core/reactor.cc:2590
  | |   |     |              (inlined by) seastar::reactor::run_some_tasks at ./build/release/seastar/./seastar/src/core/reactor.cc:3053
  | |   |     -> continued at addr=0x56bb654 above
  | |   |+[4#5/13 12%] addr=0x1e2808c total=1770 count=246 avg=7:
  | |   |              cache::cache_flat_mutation_reader::fill_buffer at ././utils/logalloc.hh:469
  | |   |              (inlined by) ?? at ././utils/logalloc.hh:520
  | |   |              (inlined by) ?? at ././row_cache.hh:481
  | |   |              (inlined by) cache::cache_flat_mutation_reader::fill_buffer()::{lambda()#1}::operator() at ././cache_flat_mutation_reader.hh:301
  | |   |              (inlined by) cache::cache_flat_mutation_reader::fill_buffer at ././cache_flat_mutation_reader.hh:309
  | |   | ++[5#1/2 67%] addr=0x1a58b53 total=1184 count=153 avg=8:
  | |   | |             operator() at ././readers/flat_mutation_reader_v2.hh:209
  | |   | | ++[6#1/1 100%] addr=0x1a5bc0a total=1190 count=154 avg=8:
  | |   | | |              seastar::internal::repeater<flat_mutation_reader_v2::impl::consume_pausable<std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<query_result_builder> > > >(std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<query_result_builder> > >)::{lambda()#1}>::run_and_dispose at ././seastar/include/seastar/core/future.hh:2006
  | |   | | |              (inlined by) seastar::futurize_invoke<flat_mutation_reader_v2::impl::consume_pausable<std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<query_result_builder> > > >(std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<query_result_builder> > >)::{lambda()#1}&> at ././seastar/include/seastar/core/future.hh:2037
  | |   | | |              (inlined by) seastar::internal::repeater<flat_mutation_reader_v2::impl::consume_pausable<std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<query_result_builder> > > >(std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<query_result_builder> > >)::{lambda()#1}>::run_and_dispose at ././seastar/include/seastar/core/loop.hh:79
  | |   | | ++           - addr=0x56bb654:
  | |   | | |              seastar::reactor::run_some_tasks at ./build/release/seastar/./seastar/src/core/reactor.cc:2590
  | |   | | |              (inlined by) seastar::reactor::run_some_tasks at ./build/release/seastar/./seastar/src/core/reactor.cc:3053
  | |   | | -> continued at addr=0x56bb654 above
  | |   | ++[5#2/2 33%] addr=0x1fcc64f total=586 count=93 avg=6:
  | |   | |             flat_mutation_reader_v2::impl::fill_buffer_from<flat_mutation_reader_v2> at ././readers/flat_mutation_reader_v2.hh:505
  | |   | |             (inlined by) ?? at ./readers/mutation_reader.cc:66
  | |   |   ++[6#1/1 100%] addr=0x1e56222 total=592 count=94 avg=6:
  | |   |   |              single_partition_populating_reader::fill_buffer()::{lambda()#3}::operator() at ./row_cache.cc:419
  | |   |   |              (inlined by) ?? at ././seastar/include/seastar/core/future.hh:2006
  | |   |   |              (inlined by) seastar::futurize_invoke<single_partition_populating_reader::fill_buffer()::{lambda()#3}&> at ././seastar/include/seastar/core/future.hh:2037
  | |   |   |              (inlined by) seastar::do_until<single_partition_populating_reader::fill_buffer()::{lambda()#3}, single_partition_populating_reader::fill_buffer()::{lambda()#2}> at ././seastar/include/seastar/core/loop.hh:349
  | |   |   |              (inlined by) ?? at ./row_cache.cc:418
  | |   |   |              (inlined by) single_partition_populating_reader::fill_buffer()::{lambda()#1}::operator() at ./row_cache.cc:415
  | |   |   |              (inlined by) std::__invoke_impl<seastar::future<void>, single_partition_populating_reader::fill_buffer()::{lambda()#1}&> at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:61
  | |   |   |              (inlined by) std::__invoke<single_partition_populating_reader::fill_buffer()::{lambda()#1}&> at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:96
  | |   |   |              (inlined by) std::invoke<single_partition_populating_reader::fill_buffer()::{lambda()#1}&> at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/functional:110
  | |   |   |              (inlined by) seastar::internal::future_invoke<single_partition_populating_reader::fill_buffer()::{lambda()#1}&, seastar::internal::monostate> at ././seastar/include/seastar/core/future.hh:1155
  | |   |   |              (inlined by) seastar::future<void>::then_impl_nrvo<single_partition_populating_reader::fill_buffer()::{lambda()#1}, seastar::future<void> >(single_partition_populating_reader::fill_buffer()::{lambda()#1}&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, {lambda()#1}&, seastar::future_state<auto:1&&::monostate>&&)#1}::operator()(seastar::internal::promise_base_with_type<void>, seastar::internal::promise_base_with_type<void>&&, seastar::future_state<single_partition_populating_reader::fill_buffer()::{lambda()#1}&&::monostate>) const::{lambda()#1}::operator() at ././seastar/include/seastar/core/future.hh:1458
  | |   |   |              (inlined by) ?? at ././seastar/include/seastar/core/future.hh:1991
  | |   |   |              (inlined by) seastar::future<void>::then_impl_nrvo<single_partition_populating_reader::fill_buffer()::{lambda()#1}, seastar::future<void> >(single_partition_populating_reader::fill_buffer()::{lambda()#1}&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, {lambda()#1}&, seastar::future_state<auto:1&&::monostate>&&)#1}::operator() at ././seastar/include/seastar/core/future.hh:1454
  | |   |   |              (inlined by) seastar::continuation<seastar::internal::promise_base_with_type<void>, single_partition_populating_reader::fill_buffer()::{lambda()#1}, seastar::future<void>::then_impl_nrvo<{lambda()#1}, seastar::future>({lambda()#1}&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, {lambda()#1}&, seastar::future_state<seastar::internal::monostate>&&)#1}, void>::run_and_dispose at ././seastar/include/seastar/core/future.hh:741
  | |   |   ++           - addr=0x56bb654:
  | |   |   |              seastar::reactor::run_some_tasks at ./build/release/seastar/./seastar/src/core/reactor.cc:2590
  | |   |   |              (inlined by) seastar::reactor::run_some_tasks at ./build/release/seastar/./seastar/src/core/reactor.cc:3053
  | |   |   -> continued at addr=0x56bb654 above
  | |   ++[4#6/13 9%] 8 more branches total=1320 count=137 avg=10
  | ++[2#2/2 46%] addr=0x1ecb4ab total=688 count=61 avg=11:
  | |             logalloc::reclaim_from_evictable at ./utils/logalloc.cc:2213
  | |             (inlined by) logalloc::reclaim_from_evictable at ./utils/logalloc.cc:2532
  |   ++[3#1/1 100%] addr=0x1ec9764 total=8296 count=693 avg=12:
  |   |              logalloc::tracker::impl::compact_and_evict_locked at ./utils/logalloc.cc:2691
  |     ++[4#1/1 100%] addr=0x1ed1330 total=8620 count=747 avg=12:
  |     |              logalloc::allocating_section::reserve at ./utils/logalloc.cc:2618
  |     |              (inlined by) ?? at ./utils/logalloc.cc:1279
  |     |              (inlined by) ?? at ./utils/logalloc.cc:1292
  |     |              (inlined by) ?? at ./utils/logalloc.cc:2871
  |     -> continued at addr=0x1ed1330 above
  |+[1#2/401 6%] addr=0x1e5a76f total=943 count=100 avg=9:
  |              standard_migrator<bplus::data<long, intrusive_array<cache_entry>, dht::raw_token_less_comparator, 16ul, (bplus::key_search)0, (bplus::with_debug)0> >::migrate at row_cache.cc:?
  |              (inlined by) standard_migrator<bplus::data<long, intrusive_array<cache_entry>, dht::raw_token_less_comparator, 16ul, (bplus::key_search)0, (bplus::with_debug)0> >::migrate at ././utils/allocation_strategy.hh:72
  |+           - addr=0x1ed4baf:
  |              logalloc::region_impl::compact_segment_locked(logalloc::segment*, logalloc::segment_descriptor&)::{lambda(logalloc::region_impl::object_descriptor const*, void*, unsigned long)#1}::operator() at ./utils/logalloc.cc:1876
  |              (inlined by) ?? at ./utils/logalloc.cc:1720
  |              (inlined by) logalloc::region_impl::compact_segment_locked at ./utils/logalloc.cc:1873
  | ++[2#1/2 61%] addr=0x1ecb4ab total=4809 count=399 avg=12:
  | |             logalloc::reclaim_from_evictable at ./utils/logalloc.cc:2213
  | |             (inlined by) logalloc::reclaim_from_evictable at ./utils/logalloc.cc:2532
  | -> continued at addr=0x1ecb4ab above
  | ++[2#2/2 39%] addr=0x1ec973b total=3059 count=455 avg=7:
  | |             logalloc::tracker::impl::compact_and_evict_locked at ./utils/logalloc.cc:2213
  | |             (inlined by) logalloc::tracker::impl::compact_and_evict_locked at ./utils/logalloc.cc:2694
  | -> continued at addr=0x1ec973b above
  ++[1#3/401 85%] 399 more branches total=13415 count=1532 avg=9

logs can be found:
db_logs
loader_logs
monitor_logs
sct_logs

[fgelcer]

@asias

[fgelcer]

Installation details
Scylla version (or git commit hash): 5.4.0~dev.20230702.1ab2bb69b8a6 with build-id 748a09783d18c97d903f62dcf4d96a6d9db4b527
Cluster size: 3
OS (RHEL/CentOS/Ubuntu/AWS AMI): ami-058e06c3ec1cdd389 (eu-west-1)
test_id: dad92baa-6fa5-4bfa-8a49-9fdca61b5fc7

this week's results is slight better than previous week's one, but we are still ~95% worst than previous runs.

add 1st node decoded stalls:

Processed 931 stalls lasting a total of 8142 milliseconds.
min=6 avg=8.7 median=6 p95=13 p99=18 p999=None max=18

This graph is printed in bottom-up order, where callees are printed first.
Use --direction=top-down to print callers first.

[level#index/out_of pct%] below denotes:
  level  - nesting level in the graph
  index  - index of node among to its siblings
  out_of - number of siblings
  pct    - percentage of total stall time of this call relative to its siblings

Command line options: --address-threshold=0x100000000 --executable=scylla/libexec/scylla --direction=bottom-up --branch-threshold=0.05 --file=mixed/add_node_1_stalls.log

++[0#1/1 100%] addr=0x56a67c3 total=8142 count=931 avg=9:
|              seastar::backtrace<seastar::backtrace_buffer::append_backtrace_oneline()::{lambda(seastar::frame)#1}> at ./build/release/seastar/./seastar/include/seastar/util/backtrace.hh:64
|              (inlined by) ?? at ./build/release/seastar/./seastar/src/core/reactor.cc:825
|              (inlined by) seastar::print_with_backtrace at ./build/release/seastar/./seastar/src/core/reactor.cc:844
++           - addr=0x56a5b70:
|              seastar::internal::cpu_stall_detector::generate_trace at ./build/release/seastar/./seastar/src/core/reactor.cc:1414
++           - addr=0x56a6f9f:
|              seastar::reactor::block_notifier at ./build/release/seastar/./seastar/src/core/reactor.cc:1151
|              (inlined by) ?? at ./build/release/seastar/./seastar/src/core/reactor.cc:1171
|              (inlined by) seastar::reactor::block_notifier at ./build/release/seastar/./seastar/src/core/reactor.cc:1451
++           - addr=0x3cb1f: ?? ??:0
  ++[1#1/263 9%] addr=0x1eabe35 total=709 count=85 avg=8:
  |              utils::uleb64_decode_forwards<void (&)(char const*, unsigned long), void (&)(char const*, unsigned long)> at ././utils/vle.hh:105
  |              (inlined by) ?? at ./utils/logalloc.cc:1619
  |              (inlined by) ?? at ./utils/logalloc.cc:1717
  |              (inlined by) logalloc::region_impl::compact_segment_locked at ./utils/logalloc.cc:1873
  | ++[2#1/2 53%] addr=0x1ea27fb total=374 count=32 avg=12:
  | |             logalloc::reclaim_from_evictable at ./utils/logalloc.cc:2213
  | |             (inlined by) logalloc::reclaim_from_evictable at ./utils/logalloc.cc:2532
  | | ++[3#1/1 100%] addr=0x1ea0ab4 total=4050 count=351 avg=12:
  | | |              logalloc::tracker::impl::compact_and_evict_locked at ./utils/logalloc.cc:2691
  | |   ++[4#1/2 100%] addr=0x1ea8680 total=4236 count=382 avg=11:
  | |   |              logalloc::allocating_section::reserve at ./utils/logalloc.cc:2618
  | |   |              (inlined by) ?? at ./utils/logalloc.cc:1279
  | |   |              (inlined by) ?? at ./utils/logalloc.cc:1292
  | |   |              (inlined by) ?? at ./utils/logalloc.cc:2871
  | |   | ++[5#1/14 35%] addr=0x1b4d033 total=2697 count=269 avg=10:
  | |   | |              replica::memtable::apply at ././utils/logalloc.hh:469
  | |   | |              (inlined by) ?? at ././utils/logalloc.hh:520
  | |   | |              (inlined by) replica::memtable::apply(frozen_mutation const&, seastar::lw_shared_ptr<schema const> const&, db::rp_handle&&)::$_9::operator() at ./replica/memtable.cc:789
  | |   | |              (inlined by) with_allocator<replica::memtable::apply(frozen_mutation const&, seastar::lw_shared_ptr<schema const> const&, db::rp_handle&&)::$_9> at ././utils/allocation_strategy.hh:313
  | |   | |              (inlined by) replica::memtable::apply at ./replica/memtable.cc:788
  | |   | | ++[6#1/1 100%] addr=0x1a9693a total=2710 count=270 avg=10:
  | |   | | |              seastar::futurize<void>::invoke<replica::table::apply(frozen_mutation const&, seastar::lw_shared_ptr<schema const>, db::rp_handle&&, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >)::$_45&> at ./replica/table.cc:2257
  | |   | | |              (inlined by) replica::table::apply(frozen_mutation const&, seastar::lw_shared_ptr<schema const>, db::rp_handle&&, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >)::$_45::operator() at ./replica/table.cc:2280
  | |   | | |              (inlined by) seastar::futurize<void>::invoke<replica::table::apply(frozen_mutation const&, seastar::lw_shared_ptr<schema const>, db::rp_handle&&, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >)::$_45&> at ././seastar/include/seastar/core/future.hh:2003
  | |   | |   ++[7#1/1 100%] addr=0x1a115f9 total=2736 count=272 avg=10:
  | |   | |   |              seastar::futurize_invoke<replica::table::apply(frozen_mutation const&, seastar::lw_shared_ptr<schema const>, db::rp_handle&&, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >)::$_45&> at ././seastar/include/seastar/core/future.hh:2037
  | |   | |   |              (inlined by) ?? at ./replica/dirty_memory_manager.hh:572
  | |   | |   |              (inlined by) replica::table::apply at ./replica/table.cc:2279
  | |   | |     ++[8#1/2 100%] addr=0x189b65d total=2723 count=271 avg=10:
  | |   | |     |              replica::database::apply_in_memory(frozen_mutation const&, seastar::lw_shared_ptr<schema const>, db::rp_handle&&, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >)::$_32::operator() at ./replica/database.cc:1849
  | |   | |     |              (inlined by) ?? at ././seastar/include/seastar/core/future.hh:2006
  | |   | |     |              (inlined by) seastar::futurize_invoke<replica::database::apply_in_memory(frozen_mutation const&, seastar::lw_shared_ptr<schema const>, db::rp_handle&&, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >)::$_32> at ././seastar/include/seastar/core/future.hh:2037
  | |   | |     |              (inlined by) seastar::internal::invoke_func_with_gate<replica::database::apply_in_memory(frozen_mutation const&, seastar::lw_shared_ptr<schema const>, db::rp_handle&&, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >)::$_32> at ././seastar/include/seastar/core/gate.hh:278
  | |   | |     |              (inlined by) seastar::with_gate<replica::database::apply_in_memory(frozen_mutation const&, seastar::lw_shared_ptr<schema const>, db::rp_handle&&, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >)::$_32> at ././seastar/include/seastar/core/gate.hh:297
  | |   | |     |              (inlined by) ?? at ./replica/database.cc:1848
  | |   | |     |              (inlined by) replica::database::do_apply at ./replica/database.cc:2072
  | |   | |     ++           - addr=0x193b38f:
  | |   | |     |              std::__invoke_impl<seastar::future<void>, seastar::future<void> (replica::database::* const&)(seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>), replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce> > at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:74
  | |   | |     |              (inlined by) std::__invoke<seastar::future<void> (replica::database::* const&)(seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>), replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce> > at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:96
  | |   | |     |              (inlined by) ?? at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/functional:167
  | |   | |     |              (inlined by) seastar::noncopyable_function<seastar::future<void> (replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>)>::direct_vtable_for<std::_Mem_fn<seastar::future<void> (replica::database::*)(seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>)> >::call at ././seastar/include/seastar/util/noncopyable_function.hh:129
  | |   | |     ++           - addr=0x195deb9:
  | |   | |     |              seastar::noncopyable_function<seastar::future<void> (replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>)>::direct_vtable_for<seastar::inheriting_concrete_execution_stage<seastar::future<void>, replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce> >::make_stage_for_group(seastar::scheduling_group)::{lambda(replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>)#1}>::call at ././seastar/include/seastar/util/noncopyable_function.hh:215
  | |   | |     |              (inlined by) seastar::inheriting_concrete_execution_stage<seastar::future<void>, replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce> >::make_stage_for_group(seastar::scheduling_group)::{lambda(replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>)#1}::operator() at ././seastar/include/seastar/core/execution_stage.hh:342
  | |   | |     |              (inlined by) seastar::noncopyable_function<seastar::future<void> (replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>)>::direct_vtable_for<seastar::inheriting_concrete_execution_stage<seastar::future<void>, replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce> >::make_stage_for_group(seastar::scheduling_group)::{lambda(replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>)#1}>::call at ././seastar/include/seastar/util/noncopyable_function.hh:129
  | |   | |     ++           - addr=0x195da8d:
  | |   | |     |              seastar::concrete_execution_stage<seastar::future<void>, replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce> >::do_flush at ././seastar/include/seastar/util/noncopyable_function.hh:215
  | |   | |     |              (inlined by) std::__invoke_impl<seastar::future<void>, seastar::noncopyable_function<seastar::future<void> (replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>)>&, replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce> > at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:61
  | |   | |     |              (inlined by) std::__invoke<seastar::noncopyable_function<seastar::future<void> (replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>)>&, replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce> > at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:96
  | |   | |     |              (inlined by) std::__apply_impl<seastar::noncopyable_function<seastar::future<void> (replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>)>&, std::tuple<replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce> >, 0ul, 1ul, 2ul, 3ul, 4ul, 5ul, 6ul> at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/tuple:1852
  | |   | |     |              (inlined by) std::apply<seastar::noncopyable_function<seastar::future<void> (replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>)>&, std::tuple<replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce> > > at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/tuple:1863
  | |   | |     |              (inlined by) ?? at ././seastar/include/seastar/core/future.hh:1973
  | |   | |     |              (inlined by) ?? at ././seastar/include/seastar/core/execution_stage.hh:251
  | |   | |     ++           - addr=0x5663d13:
  | |   | |     |              seastar::execution_stage::flush()::$_1::operator() at ./build/release/seastar/./seastar/src/core/execution_stage.cc:149
  | |   | |     |              (inlined by) ?? at ./build/release/seastar/./seastar/include/seastar/core/future.hh:2003
  | |   | |     |              (inlined by) seastar::lambda_task<seastar::execution_stage::flush()::$_1>::run_and_dispose at ./build/release/seastar/./seastar/include/seastar/core/make_task.hh:45
  | |   | |     ++           - addr=0x56b78a4:
  | |   | |     |              seastar::reactor::run_some_tasks at ./build/release/seastar/./seastar/src/core/reactor.cc:2590
  | |   | |     |              (inlined by) seastar::reactor::run_some_tasks at ./build/release/seastar/./seastar/src/core/reactor.cc:3053
  | |   | |     | ++[9#1/1 100%] addr=0x56b8b27 total=7975 count=907 avg=9:
  | |   | |     | |              seastar::reactor::do_run at ./build/release/seastar/./seastar/src/core/reactor.cc:3222
  | |   | |     |   ++[10#1/2 89%] addr=0x56db2f1 total=7076 count=807 avg=9:
  | |   | |     |   |              seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_96::operator() at ./build/release/seastar/./seastar/src/core/reactor.cc:4426
  | |   | |     |   |              (inlined by) std::__invoke_impl<void, seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_96&> at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:61
  | |   | |     |   |              (inlined by) std::__invoke_r<void, seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_96&> at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:111
  | |   | |     |   |              (inlined by) std::_Function_handler<void (), seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_96>::_M_invoke at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/std_function.h:290
  | |   | |     |   ++           - addr=0x56880ba:
  | |   | |     |   |              seastar::posix_thread::start_routine at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/std_function.h:591
  | |   | |     |   |              (inlined by) seastar::posix_thread::start_routine at ./build/release/seastar/./seastar/src/core/posix.cc:90
  | |   | |     |   ++           - addr=0x8b12c: ?? ??:0
  | |   | |     |   ++           - addr=0x10cbbf: ?? ??:0
  | |   | |     |   ++[10#2/2 11%] addr=0x56b7e69 total=899 count=100 avg=9:
  | |   | |     |   |              seastar::reactor::run at ./build/release/seastar/./seastar/src/core/reactor.cc:3105
  | |   | |     |   ++           - addr=0x565bd85:
  | |   | |     |   |              seastar::app_template::run_deprecated at ./build/release/seastar/./seastar/src/core/app-template.cc:276
  | |   | |     |   ++           - addr=0x565aef8:
  | |   | |     |   |              seastar::app_template::run at ./build/release/seastar/./seastar/src/core/app-template.cc:167
  | |   | |     |   ++           - addr=0x122cf93:
  | |   | |     |   |              scylla_main at ./main.cc:627
  | |   | |     |   ++           - addr=0x122eb20:
  | |   | |     |   |              std::function<int (int, char**)>::operator() at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/std_function.h:591
  | |   | |     |   ++           - addr=0x122b3b7:
  | |   | |     |   |              main at ./main.cc:1995
  | |   | |     |   ++           - addr=0x2750f: ?? ??:0
  | |   | |     |   ++           - addr=0x275c8: ?? ??:0
  | |   | |     |   ++           - addr=0x1228f64:
  | |   | |     |   |              _start at ??:?
  | |   | |     ++[8#2/2 0%] 1 more branches total=13 count=1 avg=13
  | |   | |+[5#2/14 21%] addr=0x2117c7f total=1612 count=124 avg=13:
  | |   | |              sstables::index_consume_entry_context<sstables::index_consumer>::process_state at ././utils/logalloc.hh:469
  | |   | |              (inlined by) ?? at ././utils/logalloc.hh:520
  | |   | |              (inlined by) ?? at ./sstables/index_reader.hh:71
  | |   | |              (inlined by) sstables::index_consume_entry_context<sstables::index_consumer>::process_state at ./sstables/index_reader.hh:271
  | |   | |+           - addr=0x21156ea:
  | |   | |              data_consumer::continuous_data_consumer<sstables::index_consume_entry_context<sstables::index_consumer> >::operator() at ././sstables/consumer.hh:542
  | |   | |              (inlined by) data_consumer::continuous_data_consumer<sstables::index_consume_entry_context<sstables::index_consumer> >::operator() at ././sstables/consumer.hh:559
  | |   | |+           - addr=0x2114706:
  | |   | |              std::__invoke_impl<seastar::future<seastar::consumption_result<char> >, sstables::index_consume_entry_context<sstables::index_consumer>&, seastar::temporary_buffer<char> > at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:61
  | |   | |              (inlined by) std::__invoke<sstables::index_consume_entry_context<sstables::index_consumer>&, seastar::temporary_buffer<char> > at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:96
  | |   | |              (inlined by) ?? at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/refwrap.h:358
  | |   | |              (inlined by) operator() at ././seastar/include/seastar/core/iostream-impl.hh:222
  | |   | |+           - addr=0x211effa:
  | |   | |              seastar::internal::repeater<seastar::input_stream<char>::consume<std::reference_wrapper<sstables::index_consume_entry_context<sstables::index_consumer> > >(std::reference_wrapper<sstables::index_consume_entry_context<sstables::index_consumer> >&&)::{lambda()#1}>::run_and_dispose at ././seastar/include/seastar/core/future.hh:2006
  | |   | |              (inlined by) seastar::futurize_invoke<seastar::input_stream<char>::consume<std::reference_wrapper<sstables::index_consume_entry_context<sstables::index_consumer> > >(std::reference_wrapper<sstables::index_consume_entry_context<sstables::index_consumer> >&&)::{lambda()#1}&> at ././seastar/include/seastar/core/future.hh:2037
  | |   | |              (inlined by) seastar::internal::repeater<seastar::input_stream<char>::consume<std::reference_wrapper<sstables::index_consume_entry_context<sstables::index_consumer> > >(std::reference_wrapper<sstables::index_consume_entry_context<sstables::index_consumer> >&&)::{lambda()#1}>::run_and_dispose at ././seastar/include/seastar/core/loop.hh:79
  | |   | |+           - addr=0x56b78a4:
  | |   | |              seastar::reactor::run_some_tasks at ./build/release/seastar/./seastar/src/core/reactor.cc:2590
  | |   | |              (inlined by) seastar::reactor::run_some_tasks at ./build/release/seastar/./seastar/src/core/reactor.cc:3053
  | |   | -> continued at addr=0x56b78a4 above
  | |   | |+[5#3/14 16%] addr=0x2129714 total=1238 count=204 avg=6:
  | |   | |              seastar::future<void>::then_impl<sstables::index_reader::advance_lower_and_check_if_present(dht::ring_position_view, std::optional<position_in_partition_view>)::{lambda()#1}::operator()() const::{lambda()#1}, seastar::future<bool> > at ././utils/logalloc.hh:469
  | |   | |              (inlined by) ?? at ././utils/logalloc.hh:520
  | |   | |              (inlined by) sstables::index_reader::advance_lower_and_check_if_present(dht::ring_position_view, std::optional<position_in_partition_view>)::{lambda()#1}::operator()() const::{lambda()#1}::operator() at ./sstables/index_reader.hh:932
  | |   | |              (inlined by) ?? at ././seastar/include/seastar/core/future.hh:2006
  | |   | |              (inlined by) ?? at ././seastar/include/seastar/core/future.hh:1848
  | |   | |              (inlined by) seastar::future<void>::then_impl<sstables::index_reader::advance_lower_and_check_if_present(dht::ring_position_view, std::optional<position_in_partition_view>)::{lambda()#1}::operator()() const::{lambda()#1}, seastar::future<bool> > at ././seastar/include/seastar/core/future.hh:1473
  | |   | |+           - addr=0x21528e7:
  | |   | |              seastar::continuation<seastar::internal::promise_base_with_type<bool>, sstables::index_reader::advance_lower_and_check_if_present(dht::ring_position_view, std::optional<position_in_partition_view>)::{lambda()#1}, seastar::future<void>::then_impl_nrvo<{lambda()#1}, sstables::index_reader::advance_lower_and_check_if_present(dht::ring_position_view, std::optional<position_in_partition_view>)::{lambda()#1}<bool> >({lambda()#1}&&)::{lambda(seastar::internal::promise_base_with_type<bool>&&, {lambda()#1}&, seastar::future_state<seastar::internal::monostate>&&)#1}, void>::run_and_dispose at ././seastar/include/seastar/core/future.hh:1399
  | |   | |              (inlined by) sstables::index_reader::advance_lower_and_check_if_present(dht::ring_position_view, std::optional<position_in_partition_view>)::{lambda()#1}::operator() at ./sstables/index_reader.hh:930
  | |   | |              (inlined by) std::__invoke_impl<seastar::future<bool>, sstables::index_reader::advance_lower_and_check_if_present(dht::ring_position_view, std::optional<position_in_partition_view>)::{lambda()#1}&> at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:61
  | |   | |              (inlined by) std::__invoke<sstables::index_reader::advance_lower_and_check_if_present(dht::ring_position_view, std::optional<position_in_partition_view>)::{lambda()#1}&> at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:96
  | |   | |              (inlined by) std::invoke<sstables::index_reader::advance_lower_and_check_if_present(dht::ring_position_view, std::optional<position_in_partition_view>)::{lambda()#1}&> at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/functional:110
  | |   | |              (inlined by) seastar::internal::future_invoke<sstables::index_reader::advance_lower_and_check_if_present(dht::ring_position_view, std::optional<position_in_partition_view>)::{lambda()#1}&, seastar::internal::monostate> at ././seastar/include/seastar/core/future.hh:1155
  | |   | |              (inlined by) seastar::future<void>::then_impl_nrvo<sstables::index_reader::advance_lower_and_check_if_present(dht::ring_position_view, std::optional<position_in_partition_view>)::{lambda()#1}, seastar::future<bool> >(sstables::index_reader::advance_lower_and_check_if_present(dht::ring_position_view, std::optional<position_in_partition_view>)::{lambda()#1}&&)::{lambda(seastar::internal::promise_base_with_type<bool>&&, {lambda()#1}&, seastar::future_state<auto:1&&::monostate>&&)#1}::operator()(seastar::internal::promise_base_with_type<bool>, seastar::internal::promise_base_with_type<bool>&&, seastar::future_state<sstables::index_reader::advance_lower_and_check_if_present(dht::ring_position_view, std::optional<position_in_partition_view>)::{lambda()#1}&&::monostate>) const::{lambda()#1}::operator() at ././seastar/include/seastar/core/future.hh:1458
  | |   | |              (inlined by) ?? at ././seastar/include/seastar/core/future.hh:1991
  | |   | |              (inlined by) seastar::future<void>::then_impl_nrvo<sstables::index_reader::advance_lower_and_check_if_present(dht::ring_position_view, std::optional<position_in_partition_view>)::{lambda()#1}, seastar::future<bool> >(sstables::index_reader::advance_lower_and_check_if_present(dht::ring_position_view, std::optional<position_in_partition_view>)::{lambda()#1}&&)::{lambda(seastar::internal::promise_base_with_type<bool>&&, {lambda()#1}&, seastar::future_state<auto:1&&::monostate>&&)#1}::operator() at ././seastar/include/seastar/core/future.hh:1454
  | |   | |              (inlined by) seastar::continuation<seastar::internal::promise_base_with_type<bool>, sstables::index_reader::advance_lower_and_check_if_present(dht::ring_position_view, std::optional<position_in_partition_view>)::{lambda()#1}, seastar::future<void>::then_impl_nrvo<{lambda()#1}, sstables::index_reader::advance_lower_and_check_if_present(dht::ring_position_view, std::optional<position_in_partition_view>)::{lambda()#1}<bool> >({lambda()#1}&&)::{lambda(seastar::internal::promise_base_with_type<bool>&&, {lambda()#1}&, seastar::future_state<seastar::internal::monostate>&&)#1}, void>::run_and_dispose at ././seastar/include/seastar/core/future.hh:741
  | |   | |+           - addr=0x56b78a4:
  | |   | |              seastar::reactor::run_some_tasks at ./build/release/seastar/./seastar/src/core/reactor.cc:2590
  | |   | |              (inlined by) seastar::reactor::run_some_tasks at ./build/release/seastar/./seastar/src/core/reactor.cc:3053
  | |   | -> continued at addr=0x56b78a4 above
  | |   | |+[5#4/14 11%] addr=0x1b49ba9 total=827 count=104 avg=8:
  | |   | |              replica::memtable::make_flat_reader_opt at ././utils/logalloc.hh:469
  | |   | |              (inlined by) ?? at ././utils/logalloc.hh:520
  | |   | |              (inlined by) ?? at ./replica/memtable.cc:709
  | |   | |+           - addr=0x19bd5f8:
  | |   | |              replica::table::add_memtables_to_reader_list(std::vector<flat_mutation_reader_v2, std::allocator<flat_mutation_reader_v2> >&, seastar::lw_shared_ptr<schema const> const&, reader_permit const&, nonwrapping_interval<dht::ring_position> const&, query::partition_slice const&, tracing::trace_state_ptr const&, seastar::bool_class<streamed_mutation::forwarding_tag>, seastar::bool_class<mutation_reader::partition_range_forwarding_tag>, std::function<void (unsigned long)>) const::$_56::operator() at ./replica/table.cc:194
  | |   | |              (inlined by) replica::table::add_memtables_to_reader_list at ./replica/table.cc:205
  | |   | |+           - addr=0x19bdca1:
  | |   | |              replica::table::make_reader_v2 at ./replica/table.cc:259
  | |   | |+           - addr=0x1a96fc2:
  | |   | |              replica::table::as_mutation_source() const::$_48::operator() at ./replica/table.cc:2484
  | |   | |              (inlined by) std::__invoke_impl<flat_mutation_reader_v2, replica::table::as_mutation_source() const::$_48&, seastar::lw_shared_ptr<schema const>, reader_permit, nonwrapping_interval<dht::ring_position> const&, query::partition_slice const&, tracing::trace_state_ptr, seastar::bool_class<streamed_mutation::forwarding_tag>, seastar::bool_class<mutation_reader::partition_range_forwarding_tag> > at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:61
  | |   | |              (inlined by) std::__invoke_r<flat_mutation_reader_v2, replica::table::as_mutation_source() const::$_48&, seastar::lw_shared_ptr<schema const>, reader_permit, nonwrapping_interval<dht::ring_position> const&, query::partition_slice const&, tracing::trace_state_ptr, seastar::bool_class<streamed_mutation::forwarding_tag>, seastar::bool_class<mutation_reader::partition_range_forwarding_tag> > at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:114
  | |   | |              (inlined by) std::_Function_handler<flat_mutation_reader_v2 (seastar::lw_shared_ptr<schema const>, reader_permit, nonwrapping_interval<dht::ring_position> const&, query::partition_slice const&, tracing::trace_state_ptr, seastar::bool_class<streamed_mutation::forwarding_tag>, seastar::bool_class<mutation_reader::partition_range_forwarding_tag>), replica::table::as_mutation_source() const::$_48>::_M_invoke at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/std_function.h:290
  | |   | |+           - addr=0x1a1a593:
  | |   | |              query::querier::querier at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/std_function.h:591
  | |   | |              (inlined by) ?? at ././readers/mutation_source.hh:143
  | |   | |              (inlined by) query::querier_base::querier_base at ././querier.hh:91
  | |   | |              (inlined by) querier at ././querier.hh:158
  | |   | |+           - addr=0x1a15d8e:
  | |   | |              replica::table::query at ./replica/table.cc:2393
  | |   | |+           - addr=0x1954b17:
  | |   | |              replica::database::query(seastar::lw_shared_ptr<schema const>, query::read_command const&, query::result_options, std::vector<nonwrapping_interval<dht::ring_position>, std::allocator<nonwrapping_interval<dht::ring_position> > > const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>)::$_28::operator() at ./replica/database.cc:1600
  | |   | |              (inlined by) seastar::noncopyable_function<seastar::future<void> (reader_permit)>::indirect_vtable_for<replica::database::query(seastar::lw_shared_ptr<schema const>, query::read_command const&, query::result_options, std::vector<nonwrapping_interval<dht::ring_position>, std::allocator<nonwrapping_interval<dht::ring_position> > > const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>)::$_28>::call at ././seastar/include/seastar/util/noncopyable_function.hh:158
  | |   | |+           - addr=0x4299c03:
  | |   | |              reader_concurrency_semaphore::execution_loop at ././seastar/include/seastar/util/noncopyable_function.hh:215
  | |   | |              (inlined by) reader_concurrency_semaphore::execution_loop at ./reader_concurrency_semaphore.cc:921
  | |   | | ++[6#1/2 82%] addr=0x12a595a total=682 count=88 avg=8:
  | |   | | |             seastar::internal::coroutine_traits_base<void>::promise_type::run_and_dispose at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/coroutine:244
  | |   | | |             (inlined by) seastar::internal::coroutine_traits_base<void>::promise_type::run_and_dispose at ././seastar/include/seastar/core/coroutine.hh:125
  | |   | | | ++[7#1/1 100%] addr=0x56b78a4 total=704 count=90 avg=8:
  | |   | | | |              seastar::reactor::run_some_tasks at ./build/release/seastar/./seastar/src/core/reactor.cc:2590
  | |   | | | |              (inlined by) seastar::reactor::run_some_tasks at ./build/release/seastar/./seastar/src/core/reactor.cc:3053
  | |   | | | -> continued at addr=0x56b78a4 above
  | |   | | ++[6#2/2 18%] addr=0x1a41496 total=145 count=16 avg=9:
  | |   | | |             seastar::coroutine::internal::maybe_yield_awaiter::run_and_dispose at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/coroutine:135
  | |   | | |             (inlined by) seastar::coroutine::internal::maybe_yield_awaiter::run_and_dispose at ././seastar/include/seastar/coroutine/maybe_yield.hh:53
  | |   | |   ++[7#1/1 100%] addr=0x56b78a4 total=476 count=71 avg=7:
  | |   | |   |              seastar::reactor::run_some_tasks at ./build/release/seastar/./seastar/src/core/reactor.cc:2590
  | |   | |   |              (inlined by) seastar::reactor::run_some_tasks at ./build/release/seastar/./seastar/src/core/reactor.cc:3053
  | |   | |   -> continued at addr=0x56b78a4 above
  | |   | |+[5#5/14 9%] addr=0x1dff3dc total=714 count=96 avg=7:
  | |   | |             cache::cache_flat_mutation_reader::fill_buffer at ././utils/logalloc.hh:469
  | |   | |             (inlined by) ?? at ././utils/logalloc.hh:520
  | |   | |             (inlined by) ?? at ././row_cache.hh:481
  | |   | |             (inlined by) cache::cache_flat_mutation_reader::fill_buffer()::{lambda()#1}::operator() at ././cache_flat_mutation_reader.hh:301
  | |   | |             (inlined by) cache::cache_flat_mutation_reader::fill_buffer at ././cache_flat_mutation_reader.hh:309
  | |   | | ++[6#1/2 62%] addr=0x1a58283 total=441 count=56 avg=8:
  | |   | | |             operator() at ././readers/flat_mutation_reader_v2.hh:209
  | |   | | ++          - addr=0x1a5b33a:
  | |   | | |             seastar::internal::repeater<flat_mutation_reader_v2::impl::consume_pausable<std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<query_result_builder> > > >(std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<query_result_builder> > >)::{lambda()#1}>::run_and_dispose at ././seastar/include/seastar/core/future.hh:2006
  | |   | | |             (inlined by) seastar::futurize_invoke<flat_mutation_reader_v2::impl::consume_pausable<std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<query_result_builder> > > >(std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<query_result_builder> > >)::{lambda()#1}&> at ././seastar/include/seastar/core/future.hh:2037
  | |   | | |             (inlined by) seastar::internal::repeater<flat_mutation_reader_v2::impl::consume_pausable<std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<query_result_builder> > > >(std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<query_result_builder> > >)::{lambda()#1}>::run_and_dispose at ././seastar/include/seastar/core/loop.hh:79
  | |   | | ++          - addr=0x56b78a4:
  | |   | | |             seastar::reactor::run_some_tasks at ./build/release/seastar/./seastar/src/core/reactor.cc:2590
  | |   | | |             (inlined by) seastar::reactor::run_some_tasks at ./build/release/seastar/./seastar/src/core/reactor.cc:3053
  | |   | | -> continued at addr=0x56b78a4 above
  | |   | | ++[6#2/2 38%] addr=0x1fa3ebf total=273 count=40 avg=7:
  | |   | | |             flat_mutation_reader_v2::impl::fill_buffer_from<flat_mutation_reader_v2> at ././readers/flat_mutation_reader_v2.hh:505
  | |   | | |             (inlined by) ?? at ./readers/mutation_reader.cc:66
  | |   | |   ++[7#1/1 100%] addr=0x1e2d572 total=279 count=41 avg=7:
  | |   | |   |              single_partition_populating_reader::fill_buffer()::{lambda()#3}::operator() at ./row_cache.cc:419
  | |   | |   |              (inlined by) ?? at ././seastar/include/seastar/core/future.hh:2006
  | |   | |   |              (inlined by) seastar::futurize_invoke<single_partition_populating_reader::fill_buffer()::{lambda()#3}&> at ././seastar/include/seastar/core/future.hh:2037
  | |   | |   |              (inlined by) seastar::do_until<single_partition_populating_reader::fill_buffer()::{lambda()#3}, single_partition_populating_reader::fill_buffer()::{lambda()#2}> at ././seastar/include/seastar/core/loop.hh:349
  | |   | |   |              (inlined by) ?? at ./row_cache.cc:418
  | |   | |   |              (inlined by) single_partition_populating_reader::fill_buffer()::{lambda()#1}::operator() at ./row_cache.cc:415
  | |   | |   |              (inlined by) std::__invoke_impl<seastar::future<void>, single_partition_populating_reader::fill_buffer()::{lambda()#1}&> at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:61
  | |   | |   |              (inlined by) std::__invoke<single_partition_populating_reader::fill_buffer()::{lambda()#1}&> at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:96
  | |   | |   |              (inlined by) std::invoke<single_partition_populating_reader::fill_buffer()::{lambda()#1}&> at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/functional:110
  | |   | |   |              (inlined by) seastar::internal::future_invoke<single_partition_populating_reader::fill_buffer()::{lambda()#1}&, seastar::internal::monostate> at ././seastar/include/seastar/core/future.hh:1155
  | |   | |   |              (inlined by) seastar::future<void>::then_impl_nrvo<single_partition_populating_reader::fill_buffer()::{lambda()#1}, seastar::future<void> >(single_partition_populating_reader::fill_buffer()::{lambda()#1}&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, {lambda()#1}&, seastar::future_state<auto:1&&::monostate>&&)#1}::operator()(seastar::internal::promise_base_with_type<void>, seastar::internal::promise_base_with_type<void>&&, seastar::future_state<single_partition_populating_reader::fill_buffer()::{lambda()#1}&&::monostate>) const::{lambda()#1}::operator() at ././seastar/include/seastar/core/future.hh:1458
  | |   | |   |              (inlined by) ?? at ././seastar/include/seastar/core/future.hh:1991
  | |   | |   |              (inlined by) seastar::future<void>::then_impl_nrvo<single_partition_populating_reader::fill_buffer()::{lambda()#1}, seastar::future<void> >(single_partition_populating_reader::fill_buffer()::{lambda()#1}&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, {lambda()#1}&, seastar::future_state<auto:1&&::monostate>&&)#1}::operator() at ././seastar/include/seastar/core/future.hh:1454
  | |   | |   |              (inlined by) seastar::continuation<seastar::internal::promise_base_with_type<void>, single_partition_populating_reader::fill_buffer()::{lambda()#1}, seastar::future<void>::then_impl_nrvo<{lambda()#1}, seastar::future>({lambda()#1}&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, {lambda()#1}&, seastar::future_state<seastar::internal::monostate>&&)#1}, void>::run_and_dispose at ././seastar/include/seastar/core/future.hh:741
  | |   | |   ++           - addr=0x56b78a4:
  | |   | |   |              seastar::reactor::run_some_tasks at ./build/release/seastar/./seastar/src/core/reactor.cc:2590
  | |   | |   |              (inlined by) seastar::reactor::run_some_tasks at ./build/release/seastar/./seastar/src/core/reactor.cc:3053
  | |   | |   -> continued at addr=0x56b78a4 above
  | |   | ++[5#6/14 8%] 9 more branches total=649 count=72 avg=9
  | |   ++[4#2/2 0%] 1 more branches total=13 count=1 avg=13
  | ++[2#2/2 47%] addr=0x1ea0a8b total=335 count=53 avg=6:
  | |             logalloc::tracker::impl::compact_and_evict_locked at ./utils/logalloc.cc:2213
  | |             (inlined by) logalloc::tracker::impl::compact_and_evict_locked at ./utils/logalloc.cc:2694
  |   ++[3#1/2 100%] addr=0x1ea8680 total=3341 count=474 avg=7:
  |   |              logalloc::allocating_section::reserve at ./utils/logalloc.cc:2618
  |   |              (inlined by) ?? at ./utils/logalloc.cc:1279
  |   |              (inlined by) ?? at ./utils/logalloc.cc:1292
  |   |              (inlined by) ?? at ./utils/logalloc.cc:2871
  |   -> continued at addr=0x1ea8680 above
  |   ++[3#2/2 0%] 1 more branches total=9 count=1 avg=9
  |+[1#2/263 5%] addr=0x1e31abf total=442 count=48 avg=9:
  |              standard_migrator<bplus::data<long, intrusive_array<cache_entry>, dht::raw_token_less_comparator, 16ul, (bplus::key_search)0, (bplus::with_debug)0> >::migrate at row_cache.cc:?
  |              (inlined by) standard_migrator<bplus::data<long, intrusive_array<cache_entry>, dht::raw_token_less_comparator, 16ul, (bplus::key_search)0, (bplus::with_debug)0> >::migrate at ././utils/allocation_strategy.hh:72
  |+           - addr=0x1eabeff:
  |              logalloc::region_impl::compact_segment_locked(logalloc::segment*, logalloc::segment_descriptor&)::{lambda(logalloc::region_impl::object_descriptor const*, void*, unsigned long)#1}::operator() at ./utils/logalloc.cc:1876
  |              (inlined by) ?? at ./utils/logalloc.cc:1720
  |              (inlined by) logalloc::region_impl::compact_segment_locked at ./utils/logalloc.cc:1873
  | ++[2#1/2 55%] addr=0x1ea27fb total=2237 count=194 avg=12:
  | |             logalloc::reclaim_from_evictable at ./utils/logalloc.cc:2213
  | |             (inlined by) logalloc::reclaim_from_evictable at ./utils/logalloc.cc:2532
  | -> continued at addr=0x1ea27fb above
  | ++[2#2/2 45%] addr=0x1ea0a8b total=1810 count=255 avg=7:
  | |             logalloc::tracker::impl::compact_and_evict_locked at ./utils/logalloc.cc:2213
  | |             (inlined by) logalloc::tracker::impl::compact_and_evict_locked at ./utils/logalloc.cc:2694
  | -> continued at addr=0x1ea0a8b above
  ++[1#3/263 86%] 261 more branches total=6991 count=798 avg=9

logs can be found:
db_logs
loader_logs
monitor_logs
sct_logs

[fgelcer]

continuation of previous comment, add 2nd node decoded stalls:

Processed 534 stalls lasting a total of 4410 milliseconds.
min=6 avg=8.3 median=6 p95=13 p99=None p999=None max=13

This graph is printed in bottom-up order, where callees are printed first.
Use --direction=top-down to print callers first.

[level#index/out_of pct%] below denotes:
  level  - nesting level in the graph
  index  - index of node among to its siblings
  out_of - number of siblings
  pct    - percentage of total stall time of this call relative to its siblings

Command line options: --address-threshold=0x100000000 --executable=scylla/libexec/scylla --direction=bottom-up --branch-threshold=0.05 --file=mixed/add_node_2_stalls.log

++[0#1/1 100%] addr=0x56a67c3 total=4410 count=534 avg=8:
|              seastar::backtrace<seastar::backtrace_buffer::append_backtrace_oneline()::{lambda(seastar::frame)#1}> at ./build/release/seastar/./seastar/include/seastar/util/backtrace.hh:64
|              (inlined by) ?? at ./build/release/seastar/./seastar/src/core/reactor.cc:825
|              (inlined by) seastar::print_with_backtrace at ./build/release/seastar/./seastar/src/core/reactor.cc:844
++           - addr=0x56a5b70:
|              seastar::internal::cpu_stall_detector::generate_trace at ./build/release/seastar/./seastar/src/core/reactor.cc:1414
++           - addr=0x56a6f9f:
|              seastar::reactor::block_notifier at ./build/release/seastar/./seastar/src/core/reactor.cc:1151
|              (inlined by) ?? at ./build/release/seastar/./seastar/src/core/reactor.cc:1171
|              (inlined by) seastar::reactor::block_notifier at ./build/release/seastar/./seastar/src/core/reactor.cc:1451
++           - addr=0x3cb1f: ?? ??:0
  ++[1#1/188 11%] addr=0x1eabe35 total=501 count=59 avg=8:
  |               utils::uleb64_decode_forwards<void (&)(char const*, unsigned long), void (&)(char const*, unsigned long)> at ././utils/vle.hh:105
  |               (inlined by) ?? at ./utils/logalloc.cc:1619
  |               (inlined by) ?? at ./utils/logalloc.cc:1717
  |               (inlined by) logalloc::region_impl::compact_segment_locked at ./utils/logalloc.cc:1873
  | ++[2#1/2 50%] addr=0x1ea27fb total=251 count=22 avg=11:
  | |             logalloc::reclaim_from_evictable at ./utils/logalloc.cc:2213
  | |             (inlined by) logalloc::reclaim_from_evictable at ./utils/logalloc.cc:2532
  | | ++[3#1/1 100%] addr=0x1ea0ab4 total=2064 count=177 avg=12:
  | | |              logalloc::tracker::impl::compact_and_evict_locked at ./utils/logalloc.cc:2691
  | |   ++[4#1/1 100%] addr=0x1ea8680 total=2166 count=194 avg=11:
  | |   |              logalloc::allocating_section::reserve at ./utils/logalloc.cc:2618
  | |   |              (inlined by) ?? at ./utils/logalloc.cc:1279
  | |   |              (inlined by) ?? at ./utils/logalloc.cc:1292
  | |   |              (inlined by) ?? at ./utils/logalloc.cc:2871
  | |     ++[5#1/11 40%] addr=0x1b4d033 total=1627 count=165 avg=10:
  | |     |              replica::memtable::apply at ././utils/logalloc.hh:469
  | |     |              (inlined by) ?? at ././utils/logalloc.hh:520
  | |     |              (inlined by) replica::memtable::apply(frozen_mutation const&, seastar::lw_shared_ptr<schema const> const&, db::rp_handle&&)::$_9::operator() at ./replica/memtable.cc:789
  | |     |              (inlined by) with_allocator<replica::memtable::apply(frozen_mutation const&, seastar::lw_shared_ptr<schema const> const&, db::rp_handle&&)::$_9> at ././utils/allocation_strategy.hh:313
  | |     |              (inlined by) replica::memtable::apply at ./replica/memtable.cc:788
  | |     ++           - addr=0x1a9693a:
  | |     |              seastar::futurize<void>::invoke<replica::table::apply(frozen_mutation const&, seastar::lw_shared_ptr<schema const>, db::rp_handle&&, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >)::$_45&> at ./replica/table.cc:2257
  | |     |              (inlined by) replica::table::apply(frozen_mutation const&, seastar::lw_shared_ptr<schema const>, db::rp_handle&&, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >)::$_45::operator() at ./replica/table.cc:2280
  | |     |              (inlined by) seastar::futurize<void>::invoke<replica::table::apply(frozen_mutation const&, seastar::lw_shared_ptr<schema const>, db::rp_handle&&, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >)::$_45&> at ././seastar/include/seastar/core/future.hh:2003
  | |     | ++[6#1/1 100%] addr=0x1a115f9 total=1633 count=166 avg=10:
  | |     | |              seastar::futurize_invoke<replica::table::apply(frozen_mutation const&, seastar::lw_shared_ptr<schema const>, db::rp_handle&&, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >)::$_45&> at ././seastar/include/seastar/core/future.hh:2037
  | |     | |              (inlined by) ?? at ./replica/dirty_memory_manager.hh:572
  | |     | |              (inlined by) replica::table::apply at ./replica/table.cc:2279
  | |     |   ++[7#1/2 100%] addr=0x189b65d total=1653 count=167 avg=10:
  | |     |   |              replica::database::apply_in_memory(frozen_mutation const&, seastar::lw_shared_ptr<schema const>, db::rp_handle&&, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >)::$_32::operator() at ./replica/database.cc:1849
  | |     |   |              (inlined by) ?? at ././seastar/include/seastar/core/future.hh:2006
  | |     |   |              (inlined by) seastar::futurize_invoke<replica::database::apply_in_memory(frozen_mutation const&, seastar::lw_shared_ptr<schema const>, db::rp_handle&&, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >)::$_32> at ././seastar/include/seastar/core/future.hh:2037
  | |     |   |              (inlined by) seastar::internal::invoke_func_with_gate<replica::database::apply_in_memory(frozen_mutation const&, seastar::lw_shared_ptr<schema const>, db::rp_handle&&, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >)::$_32> at ././seastar/include/seastar/core/gate.hh:278
  | |     |   |              (inlined by) seastar::with_gate<replica::database::apply_in_memory(frozen_mutation const&, seastar::lw_shared_ptr<schema const>, db::rp_handle&&, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >)::$_32> at ././seastar/include/seastar/core/gate.hh:297
  | |     |   |              (inlined by) ?? at ./replica/database.cc:1848
  | |     |   |              (inlined by) replica::database::do_apply at ./replica/database.cc:2072
  | |     |   ++           - addr=0x193b38f:
  | |     |   |              std::__invoke_impl<seastar::future<void>, seastar::future<void> (replica::database::* const&)(seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>), replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce> > at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:74
  | |     |   |              (inlined by) std::__invoke<seastar::future<void> (replica::database::* const&)(seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>), replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce> > at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:96
  | |     |   |              (inlined by) ?? at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/functional:167
  | |     |   |              (inlined by) seastar::noncopyable_function<seastar::future<void> (replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>)>::direct_vtable_for<std::_Mem_fn<seastar::future<void> (replica::database::*)(seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>)> >::call at ././seastar/include/seastar/util/noncopyable_function.hh:129
  | |     |   ++           - addr=0x195deb9:
  | |     |   |              seastar::noncopyable_function<seastar::future<void> (replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>)>::direct_vtable_for<seastar::inheriting_concrete_execution_stage<seastar::future<void>, replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce> >::make_stage_for_group(seastar::scheduling_group)::{lambda(replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>)#1}>::call at ././seastar/include/seastar/util/noncopyable_function.hh:215
  | |     |   |              (inlined by) seastar::inheriting_concrete_execution_stage<seastar::future<void>, replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce> >::make_stage_for_group(seastar::scheduling_group)::{lambda(replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>)#1}::operator() at ././seastar/include/seastar/core/execution_stage.hh:342
  | |     |   |              (inlined by) seastar::noncopyable_function<seastar::future<void> (replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>)>::direct_vtable_for<seastar::inheriting_concrete_execution_stage<seastar::future<void>, replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce> >::make_stage_for_group(seastar::scheduling_group)::{lambda(replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>)#1}>::call at ././seastar/include/seastar/util/noncopyable_function.hh:129
  | |     |   ++           - addr=0x195da8d:
  | |     |   |              seastar::concrete_execution_stage<seastar::future<void>, replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce> >::do_flush at ././seastar/include/seastar/util/noncopyable_function.hh:215
  | |     |   |              (inlined by) std::__invoke_impl<seastar::future<void>, seastar::noncopyable_function<seastar::future<void> (replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>)>&, replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce> > at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:61
  | |     |   |              (inlined by) std::__invoke<seastar::noncopyable_function<seastar::future<void> (replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>)>&, replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce> > at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:96
  | |     |   |              (inlined by) std::__apply_impl<seastar::noncopyable_function<seastar::future<void> (replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>)>&, std::tuple<replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce> >, 0ul, 1ul, 2ul, 3ul, 4ul, 5ul, 6ul> at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/tuple:1852
  | |     |   |              (inlined by) std::apply<seastar::noncopyable_function<seastar::future<void> (replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>)>&, std::tuple<replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce> > > at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/tuple:1863
  | |     |   |              (inlined by) ?? at ././seastar/include/seastar/core/future.hh:1973
  | |     |   |              (inlined by) ?? at ././seastar/include/seastar/core/execution_stage.hh:251
  | |     |   ++           - addr=0x5663d13:
  | |     |   |              seastar::execution_stage::flush()::$_1::operator() at ./build/release/seastar/./seastar/src/core/execution_stage.cc:149
  | |     |   |              (inlined by) ?? at ./build/release/seastar/./seastar/include/seastar/core/future.hh:2003
  | |     |   |              (inlined by) seastar::lambda_task<seastar::execution_stage::flush()::$_1>::run_and_dispose at ./build/release/seastar/./seastar/include/seastar/core/make_task.hh:45
  | |     |   ++           - addr=0x56b78a4:
  | |     |   |              seastar::reactor::run_some_tasks at ./build/release/seastar/./seastar/src/core/reactor.cc:2590
  | |     |   |              (inlined by) seastar::reactor::run_some_tasks at ./build/release/seastar/./seastar/src/core/reactor.cc:3053
  | |     |   | ++[8#1/1 100%] addr=0x56b8b27 total=4384 count=530 avg=8:
  | |     |   | |              seastar::reactor::do_run at ./build/release/seastar/./seastar/src/core/reactor.cc:3222
  | |     |   |   ++[9#1/2 84%] addr=0x56db2f1 total=3702 count=456 avg=8:
  | |     |   |   |             seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_96::operator() at ./build/release/seastar/./seastar/src/core/reactor.cc:4426
  | |     |   |   |             (inlined by) std::__invoke_impl<void, seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_96&> at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:61
  | |     |   |   |             (inlined by) std::__invoke_r<void, seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_96&> at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:111
  | |     |   |   |             (inlined by) std::_Function_handler<void (), seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_96>::_M_invoke at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/std_function.h:290
  | |     |   |   ++          - addr=0x56880ba:
  | |     |   |   |             seastar::posix_thread::start_routine at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/std_function.h:591
  | |     |   |   |             (inlined by) seastar::posix_thread::start_routine at ./build/release/seastar/./seastar/src/core/posix.cc:90
  | |     |   |   ++          - addr=0x8b12c: ?? ??:0
  | |     |   |   ++          - addr=0x10cbbf: ?? ??:0
  | |     |   |   ++[9#2/2 16%] addr=0x56b7e69 total=682 count=74 avg=9:
  | |     |   |   |             seastar::reactor::run at ./build/release/seastar/./seastar/src/core/reactor.cc:3105
  | |     |   |   ++          - addr=0x565bd85:
  | |     |   |   |             seastar::app_template::run_deprecated at ./build/release/seastar/./seastar/src/core/app-template.cc:276
  | |     |   |   ++          - addr=0x565aef8:
  | |     |   |   |             seastar::app_template::run at ./build/release/seastar/./seastar/src/core/app-template.cc:167
  | |     |   |   ++          - addr=0x122cf93:
  | |     |   |   |             scylla_main at ./main.cc:627
  | |     |   |   ++          - addr=0x122eb20:
  | |     |   |   |             std::function<int (int, char**)>::operator() at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/std_function.h:591
  | |     |   |   ++          - addr=0x122b3b7:
  | |     |   |   |             main at ./main.cc:1995
  | |     |   |   ++          - addr=0x2750f: ?? ??:0
  | |     |   |   ++          - addr=0x275c8: ?? ??:0
  | |     |   |   ++          - addr=0x1228f64:
  | |     |   |   |             _start at ??:?
  | |     |   ++[7#2/2 0%] 1 more branches total=6 count=1 avg=6
  | |     |+[5#2/11 15%] addr=0x2129714 total=630 count=105 avg=6:
  | |     |              seastar::future<void>::then_impl<sstables::index_reader::advance_lower_and_check_if_present(dht::ring_position_view, std::optional<position_in_partition_view>)::{lambda()#1}::operator()() const::{lambda()#1}, seastar::future<bool> > at ././utils/logalloc.hh:469
  | |     |              (inlined by) ?? at ././utils/logalloc.hh:520
  | |     |              (inlined by) sstables::index_reader::advance_lower_and_check_if_present(dht::ring_position_view, std::optional<position_in_partition_view>)::{lambda()#1}::operator()() const::{lambda()#1}::operator() at ./sstables/index_reader.hh:932
  | |     |              (inlined by) ?? at ././seastar/include/seastar/core/future.hh:2006
  | |     |              (inlined by) ?? at ././seastar/include/seastar/core/future.hh:1848
  | |     |              (inlined by) seastar::future<void>::then_impl<sstables::index_reader::advance_lower_and_check_if_present(dht::ring_position_view, std::optional<position_in_partition_view>)::{lambda()#1}::operator()() const::{lambda()#1}, seastar::future<bool> > at ././seastar/include/seastar/core/future.hh:1473
  | |     |+           - addr=0x21528e7:
  | |     |              seastar::continuation<seastar::internal::promise_base_with_type<bool>, sstables::index_reader::advance_lower_and_check_if_present(dht::ring_position_view, std::optional<position_in_partition_view>)::{lambda()#1}, seastar::future<void>::then_impl_nrvo<{lambda()#1}, sstables::index_reader::advance_lower_and_check_if_present(dht::ring_position_view, std::optional<position_in_partition_view>)::{lambda()#1}<bool> >({lambda()#1}&&)::{lambda(seastar::internal::promise_base_with_type<bool>&&, {lambda()#1}&, seastar::future_state<seastar::internal::monostate>&&)#1}, void>::run_and_dispose at ././seastar/include/seastar/core/future.hh:1399
  | |     |              (inlined by) sstables::index_reader::advance_lower_and_check_if_present(dht::ring_position_view, std::optional<position_in_partition_view>)::{lambda()#1}::operator() at ./sstables/index_reader.hh:930
  | |     |              (inlined by) std::__invoke_impl<seastar::future<bool>, sstables::index_reader::advance_lower_and_check_if_present(dht::ring_position_view, std::optional<position_in_partition_view>)::{lambda()#1}&> at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:61
  | |     |              (inlined by) std::__invoke<sstables::index_reader::advance_lower_and_check_if_present(dht::ring_position_view, std::optional<position_in_partition_view>)::{lambda()#1}&> at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:96
  | |     |              (inlined by) std::invoke<sstables::index_reader::advance_lower_and_check_if_present(dht::ring_position_view, std::optional<position_in_partition_view>)::{lambda()#1}&> at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/functional:110
  | |     |              (inlined by) seastar::internal::future_invoke<sstables::index_reader::advance_lower_and_check_if_present(dht::ring_position_view, std::optional<position_in_partition_view>)::{lambda()#1}&, seastar::internal::monostate> at ././seastar/include/seastar/core/future.hh:1155
  | |     |              (inlined by) seastar::future<void>::then_impl_nrvo<sstables::index_reader::advance_lower_and_check_if_present(dht::ring_position_view, std::optional<position_in_partition_view>)::{lambda()#1}, seastar::future<bool> >(sstables::index_reader::advance_lower_and_check_if_present(dht::ring_position_view, std::optional<position_in_partition_view>)::{lambda()#1}&&)::{lambda(seastar::internal::promise_base_with_type<bool>&&, {lambda()#1}&, seastar::future_state<auto:1&&::monostate>&&)#1}::operator()(seastar::internal::promise_base_with_type<bool>, seastar::internal::promise_base_with_type<bool>&&, seastar::future_state<sstables::index_reader::advance_lower_and_check_if_present(dht::ring_position_view, std::optional<position_in_partition_view>)::{lambda()#1}&&::monostate>) const::{lambda()#1}::operator() at ././seastar/include/seastar/core/future.hh:1458
  | |     |              (inlined by) ?? at ././seastar/include/seastar/core/future.hh:1991
  | |     |              (inlined by) seastar::future<void>::then_impl_nrvo<sstables::index_reader::advance_lower_and_check_if_present(dht::ring_position_view, std::optional<position_in_partition_view>)::{lambda()#1}, seastar::future<bool> >(sstables::index_reader::advance_lower_and_check_if_present(dht::ring_position_view, std::optional<position_in_partition_view>)::{lambda()#1}&&)::{lambda(seastar::internal::promise_base_with_type<bool>&&, {lambda()#1}&, seastar::future_state<auto:1&&::monostate>&&)#1}::operator() at ././seastar/include/seastar/core/future.hh:1454
  | |     |              (inlined by) seastar::continuation<seastar::internal::promise_base_with_type<bool>, sstables::index_reader::advance_lower_and_check_if_present(dht::ring_position_view, std::optional<position_in_partition_view>)::{lambda()#1}, seastar::future<void>::then_impl_nrvo<{lambda()#1}, sstables::index_reader::advance_lower_and_check_if_present(dht::ring_position_view, std::optional<position_in_partition_view>)::{lambda()#1}<bool> >({lambda()#1}&&)::{lambda(seastar::internal::promise_base_with_type<bool>&&, {lambda()#1}&, seastar::future_state<seastar::internal::monostate>&&)#1}, void>::run_and_dispose at ././seastar/include/seastar/core/future.hh:741
  | |     |+           - addr=0x56b78a4:
  | |     |              seastar::reactor::run_some_tasks at ./build/release/seastar/./seastar/src/core/reactor.cc:2590
  | |     |              (inlined by) seastar::reactor::run_some_tasks at ./build/release/seastar/./seastar/src/core/reactor.cc:3053
  | |     -> continued at addr=0x56b78a4 above
  | |     |+[5#3/11 14%] addr=0x1b49ba9 total=572 count=79 avg=7:
  | |     |              replica::memtable::make_flat_reader_opt at ././utils/logalloc.hh:469
  | |     |              (inlined by) ?? at ././utils/logalloc.hh:520
  | |     |              (inlined by) ?? at ./replica/memtable.cc:709
  | |     |+           - addr=0x19bd5f8:
  | |     |              replica::table::add_memtables_to_reader_list(std::vector<flat_mutation_reader_v2, std::allocator<flat_mutation_reader_v2> >&, seastar::lw_shared_ptr<schema const> const&, reader_permit const&, nonwrapping_interval<dht::ring_position> const&, query::partition_slice const&, tracing::trace_state_ptr const&, seastar::bool_class<streamed_mutation::forwarding_tag>, seastar::bool_class<mutation_reader::partition_range_forwarding_tag>, std::function<void (unsigned long)>) const::$_56::operator() at ./replica/table.cc:194
  | |     |              (inlined by) replica::table::add_memtables_to_reader_list at ./replica/table.cc:205
  | |     |+           - addr=0x19bdca1:
  | |     |              replica::table::make_reader_v2 at ./replica/table.cc:259
  | |     |+           - addr=0x1a96fc2:
  | |     |              replica::table::as_mutation_source() const::$_48::operator() at ./replica/table.cc:2484
  | |     |              (inlined by) std::__invoke_impl<flat_mutation_reader_v2, replica::table::as_mutation_source() const::$_48&, seastar::lw_shared_ptr<schema const>, reader_permit, nonwrapping_interval<dht::ring_position> const&, query::partition_slice const&, tracing::trace_state_ptr, seastar::bool_class<streamed_mutation::forwarding_tag>, seastar::bool_class<mutation_reader::partition_range_forwarding_tag> > at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:61
  | |     |              (inlined by) std::__invoke_r<flat_mutation_reader_v2, replica::table::as_mutation_source() const::$_48&, seastar::lw_shared_ptr<schema const>, reader_permit, nonwrapping_interval<dht::ring_position> const&, query::partition_slice const&, tracing::trace_state_ptr, seastar::bool_class<streamed_mutation::forwarding_tag>, seastar::bool_class<mutation_reader::partition_range_forwarding_tag> > at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:114
  | |     |              (inlined by) std::_Function_handler<flat_mutation_reader_v2 (seastar::lw_shared_ptr<schema const>, reader_permit, nonwrapping_interval<dht::ring_position> const&, query::partition_slice const&, tracing::trace_state_ptr, seastar::bool_class<streamed_mutation::forwarding_tag>, seastar::bool_class<mutation_reader::partition_range_forwarding_tag>), replica::table::as_mutation_source() const::$_48>::_M_invoke at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/std_function.h:290
  | |     |+           - addr=0x1a1a593:
  | |     |              query::querier::querier at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/std_function.h:591
  | |     |              (inlined by) ?? at ././readers/mutation_source.hh:143
  | |     |              (inlined by) query::querier_base::querier_base at ././querier.hh:91
  | |     |              (inlined by) querier at ././querier.hh:158
  | |     |+           - addr=0x1a15d8e:
  | |     |              replica::table::query at ./replica/table.cc:2393
  | |     |+           - addr=0x1954b17:
  | |     |              replica::database::query(seastar::lw_shared_ptr<schema const>, query::read_command const&, query::result_options, std::vector<nonwrapping_interval<dht::ring_position>, std::allocator<nonwrapping_interval<dht::ring_position> > > const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>)::$_28::operator() at ./replica/database.cc:1600
  | |     |              (inlined by) seastar::noncopyable_function<seastar::future<void> (reader_permit)>::indirect_vtable_for<replica::database::query(seastar::lw_shared_ptr<schema const>, query::read_command const&, query::result_options, std::vector<nonwrapping_interval<dht::ring_position>, std::allocator<nonwrapping_interval<dht::ring_position> > > const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>)::$_28>::call at ././seastar/include/seastar/util/noncopyable_function.hh:158
  | |     |+           - addr=0x4299c03:
  | |     |              reader_concurrency_semaphore::execution_loop at ././seastar/include/seastar/util/noncopyable_function.hh:215
  | |     |              (inlined by) reader_concurrency_semaphore::execution_loop at ./reader_concurrency_semaphore.cc:921
  | |     | ++[6#1/2 94%] addr=0x12a595a total=536 count=73 avg=7:
  | |     | |             seastar::internal::coroutine_traits_base<void>::promise_type::run_and_dispose at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/coroutine:244
  | |     | |             (inlined by) seastar::internal::coroutine_traits_base<void>::promise_type::run_and_dispose at ././seastar/include/seastar/core/coroutine.hh:125
  | |     | | ++[7#1/1 100%] addr=0x56b78a4 total=542 count=74 avg=7:
  | |     | | |              seastar::reactor::run_some_tasks at ./build/release/seastar/./seastar/src/core/reactor.cc:2590
  | |     | | |              (inlined by) seastar::reactor::run_some_tasks at ./build/release/seastar/./seastar/src/core/reactor.cc:3053
  | |     | | -> continued at addr=0x56b78a4 above
  | |     | ++[6#2/2 6%] addr=0x1a41496 total=36 count=6 avg=6:
  | |     | |            seastar::coroutine::internal::maybe_yield_awaiter::run_and_dispose at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/coroutine:135
  | |     | |            (inlined by) seastar::coroutine::internal::maybe_yield_awaiter::run_and_dispose at ././seastar/include/seastar/coroutine/maybe_yield.hh:53
  | |     |   ++[7#1/1 100%] addr=0x56b78a4 total=306 count=51 avg=6:
  | |     |   |              seastar::reactor::run_some_tasks at ./build/release/seastar/./seastar/src/core/reactor.cc:2590
  | |     |   |              (inlined by) seastar::reactor::run_some_tasks at ./build/release/seastar/./seastar/src/core/reactor.cc:3053
  | |     |   -> continued at addr=0x56b78a4 above
  | |     |+[5#4/11 11%] addr=0x2117c7f total=468 count=36 avg=13:
  | |     |              sstables::index_consume_entry_context<sstables::index_consumer>::process_state at ././utils/logalloc.hh:469
  | |     |              (inlined by) ?? at ././utils/logalloc.hh:520
  | |     |              (inlined by) ?? at ./sstables/index_reader.hh:71
  | |     |              (inlined by) sstables::index_consume_entry_context<sstables::index_consumer>::process_state at ./sstables/index_reader.hh:271
  | |     |+           - addr=0x21156ea:
  | |     |              data_consumer::continuous_data_consumer<sstables::index_consume_entry_context<sstables::index_consumer> >::operator() at ././sstables/consumer.hh:542
  | |     |              (inlined by) data_consumer::continuous_data_consumer<sstables::index_consume_entry_context<sstables::index_consumer> >::operator() at ././sstables/consumer.hh:559
  | |     |+           - addr=0x2114706:
  | |     |              std::__invoke_impl<seastar::future<seastar::consumption_result<char> >, sstables::index_consume_entry_context<sstables::index_consumer>&, seastar::temporary_buffer<char> > at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:61
  | |     |              (inlined by) std::__invoke<sstables::index_consume_entry_context<sstables::index_consumer>&, seastar::temporary_buffer<char> > at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:96
  | |     |              (inlined by) ?? at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/refwrap.h:358
  | |     |              (inlined by) operator() at ././seastar/include/seastar/core/iostream-impl.hh:222
  | |     |+           - addr=0x211effa:
  | |     |              seastar::internal::repeater<seastar::input_stream<char>::consume<std::reference_wrapper<sstables::index_consume_entry_context<sstables::index_consumer> > >(std::reference_wrapper<sstables::index_consume_entry_context<sstables::index_consumer> >&&)::{lambda()#1}>::run_and_dispose at ././seastar/include/seastar/core/future.hh:2006
  | |     |              (inlined by) seastar::futurize_invoke<seastar::input_stream<char>::consume<std::reference_wrapper<sstables::index_consume_entry_context<sstables::index_consumer> > >(std::reference_wrapper<sstables::index_consume_entry_context<sstables::index_consumer> >&&)::{lambda()#1}&> at ././seastar/include/seastar/core/future.hh:2037
  | |     |              (inlined by) seastar::internal::repeater<seastar::input_stream<char>::consume<std::reference_wrapper<sstables::index_consume_entry_context<sstables::index_consumer> > >(std::reference_wrapper<sstables::index_consume_entry_context<sstables::index_consumer> >&&)::{lambda()#1}>::run_and_dispose at ././seastar/include/seastar/core/loop.hh:79
  | |     |+           - addr=0x56b78a4:
  | |     |              seastar::reactor::run_some_tasks at ./build/release/seastar/./seastar/src/core/reactor.cc:2590
  | |     |              (inlined by) seastar::reactor::run_some_tasks at ./build/release/seastar/./seastar/src/core/reactor.cc:3053
  | |     -> continued at addr=0x56b78a4 above
  | |     |+[5#5/11 11%] addr=0x1dff3dc total=460 count=65 avg=7:
  | |     |              cache::cache_flat_mutation_reader::fill_buffer at ././utils/logalloc.hh:469
  | |     |              (inlined by) ?? at ././utils/logalloc.hh:520
  | |     |              (inlined by) ?? at ././row_cache.hh:481
  | |     |              (inlined by) cache::cache_flat_mutation_reader::fill_buffer()::{lambda()#1}::operator() at ././cache_flat_mutation_reader.hh:301
  | |     |              (inlined by) cache::cache_flat_mutation_reader::fill_buffer at ././cache_flat_mutation_reader.hh:309
  | |     | ++[6#1/2 70%] addr=0x1a58283 total=321 count=43 avg=7:
  | |     | |             operator() at ././readers/flat_mutation_reader_v2.hh:209
  | |     | ++          - addr=0x1a5b33a:
  | |     | |             seastar::internal::repeater<flat_mutation_reader_v2::impl::consume_pausable<std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<query_result_builder> > > >(std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<query_result_builder> > >)::{lambda()#1}>::run_and_dispose at ././seastar/include/seastar/core/future.hh:2006
  | |     | |             (inlined by) seastar::futurize_invoke<flat_mutation_reader_v2::impl::consume_pausable<std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<query_result_builder> > > >(std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<query_result_builder> > >)::{lambda()#1}&> at ././seastar/include/seastar/core/future.hh:2037
  | |     | |             (inlined by) seastar::internal::repeater<flat_mutation_reader_v2::impl::consume_pausable<std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<query_result_builder> > > >(std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<query_result_builder> > >)::{lambda()#1}>::run_and_dispose at ././seastar/include/seastar/core/loop.hh:79
  | |     | ++          - addr=0x56b78a4:
  | |     | |             seastar::reactor::run_some_tasks at ./build/release/seastar/./seastar/src/core/reactor.cc:2590
  | |     | |             (inlined by) seastar::reactor::run_some_tasks at ./build/release/seastar/./seastar/src/core/reactor.cc:3053
  | |     | -> continued at addr=0x56b78a4 above
  | |     | ++[6#2/2 30%] addr=0x1fa3ebf total=139 count=22 avg=6:
  | |     | |             flat_mutation_reader_v2::impl::fill_buffer_from<flat_mutation_reader_v2> at ././readers/flat_mutation_reader_v2.hh:505
  | |     | |             (inlined by) ?? at ./readers/mutation_reader.cc:66
  | |     | ++          - addr=0x1e2d572:
  | |     | |             single_partition_populating_reader::fill_buffer()::{lambda()#3}::operator() at ./row_cache.cc:419
  | |     | |             (inlined by) ?? at ././seastar/include/seastar/core/future.hh:2006
  | |     | |             (inlined by) seastar::futurize_invoke<single_partition_populating_reader::fill_buffer()::{lambda()#3}&> at ././seastar/include/seastar/core/future.hh:2037
  | |     | |             (inlined by) seastar::do_until<single_partition_populating_reader::fill_buffer()::{lambda()#3}, single_partition_populating_reader::fill_buffer()::{lambda()#2}> at ././seastar/include/seastar/core/loop.hh:349
  | |     | |             (inlined by) ?? at ./row_cache.cc:418
  | |     | |             (inlined by) single_partition_populating_reader::fill_buffer()::{lambda()#1}::operator() at ./row_cache.cc:415
  | |     | |             (inlined by) std::__invoke_impl<seastar::future<void>, single_partition_populating_reader::fill_buffer()::{lambda()#1}&> at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:61
  | |     | |             (inlined by) std::__invoke<single_partition_populating_reader::fill_buffer()::{lambda()#1}&> at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:96
  | |     | |             (inlined by) std::invoke<single_partition_populating_reader::fill_buffer()::{lambda()#1}&> at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/functional:110
  | |     | |             (inlined by) seastar::internal::future_invoke<single_partition_populating_reader::fill_buffer()::{lambda()#1}&, seastar::internal::monostate> at ././seastar/include/seastar/core/future.hh:1155
  | |     | |             (inlined by) seastar::future<void>::then_impl_nrvo<single_partition_populating_reader::fill_buffer()::{lambda()#1}, seastar::future<void> >(single_partition_populating_reader::fill_buffer()::{lambda()#1}&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, {lambda()#1}&, seastar::future_state<auto:1&&::monostate>&&)#1}::operator()(seastar::internal::promise_base_with_type<void>, seastar::internal::promise_base_with_type<void>&&, seastar::future_state<single_partition_populating_reader::fill_buffer()::{lambda()#1}&&::monostate>) const::{lambda()#1}::operator() at ././seastar/include/seastar/core/future.hh:1458
  | |     | |             (inlined by) ?? at ././seastar/include/seastar/core/future.hh:1991
  | |     | |             (inlined by) seastar::future<void>::then_impl_nrvo<single_partition_populating_reader::fill_buffer()::{lambda()#1}, seastar::future<void> >(single_partition_populating_reader::fill_buffer()::{lambda()#1}&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, {lambda()#1}&, seastar::future_state<auto:1&&::monostate>&&)#1}::operator() at ././seastar/include/seastar/core/future.hh:1454
  | |     | |             (inlined by) seastar::continuation<seastar::internal::promise_base_with_type<void>, single_partition_populating_reader::fill_buffer()::{lambda()#1}, seastar::future<void>::then_impl_nrvo<{lambda()#1}, seastar::future>({lambda()#1}&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, {lambda()#1}&, seastar::future_state<seastar::internal::monostate>&&)#1}, void>::run_and_dispose at ././seastar/include/seastar/core/future.hh:741
  | |     | ++          - addr=0x56b78a4:
  | |     | |             seastar::reactor::run_some_tasks at ./build/release/seastar/./seastar/src/core/reactor.cc:2590
  | |     | |             (inlined by) seastar::reactor::run_some_tasks at ./build/release/seastar/./seastar/src/core/reactor.cc:3053
  | |     | -> continued at addr=0x56b78a4 above
  | |     ++[5#6/11 8%] 6 more branches total=338 count=34 avg=10
  | ++[2#2/2 50%] addr=0x1ea0a8b total=250 count=37 avg=7:
  | |             logalloc::tracker::impl::compact_and_evict_locked at ./utils/logalloc.cc:2213
  | |             (inlined by) logalloc::tracker::impl::compact_and_evict_locked at ./utils/logalloc.cc:2694
  |   ++[3#1/1 100%] addr=0x1ea8680 total=1839 count=282 avg=7:
  |   |              logalloc::allocating_section::reserve at ./utils/logalloc.cc:2618
  |   |              (inlined by) ?? at ./utils/logalloc.cc:1279
  |   |              (inlined by) ?? at ./utils/logalloc.cc:1292
  |   |              (inlined by) ?? at ./utils/logalloc.cc:2871
  |   -> continued at addr=0x1ea8680 above
  |+[1#2/188 7%] addr=0x1e31abf total=290 count=32 avg=9:
  |              standard_migrator<bplus::data<long, intrusive_array<cache_entry>, dht::raw_token_less_comparator, 16ul, (bplus::key_search)0, (bplus::with_debug)0> >::migrate at row_cache.cc:?
  |              (inlined by) standard_migrator<bplus::data<long, intrusive_array<cache_entry>, dht::raw_token_less_comparator, 16ul, (bplus::key_search)0, (bplus::with_debug)0> >::migrate at ././utils/allocation_strategy.hh:72
  |+           - addr=0x1eabeff:
  |              logalloc::region_impl::compact_segment_locked(logalloc::segment*, logalloc::segment_descriptor&)::{lambda(logalloc::region_impl::object_descriptor const*, void*, unsigned long)#1}::operator() at ./utils/logalloc.cc:1876
  |              (inlined by) ?? at ./utils/logalloc.cc:1720
  |              (inlined by) logalloc::region_impl::compact_segment_locked at ./utils/logalloc.cc:1873
  | ++[2#1/2 51%] addr=0x1ea27fb total=973 count=84 avg=12:
  | |             logalloc::reclaim_from_evictable at ./utils/logalloc.cc:2213
  | |             (inlined by) logalloc::reclaim_from_evictable at ./utils/logalloc.cc:2532
  | -> continued at addr=0x1ea27fb above
  | ++[2#2/2 49%] addr=0x1ea0a8b total=944 count=148 avg=6:
  | |             logalloc::tracker::impl::compact_and_evict_locked at ./utils/logalloc.cc:2213
  | |             (inlined by) logalloc::tracker::impl::compact_and_evict_locked at ./utils/logalloc.cc:2694
  | -> continued at addr=0x1ea0a8b above
  ++[1#3/188 82%] 186 more branches total=3619 count=443 avg=8

[fgelcer]

continuation of last 2 comments, add 3rd node decoded stalls:

Processed 346 stalls lasting a total of 2909 milliseconds.
min=6 avg=8.4 median=6 p95=13 p99=None p999=None max=13

This graph is printed in bottom-up order, where callees are printed first.
Use --direction=top-down to print callers first.

[level#index/out_of pct%] below denotes:
  level  - nesting level in the graph
  index  - index of node among to its siblings
  out_of - number of siblings
  pct    - percentage of total stall time of this call relative to its siblings

Command line options: --address-threshold=0x100000000 --executable=scylla/libexec/scylla --direction=bottom-up --branch-threshold=0.05 --file=mixed/add_node_3_stalls.log

++[0#1/1 100%] addr=0x56a67c3 total=2909 count=346 avg=8:
|              seastar::backtrace<seastar::backtrace_buffer::append_backtrace_oneline()::{lambda(seastar::frame)#1}> at ./build/release/seastar/./seastar/include/seastar/util/backtrace.hh:64
|              (inlined by) ?? at ./build/release/seastar/./seastar/src/core/reactor.cc:825
|              (inlined by) seastar::print_with_backtrace at ./build/release/seastar/./seastar/src/core/reactor.cc:844
++           - addr=0x56a5b70:
|              seastar::internal::cpu_stall_detector::generate_trace at ./build/release/seastar/./seastar/src/core/reactor.cc:1414
++           - addr=0x56a6f9f:
|              seastar::reactor::block_notifier at ./build/release/seastar/./seastar/src/core/reactor.cc:1151
|              (inlined by) ?? at ./build/release/seastar/./seastar/src/core/reactor.cc:1171
|              (inlined by) seastar::reactor::block_notifier at ./build/release/seastar/./seastar/src/core/reactor.cc:1451
++           - addr=0x3cb1f: ?? ??:0
  ++[1#1/164 8%] addr=0x1eabe35 total=242 count=31 avg=8:
  |              utils::uleb64_decode_forwards<void (&)(char const*, unsigned long), void (&)(char const*, unsigned long)> at ././utils/vle.hh:105
  |              (inlined by) ?? at ./utils/logalloc.cc:1619
  |              (inlined by) ?? at ./utils/logalloc.cc:1717
  |              (inlined by) logalloc::region_impl::compact_segment_locked at ./utils/logalloc.cc:1873
  | ++[2#1/2 63%] addr=0x1ea0a8b total=152 count=23 avg=7:
  | |             logalloc::tracker::impl::compact_and_evict_locked at ./utils/logalloc.cc:2213
  | |             (inlined by) logalloc::tracker::impl::compact_and_evict_locked at ./utils/logalloc.cc:2694
  | | ++[3#1/1 100%] addr=0x1ea8680 total=1059 count=166 avg=6:
  | | |              logalloc::allocating_section::reserve at ./utils/logalloc.cc:2618
  | | |              (inlined by) ?? at ./utils/logalloc.cc:1279
  | | |              (inlined by) ?? at ./utils/logalloc.cc:1292
  | | |              (inlined by) ?? at ./utils/logalloc.cc:2871
  | |   ++[4#1/8 49%] addr=0x1b4d033 total=1307 count=121 avg=11:
  | |   |             replica::memtable::apply at ././utils/logalloc.hh:469
  | |   |             (inlined by) ?? at ././utils/logalloc.hh:520
  | |   |             (inlined by) replica::memtable::apply(frozen_mutation const&, seastar::lw_shared_ptr<schema const> const&, db::rp_handle&&)::$_9::operator() at ./replica/memtable.cc:789
  | |   |             (inlined by) with_allocator<replica::memtable::apply(frozen_mutation const&, seastar::lw_shared_ptr<schema const> const&, db::rp_handle&&)::$_9> at ././utils/allocation_strategy.hh:313
  | |   |             (inlined by) replica::memtable::apply at ./replica/memtable.cc:788
  | |   ++          - addr=0x1a9693a:
  | |   |             seastar::futurize<void>::invoke<replica::table::apply(frozen_mutation const&, seastar::lw_shared_ptr<schema const>, db::rp_handle&&, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >)::$_45&> at ./replica/table.cc:2257
  | |   |             (inlined by) replica::table::apply(frozen_mutation const&, seastar::lw_shared_ptr<schema const>, db::rp_handle&&, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >)::$_45::operator() at ./replica/table.cc:2280
  | |   |             (inlined by) seastar::futurize<void>::invoke<replica::table::apply(frozen_mutation const&, seastar::lw_shared_ptr<schema const>, db::rp_handle&&, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >)::$_45&> at ././seastar/include/seastar/core/future.hh:2003
  | |   | ++[5#1/1 100%] addr=0x1a115f9 total=1320 count=122 avg=11:
  | |   | |              seastar::futurize_invoke<replica::table::apply(frozen_mutation const&, seastar::lw_shared_ptr<schema const>, db::rp_handle&&, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >)::$_45&> at ././seastar/include/seastar/core/future.hh:2037
  | |   | |              (inlined by) ?? at ./replica/dirty_memory_manager.hh:572
  | |   | |              (inlined by) replica::table::apply at ./replica/table.cc:2279
  | |   |   ++[6#1/2 99%] addr=0x189b65d total=1320 count=122 avg=11:
  | |   |   |             replica::database::apply_in_memory(frozen_mutation const&, seastar::lw_shared_ptr<schema const>, db::rp_handle&&, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >)::$_32::operator() at ./replica/database.cc:1849
  | |   |   |             (inlined by) ?? at ././seastar/include/seastar/core/future.hh:2006
  | |   |   |             (inlined by) seastar::futurize_invoke<replica::database::apply_in_memory(frozen_mutation const&, seastar::lw_shared_ptr<schema const>, db::rp_handle&&, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >)::$_32> at ././seastar/include/seastar/core/future.hh:2037
  | |   |   |             (inlined by) seastar::internal::invoke_func_with_gate<replica::database::apply_in_memory(frozen_mutation const&, seastar::lw_shared_ptr<schema const>, db::rp_handle&&, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >)::$_32> at ././seastar/include/seastar/core/gate.hh:278
  | |   |   |             (inlined by) seastar::with_gate<replica::database::apply_in_memory(frozen_mutation const&, seastar::lw_shared_ptr<schema const>, db::rp_handle&&, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >)::$_32> at ././seastar/include/seastar/core/gate.hh:297
  | |   |   |             (inlined by) ?? at ./replica/database.cc:1848
  | |   |   |             (inlined by) replica::database::do_apply at ./replica/database.cc:2072
  | |   |   ++          - addr=0x193b38f:
  | |   |   |             std::__invoke_impl<seastar::future<void>, seastar::future<void> (replica::database::* const&)(seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>), replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce> > at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:74
  | |   |   |             (inlined by) std::__invoke<seastar::future<void> (replica::database::* const&)(seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>), replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce> > at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:96
  | |   |   |             (inlined by) ?? at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/functional:167
  | |   |   |             (inlined by) seastar::noncopyable_function<seastar::future<void> (replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>)>::direct_vtable_for<std::_Mem_fn<seastar::future<void> (replica::database::*)(seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>)> >::call at ././seastar/include/seastar/util/noncopyable_function.hh:129
  | |   |   ++          - addr=0x195deb9:
  | |   |   |             seastar::noncopyable_function<seastar::future<void> (replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>)>::direct_vtable_for<seastar::inheriting_concrete_execution_stage<seastar::future<void>, replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce> >::make_stage_for_group(seastar::scheduling_group)::{lambda(replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>)#1}>::call at ././seastar/include/seastar/util/noncopyable_function.hh:215
  | |   |   |             (inlined by) seastar::inheriting_concrete_execution_stage<seastar::future<void>, replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce> >::make_stage_for_group(seastar::scheduling_group)::{lambda(replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>)#1}::operator() at ././seastar/include/seastar/core/execution_stage.hh:342
  | |   |   |             (inlined by) seastar::noncopyable_function<seastar::future<void> (replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>)>::direct_vtable_for<seastar::inheriting_concrete_execution_stage<seastar::future<void>, replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce> >::make_stage_for_group(seastar::scheduling_group)::{lambda(replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>)#1}>::call at ././seastar/include/seastar/util/noncopyable_function.hh:129
  | |   |   ++          - addr=0x195da8d:
  | |   |   |             seastar::concrete_execution_stage<seastar::future<void>, replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce> >::do_flush at ././seastar/include/seastar/util/noncopyable_function.hh:215
  | |   |   |             (inlined by) std::__invoke_impl<seastar::future<void>, seastar::noncopyable_function<seastar::future<void> (replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>)>&, replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce> > at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:61
  | |   |   |             (inlined by) std::__invoke<seastar::noncopyable_function<seastar::future<void> (replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>)>&, replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce> > at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:96
  | |   |   |             (inlined by) std::__apply_impl<seastar::noncopyable_function<seastar::future<void> (replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>)>&, std::tuple<replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce> >, 0ul, 1ul, 2ul, 3ul, 4ul, 5ul, 6ul> at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/tuple:1852
  | |   |   |             (inlined by) std::apply<seastar::noncopyable_function<seastar::future<void> (replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>)>&, std::tuple<replica::database*, seastar::lw_shared_ptr<schema const>, frozen_mutation const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::bool_class<db::force_sync_tag>, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce> > > at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/tuple:1863
  | |   |   |             (inlined by) ?? at ././seastar/include/seastar/core/future.hh:1973
  | |   |   |             (inlined by) ?? at ././seastar/include/seastar/core/execution_stage.hh:251
  | |   |   ++          - addr=0x5663d13:
  | |   |   |             seastar::execution_stage::flush()::$_1::operator() at ./build/release/seastar/./seastar/src/core/execution_stage.cc:149
  | |   |   |             (inlined by) ?? at ./build/release/seastar/./seastar/include/seastar/core/future.hh:2003
  | |   |   |             (inlined by) seastar::lambda_task<seastar::execution_stage::flush()::$_1>::run_and_dispose at ./build/release/seastar/./seastar/include/seastar/core/make_task.hh:45
  | |   |   ++          - addr=0x56b78a4:
  | |   |   |             seastar::reactor::run_some_tasks at ./build/release/seastar/./seastar/src/core/reactor.cc:2590
  | |   |   |             (inlined by) seastar::reactor::run_some_tasks at ./build/release/seastar/./seastar/src/core/reactor.cc:3053
  | |   |   | ++[7#1/1 100%] addr=0x56b8b27 total=2909 count=346 avg=8:
  | |   |   | |              seastar::reactor::do_run at ./build/release/seastar/./seastar/src/core/reactor.cc:3222
  | |   |   |   ++[8#1/2 83%] addr=0x56db2f1 total=2402 count=286 avg=8:
  | |   |   |   |             seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_96::operator() at ./build/release/seastar/./seastar/src/core/reactor.cc:4426
  | |   |   |   |             (inlined by) std::__invoke_impl<void, seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_96&> at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:61
  | |   |   |   |             (inlined by) std::__invoke_r<void, seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_96&> at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:111
  | |   |   |   |             (inlined by) std::_Function_handler<void (), seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_96>::_M_invoke at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/std_function.h:290
  | |   |   |   ++          - addr=0x56880ba:
  | |   |   |   |             seastar::posix_thread::start_routine at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/std_function.h:591
  | |   |   |   |             (inlined by) seastar::posix_thread::start_routine at ./build/release/seastar/./seastar/src/core/posix.cc:90
  | |   |   |   ++          - addr=0x8b12c: ?? ??:0
  | |   |   |   ++          - addr=0x10cbbf: ?? ??:0
  | |   |   |   ++[8#2/2 17%] addr=0x56b7e69 total=507 count=60 avg=8:
  | |   |   |   |             seastar::reactor::run at ./build/release/seastar/./seastar/src/core/reactor.cc:3105
  | |   |   |   ++          - addr=0x565bd85:
  | |   |   |   |             seastar::app_template::run_deprecated at ./build/release/seastar/./seastar/src/core/app-template.cc:276
  | |   |   |   ++          - addr=0x565aef8:
  | |   |   |   |             seastar::app_template::run at ./build/release/seastar/./seastar/src/core/app-template.cc:167
  | |   |   |   ++          - addr=0x122cf93:
  | |   |   |   |             scylla_main at ./main.cc:627
  | |   |   |   ++          - addr=0x122eb20:
  | |   |   |   |             std::function<int (int, char**)>::operator() at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/std_function.h:591
  | |   |   |   ++          - addr=0x122b3b7:
  | |   |   |   |             main at ./main.cc:1995
  | |   |   |   ++          - addr=0x2750f: ?? ??:0
  | |   |   |   ++          - addr=0x275c8: ?? ??:0
  | |   |   |   ++          - addr=0x1228f64:
  | |   |   |   |             _start at ??:?
  | |   |   ++[6#2/2 1%] 1 more branches total=13 count=1 avg=13
  | |   |+[4#2/8 16%] addr=0x2129714 total=415 count=68 avg=6:
  | |   |             seastar::future<void>::then_impl<sstables::index_reader::advance_lower_and_check_if_present(dht::ring_position_view, std::optional<position_in_partition_view>)::{lambda()#1}::operator()() const::{lambda()#1}, seastar::future<bool> > at ././utils/logalloc.hh:469
  | |   |             (inlined by) ?? at ././utils/logalloc.hh:520
  | |   |             (inlined by) sstables::index_reader::advance_lower_and_check_if_present(dht::ring_position_view, std::optional<position_in_partition_view>)::{lambda()#1}::operator()() const::{lambda()#1}::operator() at ./sstables/index_reader.hh:932
  | |   |             (inlined by) ?? at ././seastar/include/seastar/core/future.hh:2006
  | |   |             (inlined by) ?? at ././seastar/include/seastar/core/future.hh:1848
  | |   |             (inlined by) seastar::future<void>::then_impl<sstables::index_reader::advance_lower_and_check_if_present(dht::ring_position_view, std::optional<position_in_partition_view>)::{lambda()#1}::operator()() const::{lambda()#1}, seastar::future<bool> > at ././seastar/include/seastar/core/future.hh:1473
  | |   |+          - addr=0x21528e7:
  | |   |             seastar::continuation<seastar::internal::promise_base_with_type<bool>, sstables::index_reader::advance_lower_and_check_if_present(dht::ring_position_view, std::optional<position_in_partition_view>)::{lambda()#1}, seastar::future<void>::then_impl_nrvo<{lambda()#1}, sstables::index_reader::advance_lower_and_check_if_present(dht::ring_position_view, std::optional<position_in_partition_view>)::{lambda()#1}<bool> >({lambda()#1}&&)::{lambda(seastar::internal::promise_base_with_type<bool>&&, {lambda()#1}&, seastar::future_state<seastar::internal::monostate>&&)#1}, void>::run_and_dispose at ././seastar/include/seastar/core/future.hh:1399
  | |   |             (inlined by) sstables::index_reader::advance_lower_and_check_if_present(dht::ring_position_view, std::optional<position_in_partition_view>)::{lambda()#1}::operator() at ./sstables/index_reader.hh:930
  | |   |             (inlined by) std::__invoke_impl<seastar::future<bool>, sstables::index_reader::advance_lower_and_check_if_present(dht::ring_position_view, std::optional<position_in_partition_view>)::{lambda()#1}&> at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:61
  | |   |             (inlined by) std::__invoke<sstables::index_reader::advance_lower_and_check_if_present(dht::ring_position_view, std::optional<position_in_partition_view>)::{lambda()#1}&> at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:96
  | |   |             (inlined by) std::invoke<sstables::index_reader::advance_lower_and_check_if_present(dht::ring_position_view, std::optional<position_in_partition_view>)::{lambda()#1}&> at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/functional:110
  | |   |             (inlined by) seastar::internal::future_invoke<sstables::index_reader::advance_lower_and_check_if_present(dht::ring_position_view, std::optional<position_in_partition_view>)::{lambda()#1}&, seastar::internal::monostate> at ././seastar/include/seastar/core/future.hh:1155
  | |   |             (inlined by) seastar::future<void>::then_impl_nrvo<sstables::index_reader::advance_lower_and_check_if_present(dht::ring_position_view, std::optional<position_in_partition_view>)::{lambda()#1}, seastar::future<bool> >(sstables::index_reader::advance_lower_and_check_if_present(dht::ring_position_view, std::optional<position_in_partition_view>)::{lambda()#1}&&)::{lambda(seastar::internal::promise_base_with_type<bool>&&, {lambda()#1}&, seastar::future_state<auto:1&&::monostate>&&)#1}::operator()(seastar::internal::promise_base_with_type<bool>, seastar::internal::promise_base_with_type<bool>&&, seastar::future_state<sstables::index_reader::advance_lower_and_check_if_present(dht::ring_position_view, std::optional<position_in_partition_view>)::{lambda()#1}&&::monostate>) const::{lambda()#1}::operator() at ././seastar/include/seastar/core/future.hh:1458
  | |   |             (inlined by) ?? at ././seastar/include/seastar/core/future.hh:1991
  | |   |             (inlined by) seastar::future<void>::then_impl_nrvo<sstables::index_reader::advance_lower_and_check_if_present(dht::ring_position_view, std::optional<position_in_partition_view>)::{lambda()#1}, seastar::future<bool> >(sstables::index_reader::advance_lower_and_check_if_present(dht::ring_position_view, std::optional<position_in_partition_view>)::{lambda()#1}&&)::{lambda(seastar::internal::promise_base_with_type<bool>&&, {lambda()#1}&, seastar::future_state<auto:1&&::monostate>&&)#1}::operator() at ././seastar/include/seastar/core/future.hh:1454
  | |   |             (inlined by) seastar::continuation<seastar::internal::promise_base_with_type<bool>, sstables::index_reader::advance_lower_and_check_if_present(dht::ring_position_view, std::optional<position_in_partition_view>)::{lambda()#1}, seastar::future<void>::then_impl_nrvo<{lambda()#1}, sstables::index_reader::advance_lower_and_check_if_present(dht::ring_position_view, std::optional<position_in_partition_view>)::{lambda()#1}<bool> >({lambda()#1}&&)::{lambda(seastar::internal::promise_base_with_type<bool>&&, {lambda()#1}&, seastar::future_state<seastar::internal::monostate>&&)#1}, void>::run_and_dispose at ././seastar/include/seastar/core/future.hh:741
  | |   |+          - addr=0x56b78a4:
  | |   |             seastar::reactor::run_some_tasks at ./build/release/seastar/./seastar/src/core/reactor.cc:2590
  | |   |             (inlined by) seastar::reactor::run_some_tasks at ./build/release/seastar/./seastar/src/core/reactor.cc:3053
  | |   -> continued at addr=0x56b78a4 above
  | |   |+[4#3/8 12%] addr=0x1dff3dc total=324 count=47 avg=7:
  | |   |             cache::cache_flat_mutation_reader::fill_buffer at ././utils/logalloc.hh:469
  | |   |             (inlined by) ?? at ././utils/logalloc.hh:520
  | |   |             (inlined by) ?? at ././row_cache.hh:481
  | |   |             (inlined by) cache::cache_flat_mutation_reader::fill_buffer()::{lambda()#1}::operator() at ././cache_flat_mutation_reader.hh:301
  | |   |             (inlined by) cache::cache_flat_mutation_reader::fill_buffer at ././cache_flat_mutation_reader.hh:309
  | |   | ++[5#1/2 80%] addr=0x1a58283 total=258 count=36 avg=7:
  | |   | |             operator() at ././readers/flat_mutation_reader_v2.hh:209
  | |   | ++          - addr=0x1a5b33a:
  | |   | |             seastar::internal::repeater<flat_mutation_reader_v2::impl::consume_pausable<std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<query_result_builder> > > >(std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<query_result_builder> > >)::{lambda()#1}>::run_and_dispose at ././seastar/include/seastar/core/future.hh:2006
  | |   | |             (inlined by) seastar::futurize_invoke<flat_mutation_reader_v2::impl::consume_pausable<std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<query_result_builder> > > >(std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<query_result_builder> > >)::{lambda()#1}&> at ././seastar/include/seastar/core/future.hh:2037
  | |   | |             (inlined by) seastar::internal::repeater<flat_mutation_reader_v2::impl::consume_pausable<std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<query_result_builder> > > >(std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<query_result_builder> > >)::{lambda()#1}>::run_and_dispose at ././seastar/include/seastar/core/loop.hh:79
  | |   | ++          - addr=0x56b78a4:
  | |   | |             seastar::reactor::run_some_tasks at ./build/release/seastar/./seastar/src/core/reactor.cc:2590
  | |   | |             (inlined by) seastar::reactor::run_some_tasks at ./build/release/seastar/./seastar/src/core/reactor.cc:3053
  | |   | -> continued at addr=0x56b78a4 above
  | |   | ++[5#2/2 20%] addr=0x1fa3ebf total=66 count=11 avg=6:
  | |   | |             flat_mutation_reader_v2::impl::fill_buffer_from<flat_mutation_reader_v2> at ././readers/flat_mutation_reader_v2.hh:505
  | |   | |             (inlined by) ?? at ./readers/mutation_reader.cc:66
  | |   | ++          - addr=0x1e2d572:
  | |   | |             single_partition_populating_reader::fill_buffer()::{lambda()#3}::operator() at ./row_cache.cc:419
  | |   | |             (inlined by) ?? at ././seastar/include/seastar/core/future.hh:2006
  | |   | |             (inlined by) seastar::futurize_invoke<single_partition_populating_reader::fill_buffer()::{lambda()#3}&> at ././seastar/include/seastar/core/future.hh:2037
  | |   | |             (inlined by) seastar::do_until<single_partition_populating_reader::fill_buffer()::{lambda()#3}, single_partition_populating_reader::fill_buffer()::{lambda()#2}> at ././seastar/include/seastar/core/loop.hh:349
  | |   | |             (inlined by) ?? at ./row_cache.cc:418
  | |   | |             (inlined by) single_partition_populating_reader::fill_buffer()::{lambda()#1}::operator() at ./row_cache.cc:415
  | |   | |             (inlined by) std::__invoke_impl<seastar::future<void>, single_partition_populating_reader::fill_buffer()::{lambda()#1}&> at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:61
  | |   | |             (inlined by) std::__invoke<single_partition_populating_reader::fill_buffer()::{lambda()#1}&> at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:96
  | |   | |             (inlined by) std::invoke<single_partition_populating_reader::fill_buffer()::{lambda()#1}&> at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/functional:110
  | |   | |             (inlined by) seastar::internal::future_invoke<single_partition_populating_reader::fill_buffer()::{lambda()#1}&, seastar::internal::monostate> at ././seastar/include/seastar/core/future.hh:1155
  | |   | |             (inlined by) seastar::future<void>::then_impl_nrvo<single_partition_populating_reader::fill_buffer()::{lambda()#1}, seastar::future<void> >(single_partition_populating_reader::fill_buffer()::{lambda()#1}&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, {lambda()#1}&, seastar::future_state<auto:1&&::monostate>&&)#1}::operator()(seastar::internal::promise_base_with_type<void>, seastar::internal::promise_base_with_type<void>&&, seastar::future_state<single_partition_populating_reader::fill_buffer()::{lambda()#1}&&::monostate>) const::{lambda()#1}::operator() at ././seastar/include/seastar/core/future.hh:1458
  | |   | |             (inlined by) ?? at ././seastar/include/seastar/core/future.hh:1991
  | |   | |             (inlined by) seastar::future<void>::then_impl_nrvo<single_partition_populating_reader::fill_buffer()::{lambda()#1}, seastar::future<void> >(single_partition_populating_reader::fill_buffer()::{lambda()#1}&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, {lambda()#1}&, seastar::future_state<auto:1&&::monostate>&&)#1}::operator() at ././seastar/include/seastar/core/future.hh:1454
  | |   | |             (inlined by) seastar::continuation<seastar::internal::promise_base_with_type<void>, single_partition_populating_reader::fill_buffer()::{lambda()#1}, seastar::future<void>::then_impl_nrvo<{lambda()#1}, seastar::future>({lambda()#1}&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, {lambda()#1}&, seastar::future_state<seastar::internal::monostate>&&)#1}, void>::run_and_dispose at ././seastar/include/seastar/core/future.hh:741
  | |   | ++          - addr=0x56b78a4:
  | |   | |             seastar::reactor::run_some_tasks at ./build/release/seastar/./seastar/src/core/reactor.cc:2590
  | |   | |             (inlined by) seastar::reactor::run_some_tasks at ./build/release/seastar/./seastar/src/core/reactor.cc:3053
  | |   | -> continued at addr=0x56b78a4 above
  | |   |+[4#4/8 12%] addr=0x1b49ba9 total=322 count=42 avg=8:
  | |   |             replica::memtable::make_flat_reader_opt at ././utils/logalloc.hh:469
  | |   |             (inlined by) ?? at ././utils/logalloc.hh:520
  | |   |             (inlined by) ?? at ./replica/memtable.cc:709
  | |   |+          - addr=0x19bd5f8:
  | |   |             replica::table::add_memtables_to_reader_list(std::vector<flat_mutation_reader_v2, std::allocator<flat_mutation_reader_v2> >&, seastar::lw_shared_ptr<schema const> const&, reader_permit const&, nonwrapping_interval<dht::ring_position> const&, query::partition_slice const&, tracing::trace_state_ptr const&, seastar::bool_class<streamed_mutation::forwarding_tag>, seastar::bool_class<mutation_reader::partition_range_forwarding_tag>, std::function<void (unsigned long)>) const::$_56::operator() at ./replica/table.cc:194
  | |   |             (inlined by) replica::table::add_memtables_to_reader_list at ./replica/table.cc:205
  | |   |+          - addr=0x19bdca1:
  | |   |             replica::table::make_reader_v2 at ./replica/table.cc:259
  | |   |+          - addr=0x1a96fc2:
  | |   |             replica::table::as_mutation_source() const::$_48::operator() at ./replica/table.cc:2484
  | |   |             (inlined by) std::__invoke_impl<flat_mutation_reader_v2, replica::table::as_mutation_source() const::$_48&, seastar::lw_shared_ptr<schema const>, reader_permit, nonwrapping_interval<dht::ring_position> const&, query::partition_slice const&, tracing::trace_state_ptr, seastar::bool_class<streamed_mutation::forwarding_tag>, seastar::bool_class<mutation_reader::partition_range_forwarding_tag> > at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:61
  | |   |             (inlined by) std::__invoke_r<flat_mutation_reader_v2, replica::table::as_mutation_source() const::$_48&, seastar::lw_shared_ptr<schema const>, reader_permit, nonwrapping_interval<dht::ring_position> const&, query::partition_slice const&, tracing::trace_state_ptr, seastar::bool_class<streamed_mutation::forwarding_tag>, seastar::bool_class<mutation_reader::partition_range_forwarding_tag> > at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:114
  | |   |             (inlined by) std::_Function_handler<flat_mutation_reader_v2 (seastar::lw_shared_ptr<schema const>, reader_permit, nonwrapping_interval<dht::ring_position> const&, query::partition_slice const&, tracing::trace_state_ptr, seastar::bool_class<streamed_mutation::forwarding_tag>, seastar::bool_class<mutation_reader::partition_range_forwarding_tag>), replica::table::as_mutation_source() const::$_48>::_M_invoke at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/std_function.h:290
  | |   |+          - addr=0x1a1a593:
  | |   |             query::querier::querier at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/std_function.h:591
  | |   |             (inlined by) ?? at ././readers/mutation_source.hh:143
  | |   |             (inlined by) query::querier_base::querier_base at ././querier.hh:91
  | |   |             (inlined by) querier at ././querier.hh:158
  | |   |+          - addr=0x1a15d8e:
  | |   |             replica::table::query at ./replica/table.cc:2393
  | |   |+          - addr=0x1954b17:
  | |   |             replica::database::query(seastar::lw_shared_ptr<schema const>, query::read_command const&, query::result_options, std::vector<nonwrapping_interval<dht::ring_position>, std::allocator<nonwrapping_interval<dht::ring_position> > > const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>)::$_28::operator() at ./replica/database.cc:1600
  | |   |             (inlined by) seastar::noncopyable_function<seastar::future<void> (reader_permit)>::indirect_vtable_for<replica::database::query(seastar::lw_shared_ptr<schema const>, query::read_command const&, query::result_options, std::vector<nonwrapping_interval<dht::ring_position>, std::allocator<nonwrapping_interval<dht::ring_position> > > const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>)::$_28>::call at ././seastar/include/seastar/util/noncopyable_function.hh:158
  | |   |+          - addr=0x4299c03:
  | |   |             reader_concurrency_semaphore::execution_loop at ././seastar/include/seastar/util/noncopyable_function.hh:215
  | |   |             (inlined by) reader_concurrency_semaphore::execution_loop at ./reader_concurrency_semaphore.cc:921
  | |   |+          - addr=0x12a595a:
  | |   |             seastar::internal::coroutine_traits_base<void>::promise_type::run_and_dispose at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/coroutine:244
  | |   |             (inlined by) seastar::internal::coroutine_traits_base<void>::promise_type::run_and_dispose at ././seastar/include/seastar/core/coroutine.hh:125
  | |   | ++[5#1/1 100%] addr=0x56b78a4 total=335 count=43 avg=8:
  | |   | |              seastar::reactor::run_some_tasks at ./build/release/seastar/./seastar/src/core/reactor.cc:2590
  | |   | |              (inlined by) seastar::reactor::run_some_tasks at ./build/release/seastar/./seastar/src/core/reactor.cc:3053
  | |   | -> continued at addr=0x56b78a4 above
  | |   ++[4#5/8 11%] 4 more branches total=293 count=29 avg=10
  | ++[2#2/2 37%] addr=0x1ea27fb total=90 count=8 avg=11:
  | |             logalloc::reclaim_from_evictable at ./utils/logalloc.cc:2213
  | |             (inlined by) logalloc::reclaim_from_evictable at ./utils/logalloc.cc:2532
  |   ++[3#1/1 100%] addr=0x1ea0ab4 total=1471 count=125 avg=12:
  |   |              logalloc::tracker::impl::compact_and_evict_locked at ./utils/logalloc.cc:2691
  |     ++[4#1/1 100%] addr=0x1ea8680 total=1537 count=136 avg=11:
  |     |              logalloc::allocating_section::reserve at ./utils/logalloc.cc:2618
  |     |              (inlined by) ?? at ./utils/logalloc.cc:1279
  |     |              (inlined by) ?? at ./utils/logalloc.cc:1292
  |     |              (inlined by) ?? at ./utils/logalloc.cc:2871
  |     -> continued at addr=0x1ea8680 above
  |+[1#2/164 7%] addr=0x1e31abf total=191 count=19 avg=10:
  |              standard_migrator<bplus::data<long, intrusive_array<cache_entry>, dht::raw_token_less_comparator, 16ul, (bplus::key_search)0, (bplus::with_debug)0> >::migrate at row_cache.cc:?
  |              (inlined by) standard_migrator<bplus::data<long, intrusive_array<cache_entry>, dht::raw_token_less_comparator, 16ul, (bplus::key_search)0, (bplus::with_debug)0> >::migrate at ././utils/allocation_strategy.hh:72
  |+           - addr=0x1eabeff:
  |              logalloc::region_impl::compact_segment_locked(logalloc::segment*, logalloc::segment_descriptor&)::{lambda(logalloc::region_impl::object_descriptor const*, void*, unsigned long)#1}::operator() at ./utils/logalloc.cc:1876
  |              (inlined by) ?? at ./utils/logalloc.cc:1720
  |              (inlined by) logalloc::region_impl::compact_segment_locked at ./utils/logalloc.cc:1873
  | ++[2#1/2 60%] addr=0x1ea27fb total=820 count=69 avg=12:
  | |             logalloc::reclaim_from_evictable at ./utils/logalloc.cc:2213
  | |             (inlined by) logalloc::reclaim_from_evictable at ./utils/logalloc.cc:2532
  | -> continued at addr=0x1ea27fb above
  | ++[2#2/2 40%] addr=0x1ea0a8b total=550 count=87 avg=6:
  | |             logalloc::tracker::impl::compact_and_evict_locked at ./utils/logalloc.cc:2213
  | |             (inlined by) logalloc::tracker::impl::compact_and_evict_locked at ./utils/logalloc.cc:2694
  | -> continued at addr=0x1ea0a8b above
  ++[1#3/164 85%] 162 more branches total=2476 count=296 avg=8

[temichus]

@avikivity

[fruch]

this is still seem to be hapanning on our weekly runs:

Cycle	Scylla build	Latency percentile_90 R	Latency percentile_90 W	Steady State percentile_90 R	Steady State percentile_90 W	Latency percentile_99 R	Latency percentile_99 W	Steady State percentile_99 R	Steady State percentile_99 W	Duration (sec)	Commit id, date
Cycle #1	current build	2.65	5.52	1.92	2.4	5.8	21.92	2.48	4.35	0:39:07	5.5.0~dev.20231122.65e42e4166ce
Cycle #2	current build	2.44	4.34	1.92	2.4	4.3	12.03	2.48	4.35	0:26:53	5.5.0~dev.20231122.65e42e4166ce
Cycle #3	current build	2.14	3.44	1.92	2.4	3.47	7.0	2.48	4.35	0:20:54	5.5.0~dev.20231122.65e42e4166ce

Compare current average cycles results with previous version

Cycle	Scylla build	Latency percentile_90 R	Latency percentile_90 W	Steady State percentile_90 R	Steady State percentile_90 W	Latency percentile_99 R	Latency percentile_99 W	Steady State percentile_99 R	Steady State percentile_99 W	Duration (sec)	Commit id, date
average	current build	2.41	4.43	1.92	2.4	4.52	13.65	2.48	4.35	0:28:58	5.5.0~dev.20231122.65e42e4166ce

https://argus.scylladb.com/test/66844056-081d-4a4e-87b9-df7c8ee3dca6/runs?additionalRuns[]=06be77f7-84be-49b2-8d1c-8df73117af6a

[kostja]

I don't think we will be looking into this in dev since the problem will be addressed in a more systematic way by the tablet load balancer. Generallly it's a domain of streaming and workload prioritization @bhalevy @eliransin

[avikivity]

I don't think we will be looking into this in dev since the problem will be addressed in a more systematic way by the tablet load balancer. Generallly it's a domain of streaming and workload prioritization @bhalevy @eliransin

Ignoring problems doesn't make them go away, and we'll be supporting vnodes for a long time.

[kostja]

I don't think we will be looking into this in dev since the problem will be addressed in a more systematic way by the tablet load balancer. Generallly it's a domain of streaming and workload prioritization @bhalevy @eliransin

Ignoring problems doesn't make them go away, and we'll be supporting vnodes for a long time.

It's complicated, as the recent discussion with @dorlaor highlights. I guess this shouldn't be looked at before maturing tablets, and once this happens, there will be a viable workaround.

[avikivity]

I don't think we will be looking into this in dev since the problem will be addressed in a more systematic way by the tablet load balancer. Generallly it's a domain of streaming and workload prioritization @bhalevy @eliransin

Ignoring problems doesn't make them go away, and we'll be supporting vnodes for a long time.

It's complicated, as the recent discussion with @dorlaor highlights. I guess this shouldn't be looked at before maturing tablets, and once this happens, there will be a viable workaround.

That's not clear at all.