1171 reactor stalls messages with >800ms in ics-longevity-toggle-strategy-large-partitions-24h-test #13296

temichus · 2023-03-23T11:22:45Z

Issue description

there are 1171 reactor stalls messages with about ~877ms during ics-longevity-toggle-strategy-large-partitions-24h-test

Installation details

Kernel Version: 5.15.0-1031-aws
Scylla version (or git commit hash): 2023.1.0~rc2-20230302.726f8a090337 with build-id efcc950cda22875aa488c5295c6cc68f35d9cc6f

Cluster size: 5 nodes (i3en.3xlarge)

Scylla Nodes used in this run:

ics-longevity-toggle-strategy-large-db-node-440d8fc6-5 (52.16.28.157 | 10.4.0.37) (shards: 10)
ics-longevity-toggle-strategy-large-db-node-440d8fc6-4 (3.250.104.253 | 10.4.3.66) (shards: 10)
ics-longevity-toggle-strategy-large-db-node-440d8fc6-3 (34.242.52.188 | 10.4.1.90) (shards: 10)
ics-longevity-toggle-strategy-large-db-node-440d8fc6-2 (3.252.198.163 | 10.4.3.232) (shards: 10)
ics-longevity-toggle-strategy-large-db-node-440d8fc6-1 (34.241.45.170 | 10.4.2.125) (shards: 10)

OS / Image: ami-0336a5a646b08dc25 (aws: eu-west-1)

Test: ics-longevity-toggle-strategy-large-partitions-24h-test
Test id: 440d8fc6-70d9-499c-8a38-388304731097
Test name: enterprise-2023.1/SCT_Enterprise_Features/ICS/ics-longevity-toggle-strategy-large-partitions-24h-test
Test config file(s):

longevity-large-partition-200k_pks-4days.yaml

Logs and commands

Restore Monitor Stack command: $ hydra investigate show-monitor 440d8fc6-70d9-499c-8a38-388304731097
Restore monitor on AWS instance using Jenkins job
Show all stored logs command: $ hydra investigate show-logs 440d8fc6-70d9-499c-8a38-388304731097

Logs:

db-cluster-440d8fc6.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/440d8fc6-70d9-499c-8a38-388304731097/20230319_214547/db-cluster-440d8fc6.tar.gz
output-440d8fc6.log.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/440d8fc6-70d9-499c-8a38-388304731097/20230319_214547/output-440d8fc6.log.tar.gz
debug-440d8fc6.log.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/440d8fc6-70d9-499c-8a38-388304731097/20230319_214547/debug-440d8fc6.log.tar.gz
events-440d8fc6.log.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/440d8fc6-70d9-499c-8a38-388304731097/20230319_214547/events-440d8fc6.log.tar.gz
sct-440d8fc6.log.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/440d8fc6-70d9-499c-8a38-388304731097/20230319_214547/sct-440d8fc6.log.tar.gz
normal-440d8fc6.log.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/440d8fc6-70d9-499c-8a38-388304731097/20230319_214547/normal-440d8fc6.log.tar.gz
argus-440d8fc6.log.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/440d8fc6-70d9-499c-8a38-388304731097/20230319_214547/argus-440d8fc6.log.tar.gz
raw_events-440d8fc6.log.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/440d8fc6-70d9-499c-8a38-388304731097/20230319_214547/raw_events-440d8fc6.log.tar.gz
critical-440d8fc6.log.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/440d8fc6-70d9-499c-8a38-388304731097/20230319_214547/critical-440d8fc6.log.tar.gz
warning-440d8fc6.log.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/440d8fc6-70d9-499c-8a38-388304731097/20230319_214547/warning-440d8fc6.log.tar.gz
summary-440d8fc6.log.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/440d8fc6-70d9-499c-8a38-388304731097/20230319_214547/summary-440d8fc6.log.tar.gz
error-440d8fc6.log.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/440d8fc6-70d9-499c-8a38-388304731097/20230319_214547/error-440d8fc6.log.tar.gz
monitor-set-440d8fc6.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/440d8fc6-70d9-499c-8a38-388304731097/20230319_214547/monitor-set-440d8fc6.tar.gz
loader-set-440d8fc6.tar.gz - https://cloudius-jenkins-test.s3.amazonaws.com/440d8fc6-70d9-499c-8a38-388304731097/20230319_214547/loader-set-440d8fc6.tar.gz

Jenkins job URL

The text was updated successfully, but these errors were encountered:

temichus · 2023-03-23T11:36:04Z

decoded the stall backtraces: output.txt

avikivity · 2023-03-23T11:41:35Z

Processed 1197 stalls lasting a total of 1015764 milliseconds.
min=500 avg=848.6 median=877 p95=883 p99=903 p999=913 max=918

This graph is printed in bottom-up order, where callees are printed first.
Use --direction=top-down to print callers first.

[level#index/out_of pct%] below denotes:
  level  - nesting level in the graph
  index  - index of node among to its siblings
  out_of - number of siblings
  pct    - percentage of total stall time of this call relative to its siblings

Command line options: --address-threshold=0x100000000 --executable=scylla/libexec/scylla --direction=bottom-up --branch-threshold=0.05 --file=stails.log

++[0#1/3 72%] addr=0x51aec83 total=734826 count=866 avg=849:
|             seastar::backtrace<seastar::backtrace_buffer::append_backtrace_oneline()::{lambda(seastar::frame)#1}> at ./build/release/seastar/./seastar/include/seastar/util/backtrace.hh:59
|             (inlined by) seastar::backtrace_buffer::append_backtrace_oneline at ./build/release/seastar/./seastar/src/core/reactor.cc:797
|             (inlined by) seastar::print_with_backtrace at ./build/release/seastar/./seastar/src/core/reactor.cc:816
++          - addr=0x51ae070:
|             seastar::internal::cpu_stall_detector::generate_trace at ./build/release/seastar/./seastar/src/core/reactor.cc:1346
++          - addr=0x51af450:
|             seastar::internal::cpu_stall_detector::maybe_report at ./build/release/seastar/./seastar/src/core/reactor.cc:1123
|             (inlined by) seastar::internal::cpu_stall_detector::on_signal at ./build/release/seastar/./seastar/src/core/reactor.cc:1140
|             (inlined by) seastar::reactor::block_notifier at ./build/release/seastar/./seastar/src/core/reactor.cc:1382
++          - addr=0x3cb1f: ?? ??:0
| ++[1#1/266 9%] addr=0x13cf34e total=62923 count=74 avg=850:
| |              managed_bytes_basic_view<(mutable_view)0>::prefix at ././utils/managed_bytes.hh:422
| |              (inlined by) compound_type<(allow_prefixes)1>::iterator::read_current at ././compound.hh:170
| |              (inlined by) iterator at ././compound.hh:176
| | ++[2#1/3 56%] addr=0x1d806b7 total=35024 count=41 avg=854:
| | |             compound_type<(allow_prefixes)1>::begin at ././compound.hh:194
| | |             (inlined by) bound_view::tri_compare::operator() at ././clustering_bounds_comparator.hh:61
| | |             (inlined by) position_in_partition::tri_compare::compare<position_in_partition_view, position_in_partition_view> at ././position_in_partition.hh:526
| | |             (inlined by) position_in_partition::tri_compare::operator() at ././position_in_partition.hh:534
| | |             (inlined by) rows_entry::tri_compare::operator() at ././mutation_partition.hh:1002
| | |             (inlined by) intrusive_b::searcher<position_in_partition_view, rows_entry, &rows_entry::_link, rows_entry::tri_compare, (intrusive_b::key_search)0>::ge at ././utils/intrusive_btree.hh:1050
| | |             (inlined by) intrusive_b::node<rows_entry, &rows_entry::_link, rows_entry::tri_compare, 12ul, 20ul, (intrusive_b::key_search)0, (intrusive_b::with_debug)0>::index_for<position_in_partition_view> at ././utils/intrusive_btree.hh:1388
| | |             (inlined by) intrusive_b::tree<rows_entry, &rows_entry::_link, rows_entry::tri_compare, 12ul, 20ul, (intrusive_b::key_search)0, (intrusive_b::with_debug)0>::key_lower_bound<position_in_partition_view> at ././utils/intrusive_btree.hh:276
| | |             (inlined by) intrusive_b::tree<rows_entry, &rows_entry::_link, rows_entry::tri_compare, 12ul, 20ul, (intrusive_b::key_search)0, (intrusive_b::with_debug)0>::find<position_in_partition_view> at ././utils/intrusive_btree.hh:454
| | | ++[3#1/1 100%] addr=0x1d30708 total=107350 count=126 avg=852:
| | | |              intrusive_b::tree<rows_entry, &rows_entry::_link, rows_entry::tri_compare, 12ul, 20ul, (intrusive_b::key_search)0, (intrusive_b::with_debug)0>::find<position_in_partition_view> at ././utils/intrusive_btree.hh:464
| | | |              (inlined by) mutation_partition::clustered_row at ./mutation_partition.cc:691
| | |   ++[4#1/1 100%] addr=0x1e4353f total=381556 count=450 avg=848:
| | |   |              converting_mutation_partition_applier::accept_row at ./converting_mutation_partition_applier.cc:139
| | |     ++[5#1/1 100%] addr=0x1d278e5 total=414477 count=489 avg=848:
| | |     |              mutation_partition::accept at ./mutation_partition.cc:1921
| | |     |              (inlined by) mutation_partition::upgrade at ./mutation_partition.cc:1939
| | |       ++[6#1/1 100%] addr=0x1bd229c total=416167 count=491 avg=848:
| | |       |              partition_entry::squashed at ./partition_version.cc:503
| | |       |              (inlined by) partition_entry::upgrade at ./partition_version.cc:517
| | |         ++[7#1/8 94%] addr=0x1c5ff15 total=657479 count=775 avg=848:
| | |         |             operator() at ./row_cache.cc:1319
| | |         |             (inlined by) with_allocator<row_cache::upgrade_entry(cache_entry&)::$_28> at ././utils/allocation_strategy.hh:313
| | |         |             (inlined by) row_cache::upgrade_entry at ./row_cache.cc:1318
| | |         |             (inlined by) scanning_and_populating_reader::read_from_entry at ./row_cache.cc:573
| | |         |             (inlined by) operator() at ./row_cache.cc:601
| | |         | ++[8#1/1 100%] addr=0x1c575d1 total=947444 count=1118 avg=847:
| | |         | |              logalloc::allocating_section::with_reclaiming_disabled<scanning_and_populating_reader::do_read_from_primary()::{lambda()#1}&> at ././utils/logalloc.hh:499
| | |         | |              (inlined by) operator() at ././utils/logalloc.hh:521
| | |         | |              (inlined by) logalloc::allocating_section::with_reserve<logalloc::allocating_section::operator()<scanning_and_populating_reader::do_read_from_primary()::{lambda()#1}>(logalloc::region&, scanning_and_populating_reader::do_read_from_primary()::{lambda()#1}&&)::{lambda()#1}> at ././utils/logalloc.hh:470
| | |         | |              (inlined by) logalloc::allocating_section::operator()<scanning_and_populating_reader::do_read_from_primary()::{lambda()#1}> at ././utils/logalloc.hh:520
| | |         | |              (inlined by) scanning_and_populating_reader::do_read_from_primary at ./row_cache.cc:584
| | |         | |              (inlined by) scanning_and_populating_reader::read_from_primary at ./row_cache.cc:630
| | |         |   ++[9#1/4 58%] addr=0x1c56dc5 total=553721 count=662 avg=836:
| | |         |   |             operator() at ./row_cache.cc:657
| | |         |   | ++[10#1/2 95%] addr=0x1c568d0 total=619792 count=743 avg=834:
| | |         |   | |              seastar::futurize<seastar::future<void> >::invoke<scanning_and_populating_reader::read_next_partition()::{lambda()#1}> at ././seastar/include/seastar/core/future.hh:2147
| | |         |   | |              (inlined by) seastar::futurize<seastar::future<void> >::invoke<scanning_and_populating_reader::read_next_partition()::{lambda()#1}> at ././seastar/include/seastar/core/future.hh:1991
| | |         |   | |              (inlined by) seastar::future<void>::then_impl<scanning_and_populating_reader::read_next_partition()::{lambda()#1}, seastar::future<void> > at ././seastar/include/seastar/core/future.hh:1613
| | |         |   | |              (inlined by) seastar::internal::call_then_impl<seastar::future<void> >::run<scanning_and_populating_reader::read_next_partition()::{lambda()#1}> at ././seastar/include/seastar/core/future.hh:1246
| | |         |   | |              (inlined by) seastar::future<void>::then<scanning_and_populating_reader::read_next_partition()::{lambda()#1}, seastar::future<void> > at ././seastar/include/seastar/core/future.hh:1532
| | |         |   | |              (inlined by) scanning_and_populating_reader::read_next_partition at ./row_cache.cc:655
| | |         |   | |              (inlined by) operator() at ./row_cache.cc:681
| | |         |   | |              (inlined by) seastar::futurize<seastar::future<void> >::invoke<scanning_and_populating_reader::fill_buffer()::{lambda()#2}&> at ././seastar/include/seastar/core/future.hh:2147
| | |         |   | |              (inlined by) seastar::futurize_invoke<scanning_and_populating_reader::fill_buffer()::{lambda()#2}&> at ././seastar/include/seastar/core/future.hh:2178
| | |         |   | |              (inlined by) seastar::do_until<scanning_and_populating_reader::fill_buffer()::{lambda()#2}, scanning_and_populating_reader::fill_buffer()::{lambda()#1}> at ././seastar/include/seastar/core/loop.hh:343
| | |         |   | ++           - addr=0x1c544eb:
| | |         |   | |              scanning_and_populating_reader::fill_buffer at ./row_cache.cc:679
| | |         |   | ++           - addr=0x1e4c852:
| | |         |   | |              flat_mutation_reader_v2::impl::operator() at ././readers/flat_mutation_reader_v2.hh:195
| | |         |   | |              (inlined by) flat_mutation_reader_v2::operator() at ././readers/flat_mutation_reader_v2.hh:415
| | |         |   | |              (inlined by) mutation_reader_merger::prepare_one at ./readers/combined.cc:457
| | |         |   | |              (inlined by) operator() at ./readers/combined.cc:448
| | |         |   | |              (inlined by) seastar::futurize<seastar::future<void> >::invoke<mutation_reader_merger::prepare_next()::$_2, mutation_reader_merger::reader_and_last_fragment_kind&> at ././seastar/include/seastar/core/future.hh:2147
| | |         |   | |              (inlined by) seastar::futurize_invoke<mutation_reader_merger::prepare_next()::$_2, mutation_reader_merger::reader_and_last_fragment_kind&> at ././seastar/include/seastar/core/future.hh:2178
| | |         |   | |              (inlined by) seastar::parallel_for_each<mutation_reader_merger::reader_and_last_fragment_kind*, mutation_reader_merger::reader_and_last_fragment_kind*, mutation_reader_merger::prepare_next()::$_2> at ././seastar/include/seastar/core/loop.hh:569
| | |         |   | |              (inlined by) seastar::internal::parallel_for_each_impl<utils::small_vector<mutation_reader_merger::reader_and_last_fragment_kind, 4ul>&, mutation_reader_merger::prepare_next()::$_2> at ././seastar/include/seastar/core/loop.hh:622
| | |         |   | |              (inlined by) seastar::futurize<seastar::future<void> >::invoke<seastar::future<void> (*&)(utils::small_vector<mutation_reader_merger::reader_and_last_fragment_kind, 4ul>&, mutation_reader_merger::prepare_next()::$_2&&), utils::small_vector<mutation_reader_merger::reader_and_last_fragment_kind, 4ul>&, mutation_reader_merger::prepare_next()::$_2> at ././seastar/include/seastar/core/future.hh:2147
| | |         |   | |              (inlined by) seastar::futurize_invoke<seastar::future<void> (*&)(utils::small_vector<mutation_reader_merger::reader_and_last_fragment_kind, 4ul>&, mutation_reader_merger::prepare_next()::$_2&&), utils::small_vector<mutation_reader_merger::reader_and_last_fragment_kind, 4ul>&, mutation_reader_merger::prepare_next()::$_2> at ././seastar/include/seastar/core/future.hh:2178
| | |         |   | |              (inlined by) seastar::parallel_for_each<utils::small_vector<mutation_reader_merger::reader_and_last_fragment_kind, 4ul>&, mutation_reader_merger::prepare_next()::$_2> at ././seastar/include/seastar/core/loop.hh:637
| | |         |   | |              (inlined by) mutation_reader_merger::prepare_next at ./readers/combined.cc:447
| | |         |   | | ++[11#1/1 100%] addr=0x1e4d713 total=620684 count=744 avg=834:
| | |         |   | | |               mutation_reader_merger::operator() at ./readers/combined.cc:578
| | |         |   | | ++            - addr=0x1e59784:
| | |         |   | | |               operator() at ./readers/combined.cc:171
| | |         |   | | |               (inlined by) seastar::futurize<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > >::invoke<mutation_fragment_merger<mutation_reader_merger>::operator()()::{lambda()#1}&> at ././seastar/include/seastar/core/future.hh:2147
| | |         |   | | |               (inlined by) seastar::repeat<mutation_fragment_merger<mutation_reader_merger>::operator()()::{lambda()#1}> at ././seastar/include/seastar/core/loop.hh:120
| | |         |   | | |               (inlined by) mutation_fragment_merger<mutation_reader_merger>::operator() at ./readers/combined.cc:170
| | |         |   | | |               (inlined by) operator() at ./readers/combined.cc:692
| | |         |   | | ++            - addr=0x1e58422:
| | |         |   | | |               seastar::futurize<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > >::invoke<merging_reader<mutation_reader_merger>::fill_buffer()::{lambda()#1}&> at ././seastar/include/seastar/core/future.hh:2147
| | |         |   | | |               (inlined by) seastar::repeat<merging_reader<mutation_reader_merger>::fill_buffer()::{lambda()#1}> at ././seastar/include/seastar/core/loop.hh:120
| | |         |   | | |               (inlined by) merging_reader<mutation_reader_merger>::fill_buffer at ./readers/combined.cc:691
| | |         |   | |   ++[12#1/4 91%] addr=0x1e7c2cf total=563130 count=676 avg=833:
| | |         |   | |   |              flat_mutation_reader_v2::fill_buffer at ./readers/flat_mutation_reader_v2.hh:510
| | |         |   | |   |              (inlined by) evictable_reader_v2::fill_buffer at ./readers/multishard.cc:595
| | |         |   | |   | ++[13#1/3 59%] addr=0x1e9515f total=332275 count=399 avg=833:
| | |         |   | |   | |              operator() at ./readers/multishard.cc:844
| | |         |   | |   | | ++[14#1/2 80%] addr=0x1e96151 total=309178 count=375 avg=824:
| | |         |   | |   | | |              std::__invoke_impl<seastar::future<(anonymous namespace)::remote_fill_buffer_result_v2>, (anonymous namespace)::shard_reader_v2::do_fill_buffer()::$_8::operator()() const::{lambda()#2}&> at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:61
| | |         |   | |   | | |              (inlined by) std::__invoke<(anonymous namespace)::shard_reader_v2::do_fill_buffer()::$_8::operator()() const::{lambda()#2}&> at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:96
| | |         |   | |   | | |              (inlined by) std::invoke<(anonymous namespace)::shard_reader_v2::do_fill_buffer()::$_8::operator()() const::{lambda()#2}&> at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/functional:110
| | |         |   | |   | | |              (inlined by) seastar::coroutine::lambda<(anonymous namespace)::shard_reader_v2::do_fill_buffer()::$_8::operator()() const::{lambda()#2}>::operator()<> at ././seastar/include/seastar/core/coroutine.hh:259
| | |         |   | |   | | |              (inlined by) seastar::futurize<seastar::future<(anonymous namespace)::remote_fill_buffer_result_v2> >::invoke<seastar::coroutine::lambda<(anonymous namespace)::shard_reader_v2::do_fill_buffer()::$_8::operator()() const::{lambda()#2}>&> at ././seastar/include/seastar/core/future.hh:2147
| | |         |   | |   | | |              (inlined by) seastar::smp_message_queue::async_work_item<seastar::coroutine::lambda<(anonymous namespace)::shard_reader_v2::do_fill_buffer()::$_8::operator()() const::{lambda()#2}> >::run_and_dispose at ././seastar/include/seastar/core/smp.hh:235
| | |         |   | |   | | ++           - addr=0x51bf564:
| | |         |   | |   | | |              seastar::reactor::run_tasks at ./build/release/seastar/./seastar/src/core/reactor.cc:2509
| | |         |   | |   | | |              (inlined by) seastar::reactor::run_some_tasks at ./build/release/seastar/./seastar/src/core/reactor.cc:2946
| | |         |   | |   | | | ++[15#1/1 100%] addr=0x51c07e7 total=977158 count=1152 avg=848:
| | |         |   | |   | | | |               seastar::reactor::do_run at ./build/release/seastar/./seastar/src/core/reactor.cc:3115
| | |         |   | |   | | |   ++[16#1/2 80%] addr=0x51e15d1 total=786361 count=928 avg=847:
| | |         |   | |   | | |   |              operator() at ./build/release/seastar/./seastar/src/core/reactor.cc:4326
| | |         |   | |   | | |   |              (inlined by) std::__invoke_impl<void, seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_94&> at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:61
| | |         |   | |   | | |   |              (inlined by) std::__invoke_r<void, seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_94&> at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:111
| | |         |   | |   | | |   |              (inlined by) std::_Function_handler<void (), seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_94>::_M_invoke at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/std_function.h:290
| | |         |   | |   | | |   ++           - addr=0x51918ba:
| | |         |   | |   | | |   |              std::function<void ()>::operator() at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/std_function.h:591
| | |         |   | |   | | |   |              (inlined by) seastar::posix_thread::start_routine at ./build/release/seastar/./seastar/src/core/posix.cc:73
| | |         |   | |   | | |   ++           - addr=0x8b12c: ?? ??:0
| | |         |   | |   | | |   ++           - addr=0x10cbbf: ?? ??:0
| | |         |   | |   | | |   ++[16#2/2 20%] addr=0x51bfb29 total=190797 count=224 avg=852:
| | |         |   | |   | | |   |              seastar::reactor::run at ./build/release/seastar/./seastar/src/core/reactor.cc:2998
| | |         |   | |   | | |   ++           - addr=0x5166575:
| | |         |   | |   | | |   |              seastar::app_template::run_deprecated at ./build/release/seastar/./seastar/src/core/app-template.cc:266
| | |         |   | |   | | |   ++           - addr=0x51656e8:
| | |         |   | |   | | |   |              seastar::app_template::run at ./build/release/seastar/./seastar/src/core/app-template.cc:157
| | |         |   | |   | | |   ++           - addr=0x11a8ca0:
| | |         |   | |   | | |   |              scylla_main at ./main.cc:557
| | |         |   | |   | | |   ++           - addr=0x11aa800:
| | |         |   | |   | | |   |              std::function<int (int, char**)>::operator() at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/std_function.h:591
| | |         |   | |   | | |   ++           - addr=0x11a73aa:
| | |         |   | |   | | |   |              main at ./main.cc:1832
| | |         |   | |   | | |   ++           - addr=0x2750f: ?? ??:0
| | |         |   | |   | | |   ++           - addr=0x275c8: ?? ??:0
| | |         |   | |   | | |   ++           - addr=0x11a5564:
| | |         |   | |   | | |   |              _start at ??:?
| | |         |   | |   | | ++[14#2/2 20%] addr=0x1e8b5e6 total=75426 count=86 avg=877:
| | |         |   | |   | | |              std::__invoke_impl<seastar::future<(anonymous namespace)::remote_fill_buffer_result_v2>, (anonymous namespace)::shard_reader_v2::do_fill_buffer()::$_8::operator()() const::{lambda()#2}&> at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:61
| | |         |   | |   | | |              (inlined by) std::__invoke<(anonymous namespace)::shard_reader_v2::do_fill_buffer()::$_8::operator()() const::{lambda()#2}&> at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:96
| | |         |   | |   | | |              (inlined by) std::invoke<(anonymous namespace)::shard_reader_v2::do_fill_buffer()::$_8::operator()() const::{lambda()#2}&> at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/functional:110
| | |         |   | |   | | |              (inlined by) seastar::coroutine::lambda<(anonymous namespace)::shard_reader_v2::do_fill_buffer()::$_8::operator()() const::{lambda()#2}>::operator()<> at ././seastar/include/seastar/core/coroutine.hh:259
| | |         |   | |   | | |              (inlined by) seastar::futurize<seastar::future<(anonymous namespace)::remote_fill_buffer_result_v2> >::invoke<seastar::coroutine::lambda<(anonymous namespace)::shard_reader_v2::do_fill_buffer()::$_8::operator()() const::{lambda()#2}>&> at ././seastar/include/seastar/core/future.hh:2147
| | |         |   | |   | | |              (inlined by) seastar::smp::submit_to<seastar::coroutine::lambda<(anonymous namespace)::shard_reader_v2::do_fill_buffer()::$_8::operator()() const::{lambda()#2}> > at ././seastar/include/seastar/core/smp.hh:353
| | |         |   | |   | | |              (inlined by) seastar::smp::submit_to<seastar::coroutine::lambda<(anonymous namespace)::shard_reader_v2::do_fill_buffer()::$_8::operator()() const::{lambda()#2}> > at ././seastar/include/seastar/core/smp.hh:380
| | |         |   | |   | | |              (inlined by) operator() at ./readers/multishard.cc:842
| | |         |   | |   | | ++           - addr=0x1e896de:
| | |         |   | |   | | |              std::__invoke_impl<seastar::future<(anonymous namespace)::remote_fill_buffer_result_v2>, (anonymous namespace)::shard_reader_v2::do_fill_buffer()::$_8> at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:61
| | |         |   | |   | | |              (inlined by) std::__invoke<(anonymous namespace)::shard_reader_v2::do_fill_buffer()::$_8> at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:96
| | |         |   | |   | | |              (inlined by) std::invoke<(anonymous namespace)::shard_reader_v2::do_fill_buffer()::$_8> at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/functional:110
| | |         |   | |   | | |              (inlined by) (anonymous namespace)::shard_reader_v2::do_fill_buffer at ./readers/multishard.cc:807
| | |         |   | |   | |   ++[15#1/2 90%] addr=0x1eb08b1 total=80656 count=92 avg=877:
| | |         |   | |   | |   |              (anonymous namespace)::shard_reader_v2::fill_buffer at ./readers/multishard.cc:868
| | |         |   | |   | |   ++           - addr=0x1ebd5c3:
| | |         |   | |   | |   |              multishard_combining_reader_v2::handle_empty_reader_buffer at ./readers/multishard.cc:1051
| | |         |   | |   | |   |              (inlined by) operator() at ./readers/multishard.cc:1081
| | |         |   | |   | |   | ++[16#1/2 96%] addr=0x1e8517b total=77148 count=88 avg=877:
| | |         |   | |   | |   | |              seastar::futurize<seastar::future<void> >::invoke<multishard_combining_reader_v2::fill_buffer()::$_11&> at ././seastar/include/seastar/core/future.hh:2147
| | |         |   | |   | |   | |              (inlined by) seastar::futurize_invoke<multishard_combining_reader_v2::fill_buffer()::$_11&> at ././seastar/include/seastar/core/future.hh:2178
| | |         |   | |   | |   | |              (inlined by) seastar::do_until<multishard_combining_reader_v2::fill_buffer()::$_11, multishard_combining_reader_v2::fill_buffer()::$_12> at ././seastar/include/seastar/core/loop.hh:343
| | |         |   | |   | |   | |              (inlined by) multishard_combining_reader_v2::fill_buffer at ./readers/multishard.cc:1077
| | |         |   | |   | |   | | ++[17#1/2 98%] addr=0x4007cc8 total=75426 count=86 avg=877:
| | |         |   | |   | |   | | |              operator() at ././readers/flat_mutation_reader_v2.hh:210
| | |         |   | |   | |   | | |              (inlined by) seastar::futurize<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > >::invoke<flat_mutation_reader_v2::impl::consume_pausable<std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<(anonymous namespace)::data_query_result_builder> > > >(std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<(anonymous namespace)::data_query_result_builder> > >)::{lambda()#1}&> at ././seastar/include/seastar/core/future.hh:2147
| | |         |   | |   | |   | | | ++[18#1/2 58%] addr=0x400a71a total=43836 count=50 avg=877:
| | |         |   | |   | |   | | | |              seastar::futurize_invoke<flat_mutation_reader_v2::impl::consume_pausable<std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<(anonymous namespace)::data_query_result_builder> > > >(std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<(anonymous namespace)::data_query_result_builder> > >)::{lambda()#1}&> at ././seastar/include/seastar/core/future.hh:2178
| | |         |   | |   | |   | | | |              (inlined by) seastar::internal::repeater<flat_mutation_reader_v2::impl::consume_pausable<std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<(anonymous namespace)::data_query_result_builder> > > >(std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<(anonymous namespace)::data_query_result_builder> > >)::{lambda()#1}>::run_and_dispose at ././seastar/include/seastar/core/loop.hh:73
| | |         |   | |   | |   | | | ++           - addr=0x51bf564:
| | |         |   | |   | |   | | | |              seastar::reactor::run_tasks at ./build/release/seastar/./seastar/src/core/reactor.cc:2509
| | |         |   | |   | |   | | | |              (inlined by) seastar::reactor::run_some_tasks at ./build/release/seastar/./seastar/src/core/reactor.cc:2946
| | |         |   | |   | |   | | | -> continued at addr=0x51bf564 above
| | |         |   | |   | |   | | | ++[18#2/2 42%] addr=0x40042e2 total=31590 count=36 avg=878:
| | |         |   | |   | |   | | | |              seastar::repeat<flat_mutation_reader_v2::impl::consume_pausable<std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<(anonymous namespace)::data_query_result_builder> > > >(std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<(anonymous namespace)::data_query_result_builder> > >)::{lambda()#1}> at ././seastar/include/seastar/core/loop.hh:120
| | |         |   | |   | |   | | | |              (inlined by) flat_mutation_reader_v2::impl::consume_pausable<std::reference_wrapper<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<(anonymous namespace)::data_query_result_builder> > > > at ././readers/flat_mutation_reader_v2.hh:205
| | |         |   | |   | |   | | | |              (inlined by) operator() at ././readers/flat_mutation_reader_v2.hh:325
| | |         |   | |   | |   | | | |              (inlined by) std::__invoke_impl<seastar::future<query::result>, flat_mutation_reader_v2::impl::consume<compact_for_query_v2<(anonymous namespace)::data_query_result_builder> >(compact_for_query_v2<(anonymous namespace)::data_query_result_builder>)::{lambda(flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<(anonymous namespace)::data_query_result_builder> >&)#1}&, flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<(anonymous namespace)::data_query_result_builder> >&> at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:61
| | |         |   | |   | |   | | | |              (inlined by) std::__invoke<flat_mutation_reader_v2::impl::consume<compact_for_query_v2<(anonymous namespace)::data_query_result_builder> >(compact_for_query_v2<(anonymous namespace)::data_query_result_builder>)::{lambda(flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<(anonymous namespace)::data_query_result_builder> >&)#1}&, flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<(anonymous namespace)::data_query_result_builder> >&> at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:96
| | |         |   | |   | |   | | | |              (inlined by) std::__apply_impl<flat_mutation_reader_v2::impl::consume<compact_for_query_v2<(anonymous namespace)::data_query_result_builder> >(compact_for_query_v2<(anonymous namespace)::data_query_result_builder>)::{lambda(flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<(anonymous namespace)::data_query_result_builder> >&)#1}&, std::tuple<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<(anonymous namespace)::data_query_result_builder> > >&, 0ul> at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/tuple:1852
| | |         |   | |   | |   | | | |              (inlined by) std::apply<flat_mutation_reader_v2::impl::consume<compact_for_query_v2<(anonymous namespace)::data_query_result_builder> >(compact_for_query_v2<(anonymous namespace)::data_query_result_builder>)::{lambda(flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<(anonymous namespace)::data_query_result_builder> >&)#1}&, std::tuple<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<(anonymous namespace)::data_query_result_builder> > >&> at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/tuple:1863
| | |         |   | |   | |   | | | |              (inlined by) seastar::internal::do_with_impl<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<(anonymous namespace)::data_query_result_builder> >, flat_mutation_reader_v2::impl::consume<compact_for_query_v2<(anonymous namespace)::data_query_result_builder> >(compact_for_query_v2<(anonymous namespace)::data_query_result_builder>)::{lambda(flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<(anonymous namespace)::data_query_result_builder> >&)#1}> at ././seastar/include/seastar/core/do_with.hh:96
| | |         |   | |   | |   | | | |              (inlined by) seastar::futurize<seastar::future<query::result> >::invoke<seastar::future<query::result> (*&)(flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<(anonymous namespace)::data_query_result_builder> >&&, flat_mutation_reader_v2::impl::consume<compact_for_query_v2<(anonymous namespace)::data_query_result_builder> >(compact_for_query_v2<(anonymous namespace)::data_query_result_builder>)::{lambda(flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<(anonymous namespace)::data_query_result_builder> >&)#1}&&), flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<(anonymous namespace)::data_query_result_builder> >, {lambda(flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<(anonymous namespace)::data_query_result_builder> >&)#1}> at ././seastar/include/seastar/core/future.hh:2147
| | |         |   | |   | |   | | | |              (inlined by) seastar::futurize_invoke<seastar::future<query::result> (*&)(flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<(anonymous namespace)::data_query_result_builder> >&&, flat_mutation_reader_v2::impl::consume<compact_for_query_v2<(anonymous namespace)::data_query_result_builder> >(compact_for_query_v2<(anonymous namespace)::data_query_result_builder>)::{lambda(flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<(anonymous namespace)::data_query_result_builder> >&)#1}&&), flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<(anonymous namespace)::data_query_result_builder> >, {lambda(flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<(anonymous namespace)::data_query_result_builder> >&)#1}> at ././seastar/include/seastar/core/future.hh:2178
| | |         |   | |   | |   | | | |              (inlined by) seastar::do_with<flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<(anonymous namespace)::data_query_result_builder> >, flat_mutation_reader_v2::impl::consume<compact_for_query_v2<(anonymous namespace)::data_query_result_builder> >(compact_for_query_v2<(anonymous namespace)::data_query_result_builder>)::{lambda(flat_mutation_reader_v2::impl::consumer_adapter<compact_for_query_v2<(anonymous namespace)::data_query_result_builder> >&)#1}> at ././seastar/include/seastar/core/do_with.hh:131
| | |         |   | |   | |   | | | |              (inlined by) flat_mutation_reader_v2::impl::consume<compact_for_query_v2<(anonymous namespace)::data_query_result_builder> > at ././readers/flat_mutation_reader_v2.hh:324
| | |         |   | |   | |   | | | |              (inlined by) flat_mutation_reader_v2::consume<compact_for_query_v2<(anonymous namespace)::data_query_result_builder> > at ././readers/flat_mutation_reader_v2.hh:427
| | |         |   | |   | |   | | | |              (inlined by) operator() at ././querier.hh:49
| | |         |   | |   | |   | | | ++           - addr=0x400b560:
| | |         |   | |   | |   | | | |              std::__invoke_impl<seastar::future<query::result>, query::consume_page<(anonymous namespace)::data_query_result_builder>(flat_mutation_reader_v2&, seastar::lw_shared_ptr<compact_mutation_state<(compact_for_sstables)0> >, query::partition_slice const&, (anonymous namespace)::data_query_result_builder&&, unsigned long, unsigned int, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >)::{lambda(mutation_fragment_v2*)#1}&, mutation_fragment_v2*> at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:61
| | |         |   | |   | |   | | | |              (inlined by) std::__invoke<query::consume_page<(anonymous namespace)::data_query_result_builder>(flat_mutation_reader_v2&, seastar::lw_shared_ptr<compact_mutation_state<(compact_for_sstables)0> >, query::partition_slice const&, (anonymous namespace)::data_query_result_builder&&, unsigned long, unsigned int, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >)::{lambda(mutation_fragment_v2*)#1}&, mutation_fragment_v2*> at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:96
| | |         |   | |   | |   | | | |              (inlined by) std::invoke<query::consume_page<(anonymous namespace)::data_query_result_builder>(flat_mutation_reader_v2&, seastar::lw_shared_ptr<compact_mutation_state<(compact_for_sstables)0> >, query::partition_slice const&, (anonymous namespace)::data_query_result_builder&&, unsigned long, unsigned int, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >)::{lambda(mutation_fragment_v2*)#1}&, mutation_fragment_v2*> at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/functional:110
| | |         |   | |   | |   | | | |              (inlined by) seastar::internal::future_invoke<query::consume_page<(anonymous namespace)::data_query_result_builder>(flat_mutation_reader_v2&, seastar::lw_shared_ptr<compact_mutation_state<(compact_for_sstables)0> >, query::partition_slice const&, (anonymous namespace)::data_query_result_builder&&, unsigned long, unsigned int, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >)::{lambda(mutation_fragment_v2*)#1}&, mutation_fragment_v2*> at ././seastar/include/seastar/core/future.hh:1223
| | |         |   | |   | |   | | | |              (inlined by) operator() at ././seastar/include/seastar/core/future.hh:1594
| | |         |   | |   | |   | | | |              (inlined by) seastar::futurize<seastar::future<query::result> >::satisfy_with_result_of<seastar::future<mutation_fragment_v2*>::then_impl_nrvo<query::consume_page<(anonymous namespace)::data_query_result_builder>(flat_mutation_reader_v2&, seastar::lw_shared_ptr<compact_mutation_state<(compact_for_sstables)0> >, query::partition_slice const&, (anonymous namespace)::data_query_result_builder&&, unsigned long, unsigned int, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >)::{lambda(mutation_fragment_v2*)#1}, seastar::future<query::result> >((anonymous namespace)::data_query_result_builder&&)::{lambda(seastar::internal::promise_base_with_type<query::result>&&, {lambda(mutation_fragment_v2*)#1}&, seastar::future_state<mutation_fragment_v2*>&&)#1}::operator()(seastar::internal::promise_base_with_type<query::result>, seastar::internal::promise_base_with_type<query::result>&&, seastar::future_state<mutation_fragment_v2*>) const::{lambda()#1}> at ././seastar/include/seastar/core/future.hh:2132
| | |         |   | |   | |   | | | |              (inlined by) operator() at ././seastar/include/seastar/core/future.hh:1587
| | |         |   | |   | |   | | | |              (inlined by) seastar::continuation<seastar::internal::promise_base_with_type<query::result>, query::consume_page<(anonymous namespace)::data_query_result_builder>(flat_mutation_reader_v2&, seastar::lw_shared_ptr<compact_mutation_state<(compact_for_sstables)0> >, query::partition_slice const&, (anonymous namespace)::data_query_result_builder&&, unsigned long, unsigned int, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >)::{lambda(mutation_fragment_v2*)#1}, seastar::future<mutation_fragment_v2*>::then_impl_nrvo<{lambda(mutation_fragment_v2*)#1}, query::consume_page<(anonymous namespace)::data_query_result_builder>(flat_mutation_reader_v2&, seastar::lw_shared_ptr<compact_mutation_state<(compact_for_sstables)0> >, query::partition_slice const&, (anonymous namespace)::data_query_result_builder&&, unsigned long, unsigned int, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >)::{lambda(mutation_fragment_v2*)#1}<query::result> >((anonymous namespace)::data_query_result_builder&&)::{lambda(seastar::internal::promise_base_with_type<query::result>&&, {lambda(mutation_fragment_v2*)#1}&, seastar::future_state<mutation_fragment_v2*>&&)#1}, mutation_fragment_v2*>::run_and_dispose at ././seastar/include/seastar/core/future.hh:781
| | |         |   | |   | |   | | | ++           - addr=0x51bf564:
| | |         |   | |   | |   | | | |              seastar::reactor::run_tasks at ./build/release/seastar/./seastar/src/core/reactor.cc:2509
| | |         |   | |   | |   | | | |              (inlined by) seastar::reactor::run_some_tasks at ./build/release/seastar/./seastar/src/core/reactor.cc:2946
| | |         |   | |   | |   | | | -> continued at addr=0x51bf564 above
| | |         |   | |   | |   | | ++[17#2/2 2%] 1 more branches total=1722 count=2 avg=861
| | |         |   | |   | |   | ++[16#2/2 4%] 1 more branches total=3508 count=4 avg=877
| | |         |   | |   | |   ++[15#2/2 10%] addr=0x1ebd520 total=8770 count=10 avg=877:
| | |         |   | |   | |   |              (anonymous namespace)::shard_reader_v2::read_ahead at ./readers/multishard.cc:924
| | |         |   | |   | |   |              (inlined by) multishard_combining_reader_v2::handle_empty_reader_buffer at ./readers/multishard.cc:1048
| | |         |   | |   | |   |              (inlined by) operator() at ./readers/multishard.cc:1081
| | |         |   | |   | |   ++           - addr=0x1ebdb77:
| | |         |   | |   | |   |              seastar::internal::do_until_state<multishard_combining_reader_v2::fill_buffer()::$_12, multishard_combining_reader_v2::fill_buffer()::$_11>::run_and_dispose at ././seastar/include/seastar/core/loop.hh:297
| | |         |   | |   | |     ++[16#1/1 100%] addr=0x51bf564 total=12278 count=14 avg=877:
| | |         |   | |   | |     |               seastar::reactor::run_tasks at ./build/release/seastar/./seastar/src/core/reactor.cc:2509
| | |         |   | |   | |     |               (inlined by) seastar::reactor::run_some_tasks at ./build/release/seastar/./seastar/src/core/reactor.cc:2946
| | |         |   | |   | |     -> continued at addr=0x51bf564 above
| | |         |   | |   | |+[13#2/3 33%] addr=0x1e8f74d total=187322 count=227 avg=825:
| | |         |   | |   | |              operator() at ./readers/multishard.cc:830
| | |         |   | |   | | ++[14#1/2 93%] addr=0x1e91ff1 total=173322 count=211 avg=821:
| | |         |   | |   | | |              std::__invoke_impl<seastar::future<(anonymous namespace)::shard_reader_v2::do_fill_buffer()::reader_and_buffer_fill_result>, (anonymous namespace)::shard_reader_v2::do_fill_buffer()::$_8::operator()() const::{lambda()#1}&> at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:61
| | |         |   | |   | | |              (inlined by) std::__invoke<(anonymous namespace)::shard_reader_v2::do_fill_buffer()::$_8::operator()() const::{lambda()#1}&> at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:96
| | |         |   | |   | | |              (inlined by) std::invoke<(anonymous namespace)::shard_reader_v2::do_fill_buffer()::$_8::operator()() const::{lambda()#1}&> at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/functional:110
| | |         |   | |   | | |              (inlined by) seastar::coroutine::lambda<(anonymous namespace)::shard_reader_v2::do_fill_buffer()::$_8::operator()() const::{lambda()#1}>::operator()<> at ././seastar/include/seastar/core/coroutine.hh:259
| | |         |   | |   | | |              (inlined by) seastar::futurize<seastar::future<(anonymous namespace)::shard_reader_v2::do_fill_buffer()::reader_and_buffer_fill_result> >::invoke<seastar::coroutine::lambda<(anonymous namespace)::shard_reader_v2::do_fill_buffer()::$_8::operator()() const::{lambda()#1}>&> at ././seastar/include/seastar/core/future.hh:2147
| | |         |   | |   | | |              (inlined by) seastar::smp_message_queue::async_work_item<seastar::coroutine::lambda<(anonymous namespace)::shard_reader_v2::do_fill_buffer()::$_8::operator()() const::{lambda()#1}> >::run_and_dispose at ././seastar/include/seastar/core/smp.hh:235
| | |         |   | |   | | ++           - addr=0x51bf564:
| | |         |   | |   | | |              seastar::reactor::run_tasks at ./build/release/seastar/./seastar/src/core/reactor.cc:2509
| | |         |   | |   | | |              (inlined by) seastar::reactor::run_some_tasks at ./build/release/seastar/./seastar/src/core/reactor.cc:2946
| | |         |   | |   | | -> continued at addr=0x51bf564 above
| | |         |   | |   | | ++[14#2/2 7%] addr=0x1e8b6d6 total=14000 count=16 avg=875:
| | |         |   | |   | | |             std::__invoke_impl<seastar::future<(anonymous namespace)::shard_reader_v2::do_fill_buffer()::reader_and_buffer_fill_result>, (anonymous namespace)::shard_reader_v2::do_fill_buffer()::$_8::operator()() const::{lambda()#1}&> at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:61
| | |         |   | |   | | |             (inlined by) std::__invoke<(anonymous namespace)::shard_reader_v2::do_fill_buffer()::$_8::operator()() const::{lambda()#1}&> at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:96
| | |         |   | |   | | |             (inlined by) std::invoke<(anonymous namespace)::shard_reader_v2::do_fill_buffer()::$_8::operator()() const::{lambda()#1}&> at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/functional:110
| | |         |   | |   | | |             (inlined by) seastar::coroutine::lambda<(anonymous namespace)::shard_reader_v2::do_fill_buffer()::$_8::operator()() const::{lambda()#1}>::operator()<> at ././seastar/include/seastar/core/coroutine.hh:259
| | |         |   | |   | | |             (inlined by) seastar::futurize<seastar::future<(anonymous namespace)::shard_reader_v2::do_fill_buffer()::reader_and_buffer_fill_result> >::invoke<seastar::coroutine::lambda<(anonymous namespace)::shard_reader_v2::do_fill_buffer()::$_8::operator()() const::{lambda()#1}>&> at ././seastar/include/seastar/core/future.hh:2147
| | |         |   | |   | | |             (inlined by) seastar::smp::submit_to<seastar::coroutine::lambda<(anonymous namespace)::shard_reader_v2::do_fill_buffer()::$_8::operator()() const::{lambda()#1}> > at ././seastar/include/seastar/core/smp.hh:353
| | |         |   | |   | | |             (inlined by) seastar::smp::submit_to<seastar::coroutine::lambda<(anonymous namespace)::shard_reader_v2::do_fill_buffer()::$_8::operator()() const::{lambda()#1}> > at ././seastar/include/seastar/core/smp.hh:380
| | |         |   | |   | | |             (inlined by) operator() at ./readers/multishard.cc:809
| | |         |   | |   | | ++          - addr=0x1e896de:
| | |         |   | |   | | |             std::__invoke_impl<seastar::future<(anonymous namespace)::remote_fill_buffer_result_v2>, (anonymous namespace)::shard_reader_v2::do_fill_buffer()::$_8> at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:61
| | |         |   | |   | | |             (inlined by) std::__invoke<(anonymous namespace)::shard_reader_v2::do_fill_buffer()::$_8> at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:96
| | |         |   | |   | | |             (inlined by) std::invoke<(anonymous namespace)::shard_reader_v2::do_fill_buffer()::$_8> at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/functional:110
| | |         |   | |   | | |             (inlined by) (anonymous namespace)::shard_reader_v2::do_fill_buffer at ./readers/multishard.cc:807
| | |         |   | |   | | -> continued at addr=0x1e896de above
| | |         |   | |   | ++[13#3/3 8%] addr=0x1ec6fbf total=43533 count=50 avg=871:
| | |         |   | |   | |             operator() at ./readers/multishard.cc:830
| | |         |   | |   | ++          - addr=0x1e9114a:
| | |         |   | |   | |             std::__n4861::coroutine_handle<seastar::internal::coroutine_traits_base<(anonymous namespace)::shard_reader_v2::do_fill_buffer()::reader_and_buffer_fill_result>::promise_type>::resume at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/coroutine:244
| | |         |   | |   | |             (inlined by) seastar::internal::coroutine_traits_base<(anonymous namespace)::shard_reader_v2::do_fill_buffer()::reader_and_buffer_fill_result>::promise_type::run_and_dispose at ././seastar/include/seastar/core/coroutine.hh:78
| | |         |   | |   | ++          - addr=0x51bf564:
| | |         |   | |   | |             seastar::reactor::run_tasks at ./build/release/seastar/./seastar/src/core/reactor.cc:2509
| | |         |   | |   | |             (inlined by) seastar::reactor::run_some_tasks at ./build/release/seastar/./seastar/src/core/reactor.cc:2946
| | |         |   | |   | -> continued at addr=0x51bf564 above
| | |         |   | |   |+[12#2/4 8%] addr=0x1e7d4e8 total=52329 count=62 avg=844:
| | |         |   | |   |             flat_mutation_reader_v2::impl::operator() at ./readers/flat_mutation_reader_v2.hh:195
| | |         |   | |   |             (inlined by) flat_mutation_reader_v2::operator() at ./readers/flat_mutation_reader_v2.hh:415
| | |         |   | |   |             (inlined by) evictable_reader_v2::fill_buffer at ./readers/multishard.cc:580
| | |         |   | |   |+          - addr=0x1e9515f:
| | |         |   | |   |             operator() at ./readers/multishard.cc:844
| | |         |   | |   -> continued at addr=0x1e9515f above
| | |         |   | |   ++[12#3/4 1%] 2 more branches total=5225 count=6 avg=871
| | |         |   | ++[10#2/2 5%] 1 more branches total=31696 count=36 avg=880
| | |         |   |+[9#2/4 31%] addr=0x1c5f92c total=295956 count=339 avg=873:
| | |         |   |             operator() at ./row_cache.cc:649
| | |         |   |             (inlined by) std::__invoke_impl<seastar::future<seastar::optimized_optional<flat_mutation_reader_v2> >, scanning_and_populating_reader::read_from_secondary()::{lambda(std::tuple<seastar::optimized_optional<flat_mutation_reader_v2>, seastar::optimized_optional<mutation_fragment_v2> >&&)#1}&, std::tuple<seastar::optimized_optional<flat_mutation_reader_v2>, seastar::optimized_optional<mutation_fragment_v2> > > at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:61
| | |         |   |             (inlined by) std::__invoke<scanning_and_populating_reader::read_from_secondary()::{lambda(std::tuple<seastar::optimized_optional<flat_mutation_reader_v2>, seastar::optimized_optional<mutation_fragment_v2> >&&)#1}&, std::tuple<seastar::optimized_optional<flat_mutation_reader_v2>, seastar::optimized_optional<mutation_fragment_v2> > > at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:96
| | |         |   |             (inlined by) std::invoke<scanning_and_populating_reader::read_from_secondary()::{lambda(std::tuple<seastar::optimized_optional<flat_mutation_reader_v2>, seastar::optimized_optional<mutation_fragment_v2> >&&)#1}&, std::tuple<seastar::optimized_optional<flat_mutation_reader_v2>, seastar::optimized_optional<mutation_fragment_v2> > > at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/functional:110
| | |         |   |             (inlined by) seastar::internal::future_invoke<scanning_and_populating_reader::read_from_secondary()::{lambda(std::tuple<seastar::optimized_optional<flat_mutation_reader_v2>, seastar::optimized_optional<mutation_fragment_v2> >&&)#1}&, std::tuple<seastar::optimized_optional<flat_mutation_reader_v2>, seastar::optimized_optional<mutation_fragment_v2> > > at ././seastar/include/seastar/core/future.hh:1223
| | |         |   |             (inlined by) operator() at ././seastar/include/seastar/core/future.hh:1594
| | |         |   |             (inlined by) seastar::futurize<seastar::future<seastar::optimized_optional<flat_mutation_reader_v2> > >::satisfy_with_result_of<seastar::future<std::tuple<seastar::optimized_optional<flat_mutation_reader_v2>, seastar::optimized_optional<mutation_fragment_v2> > >::then_impl_nrvo<scanning_and_populating_reader::read_from_secondary()::{lambda(std::tuple<seastar::optimized_optional<flat_mutation_reader_v2>, seastar::optimized_optional<mutation_fragment_v2> >&&)#1}, seastar::future<seastar::optimized_optional<flat_mutation_reader_v2> > >(scanning_and_populating_reader::read_from_secondary()::{lambda(std::tuple<seastar::optimized_optional<flat_mutation_reader_v2>, seastar::optimized_optional<mutation_fragment_v2> >&&)#1}&&)::{lambda(seastar::internal::promise_base_with_type<seastar::optimized_optional<flat_mutation_reader_v2> >&&, {lambda(std::tuple<seastar::optimized_optional<flat_mutation_reader_v2>, seastar::optimized_optional<mutation_fragment_v2> >&&)#1}&, seastar::future_state<std::tuple<seastar::optimized_optional<flat_mutation_reader_v2>, seastar::optimized_optional<mutation_fragment_v2> > >&&)#1}::operator()(seastar::internal::promise_base_with_type<seastar::optimized_optional<flat_mutation_reader_v2> >, seastar::internal::promise_base_with_type<seastar::optimized_optional<flat_mutation_reader_v2> >&&, seastar::future_state<std::tuple<seastar::optimized_optional<flat_mutation_reader_v2>, seastar::optimized_optional<mutation_fragment_v2> > >) const::{lambda()#1}> at ././seastar/include/seastar/core/future.hh:2132
| | |         |   |             (inlined by) operator() at ././seastar/include/seastar/core/future.hh:1587
| | |         |   |             (inlined by) seastar::continuation<seastar::internal::promise_base_with_type<seastar::optimized_optional<flat_mutation_reader_v2> >, scanning_and_populating_reader::read_from_secondary()::{lambda(std::tuple<seastar::optimized_optional<flat_mutation_reader_v2>, seastar::optimized_optional<mutation_fragment_v2> >&&)#1}, seastar::future<std::tuple<seastar::optimized_optional<flat_mutation_reader_v2>, seastar::optimized_optional<mutation_fragment_v2> > >::then_impl_nrvo<{lambda(std::tuple<seastar::optimized_optional<flat_mutation_reader_v2>, seastar::optimized_optional<mutation_fragment_v2> >&&)#1}, scanning_and_populating_reader::read_from_secondary()::{lambda(std::tuple<seastar::optimized_optional<flat_mutation_reader_v2>, seastar::optimized_optional<mutation_fragment_v2> >&&)#1}<seastar::optimized_optional<flat_mutation_reader_v2> > >({lambda(std::tuple<seastar::optimized_optional<flat_mutation_reader_v2>, seastar::optimized_optional<mutation_fragment_v2> >&&)#1}&&)::{lambda(seastar::internal::promise_base_with_type<seastar::optimized_optional<flat_mutation_reader_v2> >&&, {lambda(std::tuple<seastar::optimized_optional<flat_mutation_reader_v2>, seastar::optimized_optional<mutation_fragment_v2> >&&)#1}&, seastar::future_state<std::tuple<seastar::optimized_optional<flat_mutation_reader_v2>, seastar::optimized_optional<mutation_fragment_v2> > >&&)#1}, std::tuple<seastar::optimized_optional<flat_mutation_reader_v2>, seastar::optimized_optional<mutation_fragment_v2> > >::run_and_dispose at ././seastar/include/seastar/core/future.hh:781
| | |         |   | ++[10#1/1 100%] addr=0x51bf564 total=297710 count=341 avg=873:
| | |         |   | |               seastar::reactor::run_tasks at ./build/release/seastar/./seastar/src/core/reactor.cc:2509
| | |         |   | |               (inlined by) seastar::reactor::run_some_tasks at ./build/release/seastar/./seastar/src/core/reactor.cc:2946
| | |         |   | -> continued at addr=0x51bf564 above
| | |         |   |+[9#3/4 10%] addr=0x1c583dd total=96889 count=116 avg=835:
| | |         |   |             operator() at ./row_cache.cc:649
| | |         |   |             (inlined by) seastar::futurize<seastar::future<seastar::optimized_optional<flat_mutation_reader_v2> > >::invoke<scanning_and_populating_reader::read_from_secondary()::{lambda(std::tuple<seastar::optimized_optional<flat_mutation_reader_v2>, seastar::optimized_optional<mutation_fragment_v2> >&&)#1}, std::tuple<seastar::optimized_optional<flat_mutation_reader_v2>, seastar::optimized_optional<mutation_fragment_v2> > > at ././seastar/include/seastar/core/future.hh:2147
| | |         |   |             (inlined by) seastar::future<std::tuple<seastar::optimized_optional<flat_mutation_reader_v2>, seastar::optimized_optional<mutation_fragment_v2> > >::then_impl<scanning_and_populating_reader::read_from_secondary()::{lambda(std::tuple<seastar::optimized_optional<flat_mutation_reader_v2>, seastar::optimized_optional<mutation_fragment_v2> >&&)#1}, seastar::future<seastar::optimized_optional<flat_mutation_reader_v2> > > at ././seastar/include/seastar/core/future.hh:1613
| | |         |   |             (inlined by) seastar::internal::call_then_impl<seastar::future<std::tuple<seastar::optimized_optional<flat_mutation_reader_v2>, seastar::optimized_optional<mutation_fragment_v2> > > >::run<scanning_and_populating_reader::read_from_secondary()::{lambda(std::tuple<seastar::optimized_optional<flat_mutation_reader_v2>, seastar::optimized_optional<mutation_fragment_v2> >&&)#1}> at ././seastar/include/seastar/core/future.hh:1246
| | |         |   |             (inlined by) seastar::future<std::tuple<seastar::optimized_optional<flat_mutation_reader_v2>, seastar::optimized_optional<mutation_fragment_v2> > >::then<scanning_and_populating_reader::read_from_secondary()::{lambda(std::tuple<seastar::optimized_optional<flat_mutation_reader_v2>, seastar::optimized_optional<mutation_fragment_v2> >&&)#1}, seastar::future<seastar::optimized_optional<flat_mutation_reader_v2> > > at ././seastar/include/seastar/core/future.hh:1532
| | |         |   |             (inlined by) scanning_and_populating_reader::read_from_secondary at ./row_cache.cc:640
| | |         |   |             (inlined by) operator() at ./row_cache.cc:635
| | |         |   |             (inlined by) seastar::futurize<seastar::future<seastar::optimized_optional<flat_mutation_reader_v2> > >::invoke<scanning_and_populating_reader::read_from_primary()::{lambda()#1}> at ././seastar/include/seastar/core/future.hh:2147
| | |         |   |             (inlined by) seastar::futurize<seastar::future<seastar::optimized_optional<flat_mutation_reader_v2> > >::invoke<scanning_and_populating_reader::read_from_primary()::{lambda()#1}> at ././seastar/include/seastar/core/future.hh:1991
| | |         |   |             (inlined by) seastar::future<void>::then_impl<scanning_and_populating_reader::read_from_primary()::{lambda()#1}, seastar::future<seastar::optimized_optional<flat_mutation_reader_v2> > > at ././seastar/include/seastar/core/future.hh:1613
| | |         |   |             (inlined by) seastar::internal::call_then_impl<seastar::future<void> >::run<scanning_and_populating_reader::read_from_primary()::{lambda()#1}> at ././seastar/include/seastar/core/future.hh:1246
| | |         |   |             (inlined by) seastar::future<void>::then<scanning_and_populating_reader::read_from_primary()::{lambda()#1}, seastar::future<seastar::optimized_optional<flat_mutation_reader_v2> > > at ././seastar/include/seastar/core/future.hh:1532
| | |         |   |             (inlined by) scanning_and_populating_reader::read_from_primary at ./row_cache.cc:634
| | |         |   |+          - addr=0x1c56dc5:
| | |         |   |             operator() at ./row_cache.cc:657
| | |         |   -> continued at addr=0x1c56dc5 above
| | |         |   ++[9#4/4 0%] 1 more branches total=878 count=1 avg=878
| | |         ++[7#2/8 6%] 7 more branches total=41939 count=49 avg=856
| | |+[2#2/3 42%] addr=0x1d8065c total=26144 count=31 avg=843:
| | |             compound_type<(allow_prefixes)1>::begin at ././compound.hh:194
| | |             (inlined by) bound_view::tri_compare::operator() at ././clustering_bounds_comparator.hh:60
| | |             (inlined by) position_in_partition::tri_compare::compare<position_in_partition_view, position_in_partition_view> at ././position_in_partition.hh:526
| | |             (inlined by) position_in_partition::tri_compare::operator() at ././position_in_partition.hh:534
| | |             (inlined by) rows_entry::tri_compare::operator() at ././mutation_partition.hh:1002
| | |             (inlined by) intrusive_b::searcher<position_in_partition_view, rows_entry, &rows_entry::_link, rows_entry::tri_compare, (intrusive_b::key_search)0>::ge at ././utils/intrusive_btree.hh:1050
| | |             (inlined by) intrusive_b::node<rows_entry, &rows_entry::_link, rows_entry::tri_compare, 12ul, 20ul, (intrusive_b::key_search)0, (intrusive_b::with_debug)0>::index_for<position_in_partition_view> at ././utils/intrusive_btree.hh:1388
| | |             (inlined by) intrusive_b::tree<rows_entry, &rows_entry::_link, rows_entry::tri_compare, 12ul, 20ul, (intrusive_b::key_search)0, (intrusive_b::with_debug)0>::key_lower_bound<position_in_partition_view> at ././utils/intrusive_btree.hh:276
| | |             (inlined by) intrusive_b::tree<rows_entry, &rows_entry::_link, rows_entry::tri_compare, 12ul, 20ul, (intrusive_b::key_search)0, (intrusive_b::with_debug)0>::find<position_in_partition_view> at ././utils/intrusive_btree.hh:454
| | | ++[3#1/1 100%] addr=0x1d30708 total=110654 count=131 avg=845:
| | | |              intrusive_b::tree<rows_entry, &rows_entry::_link, rows_entry::tri_compare, 12ul, 20ul, (intrusive_b::key_search)0, (intrusive_b::with_debug)0>::find<position_in_partition_view> at ././utils/intrusive_btree.hh:464
| | | |              (inlined by) mutation_partition::clustered_row at ./mutation_partition.cc:691
| | | -> continued at addr=0x1d30708 above
| | ++[2#3/3 3%] 1 more branches total=1755 count=2 avg=878
| |+[1#2/266 7%] addr=0x13cf293 total=48965 count=58 avg=844:
| |              iterator at ././compound.hh:175
| | ++[2#1/3 49%] addr=0x1d8065c total=23970 count=28 avg=856:
| | |             compound_type<(allow_prefixes)1>::begin at ././compound.hh:194
| | |             (inlined by) bound_view::tri_compare::operator() at ././clustering_bounds_comparator.hh:60
| | |             (inlined by) position_in_partition::tri_compare::compare<position_in_partition_view, position_in_partition_view> at ././position_in_partition.hh:526
| | |             (inlined by) position_in_partition::tri_compare::operator() at ././position_in_partition.hh:534
| | |             (inlined by) rows_entry::tri_compare::operator() at ././mutation_partition.hh:1002
| | |             (inlined by) intrusive_b::searcher<position_in_partition_view, rows_entry, &rows_entry::_link, rows_entry::tri_compare, (intrusive_b::key_search)0>::ge at ././utils/intrusive_btree.hh:1050
| | |             (inlined by) intrusive_b::node<rows_entry, &rows_entry::_link, rows_entry::tri_compare, 12ul, 20ul, (intrusive_b::key_search)0, (intrusive_b::with_debug)0>::index_for<position_in_partition_view> at ././utils/intrusive_btree.hh:1388
| | |             (inlined by) intrusive_b::tree<rows_entry, &rows_entry::_link, rows_entry::tri_compare, 12ul, 20ul, (intrusive_b::key_search)0, (intrusive_b::with_debug)0>::key_lower_bound<position_in_partition_view> at ././utils/intrusive_btree.hh:276
| | |             (inlined by) intrusive_b::tree<rows_entry, &rows_entry::_link, rows_entry::tri_compare, 12ul, 20ul, (intrusive_b::key_search)0, (intrusive_b::with_debug)0>::find<position_in_partition_view> at ././utils/intrusive_btree.hh:454
| | -> continued at addr=0x1d8065c above
| | |+[2#2/3 48%] addr=0x1d806b7 total=23371 count=28 avg=835:
| | |             compound_type<(allow_prefixes)1>::begin at ././compound.hh:194
| | |             (inlined by) bound_view::tri_compare::operator() at ././clustering_bounds_comparator.hh:61
| | |             (inlined by) position_in_partition::tri_compare::compare<position_in_partition_view, position_in_partition_view> at ././position_in_partition.hh:526
| | |             (inlined by) position_in_partition::tri_compare::operator() at ././position_in_partition.hh:534
| | |             (inlined by) rows_entry::tri_compare::operator() at ././mutation_partition.hh:1002
| | |             (inlined by) intrusive_b::searcher<position_in_partition_view, rows_entry, &rows_entry::_link, rows_entry::tri_compare, (intrusive_b::key_search)0>::ge at ././utils/intrusive_btree.hh:1050
| | |             (inlined by) intrusive_b::node<rows_entry, &rows_entry::_link, rows_entry::tri_compare, 12ul, 20ul, (intrusive_b::key_search)0, (intrusive_b::with_debug)0>::index_for<position_in_partition_view> at ././utils/intrusive_btree.hh:1388
| | |             (inlined by) intrusive_b::tree<rows_entry, &rows_entry::_link, rows_entry::tri_compare, 12ul, 20ul, (intrusive_b::key_search)0, (intrusive_b::with_debug)0>::key_lower_bound<position_in_partition_view> at ././utils/intrusive_btree.hh:276
| | |             (inlined by) intrusive_b::tree<rows_entry, &rows_entry::_link, rows_entry::tri_compare, 12ul, 20ul, (intrusive_b::key_search)0, (intrusive_b::with_debug)0>::find<position_in_partition_view> at ././utils/intrusive_btree.hh:454
| | -> continued at addr=0x1d806b7 above
| | ++[2#3/3 3%] 1 more branches total=1624 count=2 avg=812
| |+[1#3/266 5%] addr=0x1d7dc99 total=37165 count=44 avg=845:
| |              compact_radix_tree::tree<cell_and_hash, unsigned int>::node_head_ptr::is at ././utils/compact-radix-tree.hh:550
| |              (inlined by) compact_radix_tree::tree<cell_and_hash, unsigned int>::visit<compact_radix_tree::tree<cell_and_hash, unsigned int>::visit<compact_radix_tree::tree<cell_and_hash, unsigned int>::clear()::clearing_visitor>(compact_radix_tree::tree<cell_and_hash, unsigned int>::clear()::clearing_visitor&&)::adaptor> at ././utils/compact-radix-tree.hh:1627
| |              (inlined by) compact_radix_tree::tree<cell_and_hash, unsigned int>::visit<compact_radix_tree::tree<cell_and_hash, unsigned int>::clear()::clearing_visitor> at ././utils/compact-radix-tree.hh:1647
| |              (inlined by) compact_radix_tree::tree<cell_and_hash, unsigned int>::clear at ././utils/compact-radix-tree.hh:1844
| |              (inlined by) ~tree at ././utils/compact-radix-tree.hh:1777
| |              (inlined by) ~row at ./mutation_partition.cc:1577
| |              (inlined by) ~deletable_row at ././mutation_partition.hh:813
| |              (inlined by) ~rows_entry at ././mutation_partition.hh:901
| |              (inlined by) allocation_strategy::destroy<rows_entry> at ././utils/allocation_strategy.hh:167
| |              (inlined by) operator() at ././utils/allocation_strategy.hh:244
| |              (inlined by) intrusive_b::node<rows_entry, &rows_entry::_link, rows_entry::tri_compare, 12ul, 20ul, (intrusive_b::key_search)0, (intrusive_b::with_debug)0>::dispose_key<current_deleter<rows_entry>()::{lambda(rows_entry*)#1}&> at ././utils/intrusive_btree.hh:1300
| |              (inlined by) operator() at ././utils/intrusive_btree.hh:613
| |              (inlined by) intrusive_b::node<rows_entry, &rows_entry::_link, rows_entry::tri_compare, 12ul, 20ul, (intrusive_b::key_search)0, (intrusive_b::with_debug)0>::clear<intrusive_b::tree<rows_entry, &rows_entry::_link, rows_entry::tri_compare, 12ul, 20ul, (intrusive_b::key_search)0, (intrusive_b::with_debug)0>::clear_and_dispose<current_deleter<rows_entry>()::{lambda(rows_entry*)#1}>(current_deleter<rows_entry>()::{lambda(rows_entry*)#1}&&)::{lambda(intrusive_b::member_hook*)#1}&> at ././utils/intrusive_btree.hh:2063
| |+           - addr=0x1d7dbec:
| |              intrusive_b::node<rows_entry, &rows_entry::_link, rows_entry::tri_compare, 12ul, 20ul, (intrusive_b::key_search)0, (intrusive_b::with_debug)0>::clear<intrusive_b::tree<rows_entry, &rows_entry::_link, rows_entry::tri_compare, 12ul, 20ul, (intrusive_b::key_search)0, (intrusive_b::with_debug)0>::clear_and_dispose<current_deleter<rows_entry>()::{lambda(rows_entry*)#1}>(current_deleter<rows_entry>()::{lambda(rows_entry*)#1}&&)::{lambda(intrusive_b::member_hook*)#1}&> at ././utils/intrusive_btree.hh:2057
| | ++[2#1/2 66%] addr=0x1d7dbec total=306650 count=362 avg=847:
| | |             intrusive_b::node<rows_entry, &rows_entry::_link, rows_entry::tri_compare, 12ul, 20ul, (intrusive_b::key_search)0, (intrusive_b::with_debug)0>::clear<intrusive_b::tree<rows_entry, &rows_entry::_link, rows_entry::tri_compare, 12ul, 20ul, (intrusive_b::key_search)0, (intrusive_b::with_debug)0>::clear_and_dispose<current_deleter<rows_entry>()::{lambda(rows_entry*)#1}>(current_deleter<rows_entry>()::{lambda(rows_entry*)#1}&&)::{lambda(intrusive_b::member_hook*)#1}&> at ././utils/intrusive_btree.hh:2057
| | -> continued at addr=0x1d7dbec above
| | ++[2#2/2 34%] addr=0x1d1cd5c total=158393 count=187 avg=847:
| | |             intrusive_b::node<rows_entry, &rows_entry::_link, rows_entry::tri_compare, 12ul, 20ul, (intrusive_b::key_search)0, (intrusive_b::with_debug)0>::clear<intrusive_b::tree<rows_entry, &rows_entry::_link, rows_entry::tri_compare, 12ul, 20ul, (intrusive_b::key_search)0, (intrusive_b::with_debug)0>::clear_and_dispose<current_deleter<rows_entry>()::{lambda(rows_entry*)#1}>(current_deleter<rows_entry>()::{lambda(rows_entry*)#1}&&)::{lambda(intrusive_b::member_hook*)#1}> at ././utils/intrusive_btree.hh:2057
| | |             (inlined by) intrusive_b::tree<rows_entry, &rows_entry::_link, rows_entry::tri_compare, 12ul, 20ul, (intrusive_b::key_search)0, (intrusive_b::with_debug)0>::clear_and_dispose<current_deleter<rows_entry>()::{lambda(rows_entry*)#1}> at ././utils/intrusive_btree.hh:613
| |   ++[3#1/2 57%] addr=0x1d1f3ac total=90982 count=107 avg=850:
| |   |             ~mutation_partition at ./mutation_partition.cc:224
| |   | ++[4#1/2 59%] addr=0x1bd2c07 total=53435 count=63 avg=848:
| |   | |             partition_entry::squashed at ./partition_version.cc:506
| |   | |             (inlined by) partition_entry::upgrade at ./partition_version.cc:517
| |   | | ++[5#1/3 97%] addr=0x1c5ff15 total=51672 count=61 avg=847:
| |   | | |             operator() at ./row_cache.cc:1319
| |   | | |             (inlined by) with_allocator<row_cache::upgrade_entry(cache_entry&)::$_28> at ././utils/allocation_strategy.hh:313
| |   | | |             (inlined by) row_cache::upgrade_entry at ./row_cache.cc:1318
| |   | | |             (inlined by) scanning_and_populating_reader::read_from_entry at ./row_cache.cc:573
| |   | | |             (inlined by) operator() at ./row_cache.cc:601
| |   | | -> continued at addr=0x1c5ff15 above
| |   | | ++[5#2/3 3%] 2 more branches total=1763 count=2 avg=882
| |   | ++[4#2/2 41%] addr=0x1d28840 total=37547 count=44 avg=853:
| |   | |             mutation_partition::upgrade at ./mutation_partition.cc:1941
| |   | ++          - addr=0x1bd229c:
| |   | |             partition_entry::squashed at ./partition_version.cc:503
| |   | |             (inlined by) partition_entry::upgrade at ./partition_version.cc:517
| |   | -> continued at addr=0x1bd229c above
| |   ++[3#2/2 43%] addr=0x1d27a79 total=67411 count=80 avg=843:
| |   |             ~mutation_partition at ./mutation_partition.cc:224
| |   |             (inlined by) mutation_partition::operator= at ./mutation_partition.cc:230
| |   |             (inlined by) mutation_partition::upgrade at ./mutation_partition.cc:1940
| |   ++          - addr=0x1bd229c:
| |   |             partition_entry::squashed at ./partition_version.cc:503
| |   |             (inlined by) partition_entry::upgrade at ./partition_version.cc:517
| |   -> continued at addr=0x1bd229c above
| ++[1#4/266 80%] 263 more branches total=585773 count=690 avg=849
|+[0#2/3 16%] addr=0x1b0c248 total=161014 count=190 avg=847:
|             managed_bytes at ././utils/managed_bytes.hh:223
|             (inlined by) atomic_cell_or_collection::copy at ./atomic_cell.cc:116
|+          - addr=0x1d9ae38:
|             operator() at ./mutation_partition.cc:1570
|             (inlined by) compact_radix_tree::tree<cell_and_hash, unsigned int>::copy_slots<compact_radix_tree::tree<cell_and_hash, unsigned int>::indirect_layout<cell_and_hash, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)1, 4u, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)2, 0u, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)0, 0u>::clone<compact_radix_tree::tree<cell_and_hash, unsigned int>::leaf_node, row::row(schema const&, column_kind, row const&)::$_14&>(compact_radix_tree::tree<cell_and_hash, unsigned int>::node_head const&, row::row(schema const&, column_kind, row const&)::$_14&, unsigned int) const::{lambda(unsigned int)#1}, row::row(schema const&, column_kind, row const&)::$_14&, compact_radix_tree::tree<cell_and_hash, unsigned int>::node_base<cell_and_hash, compact_radix_tree::tree<cell_and_hash, unsigned int>::indirect_layout<cell_and_hash, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)1, 4u, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)2, 0u, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)0, 0u>, compact_radix_tree::tree<cell_and_hash, unsigned int>::indirect_layout<cell_and_hash, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)2, 8u, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)3, 0u, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)1, 4u>, compact_radix_tree::tree<cell_and_hash, unsigned int>::indirect_layout<cell_and_hash, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)3, 16u, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)4, 0u, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)2, 8u>, compact_radix_tree::tree<cell_and_hash, unsigned int>::indirect_layout<cell_and_hash, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)4, 32u, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)6, 0u, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)3, 16u>, compact_radix_tree::tree<cell_and_hash, unsigned int>::direct_layout<cell_and_hash, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)6, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)0, 0u, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)4, 32u> > > at ././utils/compact-radix-tree.hh:1397
|             (inlined by) compact_radix_tree::tree<cell_and_hash, unsigned int>::indirect_layout<cell_and_hash, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)1, 4u, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)2, 0u, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)0, 0u>::clone<compact_radix_tree::tree<cell_and_hash, unsigned int>::leaf_node, row::row(schema const&, column_kind, row const&)::$_14&> at ././utils/compact-radix-tree.hh:1284
|             (inlined by) compact_radix_tree::tree<cell_and_hash, unsigned int>::node_base<cell_and_hash, compact_radix_tree::tree<cell_and_hash, unsigned int>::indirect_layout<cell_and_hash, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)1, 4u, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)2, 0u, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)0, 0u>, compact_radix_tree::tree<cell_and_hash, unsigned int>::indirect_layout<cell_and_hash, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)2, 8u, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)3, 0u, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)1, 4u>, compact_radix_tree::tree<cell_and_hash, unsigned int>::indirect_layout<cell_and_hash, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)3, 16u, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)4, 0u, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)2, 8u>, compact_radix_tree::tree<cell_and_hash, unsigned int>::indirect_layout<cell_and_hash, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)4, 32u, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)6, 0u, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)3, 16u>, compact_radix_tree::tree<cell_and_hash, unsigned int>::direct_layout<cell_and_hash, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)6, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)0, 0u, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)4, 32u> >::clone<compact_radix_tree::tree<cell_and_hash, unsigned int>::leaf_node, row::row(schema const&, column_kind, row const&)::$_14&, compact_radix_tree::tree<cell_and_hash, unsigned int>::indirect_layout<cell_and_hash, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)1, 4u, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)2, 0u, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)0, 0u>, compact_radix_tree::tree<cell_and_hash, unsigned int>::indirect_layout<cell_and_hash, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)2, 8u, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)3, 0u, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)1, 4u>, compact_radix_tree::tree<cell_and_hash, unsigned int>::indirect_layout<cell_and_hash, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)3, 16u, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)4, 0u, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)2, 8u>, compact_radix_tree::tree<cell_and_hash, unsigned int>::indirect_layout<cell_and_hash, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)4, 32u, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)6, 0u, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)3, 16u>, compact_radix_tree::tree<cell_and_hash, unsigned int>::direct_layout<cell_and_hash, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)6, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)0, 0u, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)4, 32u> > at ././utils/compact-radix-tree.hh:820
|             (inlined by) compact_radix_tree::tree<cell_and_hash, unsigned int>::node_base<cell_and_hash, compact_radix_tree::tree<cell_and_hash, unsigned int>::indirect_layout<cell_and_hash, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)1, 4u, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)2, 0u, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)0, 0u>, compact_radix_tree::tree<cell_and_hash, unsigned int>::indirect_layout<cell_and_hash, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)2, 8u, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)3, 0u, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)1, 4u>, compact_radix_tree::tree<cell_and_hash, unsigned int>::indirect_layout<cell_and_hash, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)3, 16u, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)4, 0u, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)2, 8u>, compact_radix_tree::tree<cell_and_hash, unsigned int>::indirect_layout<cell_and_hash, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)4, 32u, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)6, 0u, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)3, 16u>, compact_radix_tree::tree<cell_and_hash, unsigned int>::direct_layout<cell_and_hash, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)6, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)0, 0u, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)4, 32u> >::clone<compact_radix_tree::tree<cell_and_hash, unsigned int>::leaf_node, row::row(schema const&, column_kind, row const&)::$_14&> at ././utils/compact-radix-tree.hh:828
|             (inlined by) compact_radix_tree::tree<cell_and_hash, unsigned int>::node_head::clone<row::row(schema const&, column_kind, row const&)::$_14&> at ././utils/compact-radix-tree.hh:486
| ++[1#1/1 100%] addr=0x1d66c08 total=168453 count=199 avg=846:
| |              compact_radix_tree::tree<cell_and_hash, unsigned int>::clone_from<row::row(schema const&, column_kind, row const&)::$_14&> at ././utils/compact-radix-tree.hh:1853
| |              (inlined by) row at ./mutation_partition.cc:1573
| |              (inlined by) deletable_row at ././mutation_partition.hh:822
|   ++[2#1/1 100%] addr=0x1d7baaa total=188123 count=222 avg=847:
|   |              rows_entry at ././mutation_partition.hh:946
|   |              (inlined by) allocation_strategy::construct<rows_entry, schema const&, rows_entry const&> at ././utils/allocation_strategy.hh:155
|   |              (inlined by) operator() at ./mutation_partition.cc:152
|   |              (inlined by) intrusive_b::node<rows_entry, &rows_entry::_link, rows_entry::tri_compare, 12ul, 20ul, (intrusive_b::key_search)0, (intrusive_b::with_debug)0>::clone<mutation_partition::mutation_partition(schema const&, mutation_partition const&)::$_0&, current_deleter<rows_entry>()::{lambda(rows_entry*)#1}&> at ././utils/intrusive_btree.hh:2084
|     ++[3#1/1 100%] addr=0x1d7bc26 total=194070 count=229 avg=847:
|     |              intrusive_b::node<rows_entry, &rows_entry::_link, rows_entry::tri_compare, 12ul, 20ul, (intrusive_b::key_search)0, (intrusive_b::with_debug)0>::clone<mutation_partition::mutation_partition(schema const&, mutation_partition const&)::$_0&, current_deleter<rows_entry>()::{lambda(rows_entry*)#1}&> at ././utils/intrusive_btree.hh:2095
|       ++[4#1/2 74%] addr=0x1d7bc26 total=641709 count=757 avg=848:
|       |             intrusive_b::node<rows_entry, &rows_entry::_link, rows_entry::tri_compare, 12ul, 20ul, (intrusive_b::key_search)0, (intrusive_b::with_debug)0>::clone<mutation_partition::mutation_partition(schema const&, mutation_partition const&)::$_0&, current_deleter<rows_entry>()::{lambda(rows_entry*)#1}&> at ././utils/intrusive_btree.hh:2095
|       -> continued at addr=0x1d7bc26 above
|       ++[4#2/2 26%] addr=0x1d1bb82 total=221226 count=261 avg=848:
|       |             intrusive_b::tree<rows_entry, &rows_entry::_link, rows_entry::tri_compare, 12ul, 20ul, (intrusive_b::key_search)0, (intrusive_b::with_debug)0>::clone_from<mutation_partition::mutation_partition(schema const&, mutation_partition const&)::$_0&, current_deleter<rows_entry>()::{lambda(rows_entry*)#1}> at ././utils/intrusive_btree.hh:638
|       |             (inlined by) mutation_partition at ./mutation_partition.cc:154
|         ++[5#1/1 100%] addr=0x1bd226e total=234873 count=277 avg=848:
|         |              partition_entry::squashed at ./partition_version.cc:501
|         |              (inlined by) partition_entry::upgrade at ./partition_version.cc:517
|           ++[6#1/4 92%] addr=0x1c5ff15 total=215541 count=255 avg=845:
|           |             operator() at ./row_cache.cc:1319
|           |             (inlined by) with_allocator<row_cache::upgrade_entry(cache_entry&)::$_28> at ././utils/allocation_strategy.hh:313
|           |             (inlined by) row_cache::upgrade_entry at ./row_cache.cc:1318
|           |             (inlined by) scanning_and_populating_reader::read_from_entry at ./row_cache.cc:573
|           |             (inlined by) operator() at ./row_cache.cc:601
|           -> continued at addr=0x1c5ff15 above
|           ++[6#2/4 8%] 3 more branches total=19332 count=22 avg=879
++[0#3/3 12%] addr=0x1aff581 total=119924 count=141 avg=851:
|             write_fragmented<managed_bytes_basic_view<(mutable_view)1>, basic_single_fragmented_view<(mutable_view)0> > at ././utils/fragment_range.hh:303
|             (inlined by) set_value<fragment_range<managed_bytes_basic_view<(mutable_view)0> > > at ././atomic_cell.hh:54
|             (inlined by) atomic_cell_type::make_live<fragment_range<managed_bytes_basic_view<(mutable_view)0> > > at ././atomic_cell.hh:157
++          - addr=0x1aff0fe:
|             atomic_cell::make_live at ./atomic_cell.cc:23
  ++[1#1/1 100%] addr=0x1e40343 total=128455 count=151 avg=851:
  |              converting_mutation_partition_applier::upgrade_cell at ./converting_mutation_partition_applier.cc:28
    ++[2#1/1 100%] addr=0x1e43806 total=129267 count=152 avg=850:
    |              converting_mutation_partition_applier::accept_cell at ./converting_mutation_partition_applier.cc:39
    |              (inlined by) converting_mutation_partition_applier::accept_row_cell at ./converting_mutation_partition_applier.cc:155
      ++[3#1/1 100%] addr=0x1da791f total=130973 count=154 avg=850:
      |              compact_radix_tree::tree<cell_and_hash, unsigned int>::node_base<cell_and_hash, compact_radix_tree::tree<cell_and_hash, unsigned int>::indirect_layout<cell_and_hash, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)1, 4u, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)2, 0u, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)0, 0u>, compact_radix_tree::tree<cell_and_hash, unsigned int>::indirect_layout<cell_and_hash, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)2, 8u, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)3, 0u, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)1, 4u>, compact_radix_tree::tree<cell_and_hash, unsigned int>::indirect_layout<cell_and_hash, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)3, 16u, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)4, 0u, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)2, 8u>, compact_radix_tree::tree<cell_and_hash, unsigned int>::indirect_layout<cell_and_hash, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)4, 32u, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)6, 0u, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)3, 16u>, compact_radix_tree::tree<cell_and_hash, unsigned int>::direct_layout<cell_and_hash, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)6, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)0, 0u, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)4, 32u> >::visit<compact_radix_tree::tree<cell_and_hash, unsigned int>::walking_visitor<row::for_each_cell<mutation_partition::accept(schema const&, mutation_partition_visitor&) const::$_21>(mutation_partition::accept(schema const&, mutation_partition_visitor&) const::$_21&&) const::{lambda(unsigned int, cell_and_hash const&)#1}, true>&> at mutation_partition.cc:?
      |              (inlined by) compact_radix_tree::tree<cell_and_hash, unsigned int>::node_base<cell_and_hash, compact_radix_tree::tree<cell_and_hash, unsigned int>::indirect_layout<cell_and_hash, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)1, 4u, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)2, 0u, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)0, 0u>, compact_radix_tree::tree<cell_and_hash, unsigned int>::indirect_layout<cell_and_hash, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)2, 8u, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)3, 0u, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)1, 4u>, compact_radix_tree::tree<cell_and_hash, unsigned int>::indirect_layout<cell_and_hash, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)3, 16u, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)4, 0u, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)2, 8u>, compact_radix_tree::tree<cell_and_hash, unsigned int>::indirect_layout<cell_and_hash, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)4, 32u, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)6, 0u, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)3, 16u>, compact_radix_tree::tree<cell_and_hash, unsigned int>::direct_layout<cell_and_hash, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)6, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)0, 0u, (compact_radix_tree::tree<cell_and_hash, unsigned int>::layout)4, 32u> >::visit<compact_radix_tree::tree<cell_and_hash, unsigned int>::walking_visitor<row::for_each_cell<mutation_partition::accept(schema const&, mutation_partition_visitor&) const::$_21>(mutation_partition::accept(schema const&, mutation_partition_visitor&) const::$_21&&) const::{lambda(unsigned int, cell_and_hash const&)#1}, true>&> at ././utils/compact-radix-tree.hh:807
        ++[4#1/1 100%] addr=0x1d27922 total=158592 count=186 avg=853:
        |              compact_radix_tree::tree<cell_and_hash, unsigned int>::node_head::visit<compact_radix_tree::tree<cell_and_hash, unsigned int>::walking_visitor<row::for_each_cell<mutation_partition::accept(schema const&, mutation_partition_visitor&) const::$_21>(mutation_partition::accept(schema const&, mutation_partition_visitor&) const::$_21&&) const::{lambda(unsigned int, cell_and_hash const&)#1}, true> > at ././utils/compact-radix-tree.hh:473
        |              (inlined by) compact_radix_tree::tree<cell_and_hash, unsigned int>::visit<compact_radix_tree::tree<cell_and_hash, unsigned int>::walking_visitor<row::for_each_cell<mutation_partition::accept(schema const&, mutation_partition_visitor&) const::$_21>(mutation_partition::accept(schema const&, mutation_partition_visitor&) const::$_21&&) const::{lambda(unsigned int, cell_and_hash const&)#1}, true> > at ././utils/compact-radix-tree.hh:1628
        |              (inlined by) compact_radix_tree::tree<cell_and_hash, unsigned int>::walk<row::for_each_cell<mutation_partition::accept(schema const&, mutation_partition_visitor&) const::$_21>(mutation_partition::accept(schema const&, mutation_partition_visitor&) const::$_21&&) const::{lambda(unsigned int, cell_and_hash const&)#1}> at ././utils/compact-radix-tree.hh:1911
        |              (inlined by) row::for_each_cell<mutation_partition::accept(schema const&, mutation_partition_visitor&) const::$_21> at ././mutation_partition.hh:152
        |              (inlined by) mutation_partition::accept at ./mutation_partition.cc:1922
        |              (inlined by) mutation_partition::upgrade at ./mutation_partition.cc:1939
          ++[5#1/1 100%] addr=0x1bd229c total=169801 count=199 avg=853:
          |              partition_entry::squashed at ./partition_version.cc:503
          |              (inlined by) partition_entry::upgrade at ./partition_version.cc:517
          -> continued at addr=0x1bd229c above

avikivity · 2023-03-23T11:42:34Z

Duplicate of #2577

avikivity · 2023-03-23T11:43:16Z

@temichus the keyword for #2577 is mutation_partition::upgrade.

temichus changed the title ~~1171 Reactor detailed messages with >800ms in ics-longevity-toggle-strategy-large-partitions-24h-test~~ 1171 reactor stalls messages with >800ms in ics-longevity-toggle-strategy-large-partitions-24h-test Mar 23, 2023

avikivity marked this as a duplicate of #2577 Mar 23, 2023

mykaul closed this as not planned Won't fix, can't repro, duplicate, stale Mar 23, 2023

DoronArazii added this to the 5.3 milestone May 1, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

1171 reactor stalls messages with >800ms in ics-longevity-toggle-strategy-large-partitions-24h-test #13296

1171 reactor stalls messages with >800ms in ics-longevity-toggle-strategy-large-partitions-24h-test #13296

temichus commented Mar 23, 2023 •

edited

Logs:

temichus commented Mar 23, 2023

avikivity commented Mar 23, 2023

avikivity commented Mar 23, 2023

avikivity commented Mar 23, 2023

1171 reactor stalls messages with >800ms in ics-longevity-toggle-strategy-large-partitions-24h-test #13296

1171 reactor stalls messages with >800ms in ics-longevity-toggle-strategy-large-partitions-24h-test #13296

Comments

temichus commented Mar 23, 2023 • edited

Issue description

Installation details

Logs:

temichus commented Mar 23, 2023

avikivity commented Mar 23, 2023

avikivity commented Mar 23, 2023

avikivity commented Mar 23, 2023

temichus commented Mar 23, 2023 •

edited