data_read_resolver::resolve() computes for hundreds of milliseconds #2361

avikivity · 2017-05-08T08:42:48Z

Seen around this line:

                auto diff = v.par
                          ? m.partition().difference(schema, v.par->mut().unfreeze(schema).partition())
                          : m.partition();

We need both to add defer points, and to understand why this is so slow.

avikivity · 2017-05-08T08:44:01Z

/cc @pdziepak @gleb-cloudius

gleb-cloudius · 2017-05-08T09:48:44Z

Some other traces:

backtrace_buffer::append_backtrace() at /home/avi/urchin/seastar/core/reactor.cc:281
 (inlined by) print_with_backtrace at /home/avi/urchin/seastar/core/reactor.cc:292
reactor::block_notifier(int) at /home/avi/urchin/seastar/core/reactor.cc:553
?? ??:0
_ZN7service18data_read_resolver27mutation_and_live_row_countC4EOS1_ at /home/avi/urchin/service/storage_proxy.cc:1884
 (inlined by) void __gnu_cxx::new_allocator<service::data_read_resolver::mutation_and_live_row_count>::construct<service::data_read_resolver::mutation_and_live_row_count, service::data_read_resolver::mutation_and_live_row_count>(service::data_read_resolver::mutation_and_live_row_count*, service::data_read_resolver::mutation_and_live_row_count&&) at /usr/include/c++/5/ext/new_allocator.h:120
 (inlined by) void std::allocator_traits<std::allocator<service::data_read_resolver::mutation_and_live_row_count> >::construct<service::data_read_resolver::mutation_and_live_row_count, service::data_read_resolver::mutation_and_live_row_count>(std::allocator<service::data_read_resolver::mutation_and_live_row_count>&, service::data_read_resolver::mutation_and_live_row_count*, service::data_read_resolver::mutation_and_live_row_count&&) at /usr/include/c++/5/bits/alloc_traits.h:530
 (inlined by) void std::vector<service::data_read_resolver::mutation_and_live_row_count, std::allocator<service::data_read_resolver::mutation_and_live_row_count> >::emplace_back<service::data_read_resolver::mutation_and_live_row_count>(service::data_read_resolver::mutation_and_live_row_count&&) at /usr/include/c++/5/bits/vector.tcc:96
 (inlined by) std::vector<service::data_read_resolver::mutation_and_live_row_count, std::allocator<service::data_read_resolver::mutation_and_live_row_count> >::push_back(service::data_read_resolver::mutation_and_live_row_count&&) at /usr/include/c++/5/bits/stl_vector.h:932
 (inlined by) std::back_insert_iterator<std::vector<service::data_read_resolver::mutation_and_live_row_count, std::allocator<service::data_read_resolver::mutation_and_live_row_count> > >::operator=(service::data_read_resolver::mutation_and_live_row_count&&) at /usr/include/c++/5/bits/stl_iterator.h:465
 (inlined by) std::back_insert_iterator<std::vector<service::data_read_resolver::mutation_and_live_row_count, std::allocator<service::data_read_resolver::mutation_and_live_row_count> > > std::transform<__gnu_cxx::__normal_iterator<std::vector<service::data_read_resolver::version, std::allocator<service::data_read_resolver::version> >*, std::vector<std::vector<service::data_read_resolver::version, std::allocator<service::data_read_resolver::version> >, std::allocator<std::vector<service::data_read_resolver::version, std::allocator<service::data_read_resolver::version> > > > >, std::back_insert_iterator<std::vector<service::data_read_resolver::mutation_and_live_row_count, std::allocator<service::data_read_resolver::mutation_and_live_row_count> > >, service::data_read_resolver::resolve(lw_shared_ptr<schema const>, query::read_command const&, unsigned int, unsigned int, unsigned int)::{lambda(std::vector<service::data_read_resolver::version, std::allocator<service::data_read_resolver::version> >&)#3}>(__gnu_cxx::__normal_iterator<std::vector<service::data_read_resolver::version, std::allocator<service::data_read_resolver::version> >*, std::vector<std::vector<service::data_read_resolver::version, std::allocator<service::data_read_resolver::version> >, std::allocator<std::vector<service::data_read_resolver::version, std::allocator<service::data_read_resolver::version> > > > >, std::back_insert_iterator<std::vector<service::data_read_resolver::mutation_and_live_row_count, std::allocator<service::data_read_resolver::mutation_and_live_row_count> > >, service::data_read_resolver::resolve(lw_shared_ptr<schema const>, query::read_command const&, unsigned int, unsigned int, unsigned int)::{lambda(std::vector<service::data_read_resolver::version, std::allocator<service::data_read_resolver::version> >&)#3}, service::data_read_resolver::resolve(lw_shared_ptr<schema const>, query::read_command const&, unsigned int, unsigned int, unsigned int)::{lambda(std::vector<service::data_read_resolver::version, std::allocator<service::data_read_resolver::version> >&)#3}) at /usr/include/c++/5/bits/stl_algo.h:4175
 (inlined by) std::back_insert_iterator<std::vector<service::data_read_resolver::mutation_and_live_row_count, std::allocator<service::data_read_resolver::mutation_and_live_row_count> > > boost::range::transform<boost::iterator_range<__gnu_cxx::__normal_iterator<std::vector<service::data_read_resolver::version, std::allocator<service::data_read_resolver::version> >*, std::vector<std::vector<service::data_read_resolver::version, std::allocator<service::data_read_resolver::version> >, std::allocator<std::vector<service::data_read_resolver::version, std::allocator<service::data_read_resolver::version> > > > > >, std::back_insert_iterator<std::vector<service::data_read_resolver::mutation_and_live_row_count, std::allocator<service::data_read_resolver::mutation_and_live_row_count> > >, service::data_read_resolver::resolve(lw_shared_ptr<schema const>, query::read_command const&, unsigned int, unsigned int, unsigned int)::{lambda(std::vector<service::data_read_resolver::version, std::allocator<service::data_read_resolver::version> >&)#3}>(boost::iterator_range<__gnu_cxx::__normal_iterator<std::vector<service::data_read_resolver::version, std::allocator<service::data_read_resolver::version> >*, std::vector<std::vector<service::data_read_resolver::version, std::allocator<service::data_read_resolver::version> >, std::allocator<std::vector<service::data_read_resolver::version, std::allocator<service::data_read_resolver::version> > > > > > const&, service::data_read_resolver::resolve(lw_shared_ptr<schema const>, query::read_command const&, unsigned int, unsigned int, unsigned int)::{lambda(std::vector<service::data_read_resolver::version, std::allocator<service::data_read_resolver::version> >&)#3}, service::data_read_resolver::resolve(lw_shared_ptr<schema const>, query::read_command const&, unsigned int, unsigned int, unsigned int)::{lambda(std::vector<service::data_read_resolver::version, std::allocator<service::data_read_resolver::version> >&)#3}) at /usr/include/boost/range/algorithm/transform.hpp:42
 (inlined by) service::data_read_resolver::resolve(lw_shared_ptr<schema const>, query::read_command const&, unsigned int, unsigned int, unsigned int) at /home/avi/urchin/service/storage_proxy.cc:2259
service::abstract_read_executor::reconcile(db::consistency_level, std::chrono::time_point<lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >, lw_shared_ptr<query::read_command>)::{lambda(future<>)#2}::operator()(future<>) const at /home/avi/urchin/service/storage_proxy.cc:2474
std::enable_if<!is_future<std::result_of<service::abstract_read_executor::reconcile(db::consistency_level, std::chrono::time_point<lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >, lw_shared_ptr<query::read_command>)::{lambda(future<>)#2} (future<>&&)>::type>::value, future<> >::type do_void_futurize_apply<service::abstract_read_executor::reconcile(db::consistency_level, std::chrono::time_point<lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >, lw_shared_ptr<query::read_command>)::{lambda(future<>)#2}, future<> >(std::result_of&&, future<>&&) at /home/avi/urchin/seastar/core/future.hh:1224
future<> futurize<void>::apply<service::abstract_read_executor::reconcile(db::consistency_level, std::chrono::time_point<lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >, lw_shared_ptr<query::read_command>)::{lambda(future<>)#2}, future<> >(service::abstract_read_executor::reconcile(db::consistency_level, std::chrono::time_point<lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >, lw_shared_ptr<query::read_command>)::{lambda(future<>)#2}&&, future<>&&) at /home/avi/urchin/seastar/core/future.hh:1272
 (inlined by) _ZZN6futureIJEE12then_wrappedIZN7service22abstract_read_executor9reconcileEN2db17consistency_levelENSt6chrono10time_pointI12lowres_clockNS6_8durationIlSt5ratioILl1ELl1000EEEEEE13lw_shared_ptrIN5query12read_commandEEEUlS0_E0_S0_EET0_OT_ENUlOSI_SL_E_clI12future_stateIJEEEEDaSM_SL_ at /home/avi/urchin/seastar/core/future.hh:923
 (inlined by) _ZZN7seastar4impl25rebind_scheduled_functionIZN6futureIJEE12then_wrappedIZN7service22abstract_read_executor9reconcileEN2db17consistency_levelENSt6chrono10time_pointI12lowres_clockNS9_8durationIlSt5ratioILl1ELl1000EEEEEE13lw_shared_ptrIN5query12read_commandEEEUlS3_E0_S3_EET0_OT_EUlOSL_SO_E_SL_EEDaOSM_SO_ENUlDpOT_E_clIJ12future_stateIJEEEEEDaSU_ at /home/avi/urchin/seastar/core/scheduling.hh:157
 (inlined by) _ZN12continuationIZN7seastar4impl25rebind_scheduled_functionIZN6futureIJEE12then_wrappedIZN7service22abstract_read_executor9reconcileEN2db17consistency_levelENSt6chrono10time_pointI12lowres_clockNSA_8durationIlSt5ratioILl1ELl1000EEEEEE13lw_shared_ptrIN5query12read_commandEEEUlS4_E0_S4_EET0_OT_EUlOSM_SP_E_SM_EEDaOSN_SP_EUlDpOT_E_JEE3runEv at /home/avi/urchin/seastar/core/future.hh:391
reactor::run_tasks(reactor::task_queue&) at /home/avi/urchin/seastar/core/reactor.cc:2270
reactor::run_some_tasks(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >&) at /home/avi/urchin/seastar/core/reactor.cc:2634
reactor::run_some_tasks(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >&) at /home/avi/urchin/seastar/core/reactor.cc:2620
 (inlined by) reactor::run() at /home/avi/urchin/seastar/core/reactor.cc:2816
operator() at /home/avi/urchin/seastar/core/reactor.cc:3742
std::function<void ()>::operator()() const at /usr/include/c++/5/functional:2267
 (inlined by) posix_thread::start_routine(void*) at /home/avi/urchin/seastar/core/posix.cc:50

gleb-cloudius · 2017-05-08T09:50:07Z

_ZN13lw_shared_ptrI13compound_typeIL14allow_prefixes1EEED4Ev at /home/avi/urchin/seastar/core/shared_ptr.hh:185
 (inlined by) compound_wrapper<clustering_key_prefix, clustering_key_prefix_view>::less_compare::~less_compare() at /home/avi/urchin/./keys.hh:237
_ZN10rows_entry7compareD4Ev at /home/avi/urchin/mutation_partition.hh:500
 (inlined by) _ZN33intrusive_set_external_comparatorI10rows_entryXadL_ZNS0_5_linkEEEE19key_node_comparatorINS0_7compareEED4Ev at /home/avi/urchin/intrusive_set_external_comparator.hh:70
 (inlined by) boost::intrusive::compact_rbtree_node<void*>* boost::intrusive::bstree_algorithms<boost::intrusive::rbtree_node_traits<void*, true> >::find<clustering_key_prefix_view, intrusive_set_external_comparator<rows_entry, &rows_entry::_link>::key_node_comparator<rows_entry::compare> >(boost::intrusive::compact_rbtree_node<void*> const* const&, clustering_key_prefix_view const&, intrusive_set_external_comparator<rows_entry, &rows_entry::_link>::key_node_comparator<rows_entry::compare>) at /usr/include/boost/intrusive/bstree_algorithms.hpp:780
 (inlined by) boost::intrusive::tree_iterator<boost::intrusive::mhtraits<rows_entry, intrusive_set_external_comparator_member_hook, &rows_entry::_link>, false> intrusive_set_external_comparator<rows_entry, &rows_entry::_link>::find<clustering_key_prefix_view, rows_entry::compare>(clustering_key_prefix_view const&, rows_entry::compare) at /home/avi/urchin/intrusive_set_external_comparator.hh:203
 (inlined by) mutation_partition::clustered_row(schema const&, clustering_key_prefix_view const&) at /home/avi/urchin/mutation_partition.cc:494
partition_builder::accept_row(clustering_key_prefix_view, tombstone, row_marker const&) at /home/avi/urchin/partition_builder.hh:60
mutation_partition_view::accept(column_mapping const&, mutation_partition_visitor&) const at /home/avi/urchin/mutation_partition_view.cc:212
mutation_partition::apply(schema const&, mutation_partition_view, schema const&) at /home/avi/urchin/mutation_partition.cc:364
service::data_read_resolver::resolve(lw_shared_ptr<schema const>, query::read_command const&, unsigned int, unsigned int, unsigned int)::{lambda(std::vector<service::data_read_resolver::version, std::allocator<service::data_read_resolver::version> >&)#3}::operator()(std::vector<service::data_read_resolver::version, std::allocator<service::data_read_resolver::version> >&) const::{lambda(mutation&, service::data_read_resolver::version const&)#1}::operator()(mutation, service::data_read_resolver::version const) const at /home/avi/urchin/service/storage_proxy.cc:2251
 (inlined by) mutation std::accumulate<__gnu_cxx::__normal_iterator<service::data_read_resolver::version const*, std::vector<service::data_read_resolver::version, std::allocator<service::data_read_resolver::version> > >, mutation, service::data_read_resolver::resolve(lw_shared_ptr<schema const>, query::read_command const&, unsigned int, unsigned int, unsigned int)::{lambda(std::vector<service::data_read_resolver::version, std::allocator<service::data_read_resolver::version> >&)#3}::operator()(std::vector<service::data_read_resolver::version, std::allocator<service::data_read_resolver::version> >&) const::{lambda(mutation&, service::data_read_resolver::version const&)#1}>(__gnu_cxx::__normal_iterator<service::data_read_resolver::version const*, std::vector<service::data_read_resolver::version, std::allocator<service::data_read_resolver::version> > >, service::data_read_resolver::resolve(lw_shared_ptr<schema const>, query::read_command const&, unsigned int, unsigned int, unsigned int)::{lambda(std::vector<service::data_read_resolver::version, std::allocator<service::data_read_resolver::version> >&)#3}::operator()(std::vector<service::data_read_resolver::version, std::allocator<service::data_read_resolver::version> >&) const::{lambda(mutation&, service::data_read_resolver::version const&)#1}, {lambda(mutation&, service::data_read_resolver::version const&)#1}, service::data_read_resolver::resolve(lw_shared_ptr<schema const>, query::read_command const&, unsigned int, unsigned int, unsigned int)::{lambda(std::vector<service::data_read_resolver::version, std::allocator<service::data_read_resolver::version> >&)#3}::operator()(std::vector<service::data_read_resolver::version, std::allocator<service::data_read_resolver::version> >&) const::{lambda(mutation&, service::data_read_resolver::version const&)#1}) at /usr/include/c++/5/bits/stl_numeric.h:154
 (inlined by) mutation boost::accumulate<std::vector<service::data_read_resolver::version, std::allocator<service::data_read_resolver::version> >, mutation, service::data_read_resolver::resolve(lw_shared_ptr<schema const>, query::read_command const&, unsigned int, unsigned int, unsigned int)::{lambda(std::vector<service::data_read_resolver::version, std::allocator<service::data_read_resolver::version> >&)#3}::operator()(std::vector<service::data_read_resolver::version, std::allocator<service::data_read_resolver::version> >&) const::{lambda(mutation&, service::data_read_resolver::version const&)#1}>(std::vector<service::data_read_resolver::version, std::allocator<service::data_read_resolver::version> > const&, {lambda(mutation&, service::data_read_resolver::version const&)#1}, service::data_read_resolver::resolve(lw_shared_ptr<schema const>, query::read_command const&, unsigned int, unsigned int, unsigned int)::{lambda(std::vector<service::data_read_resolver::version, std::allocator<service::data_read_resolver::version> >&)#3}::operator()(std::vector<service::data_read_resolver::version, std::allocator<service::data_read_resolver::version> >&) const::{lambda(mutation&, service::data_read_resolver::version const&)#1}) at /usr/include/boost/range/numeric.hpp:51
 (inlined by) service::data_read_resolver::resolve(lw_shared_ptr<schema const>, query::read_command const&, unsigned int, unsigned int, unsigned int)::{lambda(std::vector<service::data_read_resolver::version, std::allocator<service::data_read_resolver::version> >&)#3}::operator()(std::vector<service::data_read_resolver::version, std::allocator<service::data_read_resolver::version> >&) const at /home/avi/urchin/service/storage_proxy.cc:2254
std::back_insert_iterator<std::vector<service::data_read_resolver::mutation_and_live_row_count, std::allocator<service::data_read_resolver::mutation_and_live_row_count> > > std::transform<__gnu_cxx::__normal_iterator<std::vector<service::data_read_resolver::version, std::allocator<service::data_read_resolver::version> >*, std::vector<std::vector<service::data_read_resolver::version, std::allocator<service::data_read_resolver::version> >, std::allocator<std::vector<service::data_read_resolver::version, std::allocator<service::data_read_resolver::version> > > > >, std::back_insert_iterator<std::vector<service::data_read_resolver::mutation_and_live_row_count, std::allocator<service::data_read_resolver::mutation_and_live_row_count> > >, service::data_read_resolver::resolve(lw_shared_ptr<schema const>, query::read_command const&, unsigned int, unsigned int, unsigned int)::{lambda(std::vector<service::data_read_resolver::version, std::allocator<service::data_read_resolver::version> >&)#3}>(__gnu_cxx::__normal_iterator<std::vector<service::data_read_resolver::version, std::allocator<service::data_read_resolver::version> >*, std::vector<std::vector<service::data_read_resolver::version, std::allocator<service::data_read_resolver::version> >, std::allocator<std::vector<service::data_read_resolver::version, std::allocator<service::data_read_resolver::version> > > > >, std::back_insert_iterator<std::vector<service::data_read_resolver::mutation_and_live_row_count, std::allocator<service::data_read_resolver::mutation_and_live_row_count> > >, service::data_read_resolver::resolve(lw_shared_ptr<schema const>, query::read_command const&, unsigned int, unsigned int, unsigned int)::{lambda(std::vector<service::data_read_resolver::version, std::allocator<service::data_read_resolver::version> >&)#3}, service::data_read_resolver::resolve(lw_shared_ptr<schema const>, query::read_command const&, unsigned int, unsigned int, unsigned int)::{lambda(std::vector<service::data_read_resolver::version, std::allocator<service::data_read_resolver::version> >&)#3}) at /usr/include/c++/5/bits/stl_algo.h:4175
 (inlined by) std::back_insert_iterator<std::vector<service::data_read_resolver::mutation_and_live_row_count, std::allocator<service::data_read_resolver::mutation_and_live_row_count> > > boost::range::transform<boost::iterator_range<__gnu_cxx::__normal_iterator<std::vector<service::data_read_resolver::version, std::allocator<service::data_read_resolver::version> >*, std::vector<std::vector<service::data_read_resolver::version, std::allocator<service::data_read_resolver::version> >, std::allocator<std::vector<service::data_read_resolver::version, std::allocator<service::data_read_resolver::version> > > > > >, std::back_insert_iterator<std::vector<service::data_read_resolver::mutation_and_live_row_count, std::allocator<service::data_read_resolver::mutation_and_live_row_count> > >, service::data_read_resolver::resolve(lw_shared_ptr<schema const>, query::read_command const&, unsigned int, unsigned int, unsigned int)::{lambda(std::vector<service::data_read_resolver::version, std::allocator<service::data_read_resolver::version> >&)#3}>(boost::iterator_range<__gnu_cxx::__normal_iterator<std::vector<service::data_read_resolver::version, std::allocator<service::data_read_resolver::version> >*, std::vector<std::vector<service::data_read_resolver::version, std::allocator<service::data_read_resolver::version> >, std::allocator<std::vector<service::data_read_resolver::version, std::allocator<service::data_read_resolver::version> > > > > > const&, service::data_read_resolver::resolve(lw_shared_ptr<schema const>, query::read_command const&, unsigned int, unsigned int, unsigned int)::{lambda(std::vector<service::data_read_resolver::version, std::allocator<service::data_read_resolver::version> >&)#3}, service::data_read_resolver::resolve(lw_shared_ptr<schema const>, query::read_command const&, unsigned int, unsigned int, unsigned int)::{lambda(std::vector<service::data_read_resolver::version, std::allocator<service::data_read_resolver::version> >&)#3}) at /usr/include/boost/range/algorithm/transform.hpp:42
 (inlined by) service::data_read_resolver::resolve(lw_shared_ptr<schema const>, query::read_command const&, unsigned int, unsigned int, unsigned int) at /home/avi/urchin/service/storage_proxy.cc:2259
service::abstract_read_executor::reconcile(db::consistency_level, std::chrono::time_point<lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >, lw_shared_ptr<query::read_command>)::{lambda(future<>)#2}::operator()(future<>) const at /home/avi/urchin/service/storage_proxy.cc:2474
std::enable_if<!is_future<std::result_of<service::abstract_read_executor::reconcile(db::consistency_level, std::chrono::time_point<lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >, lw_shared_ptr<query::read_command>)::{lambda(future<>)#2} (future<>&&)>::type>::value, future<> >::type do_void_futurize_apply<service::abstract_read_executor::reconcile(db::consistency_level, std::chrono::time_point<lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >, lw_shared_ptr<query::read_command>)::{lambda(future<>)#2}, future<> >(std::result_of&&, future<>&&) at /home/avi/urchin/seastar/core/future.hh:1224
future<> futurize<void>::apply<service::abstract_read_executor::reconcile(db::consistency_level, std::chrono::time_point<lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >, lw_shared_ptr<query::read_command>)::{lambda(future<>)#2}, future<> >(service::abstract_read_executor::reconcile(db::consistency_level, std::chrono::time_point<lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >, lw_shared_ptr<query::read_command>)::{lambda(future<>)#2}&&, future<>&&) at /home/avi/urchin/seastar/core/future.hh:1272
 (inlined by) _ZZN6futureIJEE12then_wrappedIZN7service22abstract_read_executor9reconcileEN2db17consistency_levelENSt6chrono10time_pointI12lowres_clockNS6_8durationIlSt5ratioILl1ELl1000EEEEEE13lw_shared_ptrIN5query12read_commandEEEUlS0_E0_S0_EET0_OT_ENUlOSI_SL_E_clI12future_stateIJEEEEDaSM_SL_ at /home/avi/urchin/seastar/core/future.hh:923
 (inlined by) _ZZN7seastar4impl25rebind_scheduled_functionIZN6futureIJEE12then_wrappedIZN7service22abstract_read_executor9reconcileEN2db17consistency_levelENSt6chrono10time_pointI12lowres_clockNS9_8durationIlSt5ratioILl1ELl1000EEEEEE13lw_shared_ptrIN5query12read_commandEEEUlS3_E0_S3_EET0_OT_EUlOSL_SO_E_SL_EEDaOSM_SO_ENUlDpOT_E_clIJ12future_stateIJEEEEEDaSU_ at /home/avi/urchin/seastar/core/scheduling.hh:157
 (inlined by) _ZN12continuationIZN7seastar4impl25rebind_scheduled_functionIZN6futureIJEE12then_wrappedIZN7service22abstract_read_executor9reconcileEN2db17consistency_levelENSt6chrono10time_pointI12lowres_clockNSA_8durationIlSt5ratioILl1ELl1000EEEEEE13lw_shared_ptrIN5query12read_commandEEEUlS4_E0_S4_EET0_OT_EUlOSM_SP_E_SM_EEDaOSN_SP_EUlDpOT_E_JEE3runEv at /home/avi/urchin/seastar/core/future.hh:391
reactor::run_tasks(reactor::task_queue&) at /home/avi/urchin/seastar/core/reactor.cc:2270
reactor::run_some_tasks(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >&) at /home/avi/urchin/seastar/core/reactor.cc:2634
reactor::run_some_tasks(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >&) at /home/avi/urchin/seastar/core/reactor.cc:2620
 (inlined by) reactor::run() at /home/avi/urchin/seastar/core/reactor.cc:2816
operator() at /home/avi/urchin/seastar/core/reactor.cc:3742
std::function<void ()>::operator()() const at /usr/include/c++/5/functional:2267
 (inlined by) posix_thread::start_routine(void*) at /home/avi/urchin/seastar/core/posix.cc:50

gleb-cloudius · 2017-05-08T09:54:30Z

_ZN7service18data_read_resolver27mutation_and_live_row_countC4EOS1_ at /home/avi/urchin/service/storage_proxy.cc:1884
 (inlined by) void __gnu_cxx::new_allocator<service::data_read_resolver::mutation_and_live_row_count>::construct<service::data_read_resolver::mutation_and_live_row_count, service::data_read_resolver::mutation_and_live_row_count>(service::data_read_resolver::mutation_and_live_row_count*, service::data_read_resolver::mutation_and_live_row_count&&) at /usr/include/c++/5/ext/new_allocator.h:120
 (inlined by) void std::allocator_traits<std::allocator<service::data_read_resolver::mutation_and_live_row_count> >::construct<service::data_read_resolver::mutation_and_live_row_count, service::data_read_resolver::mutation_and_live_row_count>(std::allocator<service::data_read_resolver::mutation_and_live_row_count>&, service::data_read_resolver::mutation_and_live_row_count*, service::data_read_resolver::mutation_and_live_row_count&&) at /usr/include/c++/5/bits/alloc_traits.h:530
 (inlined by) void std::vector<service::data_read_resolver::mutation_and_live_row_count, std::allocator<service::data_read_resolver::mutation_and_live_row_count> >::emplace_back<service::data_read_resolver::mutation_and_live_row_count>(service::data_read_resolver::mutation_and_live_row_count&&) at /usr/include/c++/5/bits/vector.tcc:96
 (inlined by) std::vector<service::data_read_resolver::mutation_and_live_row_count, std::allocator<service::data_read_resolver::mutation_and_live_row_count> >::push_back(service::data_read_resolver::mutation_and_live_row_count&&) at /usr/include/c++/5/bits/stl_vector.h:932
 (inlined by) std::back_insert_iterator<std::vector<service::data_read_resolver::mutation_and_live_row_count, std::allocator<service::data_read_resolver::mutation_and_live_row_count> > >::operator=(service::data_read_resolver::mutation_and_live_row_count&&) at /usr/include/c++/5/bits/stl_iterator.h:465
 (inlined by) std::back_insert_iterator<std::vector<service::data_read_resolver::mutation_and_live_row_count, std::allocator<service::data_read_resolver::mutation_and_live_row_count> > > std::transform<__gnu_cxx::__normal_iterator<std::vector<service::data_read_resolver::version, std::allocator<service::data_read_resolver::version> >*, std::vector<std::vector<service::data_read_resolver::version, std::allocator<service::data_read_resolver::version> >, std::allocator<std::vector<service::data_read_resolver::version, std::allocator<service::data_read_resolver::version> > > > >, std::back_insert_iterator<std::vector<service::data_read_resolver::mutation_and_live_row_count, std::allocator<service::data_read_resolver::mutation_and_live_row_count> > >, service::data_read_resolver::resolve(lw_shared_ptr<schema const>, query::read_command const&, unsigned int, unsigned int, unsigned int)::{lambda(std::vector<service::data_read_resolver::version, std::allocator<service::data_read_resolver::version> >&)#3}>(__gnu_cxx::__normal_iterator<std::vector<service::data_read_resolver::version, std::allocator<service::data_read_resolver::version> >*, std::vector<std::vector<service::data_read_resolver::version, std::allocator<service::data_read_resolver::version> >, std::allocator<std::vector<service::data_read_resolver::version, std::allocator<service::data_read_resolver::version> > > > >, std::back_insert_iterator<std::vector<service::data_read_resolver::mutation_and_live_row_count, std::allocator<service::data_read_resolver::mutation_and_live_row_count> > >, service::data_read_resolver::resolve(lw_shared_ptr<schema const>, query::read_command const&, unsigned int, unsigned int, unsigned int)::{lambda(std::vector<service::data_read_resolver::version, std::allocator<service::data_read_resolver::version> >&)#3}, service::data_read_resolver::resolve(lw_shared_ptr<schema const>, query::read_command const&, unsigned int, unsigned int, unsigned int)::{lambda(std::vector<service::data_read_resolver::version, std::allocator<service::data_read_resolver::version> >&)#3}) at /usr/include/c++/5/bits/stl_algo.h:4175
 (inlined by) std::back_insert_iterator<std::vector<service::data_read_resolver::mutation_and_live_row_count, std::allocator<service::data_read_resolver::mutation_and_live_row_count> > > boost::range::transform<boost::iterator_range<__gnu_cxx::__normal_iterator<std::vector<service::data_read_resolver::version, std::allocator<service::data_read_resolver::version> >*, std::vector<std::vector<service::data_read_resolver::version, std::allocator<service::data_read_resolver::version> >, std::allocator<std::vector<service::data_read_resolver::version, std::allocator<service::data_read_resolver::version> > > > > >, std::back_insert_iterator<std::vector<service::data_read_resolver::mutation_and_live_row_count, std::allocator<service::data_read_resolver::mutation_and_live_row_count> > >, service::data_read_resolver::resolve(lw_shared_ptr<schema const>, query::read_command const&, unsigned int, unsigned int, unsigned int)::{lambda(std::vector<service::data_read_resolver::version, std::allocator<service::data_read_resolver::version> >&)#3}>(boost::iterator_range<__gnu_cxx::__normal_iterator<std::vector<service::data_read_resolver::version, std::allocator<service::data_read_resolver::version> >*, std::vector<std::vector<service::data_read_resolver::version, std::allocator<service::data_read_resolver::version> >, std::allocator<std::vector<service::data_read_resolver::version, std::allocator<service::data_read_resolver::version> > > > > > const&, service::data_read_resolver::resolve(lw_shared_ptr<schema const>, query::read_command const&, unsigned int, unsigned int, unsigned int)::{lambda(std::vector<service::data_read_resolver::version, std::allocator<service::data_read_resolver::version> >&)#3}, service::data_read_resolver::resolve(lw_shared_ptr<schema const>, query::read_command const&, unsigned int, unsigned int, unsigned int)::{lambda(std::vector<service::data_read_resolver::version, std::allocator<service::data_read_resolver::version> >&)#3}) at /usr/include/boost/range/algorithm/transform.hpp:42
 (inlined by) service::data_read_resolver::resolve(lw_shared_ptr<schema const>, query::read_command const&, unsigned int, unsigned int, unsigned int) at /home/avi/urchin/service/storage_proxy.cc:2259
service::abstract_read_executor::reconcile(db::consistency_level, std::chrono::time_point<lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >, lw_shared_ptr<query::read_command>)::{lambda(future<>)#2}::operator()(future<>) const at /home/avi/urchin/service/storage_proxy.cc:2474
std::enable_if<!is_future<std::result_of<service::abstract_read_executor::reconcile(db::consistency_level, std::chrono::time_point<lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >, lw_shared_ptr<query::read_command>)::{lambda(future<>)#2} (future<>&&)>::type>::value, future<> >::type do_void_futurize_apply<service::abstract_read_executor::reconcile(db::consistency_level, std::chrono::time_point<lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >, lw_shared_ptr<query::read_command>)::{lambda(future<>)#2}, future<> >(std::result_of&&, future<>&&) at /home/avi/urchin/seastar/core/future.hh:1224
future<> futurize<void>::apply<service::abstract_read_executor::reconcile(db::consistency_level, std::chrono::time_point<lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >, lw_shared_ptr<query::read_command>)::{lambda(future<>)#2}, future<> >(service::abstract_read_executor::reconcile(db::consistency_level, std::chrono::time_point<lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >, lw_shared_ptr<query::read_command>)::{lambda(future<>)#2}&&, future<>&&) at /home/avi/urchin/seastar/core/future.hh:1272
 (inlined by) _ZZN6futureIJEE12then_wrappedIZN7service22abstract_read_executor9reconcileEN2db17consistency_levelENSt6chrono10time_pointI12lowres_clockNS6_8durationIlSt5ratioILl1ELl1000EEEEEE13lw_shared_ptrIN5query12read_commandEEEUlS0_E0_S0_EET0_OT_ENUlOSI_SL_E_clI12future_stateIJEEEEDaSM_SL_ at /home/avi/urchin/seastar/core/future.hh:923
 (inlined by) _ZZN7seastar4impl25rebind_scheduled_functionIZN6futureIJEE12then_wrappedIZN7service22abstract_read_executor9reconcileEN2db17consistency_levelENSt6chrono10time_pointI12lowres_clockNS9_8durationIlSt5ratioILl1ELl1000EEEEEE13lw_shared_ptrIN5query12read_commandEEEUlS3_E0_S3_EET0_OT_EUlOSL_SO_E_SL_EEDaOSM_SO_ENUlDpOT_E_clIJ12future_stateIJEEEEEDaSU_ at /home/avi/urchin/seastar/core/scheduling.hh:157
 (inlined by) _ZN12continuationIZN7seastar4impl25rebind_scheduled_functionIZN6futureIJEE12then_wrappedIZN7service22abstract_read_executor9reconcileEN2db17consistency_levelENSt6chrono10time_pointI12lowres_clockNSA_8durationIlSt5ratioILl1ELl1000EEEEEE13lw_shared_ptrIN5query12read_commandEEEUlS4_E0_S4_EET0_OT_EUlOSM_SP_E_SM_EEDaOSN_SP_EUlDpOT_E_JEE3runEv at /home/avi/urchin/seastar/core/future.hh:391
reactor::run_tasks(reactor::task_queue&) at /home/avi/urchin/seastar/core/reactor.cc:2270
reactor::run_some_tasks(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >&) at /home/avi/urchin/seastar/core/reactor.cc:2634
reactor::run_some_tasks(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >&) at /home/avi/urchin/seastar/core/reactor.cc:2620
 (inlined by) reactor::run() at /home/avi/urchin/seastar/core/reactor.cc:2816
operator() at /home/avi/urchin/seastar/core/reactor.cc:3742
std::function<void ()>::operator()() const at /usr/include/c++/5/functional:2267
 (inlined by) posix_thread::start_routine(void*) at /home/avi/urchin/seastar/core/posix.cc:50

pdziepak · 2017-05-08T15:28:26Z

I don't see anything obviously wrong. We have O(N log N) apply and a bunch of linear operations, like freeze and unfreeze. Overall, it is not very surprising that resolve is an expensive operation, though hundreds of ms is probably a bit excessive.
It would be great to know what N is, perhaps, we are dealing with partitions much larger than they should be (e.g. coordinator's increase-limit-and-retry is malfunctioning).

avikivity · 2017-05-08T15:46:42Z

N was something ridiculous; there was a jmx call to get compaction history, and we implement that by internally performing a non-paging select on the compactionhistory table. Since that node was busy resharding for a while, there was lots of history.

gleb-cloudius · 2017-05-08T16:05:47Z

On Mon, May 08, 2017 at 08:46:46AM -0700, Avi Kivity wrote: N was something ridiculous; there was a jmx call to get compaction history, and we implement that by internally performing a non-paging select on the compactionhistory table. Since that node was busy resharding for a while, there was lots of history.

Note that compaction history is local table, so reconciliation step can be completely dropped.

…

-- Gleb.

avikivity · 2017-05-08T17:23:59Z

Why was it even attempted, for RF=1?

gleb-cloudius · 2017-05-08T18:37:19Z

On Mon, May 08, 2017 at 10:24:04AM -0700, Avi Kivity wrote: Why was it even attempted, for RF=1?

Range queries always goes through reconciliation code path since there is no hash matching stage. Single key read fall back there only if there is hash mismatch. Reconciliation code should be optimized for RF=1 case by skipping most of the logic and jumping directly to building a result.

…

-- Gleb.

gleb-cloudius · 2017-06-12T09:44:41Z

Another one:

backtrace_buffer::append_backtrace() at /home/avi/urchin/seastar/core/reactor.cc:281
 (inlined by) print_with_backtrace at /home/avi/urchin/seastar/core/reactor.cc:292
reactor::block_notifier(int) at /home/avi/urchin/seastar/core/reactor.cc:553
?? ??:0
memory::small_pool::trim_free_list() at /home/avi/urchin/seastar/core/memory.cc:1124
standard_allocation_strategy::free(void*) at /home/avi/urchin/./utils/allocation_strategy.hh:148
 (inlined by) void allocation_strategy::destroy<blob_storage>(blob_storage*) at /home/avi/urchin/./utils/allocation_strategy.hh:128
 (inlined by) managed_bytes::free_chain(blob_storage*) at /home/avi/urchin/./utils/managed_bytes.hh:120
 (inlined by) _ZN13managed_bytesD4Ev at /home/avi/urchin/./utils/managed_bytes.hh:207
 (inlined by) _ZN16compound_wrapperI13partition_key18partition_key_viewED4Ev at /home/avi/urchin/keys.hh:151
 (inlined by) _ZN13partition_keyD4Ev at /home/avi/urchin/keys.hh:614
 (inlined by) _ZN15frozen_mutationD4Ev at /home/avi/urchin/frozen_mutation.hh:46
 (inlined by) _ZN9partitionD4Ev at /home/avi/urchin/mutation_query.hh:33
 (inlined by) void std::_Destroy<partition>(partition*) at /usr/include/c++/5/bits/stl_construct.h:93
 (inlined by) void std::_Destroy_aux<false>::__destroy<partition*>(partition*, partition*) at /usr/include/c++/5/bits/stl_construct.h:103
 (inlined by) void std::_Destroy<partition*>(partition*, partition*) at /usr/include/c++/5/bits/stl_construct.h:126
 (inlined by) void std::_Destroy<partition*, partition>(partition*, partition*, std::allocator<partition>&) at /usr/include/c++/5/bits/stl_construct.h:151
 (inlined by) _ZNSt6vectorI9partitionSaIS0_EED4Ev at /usr/include/c++/5/bits/stl_vector.h:424
 (inlined by) reconcilable_result::~reconcilable_result() at /home/avi/urchin/mutation_query.cc:28
_ZNSt12experimental15fundamentals_v114_Optional_baseI19reconcilable_resultLb1EED4Ev at /usr/include/c++/5/experimental/optional:289
 (inlined by) _ZNSt12experimental15fundamentals_v18optionalI19reconcilable_resultED4Ev at /usr/include/c++/5/experimental/optional:477
 (inlined by) service::abstract_read_executor::reconcile(db::consistency_level, std::chrono::time_point<lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >, lw_shared_ptr<query::read_command>)::{lambda(future<>)#2}::operator()(future<>) const at /home/avi/urchin/service/storage_proxy.cc:2474
std::enable_if<!is_future<std::result_of<service::abstract_read_executor::reconcile(db::consistency_level, std::chrono::time_point<lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >, lw_shared_ptr<query::read_command>)::{lambda(future<>)#2} (future<>&&)>::type>::value, future<> >::type do_void_futurize_apply<service::abstract_read_executor::reconcile(db::consistency_level, std::chrono::time_point<lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >, lw_shared_ptr<query::read_command>)::{lambda(future<>)#2}, future<> >(std::result_of&&, future<>&&) at /home/avi/urchin/seastar/core/future.hh:1224
future<> futurize<void>::apply<service::abstract_read_executor::reconcile(db::consistency_level, std::chrono::time_point<lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >, lw_shared_ptr<query::read_command>)::{lambda(future<>)#2}, future<> >(service::abstract_read_executor::reconcile(db::consistency_level, std::chrono::time_point<lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >, lw_shared_ptr<query::read_command>)::{lambda(future<>)#2}&&, future<>&&) at /home/avi/urchin/seastar/core/future.hh:1272
 (inlined by) _ZZN6futureIJEE12then_wrappedIZN7service22abstract_read_executor9reconcileEN2db17consistency_levelENSt6chrono10time_pointI12lowres_clockNS6_8durationIlSt5ratioILl1ELl1000EEEEEE13lw_shared_ptrIN5query12read_commandEEEUlS0_E0_S0_EET0_OT_ENUlSL_OSJ_E_clISI_12future_stateIJEEEEDaSL_SM_ at /home/avi/urchin/seastar/core/future.hh:923
 (inlined by) _ZZN7seastar4impl25rebind_scheduled_functionIZN6futureIJEE12then_wrappedIZN7service22abstract_read_executor9reconcileEN2db17consistency_levelENSt6chrono10time_pointI12lowres_clockNS9_8durationIlSt5ratioILl1ELl1000EEEEEE13lw_shared_ptrIN5query12read_commandEEEUlS3_E0_S3_EET0_OT_EUlSO_OSM_E_SL_EEDaSP_SO_ENUlDpOT_E_clIJ12future_stateIJEEEEEDaST_ at /home/avi/urchin/seastar/core/scheduling.hh:157
 (inlined by) _ZN12continuationIZN7seastar4impl25rebind_scheduled_functionIZN6futureIJEE12then_wrappedIZN7service22abstract_read_executor9reconcileEN2db17consistency_levelENSt6chrono10time_pointI12lowres_clockNSA_8durationIlSt5ratioILl1ELl1000EEEEEE13lw_shared_ptrIN5query12read_commandEEEUlS4_E0_S4_EET0_OT_EUlSP_OSN_E_SM_EEDaSQ_SP_EUlDpOT_E_JEE3runEv at /home/avi/urchin/seastar/core/future.hh:391
reactor::run_tasks(reactor::task_queue&) at /home/avi/urchin/seastar/core/reactor.cc:2270
reactor::run_some_tasks(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >&) at /home/avi/urchin/seastar/core/reactor.cc:2634
reactor::run_some_tasks(std::chrono::time_point<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >&) at /home/avi/urchin/seastar/core/reactor.cc:2620
 (inlined by) reactor::run() at /home/avi/urchin/seastar/core/reactor.cc:2816
operator() at /home/avi/urchin/seastar/core/reactor.cc:3742
std::function<void ()>::operator()() const at /usr/include/c++/5/functional:2267
 (inlined by) posix_thread::start_routine(void*) at /home/avi/urchin/seastar/core/posix.cc:50

bhalevy · 2022-05-02T07:18:09Z

Recent stalls in "[scylla-perf-results] Performance Regression Compare Results - PerformanceRegressionCDCTest.test_mixed_throughput - 5.1.dev - 2022-05-01 07:16:36.190302"
build date: 2022-04-28
commit id: 333fdcb
build-id: 415dea72992f5c04c11c4171a5b878d2e1c48fa9

For example:

2022-05-01T06:44:07+00:00  perf-cdc-mixed-perf-v10-db-node-09719899-2 !INFO    | scylla[7598]: Reactor stalled for 69 ms on shard 7. Backtrace: 0x4616102 0x4614d60 0x4616010 0x7f1c92893a1f 0x173ac29 0x172002b 0x15b79a1 0x2564609 0x25600c5 0x25782bb 0x46270f4 0x46284d7 0x4647085 0x45fae7a 0x92a4 0x100322
0x4616102
0x4614d60
0x4616010
0x7f1c92893a1f
0x173ac29
0x172002b
0x15b79a1
0x2564609
0x25600c5
0x25782bb
0x46270f4
0x46284d7
0x4647085
0x45fae7a
0x92a4
0x100322

Decoded:

[Backtrace #0]
void seastar::backtrace<seastar::backtrace_buffer::append_backtrace_oneline()::{lambda(seastar::frame)#1}>(seastar::backtrace_buffer::append_backtrace_oneline()::{lambda(seastar::frame)#1}&&) at ./build/release/seastar/./seastar/include/seastar/util/backtrace.hh:59
 (inlined by) seastar::backtrace_buffer::append_backtrace_oneline() at ./build/release/seastar/./seastar/src/core/reactor.cc:764
 (inlined by) seastar::print_with_backtrace(seastar::backtrace_buffer&, bool) at ./build/release/seastar/./seastar/src/core/reactor.cc:783
seastar::internal::cpu_stall_detector::generate_trace() at ./build/release/seastar/./seastar/src/core/reactor.cc:1358
seastar::internal::cpu_stall_detector::maybe_report() at ./build/release/seastar/./seastar/src/core/reactor.cc:1100
 (inlined by) seastar::internal::cpu_stall_detector::on_signal() at ./build/release/seastar/./seastar/src/core/reactor.cc:1117
 (inlined by) seastar::reactor::block_notifier(int) at ./build/release/seastar/./seastar/src/core/reactor.cc:1341
?? ??:0
decltype(auto) seastar::memory_input_stream<bytes_ostream::fragment_iterator>::with_stream<ser::live_cell_view::created_at() const::{lambda(auto:1&)#1}>(ser::live_cell_view::created_at() const::{lambda(auto:1&)#1}&&) const at ././seastar/include/seastar/core/simple-stream.hh:478
 (inlined by) decltype(auto) seastar::with_serialized_stream<seastar::memory_input_stream<bytes_ostream::fragment_iterator> const, ser::live_cell_view::created_at() const::{lambda(auto:1&)#1}, void>(seastar::memory_input_stream<bytes_ostream::fragment_iterator> const&, ser::live_cell_view::created_at() const::{lambda(auto:1&)#1}&&) at ././seastar/include/seastar/core/simple-stream.hh:626
 (inlined by) ser::live_cell_view::created_at() const at ./build/release/gen/idl/mutation.dist.impl.hh:342
 (inlined by) operator() at ./mutation_partition_view.cc:61
 (inlined by) _ZN5boost6detail7variant14invoke_visitorIKZN12_GLOBAL__N_116read_atomic_cellERK13abstract_typeNS_7variantIN3ser14live_cell_viewEJNS8_18expiring_cell_viewENS8_14dead_cell_viewENS8_17counter_cell_viewENS8_20unknown_variant_typeEEEEN7seastar10bool_classIN11atomic_cell21collection_member_tagEEEE19atomic_cell_visitorLb0EE14internal_visitIRSA_EENS_12disable_if_cIXaaLb0Esr7is_sameIT_SQ_EE5valueESH_E4typeEOSQ_i at /usr/include/boost/variant/variant.hpp:1028
 (inlined by) boost::detail::variant::invoke_visitor<(anonymous namespace)::read_atomic_cell(abstract_type const&, boost::variant<ser::live_cell_view, ser::expiring_cell_view, ser::dead_cell_view, ser::counter_cell_view, ser::unknown_variant_type>, seastar::bool_class<atomic_cell::collection_member_tag>)::atomic_cell_visitor const, false>::result_type boost::detail::variant::visitation_impl_invoke_impl<boost::detail::variant::invoke_visitor<(anonymous namespace)::read_atomic_cell(abstract_type const&, boost::variant<ser::live_cell_view, ser::expiring_cell_view, ser::dead_cell_view, ser::counter_cell_view, ser::unknown_variant_type>, seastar::bool_class<atomic_cell::collection_member_tag>)::atomic_cell_visitor const, false>, void*, ser::expiring_cell_view>(int, boost::detail::variant::invoke_visitor<(anonymous namespace)::read_atomic_cell(abstract_type const&, boost::variant<ser::live_cell_view, ser::expiring_cell_view, ser::dead_cell_view, ser::counter_cell_view, ser::unknown_variant_type>, seastar::bool_class<atomic_cell::collection_member_tag>)::atomic_cell_visitor const, false>&, void*, ser::expiring_cell_view*, mpl_::bool_<true>) at /usr/include/boost/variant/detail/visitation_impl.hpp:117
 (inlined by) boost::detail::variant::invoke_visitor<(anonymous namespace)::read_atomic_cell(abstract_type const&, boost::variant<ser::live_cell_view, ser::expiring_cell_view, ser::dead_cell_view, ser::counter_cell_view, ser::unknown_variant_type>, seastar::bool_class<atomic_cell::collection_member_tag>)::atomic_cell_visitor const, false>::result_type boost::detail::variant::visitation_impl_invoke<boost::detail::variant::invoke_visitor<(anonymous namespace)::read_atomic_cell(abstract_type const&, boost::variant<ser::live_cell_view, ser::expiring_cell_view, ser::dead_cell_view, ser::counter_cell_view, ser::unknown_variant_type>, seastar::bool_class<atomic_cell::collection_member_tag>)::atomic_cell_visitor const, false>, void*, ser::expiring_cell_view, boost::variant<ser::live_cell_view, ser::expiring_cell_view, ser::dead_cell_view, ser::counter_cell_view, ser::unknown_variant_type>::has_fallback_type_>(int, boost::detail::variant::invoke_visitor<(anonymous namespace)::read_atomic_cell(abstract_type const&, boost::variant<ser::live_cell_view, ser::expiring_cell_view, ser::dead_cell_view, ser::counter_cell_view, ser::unknown_variant_type>, seastar::bool_class<atomic_cell::collection_member_tag>)::atomic_cell_visitor const, false>&, void*, ser::expiring_cell_view*, boost::variant<ser::live_cell_view, ser::expiring_cell_view, ser::dead_cell_view, ser::counter_cell_view, ser::unknown_variant_type>::has_fallback_type_, int) at /usr/include/boost/variant/detail/visitation_impl.hpp:157
 (inlined by) boost::detail::variant::invoke_visitor<(anonymous namespace)::read_atomic_cell(abstract_type const&, boost::variant<ser::live_cell_view, ser::expiring_cell_view, ser::dead_cell_view, ser::counter_cell_view, ser::unknown_variant_type>, seastar::bool_class<atomic_cell::collection_member_tag>)::atomic_cell_visitor const, false>::result_type boost::detail::variant::visitation_impl<mpl_::int_<0>, boost::detail::variant::visitation_impl_step<boost::mpl::l_iter<boost::mpl::l_item<mpl_::long_<5l>, ser::live_cell_view, boost::mpl::l_item<mpl_::long_<4l>, ser::expiring_cell_view, boost::mpl::l_item<mpl_::long_<3l>, ser::dead_cell_view, boost::mpl::l_item<mpl_::long_<2l>, ser::counter_cell_view, boost::mpl::l_item<mpl_::long_<1l>, ser::unknown_variant_type, boost::mpl::l_end> > > > > >, boost::mpl::l_iter<boost::mpl::l_end> >, boost::detail::variant::invoke_visitor<(anonymous namespace)::read_atomic_cell(abstract_type const&, boost::variant<ser::live_cell_view, ser::expiring_cell_view, ser::dead_cell_view, ser::counter_cell_view, ser::unknown_variant_type>, seastar::bool_class<atomic_cell::collection_member_tag>)::atomic_cell_visitor const, false>, void*, boost::variant<ser::live_cell_view, ser::expiring_cell_view, ser::dead_cell_view, ser::counter_cell_view, ser::unknown_variant_type>::has_fallback_type_>(int, int, boost::detail::variant::invoke_visitor<(anonymous namespace)::read_atomic_cell(abstract_type const&, boost::variant<ser::live_cell_view, ser::expiring_cell_view, ser::dead_cell_view, ser::counter_cell_view, ser::unknown_variant_type>, seastar::bool_class<atomic_cell::collection_member_tag>)::atomic_cell_visitor const, false>&, void*, mpl_::bool_<false>, boost::variant<ser::live_cell_view, ser::expiring_cell_view, ser::dead_cell_view, ser::counter_cell_view, ser::unknown_variant_type>::has_fallback_type_, mpl_::int_<0>*, boost::detail::variant::visitation_impl_step<boost::mpl::l_iter<boost::mpl::l_item<mpl_::long_<5l>, ser::live_cell_view, boost::mpl::l_item<mpl_::long_<4l>, ser::expiring_cell_view, boost::mpl::l_item<mpl_::long_<3l>, ser::dead_cell_view, boost::mpl::l_item<mpl_::long_<2l>, ser::counter_cell_view, boost::mpl::l_item<mpl_::long_<1l>, ser::unknown_variant_type, boost::mpl::l_end> > > > > >, boost::mpl::l_iter<boost::mpl::l_end> >*) at /usr/include/boost/variant/detail/visitation_impl.hpp:238
 (inlined by) boost::detail::variant::invoke_visitor<(anonymous namespace)::read_atomic_cell(abstract_type const&, boost::variant<ser::live_cell_view, ser::expiring_cell_view, ser::dead_cell_view, ser::counter_cell_view, ser::unknown_variant_type>, seastar::bool_class<atomic_cell::collection_member_tag>)::atomic_cell_visitor const, false>::result_type boost::variant<ser::live_cell_view, ser::expiring_cell_view, ser::dead_cell_view, ser::counter_cell_view, ser::unknown_variant_type>::internal_apply_visitor_impl<boost::detail::variant::invoke_visitor<(anonymous namespace)::read_atomic_cell(abstract_type const&, boost::variant<ser::live_cell_view, ser::expiring_cell_view, ser::dead_cell_view, ser::counter_cell_view, ser::unknown_variant_type>, seastar::bool_class<atomic_cell::collection_member_tag>)::atomic_cell_visitor const, false>, void*>(int, int, boost::detail::variant::invoke_visitor<(anonymous namespace)::read_atomic_cell(abstract_type const&, boost::variant<ser::live_cell_view, ser::expiring_cell_view, ser::dead_cell_view, ser::counter_cell_view, ser::unknown_variant_type>, seastar::bool_class<atomic_cell::collection_member_tag>)::atomic_cell_visitor const, false>&, void*) at /usr/include/boost/variant/variant.hpp:2337
 (inlined by) boost::detail::variant::invoke_visitor<(anonymous namespace)::read_atomic_cell(abstract_type const&, boost::variant<ser::live_cell_view, ser::expiring_cell_view, ser::dead_cell_view, ser::counter_cell_view, ser::unknown_variant_type>, seastar::bool_class<atomic_cell::collection_member_tag>)::atomic_cell_visitor const, false>::result_type boost::variant<ser::live_cell_view, ser::expiring_cell_view, ser::dead_cell_view, ser::counter_cell_view, ser::unknown_variant_type>::internal_apply_visitor<boost::detail::variant::invoke_visitor<(anonymous namespace)::read_atomic_cell(abstract_type const&, boost::variant<ser::live_cell_view, ser::expiring_cell_view, ser::dead_cell_view, ser::counter_cell_view, ser::unknown_variant_type>, seastar::bool_class<atomic_cell::collection_member_tag>)::atomic_cell_visitor const, false> >(boost::detail::variant::invoke_visitor<(anonymous namespace)::read_atomic_cell(abstract_type const&, boost::variant<ser::live_cell_view, ser::expiring_cell_view, ser::dead_cell_view, ser::counter_cell_view, ser::unknown_variant_type>, seastar::bool_class<atomic_cell::collection_member_tag>)::atomic_cell_visitor const, false>&) at /usr/include/boost/variant/variant.hpp:2349
 (inlined by) (anonymous namespace)::read_atomic_cell(abstract_type const&, boost::variant<ser::live_cell_view, ser::expiring_cell_view, ser::dead_cell_view, ser::counter_cell_view, ser::unknown_variant_type>, seastar::bool_class<atomic_cell::collection_member_tag>)::atomic_cell_visitor const::result_type boost::variant<ser::live_cell_view, ser::expiring_cell_view, ser::dead_cell_view, ser::counter_cell_view, ser::unknown_variant_type>::apply_visitor<(anonymous namespace)::read_atomic_cell(abstract_type const&, boost::variant<ser::live_cell_view, ser::expiring_cell_view, ser::dead_cell_view, ser::counter_cell_view, ser::unknown_variant_type>, seastar::bool_class<atomic_cell::collection_member_tag>)::atomic_cell_visitor const>((anonymous namespace)::read_atomic_cell(abstract_type const&, boost::variant<ser::live_cell_view, ser::expiring_cell_view, ser::dead_cell_view, ser::counter_cell_view, ser::unknown_variant_type>, seastar::bool_class<atomic_cell::collection_member_tag>)::atomic_cell_visitor const&) & at /usr/include/boost/variant/variant.hpp:2393
(anonymous namespace)::read_atomic_cell(abstract_type const&, boost::variant<ser::live_cell_view, ser::expiring_cell_view, ser::dead_cell_view, ser::counter_cell_view, ser::unknown_variant_type>, seastar::bool_class<atomic_cell::collection_member_tag>)::atomic_cell_visitor::result_type boost::apply_visitor<(anonymous namespace)::read_atomic_cell(abstract_type const&, boost::variant<ser::live_cell_view, ser::expiring_cell_view, ser::dead_cell_view, ser::counter_cell_view, ser::unknown_variant_type>, seastar::bool_class<atomic_cell::collection_member_tag>)::atomic_cell_visitor, boost::variant<ser::live_cell_view, ser::expiring_cell_view, ser::dead_cell_view, ser::counter_cell_view, ser::unknown_variant_type>&>((anonymous namespace)::read_atomic_cell(abstract_type const&, boost::variant<ser::live_cell_view, ser::expiring_cell_view, ser::dead_cell_view, ser::counter_cell_view, ser::unknown_variant_type>, seastar::bool_class<atomic_cell::collection_member_tag>)::atomic_cell_visitor const&, boost::variant<ser::live_cell_view, ser::expiring_cell_view, ser::dead_cell_view, ser::counter_cell_view, ser::unknown_variant_type>&) at /usr/include/boost/variant/detail/apply_visitor_unary.hpp:68
 (inlined by) (anonymous namespace)::read_atomic_cell(abstract_type const&, boost::variant<ser::live_cell_view, ser::expiring_cell_view, ser::dead_cell_view, ser::counter_cell_view, ser::unknown_variant_type>, seastar::bool_class<atomic_cell::collection_member_tag>) at ./mutation_partition_view.cc:96
 (inlined by) operator() at ./mutation_partition_view.cc:149
 (inlined by) _ZN5boost6detail7variant14invoke_visitorIKZN12_GLOBAL__N_118read_and_visit_rowIZNK23mutation_partition_view9do_acceptI17partition_builderEEvRK14column_mappingRT_E12cell_visitorEEvN3ser8row_viewESA_11column_kindOSB_E33atomic_cell_or_collection_visitorLb0EE14internal_visitIRNS_7variantINSE_14live_cell_viewEJNSE_18expiring_cell_viewENSE_14dead_cell_viewENSE_17counter_cell_viewENSE_20unknown_variant_typeEEEEEENS_12disable_if_cIXaaLb0Esr7is_sameISB_SB_EE5valueEvE4typeESH_i at /usr/include/boost/variant/variant.hpp:1028
 (inlined by) _ZN5boost6detail7variant27visitation_impl_invoke_implINS1_14invoke_visitorIKZN12_GLOBAL__N_118read_and_visit_rowIZNK23mutation_partition_view9do_acceptI17partition_builderEEvRK14column_mappingRT_E12cell_visitorEEvN3ser8row_viewESB_11column_kindOSC_E33atomic_cell_or_collection_visitorLb0EEEPvNS_7variantINSF_14live_cell_viewEJNSF_18expiring_cell_viewENSF_14dead_cell_viewENSF_17counter_cell_viewENSF_20unknown_variant_typeEEEEEENSC_11result_typeEiSD_T0_PT1_N4mpl_5bool_ILb1EEE at /usr/include/boost/variant/detail/visitation_impl.hpp:117
 (inlined by) _ZN5boost6detail7variant22visitation_impl_invokeINS1_14invoke_visitorIKZN12_GLOBAL__N_118read_and_visit_rowIZNK23mutation_partition_view9do_acceptI17partition_builderEEvRK14column_mappingRT_E12cell_visitorEEvN3ser8row_viewESB_11column_kindOSC_E33atomic_cell_or_collection_visitorLb0EEEPvNS_7variantINSF_14live_cell_viewEJNSF_18expiring_cell_viewENSF_14dead_cell_viewENSF_17counter_cell_viewENSF_20unknown_variant_typeEEEENSN_IST_JNSF_20collection_cell_viewESS_EE18has_fallback_type_EEENSC_11result_typeEiSD_T0_PT1_T2_i at /usr/include/boost/variant/detail/visitation_impl.hpp:157
 (inlined by) boost::detail::variant::invoke_visitor<(anonymous namespace)::read_and_visit_row<mutation_partition_view::do_accept<partition_builder>(column_mapping const&, partition_builder&) const::cell_visitor>(ser::row_view, column_mapping const&, column_kind, partition_builder&&)::atomic_cell_or_collection_visitor const, false>::result_type boost::detail::variant::visitation_impl<mpl_::int_<0>, boost::detail::variant::visitation_impl_step<boost::mpl::l_iter<boost::mpl::l_item<mpl_::long_<3l>, boost::variant<ser::live_cell_view, ser::expiring_cell_view, ser::dead_cell_view, ser::counter_cell_view, ser::unknown_variant_type>, boost::mpl::l_item<mpl_::long_<2l>, ser::collection_cell_view, boost::mpl::l_item<mpl_::long_<1l>, ser::unknown_variant_type, boost::mpl::l_end> > > >, boost::mpl::l_iter<boost::mpl::l_end> >, boost::detail::variant::invoke_visitor<(anonymous namespace)::read_and_visit_row<mutation_partition_view::do_accept<partition_builder>(column_mapping const&, partition_builder&) const::cell_visitor>(ser::row_view, column_mapping const&, column_kind, partition_builder&&)::atomic_cell_or_collection_visitor const, false>, void*, boost::variant<boost::variant<ser::live_cell_view, ser::expiring_cell_view, ser::dead_cell_view, ser::counter_cell_view, ser::unknown_variant_type>, ser::collection_cell_view, ser::unknown_variant_type>::has_fallback_type_>(int, int, boost::detail::variant::invoke_visitor<(anonymous namespace)::read_and_visit_row<mutation_partition_view::do_accept<partition_builder>(column_mapping const&, partition_builder&) const::cell_visitor>(ser::row_view, column_mapping const&, column_kind, partition_builder&&)::atomic_cell_or_collection_visitor const, false>&, void*, mpl_::bool_<false>, boost::variant<boost::variant<ser::live_cell_view, ser::expiring_cell_view, ser::dead_cell_view, ser::counter_cell_view, ser::unknown_variant_type>, ser::collection_cell_view, ser::unknown_variant_type>::has_fallback_type_, mpl_::int_<0>*, boost::detail::variant::visitation_impl_step<boost::mpl::l_iter<boost::mpl::l_item<mpl_::long_<3l>, boost::variant<ser::live_cell_view, ser::expiring_cell_view, ser::dead_cell_view, ser::counter_cell_view, ser::unknown_variant_type>, boost::mpl::l_item<mpl_::long_<2l>, ser::collection_cell_view, boost::mpl::l_item<mpl_::long_<1l>, ser::unknown_variant_type, boost::mpl::l_end> > > >, boost::mpl::l_iter<boost::mpl::l_end> >*) at /usr/include/boost/variant/detail/visitation_impl.hpp:238
 (inlined by) _ZN5boost7variantINS0_IN3ser14live_cell_viewEJNS1_18expiring_cell_viewENS1_14dead_cell_viewENS1_17counter_cell_viewENS1_20unknown_variant_typeEEEEJNS1_20collection_cell_viewES6_EE27internal_apply_visitor_implINS_6detail7variant14invoke_visitorIKZN12_GLOBAL__N_118read_and_visit_rowIZNK23mutation_partition_view9do_acceptI17partition_builderEEvRK14column_mappingRT_E12cell_visitorEEvNS1_8row_viewESL_11column_kindOSM_E33atomic_cell_or_collection_visitorLb0EEEPvEENSM_11result_typeEiiSN_T0_ at /usr/include/boost/variant/variant.hpp:2337
 (inlined by) _ZN5boost7variantINS0_IN3ser14live_cell_viewEJNS1_18expiring_cell_viewENS1_14dead_cell_viewENS1_17counter_cell_viewENS1_20unknown_variant_typeEEEEJNS1_20collection_cell_viewES6_EE22internal_apply_visitorINS_6detail7variant14invoke_visitorIKZN12_GLOBAL__N_118read_and_visit_rowIZNK23mutation_partition_view9do_acceptI17partition_builderEEvRK14column_mappingRT_E12cell_visitorEEvNS1_8row_viewESL_11column_kindOSM_E33atomic_cell_or_collection_visitorLb0EEEEENSM_11result_typeESN_ at /usr/include/boost/variant/variant.hpp:2349
 (inlined by) _ZNR5boost7variantINS0_IN3ser14live_cell_viewEJNS1_18expiring_cell_viewENS1_14dead_cell_viewENS1_17counter_cell_viewENS1_20unknown_variant_typeEEEEJNS1_20collection_cell_viewES6_EE13apply_visitorIKZN12_GLOBAL__N_118read_and_visit_rowIZNK23mutation_partition_view9do_acceptI17partition_builderEEvRK14column_mappingRT_E12cell_visitorEEvNS1_8row_viewESI_11column_kindOSJ_E33atomic_cell_or_collection_visitorEENSJ_11result_typeESK_ at /usr/include/boost/variant/variant.hpp:2393
 (inlined by) _ZN5boost13apply_visitorIZN12_GLOBAL__N_118read_and_visit_rowIZNK23mutation_partition_view9do_acceptI17partition_builderEEvRK14column_mappingRT_E12cell_visitorEEvN3ser8row_viewES8_11column_kindOS9_E33atomic_cell_or_collection_visitorRNS_7variantINSH_INSC_14live_cell_viewEJNSC_18expiring_cell_viewENSC_14dead_cell_viewENSC_17counter_cell_viewENSC_20unknown_variant_typeEEEEJNSC_20collection_cell_viewESM_EEEEENS9_11result_typeERKS9_OT0_ at /usr/include/boost/variant/detail/apply_visitor_unary.hpp:68
 (inlined by) void (anonymous namespace)::read_and_visit_row<mutation_partition_view::do_accept<partition_builder>(column_mapping const&, partition_builder&) const::cell_visitor>(ser::row_view, column_mapping const&, column_kind, partition_builder&&) at ./mutation_partition_view.cc:169
 (inlined by) void mutation_partition_view::do_accept<partition_builder>(column_mapping const&, partition_builder&) const at ./mutation_partition_view.cc:235
frozen_mutation::unfreeze(seastar::lw_shared_ptr<schema const>) const at ./frozen_mutation.cc:105
service::data_read_resolver::resolve(seastar::lw_shared_ptr<schema const>, query::read_command const&, unsigned long, unsigned long, unsigned int) at ./service/storage_proxy.cc:3440
operator() at ./service/storage_proxy.cc:3696
operator() at ././seastar/include/seastar/core/future.hh:1674
 (inlined by) void seastar::futurize<void>::satisfy_with_result_of<seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception>, utils::exception_container_throw_policy> >::then_wrapped_nrvo<void, service::abstract_read_executor::reconcile(db::consistency_level, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::lw_shared_ptr<query::read_command>)::{lambda(seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception>, utils::exception_container_throw_policy> >)#1}>(service::abstract_read_executor::reconcile(db::consistency_level, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::lw_shared_ptr<query::read_command>)::{lambda(seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception>, utils::exception_container_throw_policy> >)#1}&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, {lambda(seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception>, utils::exception_container_throw_policy> >)#1}&, seastar::future_state<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception>, utils::exception_container_throw_policy> >&&)#1}::operator()(seastar::internal::promise_base_with_type<void>, seastar::internal::promise_base_with_type<void>&&, seastar::future_state<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception>, utils::exception_container_throw_policy> >) const::{lambda()#1}>(seastar::internal::promise_base_with_type<void>, service::abstract_read_executor::reconcile(db::consistency_level, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::lw_shared_ptr<query::read_command>)::{lambda(seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception>, utils::exception_container_throw_policy> >)#1}&&) at ././seastar/include/seastar/core/future.hh:2123
 (inlined by) operator() at ././seastar/include/seastar/core/future.hh:1673
 (inlined by) seastar::continuation<seastar::internal::promise_base_with_type<void>, service::abstract_read_executor::reconcile(db::consistency_level, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, seastar::lw_shared_ptr<query::read_command>)::{lambda(seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception>, utils::exception_container_throw_policy> >)#1}, seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception>, utils::exception_container_throw_policy> >::then_wrapped_nrvo<void, {lambda(seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception>, utils::exception_container_throw_policy> >)#1}>({lambda(seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception>, utils::exception_container_throw_policy> >)#1}&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, {lambda(seastar::future<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception>, utils::exception_container_throw_policy> >)#1}&, seastar::future_state<boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception>, utils::exception_container_throw_policy> >&&)#1}, boost::outcome_v2::basic_result<void, utils::exception_container<exceptions::mutation_write_timeout_exception, exceptions::read_timeout_exception, exceptions::read_failure_exception>, utils::exception_container_throw_policy> >::run_and_dispose() at ././seastar/include/seastar/core/future.hh:773
seastar::reactor::run_tasks(seastar::reactor::task_queue&) at ./build/release/seastar/./seastar/src/core/reactor.cc:2344
 (inlined by) seastar::reactor::run_some_tasks() at ./build/release/seastar/./seastar/src/core/reactor.cc:2754
seastar::reactor::do_run() at ./build/release/seastar/./seastar/src/core/reactor.cc:2923
operator() at ./build/release/seastar/./seastar/src/core/reactor.cc:4135
 (inlined by) void std::__invoke_impl<void, seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_89&>(std::__invoke_other, seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_89&) at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/invoke.h:61
 (inlined by) std::enable_if<is_invocable_r_v<void, seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_89&>, void>::type std::__invoke_r<void, seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_89&>(seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_89&) at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/invoke.h:111
 (inlined by) std::_Function_handler<void (), seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_89>::_M_invoke(std::_Any_data const&) at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/std_function.h:290
std::function<void ()>::operator()() const at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/std_function.h:590
 (inlined by) seastar::posix_thread::start_routine(void*) at ./build/release/seastar/./seastar/src/core/posix.cc:60
?? ??:0
?? ??:0

Allow yielding in data_read_resolver::resolve to prevent reactor stalls. TODO: unfreeze_gently, to prevent stalls due to large partitions. Refs scylladb#2361 Signed-off-by: Benny Halevy <bhalevy@scylladb.com>

And use in data_read_resolver::resolve Fixes scylladb#2361 Signed-off-by: Benny Halevy <bhalevy@scylladb.com>

Allow yielding in data_read_resolver::resolve to prevent reactor stalls. TODO: unfreeze_gently, to prevent stalls due to large partitions. Refs scylladb#2361 Signed-off-by: Benny Halevy <bhalevy@scylladb.com>

… from Benny Halevy This series futurizes two synchronous functions used for data reconciliation: `data_read_resolver::resolve` and `to_data_query_result` and does so by introducing lower-level asynchronous infrastructure: `mutation_partition_view::accept_gently`, `frozen_mutation::unfreeze_gently` and `frozen_mutation::consume_gently`, and `mutation::consume_gently`. This trades some cycles on this cold path to prevent known reactor stalls. Fixes #2361 Fixes #10038 Closes #10482 * github.com:scylladb/scylla: mutation: add consume_gently frozen_mutation: add consume_gently query: coroutinize to_data_query_result frozen_mutation: add unfreeze_gently mutation_partition_view: add accept_gently methods storage_proxy: futurize data_read_resolver::resolve

avikivity · 2022-07-03T11:42:58Z

Performance only, and not a regression, so not backporting.

avikivity mentioned this issue May 9, 2017

compactionhistory API implementation doesn't use paging, induces latency #2366

Closed

slivne added the area/stability label Jul 5, 2017

slivne added the symptom/performance Issues causing performance problems label Dec 3, 2017

slivne added this to the 2.x milestone Dec 3, 2017

bhalevy added a commit to bhalevy/scylla that referenced this issue May 3, 2022

frozen_mutation: add unfreeze_gently

5ee721a

And use in data_read_resolver::resolve Fixes scylladb#2361 Signed-off-by: Benny Halevy <bhalevy@scylladb.com>

bhalevy mentioned this issue May 3, 2022

Futurize data_read_resolver::resolve and to_data_query_result #10482

Merged

scylladb-promoter closed this as completed in e12454f May 6, 2022

scylladb-promoter added the Backport candidate label May 6, 2022

avikivity removed the Backport candidate label Jul 3, 2022

DoronArazii modified the milestones: 5.x, 5.1 Jul 14, 2022

scylladb-promoter added the Backport candidate label Jul 21, 2022

avikivity removed the Backport candidate label Aug 16, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

data_read_resolver::resolve() computes for hundreds of milliseconds #2361

data_read_resolver::resolve() computes for hundreds of milliseconds #2361

avikivity commented May 8, 2017

avikivity commented May 8, 2017

gleb-cloudius commented May 8, 2017

gleb-cloudius commented May 8, 2017

gleb-cloudius commented May 8, 2017

pdziepak commented May 8, 2017

avikivity commented May 8, 2017

gleb-cloudius commented May 8, 2017 via email

avikivity commented May 8, 2017

gleb-cloudius commented May 8, 2017 via email

gleb-cloudius commented Jun 12, 2017

bhalevy commented May 2, 2022

avikivity commented Jul 3, 2022

data_read_resolver::resolve() computes for hundreds of milliseconds #2361

data_read_resolver::resolve() computes for hundreds of milliseconds #2361

Comments

avikivity commented May 8, 2017

avikivity commented May 8, 2017

gleb-cloudius commented May 8, 2017

gleb-cloudius commented May 8, 2017

gleb-cloudius commented May 8, 2017

pdziepak commented May 8, 2017

avikivity commented May 8, 2017

gleb-cloudius commented May 8, 2017 via email

avikivity commented May 8, 2017

gleb-cloudius commented May 8, 2017 via email

gleb-cloudius commented Jun 12, 2017

bhalevy commented May 2, 2022

avikivity commented Jul 3, 2022