Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

test_scrub_with_one_node_expect_data_loss: shared_ptr accessed on non-owner cpu #14515

Closed
bhalevy opened this issue Jul 5, 2023 · 5 comments
Closed

test_scrub_with_one_node_expect_data_loss: shared_ptr accessed on non-owner cpu #14515

bhalevy opened this issue Jul 5, 2023 · 5 comments
Assignees
@Deexie
Labels
status/regression
Milestone
5.4

Comments

@bhalevy
Copy link
Member

bhalevy commented Jul 5, 2023

Seen for example in https://jenkins.scylladb.com/job/scylla-master/job/dtest-daily-debug/242/artifact/logs-full.debug.009/1688378358842_nodetool_additional_test.py%3A%3ATestNodetool%3A%3Atest_scrub_with_one_node_expect_data_loss/node1.log

Scylla version 5.4.0~dev-0.20230703.1ab2bb69b8a6 with build-id dd05203d6175c24d1fd7a88a100d3a8e6760d2a5 starting ...

ERROR 2023-07-03 09:59:15,171 [shard 0] seastar - shared_ptr accessed on non-owner cpu, at: 0xd417f50 /jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.ccm/scylla-repository/1ab2bb69b8a6a05d9a29d3464aad03d4eabed963/libreloc/libseastar.so+0x556298c /jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.ccm/scylla-repository/1ab2bb69b8a6a05d9a29d3464aad03d4eabed963/libreloc/libseastar.so+0x55626e7 /jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.ccm/scylla-repository/1ab2bb69b8a6a05d9a29d3464aad03d4eabed963/libreloc/libseastar.so+0x5564a39 /jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.ccm/scylla-repository/1ab2bb69b8a6a05d9a29d3464aad03d4eabed963/libreloc/libseastar.so+0x55654af /jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.ccm/scylla-repository/1ab2bb69b8a6a05d9a29d3464aad03d4eabed963/libreloc/libseastar.so+0x44427c0 /jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.ccm/scylla-repository/1ab2bb69b8a6a05d9a29d3464aad03d4eabed963/libreloc/libseastar.so+0x444330e 0xd569755 0x118e9c3b 0x1195a1ad 0x11959fbd 0x11959d72 0x11959b98 /jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.ccm/scylla-repository/1ab2bb69b8a6a05d9a29d3464aad03d4eabed963/libreloc/libseastar.so+0x46755f8 /jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.ccm/scylla-repository/1ab2bb69b8a6a05d9a29d3464aad03d4eabed963/libreloc/libseastar.so+0x467d6ce /jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.ccm/scylla-repository/1ab2bb69b8a6a05d9a29d3464aad03d4eabed963/libreloc/libseastar.so+0x4681710 /jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.ccm/scylla-repository/1ab2bb69b8a6a05d9a29d3464aad03d4eabed963/libreloc/libseastar.so+0x467f336 /jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.ccm/scylla-repository/1ab2bb69b8a6a05d9a29d3464aad03d4eabed963/libreloc/libseastar.so+0x4200a40 /jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.ccm/scylla-repository/1ab2bb69b8a6a05d9a29d3464aad03d4eabed963/libreloc/libseastar.so+0x41fe2da 0xd4ad79f 0xd5600f5 0xd4ab26a /jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.ccm/scylla-repository/1ab2bb69b8a6a05d9a29d3464aad03d4eabed963/libreloc/libc.so.6+0x2750f /jenkins/workspace/scylla-master/dtest-daily-debug/scylla/.ccm/scylla-repository/1ab2bb69b8a6a05d9a29d3464aad03d4eabed963/libreloc/libc.so.6+0x275c8 0xd3ae6a4

Decoded:

[Backtrace #0]
__interceptor_backtrace.part.0 at ??:?
void seastar::backtrace<seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}>(seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}&&) at ./build/debug/seastar/./seastar/include/seastar/util/backtrace.hh:64
seastar::backtrace_buffer::append_backtrace() at ./build/debug/seastar/./seastar/src/core/reactor.cc:811
seastar::print_with_backtrace(seastar::backtrace_buffer&, bool) at ./build/debug/seastar/./seastar/src/core/reactor.cc:841
seastar::print_with_backtrace(char const*, bool) at ./build/debug/seastar/./seastar/src/core/reactor.cc:853
seastar::sigabrt_action() at ./build/debug/seastar/./seastar/src/core/reactor.cc:3969
operator() at ./build/debug/seastar/./seastar/src/core/reactor.cc:3945
__invoke at ./build/debug/seastar/./seastar/src/core/reactor.cc:3941
/data/scylla-s3-reloc.cache/by-build-id/dd05203d6175c24d1fd7a88a100d3a8e6760d2a5/extracted/scylla/libreloc/libc.so.6: ELF 64-bit LSB shared object, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=81daba31ee66dbd63efdc4252a872949d874d136, for GNU/Linux 3.2.0, not stripped

__GI___sigaction at :?
__pthread_kill_implementation at ??:?
__GI_raise at :?
__GI_abort at :?
seastar::on_fatal_internal_error(seastar::logger&, std::basic_string_view<char, std::char_traits<char> >) at ./build/debug/seastar/./seastar/src/core/on_internal_error.cc:81
seastar::debug_shared_ptr_counter_type::check() const at ././seastar/include/seastar/core/shared_ptr_debug_helper.hh:72
 (inlined by) seastar::debug_shared_ptr_counter_type::operator++() at ././seastar/include/seastar/core/shared_ptr_debug_helper.hh:52
boost::iterators::transform_iterator<api::set_column_family(api::http_context&, seastar::httpd::routes&, seastar::sharded<db::system_keyspace>&)::$_117::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(std::unordered_set<seastar::basic_sstring<char, unsigned int, 15u, true>, std::hash<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::equal_to<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::allocator<seastar::basic_sstring<char, unsigned int, 15u, true> > >, std::unordered_set<seastar::lw_shared_ptr<sstables::sstable>, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<seastar::lw_shared_ptr<sstables::sstable> > >&&)#1}::operator()(std::unordered_set<seastar::basic_sstring<char, unsigned int, 15u, true>, std::hash<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::equal_to<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::allocator<seastar::basic_sstring<char, unsigned int, 15u, true> > >, std::unordered_set<seastar::lw_shared_ptr<sstables::sstable>, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<seastar::lw_shared_ptr<sstables::sstable> > >&&)::{lambda(auto:1)#1}, std::__detail::_Node_iterator<seastar::lw_shared_ptr<sstables::sstable>, true, true>, boost::use_default, std::__detail::_Node_iterator>::dereference() const at /usr/include/boost/iterator/transform_iterator.hpp:126
 (inlined by) {lambda(std::unordered_set<seastar::basic_sstring<char, unsigned int, 15u, true>, std::hash<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::equal_to<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::allocator<seastar::basic_sstring<char, unsigned int, 15u, true> > >, std::unordered_set<seastar::lw_shared_ptr<sstables::sstable>, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<seastar::lw_shared_ptr<sstables::sstable> > >&&)#1}::reference boost::iterators::iterator_core_access::dereference<boost::iterators::transform_iterator<api::set_column_family(api::http_context&, seastar::httpd::routes&, seastar::sharded<db::system_keyspace>&)::$_117::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(std::unordered_set<seastar::basic_sstring<char, unsigned int, 15u, true>, std::hash<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::equal_to<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::allocator<seastar::basic_sstring<char, unsigned int, 15u, true> > >, std::unordered_set<seastar::lw_shared_ptr<sstables::sstable>, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<seastar::lw_shared_ptr<sstables::sstable> > >&&)#1}::operator()(std::unordered_set<seastar::basic_sstring<char, unsigned int, 15u, true>, std::hash<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::equal_to<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::allocator<seastar::basic_sstring<char, unsigned int, 15u, true> > >, std::unordered_set<seastar::lw_shared_ptr<sstables::sstable>, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<seastar::lw_shared_ptr<sstables::sstable> > >&&)::{lambda(auto:1)#1}, std::__detail::_Node_iterator<seastar::lw_shared_ptr<sstables::sstable>, true, true>, boost::use_default, std::__detail::_Node_iterator> >({lambda(std::unordered_set<seastar::basic_sstring<char, unsigned int, 15u, true>, std::hash<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::equal_to<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::allocator<seastar::basic_sstring<char, unsigned int, 15u, true> > >, std::unordered_set<seastar::lw_shared_ptr<sstables::sstable>, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<seastar::lw_shared_ptr<sstables::sstable> > >&&)#1} const&) at /usr/include/boost/iterator/iterator_facade.hpp:550
 (inlined by) boost::iterators::detail::iterator_facade_base<boost::iterators::transform_iterator<api::set_column_family(api::http_context&, seastar::httpd::routes&, seastar::sharded<db::system_keyspace>&)::$_117::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(std::unordered_set<seastar::basic_sstring<char, unsigned int, 15u, true>, std::hash<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::equal_to<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::allocator<seastar::basic_sstring<char, unsigned int, 15u, true> > >, std::unordered_set<seastar::lw_shared_ptr<sstables::sstable>, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<seastar::lw_shared_ptr<sstables::sstable> > >&&)#1}::operator()(std::unordered_set<seastar::basic_sstring<char, unsigned int, 15u, true>, std::hash<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::equal_to<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::allocator<seastar::basic_sstring<char, unsigned int, 15u, true> > >, std::unordered_set<seastar::lw_shared_ptr<sstables::sstable>, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<seastar::lw_shared_ptr<sstables::sstable> > >&&)::{lambda(auto:1)#1}, std::__detail::_Node_iterator<seastar::lw_shared_ptr<sstables::sstable>, true, true>, boost::use_default, std::__detail::_Node_iterator>, seastar::basic_sstring<char, unsigned int, 15u, true>, boost::iterators::forward_traversal_tag, seastar::basic_sstring<char, unsigned int, 15u, true>, long, false, false>::operator*() const at /usr/include/boost/iterator/iterator_facade.hpp:656
 (inlined by) void std::__detail::_Insert_base<seastar::basic_sstring<char, unsigned int, 15u, true>, seastar::basic_sstring<char, unsigned int, 15u, true>, std::allocator<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::__detail::_Identity, std::equal_to<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::hash<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, true, true> >::_M_insert_range<boost::iterators::transform_iterator<api::set_column_family(api::http_context&, seastar::httpd::routes&, seastar::sharded<db::system_keyspace>&)::$_117::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(std::unordered_set<seastar::basic_sstring<char, unsigned int, 15u, true>, std::hash<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::equal_to<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::allocator<seastar::basic_sstring<char, unsigned int, 15u, true> > >, std::unordered_set<seastar::lw_shared_ptr<sstables::sstable>, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<seastar::lw_shared_ptr<sstables::sstable> > >&&)#1}::operator()(std::unordered_set<seastar::basic_sstring<char, unsigned int, 15u, true>, std::hash<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::equal_to<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::allocator<seastar::basic_sstring<char, unsigned int, 15u, true> > >, std::unordered_set<seastar::lw_shared_ptr<sstables::sstable>, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<seastar::lw_shared_ptr<sstables::sstable> > >&&)::{lambda(auto:1)#1}, std::__detail::_Node_iterator<seastar::lw_shared_ptr<sstables::sstable>, true, true>, boost::use_default, std::__detail::_Node_iterator>, std::__detail::_AllocNode<std::allocator<std::__detail::_Hash_node<seastar::basic_sstring<char, unsigned int, 15u, true>, true> > > >({lambda(std::unordered_set<seastar::basic_sstring<char, unsigned int, 15u, true>, std::hash<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::equal_to<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::allocator<seastar::basic_sstring<char, unsigned int, 15u, true> > >, std::unordered_set<seastar::lw_shared_ptr<sstables::sstable>, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<seastar::lw_shared_ptr<sstables::sstable> > >&&)#1}, {lambda(std::unordered_set<seastar::basic_sstring<char, unsigned int, 15u, true>, std::hash<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::equal_to<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::allocator<seastar::basic_sstring<char, unsigned int, 15u, true> > >, std::unordered_set<seastar::lw_shared_ptr<sstables::sstable>, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<seastar::lw_shared_ptr<sstables::sstable> > >&&)#1}, std::__detail::_AllocNode<std::allocator<std::__detail::_Hash_node<seastar::basic_sstring<char, unsigned int, 15u, true>, true> > > const&, std::integral_constant<bool, true>) at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/hashtable_policy.h:954
 (inlined by) void std::__detail::_Insert_base<seastar::basic_sstring<char, unsigned int, 15u, true>, seastar::basic_sstring<char, unsigned int, 15u, true>, std::allocator<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::__detail::_Identity, std::equal_to<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::hash<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, true, true> >::insert<boost::iterators::transform_iterator<api::set_column_family(api::http_context&, seastar::httpd::routes&, seastar::sharded<db::system_keyspace>&)::$_117::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(std::unordered_set<seastar::basic_sstring<char, unsigned int, 15u, true>, std::hash<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::equal_to<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::allocator<seastar::basic_sstring<char, unsigned int, 15u, true> > >, std::unordered_set<seastar::lw_shared_ptr<sstables::sstable>, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<seastar::lw_shared_ptr<sstables::sstable> > >&&)#1}::operator()(std::unordered_set<seastar::basic_sstring<char, unsigned int, 15u, true>, std::hash<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::equal_to<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::allocator<seastar::basic_sstring<char, unsigned int, 15u, true> > >, std::unordered_set<seastar::lw_shared_ptr<sstables::sstable>, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<seastar::lw_shared_ptr<sstables::sstable> > >&&)::{lambda(auto:1)#1}, std::__detail::_Node_iterator<seastar::lw_shared_ptr<sstables::sstable>, true, true>, boost::use_default, std::__detail::_Node_iterator> >({lambda(std::unordered_set<seastar::basic_sstring<char, unsigned int, 15u, true>, std::hash<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::equal_to<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::allocator<seastar::basic_sstring<char, unsigned int, 15u, true> > >, std::unordered_set<seastar::lw_shared_ptr<sstables::sstable>, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<seastar::lw_shared_ptr<sstables::sstable> > >&&)#1}, {lambda(std::unordered_set<seastar::basic_sstring<char, unsigned int, 15u, true>, std::hash<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::equal_to<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::allocator<seastar::basic_sstring<char, unsigned int, 15u, true> > >, std::unordered_set<seastar::lw_shared_ptr<sstables::sstable>, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<seastar::lw_shared_ptr<sstables::sstable> > >&&)#1}) at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/hashtable_policy.h:936
 (inlined by) void std::unordered_set<seastar::basic_sstring<char, unsigned int, 15u, true>, std::hash<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::equal_to<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::allocator<seastar::basic_sstring<char, unsigned int, 15u, true> > >::insert<boost::iterators::transform_iterator<api::set_column_family(api::http_context&, seastar::httpd::routes&, seastar::sharded<db::system_keyspace>&)::$_117::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(std::unordered_set<seastar::basic_sstring<char, unsigned int, 15u, true>, std::hash<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::equal_to<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::allocator<seastar::basic_sstring<char, unsigned int, 15u, true> > >, std::unordered_set<seastar::lw_shared_ptr<sstables::sstable>, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<seastar::lw_shared_ptr<sstables::sstable> > >&&)#1}::operator()(std::unordered_set<seastar::basic_sstring<char, unsigned int, 15u, true>, std::hash<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::equal_to<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::allocator<seastar::basic_sstring<char, unsigned int, 15u, true> > >, std::unordered_set<seastar::lw_shared_ptr<sstables::sstable>, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<seastar::lw_shared_ptr<sstables::sstable> > >&&)::{lambda(auto:1)#1}, std::__detail::_Node_iterator<seastar::lw_shared_ptr<sstables::sstable>, true, true>, boost::use_default, std::__detail::_Node_iterator> >({lambda(std::unordered_set<seastar::basic_sstring<char, unsigned int, 15u, true>, std::hash<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::equal_to<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::allocator<seastar::basic_sstring<char, unsigned int, 15u, true> > >, std::unordered_set<seastar::lw_shared_ptr<sstables::sstable>, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<seastar::lw_shared_ptr<sstables::sstable> > >&&)#1}, {lambda(std::unordered_set<seastar::basic_sstring<char, unsigned int, 15u, true>, std::hash<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::equal_to<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::allocator<seastar::basic_sstring<char, unsigned int, 15u, true> > >, std::unordered_set<seastar::lw_shared_ptr<sstables::sstable>, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<seastar::lw_shared_ptr<sstables::sstable> > >&&)#1}) at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/unordered_set.h:470
 (inlined by) operator() at ./api/column_family.cc:1025
 (inlined by) operator()<seastar::future<std::unordered_set<seastar::lw_shared_ptr<sstables::sstable>, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<seastar::lw_shared_ptr<sstables::sstable> > > > > at ././seastar/include/seastar/core/map_reduce.hh:194
 (inlined by) seastar::future<void> seastar::futurize<seastar::future<void> >::invoke<seastar::map_reduce<boost::range_detail::integer_iterator<unsigned int>, seastar::sharded<replica::database>::map_reduce0<api::set_column_family(api::http_context&, seastar::httpd::routes&, seastar::sharded<db::system_keyspace>&)::$_117::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(replica::database&)#1}, std::unordered_set<seastar::basic_sstring<char, unsigned int, 15u, true>, std::hash<seastar::basic_sstring>, std::equal_to<seastar::basic_sstring>, std::allocator<seastar::basic_sstring> >, api::set_column_family(api::http_context&, seastar::httpd::routes&, seastar::sharded<db::system_keyspace>&)::$_117::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(std::allocator<seastar::basic_sstring>, api::set_column_family(api::http_context&, seastar::httpd::routes&, seastar::sharded<db::system_keyspace>&)::$_117::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(replica::database&)#1}<seastar::lw_shared_ptr<sstables::sstable>, seastar::basic_sstring<char, unsigned int, 15u, true><sstables::sstable>, std::hash<seastar::basic_sstring><sstables::sstable>, std::allocator<sstables::sstable> >&&)#1}>(api::set_column_family(api::http_context&, seastar::httpd::routes&, seastar::sharded<db::system_keyspace>&)::$_117::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(replica::database&)#1}, {lambda(std::allocator<seastar::basic_sstring>, api::set_column_family(api::http_context&, seastar::httpd::routes&, seastar::sharded<db::system_keyspace>&)::$_117::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(replica::database&)#1}<seastar::lw_shared_ptr<sstables::sstable>, seastar::basic_sstring<char, unsigned int, 15u, true><sstables::sstable>, std::hash<seastar::basic_sstring><sstables::sstable>, std::allocator<sstables::sstable> >&&)#1}, api::set_column_family(api::http_context&, seastar::httpd::routes&, seastar::sharded<db::system_keyspace>&)::$_117::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(std::allocator<seastar::basic_sstring>, api::set_column_family(api::http_context&, seastar::httpd::routes&, seastar::sharded<db::system_keyspace>&)::$_117::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(replica::database&)#1}<seastar::lw_shared_ptr<sstables::sstable>, seastar::basic_sstring<char, unsigned int, 15u, true><sstables::sstable>, std::hash<seastar::basic_sstring><sstables::sstable>, std::allocator<sstables::sstable> >&&)#1})::{lambda(unsigned int)#1}, std::allocator<seastar::basic_sstring>, api::set_column_family(api::http_context&, seastar::httpd::routes&, seastar::sharded<db::system_keyspace>&)::$_117::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(replica::database&)#1}<seastar::lw_shared_ptr<sstables::sstable>, seastar::basic_sstring<char, unsigned int, 15u, true><sstables::sstable>, std::hash<seastar::basic_sstring><sstables::sstable>, std::allocator<sstables::sstable> >&&>(seastar::sharded<replica::database>::map_reduce0<api::set_column_family(api::http_context&, seastar::httpd::routes&, seastar::sharded<db::system_keyspace>&)::$_117::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(replica::database&)#1}, std::unordered_set<seastar::basic_sstring<char, unsigned int, 15u, true>, std::hash<seastar::basic_sstring>, std::equal_to<seastar::basic_sstring>, std::allocator<seastar::basic_sstring> >, api::set_column_family(api::http_context&, seastar::httpd::routes&, seastar::sharded<db::system_keyspace>&)::$_117::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(std::allocator<seastar::basic_sstring>, api::set_column_family(api::http_context&, seastar::httpd::routes&, seastar::sharded<db::system_keyspace>&)::$_117::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(replica::database&)#1}<seastar::lw_shared_ptr<sstables::sstable>, seastar::basic_sstring<char, unsigned int, 15u, true><sstables::sstable>, std::hash<seastar::basic_sstring><sstables::sstable>, std::allocator<sstables::sstable> >&&)#1}>(api::set_column_family(api::http_context&, seastar::httpd::routes&, seastar::sharded<db::system_keyspace>&)::$_117::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(replica::database&)#1}, {lambda(std::allocator<seastar::basic_sstring>, api::set_column_family(api::http_context&, seastar::httpd::routes&, seastar::sharded<db::system_keyspace>&)::$_117::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(replica::database&)#1}<seastar::lw_shared_ptr<sstables::sstable>, seastar::basic_sstring<char, unsigned int, 15u, true><sstables::sstable>, std::hash<seastar::basic_sstring><sstables::sstable>, std::allocator<sstables::sstable> >&&)#1}, api::set_column_family(api::http_context&, seastar::httpd::routes&, seastar::sharded<db::system_keyspace>&)::$_117::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(std::allocator<seastar::basic_sstring>, api::set_column_family(api::http_context&, seastar::httpd::routes&, seastar::sharded<db::system_keyspace>&)::$_117::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(replica::database&)#1}<seastar::lw_shared_ptr<sstables::sstable>, seastar::basic_sstring<char, unsigned int, 15u, true><sstables::sstable>, std::hash<seastar::basic_sstring><sstables::sstable>, std::allocator<sstables::sstable> >&&)#1})::{lambda(unsigned int)#1}, seastar::sharded<replica::database>::map_reduce0<api::set_column_family(api::http_context&, seastar::httpd::routes&, seastar::sharded<db::system_keyspace>&)::$_117::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(replica::database&)#1}, std::unordered_set<seastar::basic_sstring<char, unsigned int, 15u, true>, std::hash<seastar::basic_sstring>, std::equal_to<seastar::basic_sstring>, std::allocator<seastar::basic_sstring> >, api::set_column_family(api::http_context&, seastar::httpd::routes&, seastar::sharded<db::system_keyspace>&)::$_117::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(std::allocator<seastar::basic_sstring>, api::set_column_family(api::http_context&, seastar::httpd::routes&, seastar::sharded<db::system_keyspace>&)::$_117::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(replica::database&)#1}<seastar::lw_shared_ptr<sstables::sstable>, seastar::basic_sstring<char, unsigned int, 15u, true><sstables::sstable>, std::hash<seastar::basic_sstring><sstables::sstable>, std::allocator<sstables::sstable> >&&)#1}>(api::set_column_family(api::http_context&, seastar::httpd::routes&, seastar::sharded<db::system_keyspace>&)::$_117::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(replica::database&)#1}, {lambda(std::allocator<seastar::basic_sstring>, api::set_column_family(api::http_context&, seastar::httpd::routes&, seastar::sharded<db::system_keyspace>&)::$_117::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(replica::database&)#1}<seastar::lw_shared_ptr<sstables::sstable>, seastar::basic_sstring<char, unsigned int, 15u, true><sstables::sstable>, std::hash<seastar::basic_sstring><sstables::sstable>, std::allocator<sstables::sstable> >&&)#1}, api::set_column_family(api::http_context&, seastar::httpd::routes&, seastar::sharded<db::system_keyspace>&)::$_117::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(std::allocator<seastar::basic_sstring>, api::set_column_family(api::http_context&, seastar::httpd::routes&, seastar::sharded<db::system_keyspace>&)::$_117::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(replica::database&)#1}<seastar::lw_shared_ptr<sstables::sstable>, seastar::basic_sstring<char, unsigned int, 15u, true><sstables::sstable>, std::hash<seastar::basic_sstring><sstables::sstable>, std::allocator<sstables::sstable> >&&)#1})::{lambda(unsigned int)#1}, {lambda(std::allocator<seastar::basic_sstring>, api::set_column_family(api::http_context&, seastar::httpd::routes&, seastar::sharded<db::system_keyspace>&)::$_117::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(replica::database&)#1}<seastar::lw_shared_ptr<sstables::sstable>, seastar::basic_sstring<char, unsigned int, 15u, true><sstables::sstable>, std::hash<seastar::basic_sstring><sstables::sstable>, std::allocator<sstables::sstable> >&&)#1}&&, seastar::future<seastar::sharded<replica::database>::map_reduce0<api::set_column_family(api::http_context&, seastar::httpd::routes&, seastar::sharded<db::system_keyspace>&)::$_117::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(replica::database&)#1}, std::unordered_set<seastar::basic_sstring<char, unsigned int, 15u, true>, std::hash<seastar::basic_sstring>, std::equal_to<seastar::basic_sstring>, std::allocator<seastar::basic_sstring> >, api::set_column_family(api::http_context&, seastar::httpd::routes&, seastar::sharded<db::system_keyspace>&)::$_117::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(std::allocator<seastar::basic_sstring>, api::set_column_family(api::http_context&, seastar::httpd::routes&, seastar::sharded<db::system_keyspace>&)::$_117::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(replica::database&)#1}<seastar::lw_shared_ptr<sstables::sstable>, seastar::basic_sstring<char, unsigned int, 15u, true><sstables::sstable>, std::hash<seastar::basic_sstring><sstables::sstable>, std::allocator<sstables::sstable> >&&)#1}>(api::set_column_family(api::http_context&, seastar::httpd::routes&, seastar::sharded<db::system_keyspace>&)::$_117::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(replica::database&)#1}, {lambda(std::allocator<seastar::basic_sstring>, api::set_column_family(api::http_context&, seastar::httpd::routes&, seastar::sharded<db::system_keyspace>&)::$_117::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(replica::database&)#1}<seastar::lw_shared_ptr<sstables::sstable>, seastar::basic_sstring<char, unsigned int, 15u, true><sstables::sstable>, std::hash<seastar::basic_sstring><sstables::sstable>, std::allocator<sstables::sstable> >&&)#1}, api::set_column_family(api::http_context&, seastar::httpd::routes&, seastar::sharded<db::system_keyspace>&)::$_117::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(std::allocator<seastar::basic_sstring>, api::set_column_family(api::http_context&, seastar::httpd::routes&, seastar::sharded<db::system_keyspace>&)::$_117::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(replica::database&)#1}<seastar::lw_shared_ptr<sstables::sstable>, seastar::basic_sstring<char, unsigned int, 15u, true><sstables::sstable>, std::hash<seastar::basic_sstring><sstables::sstable>, std::allocator<sstables::sstable> >&&)#1})::{lambda(unsigned int)#1}>, api::set_column_family(api::http_context&, seastar::httpd::routes&, seastar::sharded<db::system_keyspace>&)::$_117::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(replica::database&)#1}<seastar::lw_shared_ptr<sstables::sstable>, seastar::basic_sstring<char, unsigned int, 15u, true><sstables::sstable>, std::hash<seastar::basic_sstring><sstables::sstable>, std::allocator<sstables::sstable> >&&)::{lambda(auto:2)#1}&, seastar::future<std::allocator<sstables::sstable> > >(seastar::future<std::allocator<sstables::sstable> >&&, seastar::future<std::allocator<sstables::sstable> >&&) at ././seastar/include/seastar/core/future.hh:2006
 (inlined by) operator() at ././seastar/include/seastar/core/future.hh:1523
 (inlined by) seastar::noncopyable_function<seastar::future<void> (seastar::future<std::unordered_set<seastar::lw_shared_ptr<sstables::sstable>, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<seastar::lw_shared_ptr<sstables::sstable> > > >&&)>::direct_vtable_for<seastar::future<std::unordered_set<seastar::lw_shared_ptr<sstables::sstable>, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<seastar::lw_shared_ptr<sstables::sstable> > > >::then_wrapped_maybe_erase<true, seastar::future<void>, seastar::map_reduce<boost::range_detail::integer_iterator<unsigned int>, seastar::sharded<replica::database>::map_reduce0<api::set_column_family(api::http_context&, seastar::httpd::routes&, seastar::sharded<db::system_keyspace>&)::$_117::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(replica::database&)#1}, std::unordered_set<seastar::basic_sstring<char, unsigned int, 15u, true>, std::hash<seastar::basic_sstring>, std::equal_to<seastar::basic_sstring>, std::allocator<seastar::basic_sstring> >, api::set_column_family(api::http_context&, seastar::httpd::routes&, seastar::sharded<db::system_keyspace>&)::$_117::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(std::allocator<seastar::basic_sstring>, std::unordered_set<seastar::lw_shared_ptr<sstables::sstable>, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<seastar::lw_shared_ptr<sstables::sstable> > >&&)#1}>(api::set_column_family(api::http_context&, seastar::httpd::routes&, seastar::sharded<db::system_keyspace>&)::$_117::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(replica::database&)#1}, {lambda(std::allocator<seastar::basic_sstring>, std::unordered_set<seastar::lw_shared_ptr<sstables::sstable>, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<seastar::lw_shared_ptr<sstables::sstable> > >&&)#1}, api::set_column_family(api::http_context&, seastar::httpd::routes&, seastar::sharded<db::system_keyspace>&)::$_117::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(std::allocator<seastar::basic_sstring>, std::unordered_set<seastar::lw_shared_ptr<sstables::sstable>, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<seastar::lw_shared_ptr<sstables::sstable> > >&&)#1})::{lambda(unsigned int)#1}, std::allocator<seastar::basic_sstring>, std::unordered_set<seastar::lw_shared_ptr<sstables::sstable>, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<seastar::lw_shared_ptr<sstables::sstable> > >&&>(seastar::sharded<replica::database>::map_reduce0<api::set_column_family(api::http_context&, seastar::httpd::routes&, seastar::sharded<db::system_keyspace>&)::$_117::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(replica::database&)#1}, std::unordered_set<seastar::basic_sstring<char, unsigned int, 15u, true>, std::hash<seastar::basic_sstring>, std::equal_to<seastar::basic_sstring>, std::allocator<seastar::basic_sstring> >, api::set_column_family(api::http_context&, seastar::httpd::routes&, seastar::sharded<db::system_keyspace>&)::$_117::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(std::allocator<seastar::basic_sstring>, std::unordered_set<seastar::lw_shared_ptr<sstables::sstable>, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<seastar::lw_shared_ptr<sstables::sstable> > >&&)#1}>(api::set_column_family(api::http_context&, seastar::httpd::routes&, seastar::sharded<db::system_keyspace>&)::$_117::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(replica::database&)#1}, {lambda(std::allocator<seastar::basic_sstring>, std::unordered_set<seastar::lw_shared_ptr<sstables::sstable>, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<seastar::lw_shared_ptr<sstables::sstable> > >&&)#1}, api::set_column_family(api::http_context&, seastar::httpd::routes&, seastar::sharded<db::system_keyspace>&)::$_117::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(std::allocator<seastar::basic_sstring>, std::unordered_set<seastar::lw_shared_ptr<sstables::sstable>, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<seastar::lw_shared_ptr<sstables::sstable> > >&&)#1})::{lambda(unsigned int)#1}, seastar::sharded<replica::database>::map_reduce0<api::set_column_family(api::http_context&, seastar::httpd::routes&, seastar::sharded<db::system_keyspace>&)::$_117::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(replica::database&)#1}, std::unordered_set<seastar::basic_sstring<char, unsigned int, 15u, true>, std::hash<seastar::basic_sstring>, std::equal_to<seastar::basic_sstring>, std::allocator<seastar::basic_sstring> >, api::set_column_family(api::http_context&, seastar::httpd::routes&, seastar::sharded<db::system_keyspace>&)::$_117::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(std::allocator<seastar::basic_sstring>, std::unordered_set<seastar::lw_shared_ptr<sstables::sstable>, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<seastar::lw_shared_ptr<sstables::sstable> > >&&)#1}>(api::set_column_family(api::http_context&, seastar::httpd::routes&, seastar::sharded<db::system_keyspace>&)::$_117::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(replica::database&)#1}, {lambda(std::allocator<seastar::basic_sstring>, std::unordered_set<seastar::lw_shared_ptr<sstables::sstable>, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<seastar::lw_shared_ptr<sstables::sstable> > >&&)#1}, api::set_column_family(api::http_context&, seastar::httpd::routes&, seastar::sharded<db::system_keyspace>&)::$_117::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(std::allocator<seastar::basic_sstring>, std::unordered_set<seastar::lw_shared_ptr<sstables::sstable>, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<seastar::lw_shared_ptr<sstables::sstable> > >&&)#1})::{lambda(unsigned int)#1}, {lambda(std::allocator<seastar::basic_sstring>, std::unordered_set<seastar::lw_shared_ptr<sstables::sstable>, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<seastar::lw_shared_ptr<sstables::sstable> > >&&)#1}&&, seastar::future<seastar::sharded<replica::database>::map_reduce0<api::set_column_family(api::http_context&, seastar::httpd::routes&, seastar::sharded<db::system_keyspace>&)::$_117::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(replica::database&)#1}, std::unordered_set<seastar::basic_sstring<char, unsigned int, 15u, true>, std::hash<seastar::basic_sstring>, std::equal_to<seastar::basic_sstring>, std::allocator<seastar::basic_sstring> >, api::set_column_family(api::http_context&, seastar::httpd::routes&, seastar::sharded<db::system_keyspace>&)::$_117::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(std::allocator<seastar::basic_sstring>, std::unordered_set<seastar::lw_shared_ptr<sstables::sstable>, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<seastar::lw_shared_ptr<sstables::sstable> > >&&)#1}>(api::set_column_family(api::http_context&, seastar::httpd::routes&, seastar::sharded<db::system_keyspace>&)::$_117::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(replica::database&)#1}, {lambda(std::allocator<seastar::basic_sstring>, std::unordered_set<seastar::lw_shared_ptr<sstables::sstable>, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<seastar::lw_shared_ptr<sstables::sstable> > >&&)#1}, api::set_column_family(api::http_context&, seastar::httpd::routes&, seastar::sharded<db::system_keyspace>&)::$_117::operator()(std::unique_ptr<seastar::http::request, std::default_delete<seastar::http::request> >) const::{lambda(std::allocator<seastar::basic_sstring>, std::unordered_set<seastar::lw_shared_ptr<sstables::sstable>, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<seastar::lw_shared_ptr<sstables::sstable> > >&&)#1})::{lambda(unsigned int)#1}>, std::unordered_set<seastar::lw_shared_ptr<sstables::sstable>, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<seastar::lw_shared_ptr<sstables::sstable> > >&&)::{lambda(auto:2)#1}>(seastar::future<seastar::future<void> >&&)::{lambda(seastar::future<std::unordered_set<seastar::lw_shared_ptr<sstables::sstable>, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<seastar::lw_shared_ptr<sstables::sstable> > > >&&)#1}>::call(seastar::noncopyable_function<seastar::future<void> (seastar::future<std::unordered_set<seastar::lw_shared_ptr<sstables::sstable>, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<seastar::lw_shared_ptr<sstables::sstable> > > >&&)> const*, seastar::future<std::unordered_set<seastar::lw_shared_ptr<sstables::sstable>, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<seastar::lw_shared_ptr<sstables::sstable> > > >&&) at ././seastar/include/seastar/util/noncopyable_function.hh:129
seastar::noncopyable_function<seastar::future<void> (seastar::future<std::unordered_set<seastar::lw_shared_ptr<sstables::sstable>, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<seastar::lw_shared_ptr<sstables::sstable> > > >&&)>::operator()(seastar::future<std::unordered_set<seastar::lw_shared_ptr<sstables::sstable>, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<seastar::lw_shared_ptr<sstables::sstable> > > >&&) const at ././seastar/include/seastar/util/noncopyable_function.hh:215
operator() at ././seastar/include/seastar/core/future.hh:1539
void seastar::futurize<seastar::future<void> >::satisfy_with_result_of<seastar::future<std::unordered_set<seastar::lw_shared_ptr<sstables::sstable>, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<seastar::lw_shared_ptr<sstables::sstable> > > >::then_wrapped_nrvo<seastar::future<void>, seastar::noncopyable_function<seastar::future<void> (seastar::future<std::unordered_set<seastar::lw_shared_ptr<sstables::sstable>, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<seastar::lw_shared_ptr<sstables::sstable> > > >&&)> >(seastar::noncopyable_function<seastar::future<void> (seastar::future<std::unordered_set<seastar::lw_shared_ptr<sstables::sstable>, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<seastar::lw_shared_ptr<sstables::sstable> > > >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> (seastar::future<std::unordered_set<seastar::lw_shared_ptr<sstables::sstable>, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<seastar::lw_shared_ptr<sstables::sstable> > > >&&)>&, seastar::future_state<std::unordered_set<seastar::lw_shared_ptr<sstables::sstable>, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<seastar::lw_shared_ptr<sstables::sstable> > > >&&)#1}::operator()(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> (seastar::future<std::unordered_set<seastar::lw_shared_ptr<sstables::sstable>, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<seastar::lw_shared_ptr<sstables::sstable> > > >&&)>&, seastar::future_state<std::unordered_set<seastar::lw_shared_ptr<sstables::sstable>, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<seastar::lw_shared_ptr<sstables::sstable> > > >&&) const::{lambda()#1}>(seastar::internal::promise_base_with_type<void>&&, seastar::future<std::unordered_set<seastar::lw_shared_ptr<sstables::sstable>, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<seastar::lw_shared_ptr<sstables::sstable> > > >::then_wrapped_nrvo<seastar::future<void>, seastar::noncopyable_function<seastar::future<void> (seastar::future<std::unordered_set<seastar::lw_shared_ptr<sstables::sstable>, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<seastar::lw_shared_ptr<sstables::sstable> > > >&&)> >(seastar::noncopyable_function<seastar::future<void> (seastar::future<std::unordered_set<seastar::lw_shared_ptr<sstables::sstable>, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<seastar::lw_shared_ptr<sstables::sstable> > > >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> (seastar::future<std::unordered_set<seastar::lw_shared_ptr<sstables::sstable>, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<seastar::lw_shared_ptr<sstables::sstable> > > >&&)>&, seastar::future_state<std::unordered_set<seastar::lw_shared_ptr<sstables::sstable>, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<seastar::lw_shared_ptr<sstables::sstable> > > >&&)#1}::operator()(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> (seastar::future<std::unordered_set<seastar::lw_shared_ptr<sstables::sstable>, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<seastar::lw_shared_ptr<sstables::sstable> > > >&&)>&, seastar::future_state<std::unordered_set<seastar::lw_shared_ptr<sstables::sstable>, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<seastar::lw_shared_ptr<sstables::sstable> > > >&&) const::{lambda()#1}&&) at ././seastar/include/seastar/core/future.hh:1991
operator() at ././seastar/include/seastar/core/future.hh:1538
 (inlined by) seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::noncopyable_function<seastar::future<void> (seastar::future<std::unordered_set<seastar::lw_shared_ptr<sstables::sstable>, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<seastar::lw_shared_ptr<sstables::sstable> > > >&&)>, seastar::future<std::unordered_set<seastar::lw_shared_ptr<sstables::sstable>, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<seastar::lw_shared_ptr<sstables::sstable> > > >::then_wrapped_nrvo<seastar::future<void>, seastar::noncopyable_function<seastar::future<void> (seastar::future<std::unordered_set<seastar::lw_shared_ptr<sstables::sstable>, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<seastar::lw_shared_ptr<sstables::sstable> > > >&&)> >(seastar::noncopyable_function<seastar::future<void> (seastar::future<std::unordered_set<seastar::lw_shared_ptr<sstables::sstable>, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<seastar::lw_shared_ptr<sstables::sstable> > > >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> (seastar::future<std::unordered_set<seastar::lw_shared_ptr<sstables::sstable>, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<seastar::lw_shared_ptr<sstables::sstable> > > >&&)>&, seastar::future_state<std::unordered_set<seastar::lw_shared_ptr<sstables::sstable>, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<seastar::lw_shared_ptr<sstables::sstable> > > >&&)#1}, std::unordered_set<seastar::lw_shared_ptr<sstables::sstable>, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<seastar::lw_shared_ptr<sstables::sstable> > > >::run_and_dispose() at ././seastar/include/seastar/core/future.hh:741
seastar::reactor::run_tasks(seastar::reactor::task_queue&) at ./build/debug/seastar/./seastar/src/core/reactor.cc:2590
seastar::reactor::run_some_tasks() at ./build/debug/seastar/./seastar/src/core/reactor.cc:3053
seastar::reactor::do_run() at ./build/debug/seastar/./seastar/src/core/reactor.cc:3222
seastar::reactor::run() at ./build/debug/seastar/./seastar/src/core/reactor.cc:3105
seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) at ./build/debug/seastar/./seastar/src/core/app-template.cc:276
seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) at ./build/debug/seastar/./seastar/src/core/app-template.cc:167
scylla_main(int, char**) at ./main.cc:627
std::function<int (int, char**)>::operator()(int, char**) const at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/std_function.h:591
main at ./main.cc:1995
__libc_start_call_main at ??:?
__libc_start_main_alias_2 at :?
_start at ??:?
@bhalevy bhalevy added status/regression triage/master Looking for assignee labels Jul 5, 2023
@bhalevy
Copy link
Member Author

bhalevy commented Jul 5, 2023

@Deexie please look into this

@bhalevy
Copy link
Member Author

bhalevy commented Jul 5, 2023

Similar failure in https://jenkins.scylladb.com/job/scylla-master/job/dtest-daily-debug/242/testReport/nodetool_additional_test/TestNodetool/Run_Dtest_Parallel_Cloud_Machines___FullDtest___full_split003___test_get_sstable/

@bhalevy
Copy link
Member Author

bhalevy commented Jul 5, 2023

and https://jenkins.scylladb.com/job/scylla-master/job/dtest-daily-debug/242/testReport/migration_test/TestMigrationUpgradeSSTables/Run_Dtest_Parallel_Cloud_Machines___FullDtest___full_split004___test_migrate_sstable_with_counter_2_2_x_/

Deexie added a commit to Deexie/scylla that referenced this issue Jul 5, 2023
@Deexie
api: prevent non-owner cpu access to shared_ptr
2fe7d4a 
In get_sstables_for_key in api/column_family.cc a set of lw_shared_ptrs
to sstables is passes to reducer of map_reduce0. Reducer then accesses
these shared pointers. As reducer is invoked on the same shard
map_reduce0 is called, we have an illegal access to shared pointer
on non-owner cpu.

A set of shared pointers to sstables is trasnsformed in map function,
which is guaranteed to be invoked on a shard associated with the service.

Fixes: scylladb#14515.
@Deexie Deexie mentioned this issue Jul 5, 2023
Closed
@DoronArazii DoronArazii added this to the 5.4 milestone Jul 9, 2023
@DoronArazii DoronArazii removed the triage/master Looking for assignee label Jul 9, 2023
Deexie added a commit to Deexie/scylla that referenced this issue Jul 9, 2023
@Deexie
api: prevent non-owner cpu access to shared_ptr
d085c24 
In get_sstables_for_key in api/column_family.cc a set of lw_shared_ptrs
to sstables is passes to reducer of map_reduce0. Reducer then accesses
these shared pointers. As reducer is invoked on the same shard
map_reduce0 is called, we have an illegal access to shared pointer
on non-owner cpu.

A set of shared pointers to sstables is trasnsformed in map function,
which is guaranteed to be invoked on a shard associated with the service.

Fixes: scylladb#14515.
@denesb
Copy link
Contributor

denesb commented Dec 15, 2023

@bhalevy @Deexie should we backport this?

@bhalevy
Copy link
Member Author

bhalevy commented Dec 17, 2023

@bhalevy @Deexie should we backport this?

No need. The bug was introduced in 198bca9 so it's confined to 5.4
Cc @xemul

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Deexie Deexie

Labels
status/regression
Projects
None yet
Milestone
5.4
Development

Successfully merging a pull request may close this issue.

api: prevent non-owner cpu access to shared_ptr Deexie/scylla
5 participants
@denesb @bhalevy @scylladb-promoter @Deexie @DoronArazii