New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Log commitlog and compaction progress #12
Comments
@raphaelsc so is commitlog logging the same as origin's ? If so I suggest we close this and create another one for commitlog |
On Wed, Aug 5, 2015 at 8:21 AM, slivne notifications@github.com wrote:
Raphael S. Carvalho |
Also fixes scylladb/seastar#54 ==5658==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x6250006b7848 at pc 0x1413e02 bp 0x7fff7cd7f1e0 sp 0x7fff7cd7f1d8 WRITE of size 8 at 0x6250006b7848 thread T0 #0 0x1413e01 in unsigned long* std::__copy_move<false, false, std::random_access_iterator_tag>::__copy_m<std::_Deque_iterator<unsigned long, unsigned long&, unsigned long*>, unsigned long*>(std::_Deque_iterator<unsigned long, unsigned long&, unsigned long*>, std::_Deque_iterator<unsigned long, unsigned long&, unsigned long*>, unsigned long*) /usr/include/c++/4.9/bits/stl_algobase.h:336 #1 0x1413c59 in unsigned long* std::__copy_move_a<false, std::_Deque_iterator<unsigned long, unsigned long&, unsigned long*>, unsigned long*>(std::_Deque_iterator<unsigned long, unsigned long&, unsigned long*>, std::_Deque_iterator<unsigned long, unsigned long&, unsigned long*>, unsigned long*) /usr/include/c++/4.9/bits/stl_algobase.h:396 #2 0x1413aea in unsigned long* std::__copy_move_a2<false, std::_Deque_iterator<unsigned long, unsigned long&, unsigned long*>, unsigned long*>(std::_Deque_iterator<unsigned long, unsigned long&, unsigned long*>, std::_Deque_iterator<unsigned long, unsigned long&, unsigned long*>, unsigned long*) /usr/include/c++/4.9/bits/stl_algobase.h:434 #3 0x14138df in unsigned long* std::copy<std::_Deque_iterator<unsigned long, unsigned long&, unsigned long*>, unsigned long*>(std::_Deque_iterator<unsigned long, unsigned long&, unsigned long*>, std::_Deque_iterator<unsigned long, unsigned long&, unsigned long*>, unsigned long*) /usr/include/c++/4.9/bits/stl_algobase.h:466 #4 0x1413545 in unsigned long* std::__copy_n<std::_Deque_iterator<unsigned long, unsigned long&, unsigned long*>, unsigned long, unsigned long*>(std::_Deque_iterator<unsigned long, unsigned long&, unsigned long*>, unsigned long, unsigned long*, std::random_access_iterator_tag) /usr/include/c++/4.9/bits/stl_algo.h:779 #5 0x1412d44 in unsigned long* std::copy_n<std::_Deque_iterator<unsigned long, unsigned long&, unsigned long*>, unsigned long, unsigned long*>(std::_Deque_iterator<unsigned long, unsigned long&, unsigned long*>, unsigned long, unsigned long*) /usr/include/c++/4.9/bits/stl_algo.h:804 #6 0x14112b3 in unsigned long large_bitset::load<std::_Deque_iterator<unsigned long, unsigned long&, unsigned long*> >(std::_Deque_iterator<unsigned long, unsigned long&, unsigned long*>, std::_Deque_iterator<unsigned long, unsigned long&, unsigned long*>, unsigned long) utils/large_bitset.hh:81 #7 0x13fcfc9 in _ZZZN8sstables7sstable11read_filterEvENKUlRT_E_clINS_6filterEEEDaS2_ENKUlvE_clEv (/home/tgrabiec/src/urchin/build/debug/scylla+0x13fcfc9) #8 0x1400a50 in apply /home/tgrabiec/src/urchin/seastar/core/apply.hh:34 #9 0x1400afb in apply<sstables::sstable::read_filter()::<lambda(auto:25&)> [with auto:25 = sstables::filter]::<lambda()> > /home/tgrabiec/src/urchin/seastar/core/apply.hh:42 #10 0x1400bb2 in apply<sstables::sstable::read_filter()::<lambda(auto:25&)> [with auto:25 = sstables::filter]::<lambda()> > /home/tgrabiec/src/urchin/seastar/core/future.hh:1062 #11 0x140f1b7 in _ZZN6futureIIEE4thenIZZN8sstables7sstable11read_filterEvENKUlRT_E_clINS2_6filterEEEDaS5_EUlvE_S0_EET0_OT_ENUlOS4_E_clI12future_stateIIEEEEDaSC_ (/home/tgrabiec/src/urchin/build/debug/scylla+0x140f1b7) #12 0x140f350 in run /home/tgrabiec/src/urchin/seastar/core/future.hh:359 #13 0x426e2c in reactor::run_tasks(circular_buffer<std::unique_ptr<task, std::default_delete<task> >, std::allocator<std::unique_ptr<task, std::default_delete<task> > > >&, unsigned long) core/reactor.cc:1093 #14 0x429cb1 in reactor::run() core/reactor.cc:1190 #15 0x72bc69 in app_template::run_deprecated(int, char**, std::function<void ()>&&) core/app-template.cc:122 #16 0xa119bc in main /home/tgrabiec/src/urchin/main.cc:279 #17 0x7ffc1b6beec4 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21ec4) #18 0x412558 (/home/tgrabiec/src/urchin/build/debug/scylla+0x412558) 0x6250006b7848 is located 0 bytes to the right of 8008-byte region [0x6250006b5900,0x6250006b7848) allocated by thread T0 here: #0 0x7ffc1cf6c7df in __interceptor_malloc (/usr/lib/x86_64-linux-gnu/libasan.so.1+0x547df) #1 0x7ffc204eef17 in operator new(unsigned long) (/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0x8df17) #2 0xfa5d4f in large_bitset::large_bitset(unsigned long) utils/large_bitset.cc:15 #3 0x13fcec6 in _ZZZN8sstables7sstable11read_filterEvENKUlRT_E_clINS_6filterEEEDaS2_ENKUlvE_clEv (/home/tgrabiec/src/urchin/build/debug/scylla+0x13fcec6) #4 0x1400a50 in apply /home/tgrabiec/src/urchin/seastar/core/apply.hh:34 #5 0x1400afb in apply<sstables::sstable::read_filter()::<lambda(auto:25&)> [with auto:25 = sstables::filter]::<lambda()> > /home/tgrabiec/src/urchin/seastar/core/apply.hh:42 #6 0x1400bb2 in apply<sstables::sstable::read_filter()::<lambda(auto:25&)> [with auto:25 = sstables::filter]::<lambda()> > /home/tgrabiec/src/urchin/seastar/core/future.hh:1062 #7 0x140f1b7 in _ZZN6futureIIEE4thenIZZN8sstables7sstable11read_filterEvENKUlRT_E_clINS2_6filterEEEDaS5_EUlvE_S0_EET0_OT_ENUlOS4_E_clI12future_stateIIEEEEDaSC_ (/home/tgrabiec/src/urchin/build/debug/scylla+0x140f1b7) #8 0x140f350 in run /home/tgrabiec/src/urchin/seastar/core/future.hh:359 #9 0x426e2c in reactor::run_tasks(circular_buffer<std::unique_ptr<task, std::default_delete<task> >, std::allocator<std::unique_ptr<task, std::default_delete<task> > > >&, unsigned long) core/reactor.cc:1093 #10 0x429cb1 in reactor::run() core/reactor.cc:1190 #11 0x72bc69 in app_template::run_deprecated(int, char**, std::function<void ()>&&) core/app-template.cc:122 #12 0xa119bc in main /home/tgrabiec/src/urchin/main.cc:279 #13 0x7ffc1b6beec4 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21ec4) SUMMARY: AddressSanitizer: heap-buffer-overflow /usr/include/c++/4.9/bits/stl_algobase.h:336 unsigned long* std::__copy_move<false, false, std::random_access_iterator_tag>::__copy_m<std::_Deque_iterator<unsigned long, unsigned long&, unsigned long*>, unsigned long*>(std::_Deque_iterator<unsigned long, unsigned long&, unsigned long*>, std::_Deque_iterator<unsigned long, unsigned long&, unsigned long*>, unsigned long*) Shadow bytes around the buggy address: 0x0c4a800ceeb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0c4a800ceec0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0c4a800ceed0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0c4a800ceee0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0c4a800ceef0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 =>0x0c4a800cef00: 00 00 00 00 00 00 00 00 00[fa]fa fa fa fa fa fa 0x0c4a800cef10: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c4a800cef20: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c4a800cef30: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c4a800cef40: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c4a800cef50: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa Heap right redzone: fb Freed heap region: fd Stack left redzone: f1 Stack mid redzone: f2 Stack right redzone: f3 Stack partial redzone: f4 Stack after return: f5 Stack use after scope: f8 Global redzone: f9 Global init order: f6 Poisoned by user: f7 Contiguous container OOB:fc ASan internal: fe ==5658==ABORTING
Fixes the heap-use-after-free error in build/debug/tests/auth_test ==1415==ERROR: AddressSanitizer: heap-use-after-free on address 0x62200032cfa8 at pc 0x00000350701d bp 0x7fec96df8d40 sp 0x7fec96df8d30 READ of size 8 at 0x62200032cfa8 thread T1 #0 0x350701c in _ZZN3gms8gossiper3runEvENKUlOT_E0_clI6futureIJEEEEDaS2_ (/home/penberg/scylla/build/debug/tests/auth_test_g+0x350701c) #1 0x35795b1 in apply<gms::gossiper::run()::<lambda(auto:40&&)>, future<> > /home/penberg/scylla/seastar/core/future.hh:1203 #2 0x369103d in _ZZN6futureIJEE12then_wrappedIZN3gms8gossiper3runEvEUlOT_E0_S0_EET0_S5_ENUlS5_E_clI12future_stateIJEEEEDaS5_ (/home/penberg/scylla/build/debug/tests/auth_test_g+0x369103d) #3 0x369182a in run /home/penberg/scylla/seastar/core/future.hh:399 #4 0x435f24 in reactor::run_tasks(circular_buffer<std::unique_ptr<task, std::default_delete<task> >, std::allocator<std::unique_ptr<task, std::default_delete<task> > > >&) core/reactor.cc:1368 #5 0x43a44f in reactor::run() core/reactor.cc:1672 #6 0x952e4b in app_template::run_deprecated(int, char**, std::function<void ()>&&) core/app-template.cc:123 #7 0x58dc79d in test_runner::start(int, char**)::{lambda()#1}::operator()() (/home/penberg/scylla/build/debug/tests/auth_test_g+0x58dc79d) #8 0x58e6cd6 in _M_invoke /usr/include/c++/5.3.1/functional:1871 #9 0x688639 in std::function<void ()>::operator()() const /usr/include/c++/5.3.1/functional:2271 #10 0x8d939c in posix_thread::start_routine(void*) core/posix.cc:51 #11 0x7feca02a4609 in start_thread (/lib64/libpthread.so.0+0x7609) #12 0x7fec9ffdea4c in clone (/lib64/libc.so.6+0x102a4c) 0x62200032cfa8 is located 5800 bytes inside of 5808-byte region [0x62200032b900,0x62200032cfb0) freed by thread T1 here: #0 0x7feca4f76472 in operator delete(void*, unsigned long) (/lib64/libasan.so.2+0x9a472) #1 0x3740772 in gms::gossiper::~gossiper() (/home/penberg/scylla/build/debug/tests/auth_test_g+0x3740772) #2 0x2588ba1 in shared_ptr<gms::gossiper>::~shared_ptr() seastar/core/shared_ptr.hh:389 #3 0x4fc908c in seastar::sharded<gms::gossiper>::stop()::{lambda(unsigned int)#1}::operator()(unsigned int)::{lambda()#1}::operator()()::{lambda()#1}::~stop() (/home/penberg/scylla/build/debug/tests/auth_test_g+0x4fc908c) #4 0x4ff722a in future<> future<>::then<seastar::sharded<gms::gossiper>::stop()::{lambda(unsigned int)#1}::operator()(unsigned int)::{lambda()#1}::operator()()::{lambda()#1}, future<> >(seastar::sharded<gms::gossiper>::stop()::{lambda(unsigned int)#1}::operator()(unsigned int)::{lambda()#1}::operator()()::{lambda()#1}&&)::{lambda(seastar::sharded<gms::gossiper>::stop()::{lambda(unsigned int)#1}::operator()(unsigned int)::{lambda()#1}::operator()()::{lambda()#1})#1}::~then() (/home/penberg/scylla/build/debug/tests/auth_test_g+0x4ff722a) #5 0x509a28c in continuation<future<> future<>::then<seastar::sharded<gms::gossiper>::stop()::{lambda(unsigned int)#1}::operator()(unsigned int)::{lambda()#1}::operator()()::{lambda()#1}, future<> >(seastar::sharded<gms::gossiper>::stop()::{lambda(unsigned int)#1}::operator()(unsigned int)::{lambda()#1}::operator()()::{lambda()#1}&&)::{lambda(seastar::sharded<gms::gossiper>::stop()::{lambda(unsigned int)#1}::operator()(unsigned int)::{lambda()#1}::operator()()::{lambda()#1})#1}>::~continuation() seastar/core/future.hh:395 #6 0x509a40d in continuation<future<> Message-Id: <f8f1c92c1eb88687ab0534f5e7874d53050a5b93.1453446350.git.asias@scylladb.com>
Introduced in 4b3243f Reproducible with materialized_views_test:TestMaterializedViews.mv_populating_from_existing_data_during_node_remove_test and read_amplification_test:ReadAmplificationTest.no_read_amplification_on_repair_with_mv_test ==955382==ERROR: AddressSanitizer: heap-use-after-free on address 0x60200023de18 at pc 0x00000051d788 bp 0x7f8a0563fcc0 sp 0x7f8a0563fcb0 READ of size 8 at 0x60200023de18 thread T1 (reactor-1) #0 0x51d787 in seastar::lw_shared_ptr<sstables::sstable>::lw_shared_ptr(seastar::lw_shared_ptr<sstables::sstable> const&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:289 #1 0x10ba189 in apply<table::move_sstables_from_staging(std::vector<seastar::lw_shared_ptr<sstables::sstable> >)::<lambda()>::<lambda(std::set<seastar::basic_sstring<char, unsigned int, 15> >&)>::<lambda(sstables::shared_sstable)>&, const seastar::lw_shared_ptr<sstables::sstabl e>&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1530 #2 0x109c4f1 in apply<table::move_sstables_from_staging(std::vector<seastar::lw_shared_ptr<sstables::sstable> >)::<lambda()>::<lambda(std::set<seastar::basic_sstring<char, unsigned int, 15> >&)>::<lambda(sstables::shared_sstable)>&, const seastar::lw_shared_ptr<sstables::sstabl e>&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1556 #3 0x106941a in do_for_each<__gnu_cxx::__normal_iterator<const seastar::lw_shared_ptr<sstables::sstable>*, std::vector<seastar::lw_shared_ptr<sstables::sstable> > >, table::move_sstables_from_staging(std::vector<seastar::lw_shared_ptr<sstables::sstable> >)::<lambda()>::<lambda( std::set<seastar::basic_sstring<char, unsigned int, 15> >&)>::<lambda(sstables::shared_sstable)> > /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future-util.hh:618 #4 0x1069203 in operator() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future-util.hh:626 #5 0x10ba589 in apply /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36 #6 0x10ba668 in apply<seastar::do_for_each(Iterator, Iterator, AsyncAction) [with Iterator = __gnu_cxx::__normal_iterator<const seastar::lw_shared_ptr<sstables::sstable>*, std::vector<seastar::lw_shared_ptr<sstables::sstable> > >; AsyncAction = table::move_sstables_from_staging (std::vector<seastar::lw_shared_ptr<sstables::sstable> >)::<lambda()>::<lambda(std::set<seastar::basic_sstring<char, unsigned int, 15> >&)>::<lambda(sstables::shared_sstable)>]::<lambda()>&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:44 #7 0x10ba7c0 in apply<seastar::do_for_each(Iterator, Iterator, AsyncAction) [with Iterator = __gnu_cxx::__normal_iterator<const seastar::lw_shared_ptr<sstables::sstable>*, std::vector<seastar::lw_shared_ptr<sstables::sstable> > >; AsyncAction = table::move_sstables_from_staging (std::vector<seastar::lw_shared_ptr<sstables::sstable> >)::<lambda()>::<lambda(std::set<seastar::basic_sstring<char, unsigned int, 15> >&)>::<lambda(sstables::shared_sstable)>]::<lambda()>&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1563 ... 0x60200023de18 is located 8 bytes inside of 16-byte region [0x60200023de10,0x60200023de20) freed by thread T1 (reactor-1) here: #0 0x7f8a153b796f in operator delete(void*) (/lib64/libasan.so.5+0x11096f) #1 0x6ab4d1 in __gnu_cxx::new_allocator<seastar::lw_shared_ptr<sstables::sstable> >::deallocate(seastar::lw_shared_ptr<sstables::sstable>*, unsigned long) /usr/include/c++/9/ext/new_allocator.h:128 #2 0x612052 in std::allocator_traits<std::allocator<seastar::lw_shared_ptr<sstables::sstable> > >::deallocate(std::allocator<seastar::lw_shared_ptr<sstables::sstable> >&, seastar::lw_shared_ptr<sstables::sstable>*, unsigned long) /usr/include/c++/9/bits/alloc_traits.h:470 #3 0x58fdfb in std::_Vector_base<seastar::lw_shared_ptr<sstables::sstable>, std::allocator<seastar::lw_shared_ptr<sstables::sstable> > >::_M_deallocate(seastar::lw_shared_ptr<sstables::sstable>*, unsigned long) /usr/include/c++/9/bits/stl_vector.h:351 #4 0x52a790 in std::_Vector_base<seastar::lw_shared_ptr<sstables::sstable>, std::allocator<seastar::lw_shared_ptr<sstables::sstable> > >::~_Vector_base() /usr/include/c++/9/bits/stl_vector.h:332 #5 0x52a99b in std::vector<seastar::lw_shared_ptr<sstables::sstable>, std::allocator<seastar::lw_shared_ptr<sstables::sstable> > >::~vector() /usr/include/c++/9/bits/stl_vector.h:680 #6 0xff60fa in ~<lambda> /local/home/bhalevy/dev/scylla/table.cc:2477 #7 0xff7202 in operator() /local/home/bhalevy/dev/scylla/table.cc:2496 #8 0x106af5b in apply<table::move_sstables_from_staging(std::vector<seastar::lw_shared_ptr<sstables::sstable> >)::<lambda()> > /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1573 #9 0x102f5d5 in futurize_apply<table::move_sstables_from_staging(std::vector<seastar::lw_shared_ptr<sstables::sstable> >)::<lambda()> > /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1645 #10 0x102f9ee in operator()<seastar::semaphore_units<seastar::named_semaphore_exception_factory> > /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/semaphore.hh:488 #11 0x109d2f1 in apply /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36 #12 0x109d42c in apply<seastar::with_semaphore(seastar::basic_semaphore<ExceptionFactory, Clock>&, size_t, Func&&) [with ExceptionFactory = seastar::named_semaphore_exception_factory; Func = table::move_sstables_from_staging(std::vector<seastar::lw_shared_ptr<sstables::sstable> >)::<lambda()>; Clock = std::chrono::_V2::steady_clock]::<lambda(auto:51)>&, seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::chrono::_V2::steady_clock>&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:44 #13 0x109d595 in apply<seastar::with_semaphore(seastar::basic_semaphore<ExceptionFactory, Clock>&, size_t, Func&&) [with ExceptionFactory = seastar::named_semaphore_exception_factory; Func = table::move_sstables_from_staging(std::vector<seastar::lw_shared_ptr<sstables::sstable> >)::<lambda()>; Clock = std::chrono::_V2::steady_clock]::<lambda(auto:51)>&, seastar::semaphore_units<seastar::named_semaphore_exception_factory, std::chrono::_V2::steady_clock>&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1563 ... Fixes #5511 Signed-off-by: Benny Halevy <bhalevy@scylladb.com> Message-Id: <20191222214326.1229714-1-bhalevy@scylladb.com>
Rather than using lw_shared_ptr to keep it around, use do_with. Fixes a use-after-free seen with interrupt_build_process_with_resharding_half_to_max_test in debug mode: ==406079==ERROR: AddressSanitizer: heap-use-after-free on address 0x603001320ac0 at pc 0x00000777a062 bp 0x7ffdeb77b010 sp 0x7ffdeb77b000 READ of size 8 at 0x603001320ac0 thread T0 #0 0x777a061 in _M_next /usr/include/c++/9/bits/hashtable_policy.h:283 scylladb#1 0x779ead7 in _M_incr /usr/include/c++/9/bits/hashtable_policy.h:299 scylladb#2 0x777bbac in operator++ /usr/include/c++/9/bits/hashtable_policy.h:362 scylladb#3 0x77f8384 in run_and_dispose /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future-util.hh:568 scylladb#4 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091 scylladb#5 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506 scylladb#6 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666 scylladb#7 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199 scylladb#8 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115 scylladb#9 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486 scylladb#10 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2) scylladb#11 0x42b69d in _start (/local/home/bhalevy/.dtest/dtest-q85idok7/test/node1/bin/scylla+0x42b69d) 0x603001320ac0 is located 0 bytes inside of 32-byte region [0x603001320ac0,0x603001320ae0) freed by thread T0 here: #0 0x7fb86520996f in operator delete(void*) (/lib64/libasan.so.5+0x11096f) scylladb#1 0x77d1889 in deallocate /usr/include/c++/9/ext/new_allocator.h:128 scylladb#2 0x77bcbb2 in deallocate /usr/include/c++/9/bits/alloc_traits.h:470 scylladb#3 0x77bccc5 in _M_deallocate_node_ptr /usr/include/c++/9/bits/hashtable_policy.h:2113 scylladb#4 0x779dc59 in _M_deallocate_node /usr/include/c++/9/bits/hashtable_policy.h:2103 scylladb#5 0x77bc8b6 in _M_deallocate_nodes /usr/include/c++/9/bits/hashtable_policy.h:2124 scylladb#6 0x779d18f in clear /usr/include/c++/9/bits/hashtable.h:2028 scylladb#7 0x7779b8e in ~_Hashtable /usr/include/c++/9/bits/hashtable.h:1352 scylladb#8 0x774cfed in ~unordered_map /usr/include/c++/9/bits/unordered_map.h:102 scylladb#9 0x774d288 in ~shared_ptr_no_esft /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:157 scylladb#10 0x774d2f1 in dispose /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:213 scylladb#11 0x76f9642 in ~lw_shared_ptr /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:300 scylladb#12 0x76f9642 in ~<lambda> /local/home/bhalevy/dev/scylla/distributed_loader.cc:743 scylladb#13 0x771e0a7 in ~<lambda> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1167 scylladb#14 0x777c64c in destroy /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:133 scylladb#15 0x586059 in seastar::noncopyable_function<seastar::future<> ()>::~noncopyable_function() /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:165 scylladb#16 0x5867da in seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}::~future_state() /local/home/bha levy/dev/scylla/seastar/include/seastar/core/future.hh:1192 scylladb#17 0xab704d in seastar::continuation<seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}>::~continu ation() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:507 scylladb#18 0xab7216 in seastar::continuation<seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}>::run_and_ dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:512 scylladb#19 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091 scylladb#20 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506 scylladb#21 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666 scylladb#22 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199 scylladb#23 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115 scylladb#24 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486 scylladb#25 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2) previously allocated by thread T0 here: #0 0x7fb8652089d7 in operator new(unsigned long) (/lib64/libasan.so.5+0x10f9d7) scylladb#1 0x77d1796 in allocate /usr/include/c++/9/ext/new_allocator.h:114 scylladb#2 0x77bcab3 in allocate /usr/include/c++/9/bits/alloc_traits.h:444 scylladb#3 0x779d936 in _M_allocate_node<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable_policy.h:2081 scylladb#4 0x777a2ed in _M_emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable.h:1638 scylladb#5 0x774d707 in emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable.h:749 scylladb#6 0x771ce3e in emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/unordered_map.h:388 scylladb#7 0x76f8280 in operator() /local/home/bhalevy/dev/scylla/distributed_loader.cc:732 scylladb#8 0x774dc5d in _M_invoke /usr/include/c++/9/bits/std_function.h:286 scylladb#9 0x6cde67b in std::function<seastar::future<> (std::filesystem::__cxx11::path, seastar::directory_entry)>::operator()(std::filesystem::__cxx11::path, seastar::directory_entry) const /usr/include/c++/9/bits/std_function.h:690 scylladb#10 0x6cd345d in operator() /local/home/bhalevy/dev/scylla/lister.cc:30 scylladb#11 0x6cd96e0 in apply /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36 scylladb#12 0x6cd9854 in apply<lister::visit(seastar::directory_entry)::<lambda(seastar::directory_entry)>&, seastar::directory_entry&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:44 scylladb#13 0x6cd9953 in apply<lister::visit(seastar::directory_entry)::<lambda(seastar::directory_entry)>&, seastar::directory_entry&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1650 scylladb#14 0x6cd6ba9 in operator() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1168 scylladb#15 0x6cdbac0 in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:101 scylladb#16 0x6ce6178 in seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>::operator()(seastar::directory_entry&&) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:184 scylladb#17 0x6ce3033 in seastar::apply_helper<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, std::tuple<seastar::directory_entry>&&, std::integer_sequence<unsigned long, 0ul> >::apply(seastar::noncopyable_function<seastar::future<> (seastar::directory_e ntry&&)>&&, std::tuple<seastar::directory_entry>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36 scylladb#18 0x6ce3130 in auto seastar::apply<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::directory_entry>(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&, std::tuple<seastar::directory_entry>&&) /local/home/bha levy/dev/scylla/seastar/include/seastar/core/apply.hh:44 scylladb#19 0x6ce322f in seastar::future<> seastar::futurize<seastar::future<> >::apply<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::directory_entry>(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&, std::tuple<s eastar::directory_entry>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1650 scylladb#20 0x6cdf88d in seastar::future<seastar::directory_entry>::then_impl<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<seastar::directory_entry>&&)scylladb#1}::operator()(seastar::future_state<seastar::directory_entry>) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1196 scylladb#21 0x6cea2e0 in seastar::continuation<seastar::future<seastar::directory_entry>::then_impl<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&)::{lamb da()scylladb#1}::operator()() const::{lambda(seastar::future_state<seastar::directory_entry>&&)scylladb#1}, seastar::directory_entry>::run_and_dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:511 scylladb#22 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091 scylladb#23 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506 scylladb#24 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666 scylladb#25 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199 scylladb#26 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115 scylladb#27 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486 scylladb#28 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2) SUMMARY: AddressSanitizer: heap-use-after-free /usr/include/c++/9/bits/hashtable_policy.h:283 in _M_next Shadow bytes around the buggy address: 0x0c068025c100: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c068025c110: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa 0x0c068025c120: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd 0x0c068025c130: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c068025c140: fd fd fd fa fa fa 00 00 00 fa fa fa fd fd fd fa =>0x0c068025c150: fa fa fd fd fd fa fa fa[fd]fd fd fd fa fa fd fd 0x0c068025c160: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c068025c170: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa 0x0c068025c180: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd 0x0c068025c190: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c068025c1a0: fd fd fd fd fa fa fd fd fd fd fa fa 00 00 00 fa Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa Freed heap region: fd Stack left redzone: f1 Stack mid redzone: f2 Stack right redzone: f3 Stack after return: f5 Stack use after scope: f8 Global redzone: f9 Global init order: f6 Poisoned by user: f7 Container overflow: fc Array cookie: ac Intra object redzone: bb ASan internal: fe Left alloca redzone: ca Right alloca redzone: cb Shadow gap: cc ==406079==ABORTING Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Rather than using lw_shared_ptr to keep it around, use do_with. Fixes a use-after-free seen with interrupt_build_process_with_resharding_half_to_max_test in debug mode: ==406079==ERROR: AddressSanitizer: heap-use-after-free on address 0x603001320ac0 at pc 0x00000777a062 bp 0x7ffdeb77b010 sp 0x7ffdeb77b000 READ of size 8 at 0x603001320ac0 thread T0 #0 0x777a061 in _M_next /usr/include/c++/9/bits/hashtable_policy.h:283 scylladb#1 0x779ead7 in _M_incr /usr/include/c++/9/bits/hashtable_policy.h:299 scylladb#2 0x777bbac in operator++ /usr/include/c++/9/bits/hashtable_policy.h:362 scylladb#3 0x77f8384 in run_and_dispose /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future-util.hh:568 scylladb#4 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091 scylladb#5 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506 scylladb#6 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666 scylladb#7 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199 scylladb#8 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115 scylladb#9 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486 scylladb#10 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2) scylladb#11 0x42b69d in _start (/local/home/bhalevy/.dtest/dtest-q85idok7/test/node1/bin/scylla+0x42b69d) 0x603001320ac0 is located 0 bytes inside of 32-byte region [0x603001320ac0,0x603001320ae0) freed by thread T0 here: #0 0x7fb86520996f in operator delete(void*) (/lib64/libasan.so.5+0x11096f) scylladb#1 0x77d1889 in deallocate /usr/include/c++/9/ext/new_allocator.h:128 scylladb#2 0x77bcbb2 in deallocate /usr/include/c++/9/bits/alloc_traits.h:470 scylladb#3 0x77bccc5 in _M_deallocate_node_ptr /usr/include/c++/9/bits/hashtable_policy.h:2113 scylladb#4 0x779dc59 in _M_deallocate_node /usr/include/c++/9/bits/hashtable_policy.h:2103 scylladb#5 0x77bc8b6 in _M_deallocate_nodes /usr/include/c++/9/bits/hashtable_policy.h:2124 scylladb#6 0x779d18f in clear /usr/include/c++/9/bits/hashtable.h:2028 scylladb#7 0x7779b8e in ~_Hashtable /usr/include/c++/9/bits/hashtable.h:1352 scylladb#8 0x774cfed in ~unordered_map /usr/include/c++/9/bits/unordered_map.h:102 scylladb#9 0x774d288 in ~shared_ptr_no_esft /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:157 scylladb#10 0x774d2f1 in dispose /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:213 scylladb#11 0x76f9642 in ~lw_shared_ptr /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:300 scylladb#12 0x76f9642 in ~<lambda> /local/home/bhalevy/dev/scylla/distributed_loader.cc:743 scylladb#13 0x771e0a7 in ~<lambda> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1167 scylladb#14 0x777c64c in destroy /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:133 scylladb#15 0x586059 in seastar::noncopyable_function<seastar::future<> ()>::~noncopyable_function() /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:165 scylladb#16 0x5867da in seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}::~future_state() /local/home/bha levy/dev/scylla/seastar/include/seastar/core/future.hh:1192 scylladb#17 0xab704d in seastar::continuation<seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}>::~continu ation() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:507 scylladb#18 0xab7216 in seastar::continuation<seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}>::run_and_ dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:512 scylladb#19 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091 scylladb#20 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506 scylladb#21 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666 scylladb#22 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199 scylladb#23 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115 scylladb#24 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486 scylladb#25 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2) previously allocated by thread T0 here: #0 0x7fb8652089d7 in operator new(unsigned long) (/lib64/libasan.so.5+0x10f9d7) scylladb#1 0x77d1796 in allocate /usr/include/c++/9/ext/new_allocator.h:114 scylladb#2 0x77bcab3 in allocate /usr/include/c++/9/bits/alloc_traits.h:444 scylladb#3 0x779d936 in _M_allocate_node<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable_policy.h:2081 scylladb#4 0x777a2ed in _M_emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable.h:1638 scylladb#5 0x774d707 in emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable.h:749 scylladb#6 0x771ce3e in emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/unordered_map.h:388 scylladb#7 0x76f8280 in operator() /local/home/bhalevy/dev/scylla/distributed_loader.cc:732 scylladb#8 0x774dc5d in _M_invoke /usr/include/c++/9/bits/std_function.h:286 scylladb#9 0x6cde67b in std::function<seastar::future<> (std::filesystem::__cxx11::path, seastar::directory_entry)>::operator()(std::filesystem::__cxx11::path, seastar::directory_entry) const /usr/include/c++/9/bits/std_function.h:690 scylladb#10 0x6cd345d in operator() /local/home/bhalevy/dev/scylla/lister.cc:30 scylladb#11 0x6cd96e0 in apply /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36 scylladb#12 0x6cd9854 in apply<lister::visit(seastar::directory_entry)::<lambda(seastar::directory_entry)>&, seastar::directory_entry&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:44 scylladb#13 0x6cd9953 in apply<lister::visit(seastar::directory_entry)::<lambda(seastar::directory_entry)>&, seastar::directory_entry&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1650 scylladb#14 0x6cd6ba9 in operator() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1168 scylladb#15 0x6cdbac0 in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:101 scylladb#16 0x6ce6178 in seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>::operator()(seastar::directory_entry&&) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:184 scylladb#17 0x6ce3033 in seastar::apply_helper<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, std::tuple<seastar::directory_entry>&&, std::integer_sequence<unsigned long, 0ul> >::apply(seastar::noncopyable_function<seastar::future<> (seastar::directory_e ntry&&)>&&, std::tuple<seastar::directory_entry>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36 scylladb#18 0x6ce3130 in auto seastar::apply<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::directory_entry>(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&, std::tuple<seastar::directory_entry>&&) /local/home/bha levy/dev/scylla/seastar/include/seastar/core/apply.hh:44 scylladb#19 0x6ce322f in seastar::future<> seastar::futurize<seastar::future<> >::apply<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::directory_entry>(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&, std::tuple<s eastar::directory_entry>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1650 scylladb#20 0x6cdf88d in seastar::future<seastar::directory_entry>::then_impl<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<seastar::directory_entry>&&)scylladb#1}::operator()(seastar::future_state<seastar::directory_entry>) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1196 scylladb#21 0x6cea2e0 in seastar::continuation<seastar::future<seastar::directory_entry>::then_impl<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&)::{lamb da()scylladb#1}::operator()() const::{lambda(seastar::future_state<seastar::directory_entry>&&)scylladb#1}, seastar::directory_entry>::run_and_dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:511 scylladb#22 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091 scylladb#23 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506 scylladb#24 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666 scylladb#25 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199 scylladb#26 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115 scylladb#27 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486 scylladb#28 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2) SUMMARY: AddressSanitizer: heap-use-after-free /usr/include/c++/9/bits/hashtable_policy.h:283 in _M_next Shadow bytes around the buggy address: 0x0c068025c100: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c068025c110: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa 0x0c068025c120: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd 0x0c068025c130: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c068025c140: fd fd fd fa fa fa 00 00 00 fa fa fa fd fd fd fa =>0x0c068025c150: fa fa fd fd fd fa fa fa[fd]fd fd fd fa fa fd fd 0x0c068025c160: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c068025c170: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa 0x0c068025c180: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd 0x0c068025c190: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c068025c1a0: fd fd fd fd fa fa fd fd fd fd fa fa 00 00 00 fa Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa Freed heap region: fd Stack left redzone: f1 Stack mid redzone: f2 Stack right redzone: f3 Stack after return: f5 Stack use after scope: f8 Global redzone: f9 Global init order: f6 Poisoned by user: f7 Container overflow: fc Array cookie: ac Intra object redzone: bb ASan internal: fe Left alloca redzone: ca Right alloca redzone: cb Shadow gap: cc ==406079==ABORTING Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Rather than using lw_shared_ptr to keep it around, use do_with. Fixes a use-after-free seen with interrupt_build_process_with_resharding_half_to_max_test in debug mode: ==406079==ERROR: AddressSanitizer: heap-use-after-free on address 0x603001320ac0 at pc 0x00000777a062 bp 0x7ffdeb77b010 sp 0x7ffdeb77b000 READ of size 8 at 0x603001320ac0 thread T0 #0 0x777a061 in _M_next /usr/include/c++/9/bits/hashtable_policy.h:283 scylladb#1 0x779ead7 in _M_incr /usr/include/c++/9/bits/hashtable_policy.h:299 scylladb#2 0x777bbac in operator++ /usr/include/c++/9/bits/hashtable_policy.h:362 scylladb#3 0x77f8384 in run_and_dispose /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future-util.hh:568 scylladb#4 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091 scylladb#5 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506 scylladb#6 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666 scylladb#7 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199 scylladb#8 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115 scylladb#9 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486 scylladb#10 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2) scylladb#11 0x42b69d in _start (/local/home/bhalevy/.dtest/dtest-q85idok7/test/node1/bin/scylla+0x42b69d) 0x603001320ac0 is located 0 bytes inside of 32-byte region [0x603001320ac0,0x603001320ae0) freed by thread T0 here: #0 0x7fb86520996f in operator delete(void*) (/lib64/libasan.so.5+0x11096f) scylladb#1 0x77d1889 in deallocate /usr/include/c++/9/ext/new_allocator.h:128 scylladb#2 0x77bcbb2 in deallocate /usr/include/c++/9/bits/alloc_traits.h:470 scylladb#3 0x77bccc5 in _M_deallocate_node_ptr /usr/include/c++/9/bits/hashtable_policy.h:2113 scylladb#4 0x779dc59 in _M_deallocate_node /usr/include/c++/9/bits/hashtable_policy.h:2103 scylladb#5 0x77bc8b6 in _M_deallocate_nodes /usr/include/c++/9/bits/hashtable_policy.h:2124 scylladb#6 0x779d18f in clear /usr/include/c++/9/bits/hashtable.h:2028 scylladb#7 0x7779b8e in ~_Hashtable /usr/include/c++/9/bits/hashtable.h:1352 scylladb#8 0x774cfed in ~unordered_map /usr/include/c++/9/bits/unordered_map.h:102 scylladb#9 0x774d288 in ~shared_ptr_no_esft /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:157 scylladb#10 0x774d2f1 in dispose /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:213 scylladb#11 0x76f9642 in ~lw_shared_ptr /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:300 scylladb#12 0x76f9642 in ~<lambda> /local/home/bhalevy/dev/scylla/distributed_loader.cc:743 scylladb#13 0x771e0a7 in ~<lambda> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1167 scylladb#14 0x777c64c in destroy /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:133 scylladb#15 0x586059 in seastar::noncopyable_function<seastar::future<> ()>::~noncopyable_function() /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:165 scylladb#16 0x5867da in seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}::~future_state() /local/home/bha levy/dev/scylla/seastar/include/seastar/core/future.hh:1192 scylladb#17 0xab704d in seastar::continuation<seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}>::~continu ation() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:507 scylladb#18 0xab7216 in seastar::continuation<seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}>::run_and_ dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:512 scylladb#19 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091 scylladb#20 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506 scylladb#21 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666 scylladb#22 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199 scylladb#23 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115 scylladb#24 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486 scylladb#25 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2) previously allocated by thread T0 here: #0 0x7fb8652089d7 in operator new(unsigned long) (/lib64/libasan.so.5+0x10f9d7) scylladb#1 0x77d1796 in allocate /usr/include/c++/9/ext/new_allocator.h:114 scylladb#2 0x77bcab3 in allocate /usr/include/c++/9/bits/alloc_traits.h:444 scylladb#3 0x779d936 in _M_allocate_node<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable_policy.h:2081 scylladb#4 0x777a2ed in _M_emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable.h:1638 scylladb#5 0x774d707 in emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable.h:749 scylladb#6 0x771ce3e in emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/unordered_map.h:388 scylladb#7 0x76f8280 in operator() /local/home/bhalevy/dev/scylla/distributed_loader.cc:732 scylladb#8 0x774dc5d in _M_invoke /usr/include/c++/9/bits/std_function.h:286 scylladb#9 0x6cde67b in std::function<seastar::future<> (std::filesystem::__cxx11::path, seastar::directory_entry)>::operator()(std::filesystem::__cxx11::path, seastar::directory_entry) const /usr/include/c++/9/bits/std_function.h:690 scylladb#10 0x6cd345d in operator() /local/home/bhalevy/dev/scylla/lister.cc:30 scylladb#11 0x6cd96e0 in apply /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36 scylladb#12 0x6cd9854 in apply<lister::visit(seastar::directory_entry)::<lambda(seastar::directory_entry)>&, seastar::directory_entry&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:44 scylladb#13 0x6cd9953 in apply<lister::visit(seastar::directory_entry)::<lambda(seastar::directory_entry)>&, seastar::directory_entry&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1650 scylladb#14 0x6cd6ba9 in operator() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1168 scylladb#15 0x6cdbac0 in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:101 scylladb#16 0x6ce6178 in seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>::operator()(seastar::directory_entry&&) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:184 scylladb#17 0x6ce3033 in seastar::apply_helper<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, std::tuple<seastar::directory_entry>&&, std::integer_sequence<unsigned long, 0ul> >::apply(seastar::noncopyable_function<seastar::future<> (seastar::directory_e ntry&&)>&&, std::tuple<seastar::directory_entry>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36 scylladb#18 0x6ce3130 in auto seastar::apply<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::directory_entry>(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&, std::tuple<seastar::directory_entry>&&) /local/home/bha levy/dev/scylla/seastar/include/seastar/core/apply.hh:44 scylladb#19 0x6ce322f in seastar::future<> seastar::futurize<seastar::future<> >::apply<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::directory_entry>(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&, std::tuple<s eastar::directory_entry>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1650 scylladb#20 0x6cdf88d in seastar::future<seastar::directory_entry>::then_impl<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<seastar::directory_entry>&&)scylladb#1}::operator()(seastar::future_state<seastar::directory_entry>) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1196 scylladb#21 0x6cea2e0 in seastar::continuation<seastar::future<seastar::directory_entry>::then_impl<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&)::{lamb da()scylladb#1}::operator()() const::{lambda(seastar::future_state<seastar::directory_entry>&&)scylladb#1}, seastar::directory_entry>::run_and_dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:511 scylladb#22 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091 scylladb#23 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506 scylladb#24 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666 scylladb#25 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199 scylladb#26 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115 scylladb#27 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486 scylladb#28 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2) SUMMARY: AddressSanitizer: heap-use-after-free /usr/include/c++/9/bits/hashtable_policy.h:283 in _M_next Shadow bytes around the buggy address: 0x0c068025c100: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c068025c110: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa 0x0c068025c120: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd 0x0c068025c130: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c068025c140: fd fd fd fa fa fa 00 00 00 fa fa fa fd fd fd fa =>0x0c068025c150: fa fa fd fd fd fa fa fa[fd]fd fd fd fa fa fd fd 0x0c068025c160: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c068025c170: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa 0x0c068025c180: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd 0x0c068025c190: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c068025c1a0: fd fd fd fd fa fa fd fd fd fd fa fa 00 00 00 fa Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa Freed heap region: fd Stack left redzone: f1 Stack mid redzone: f2 Stack right redzone: f3 Stack after return: f5 Stack use after scope: f8 Global redzone: f9 Global init order: f6 Poisoned by user: f7 Container overflow: fc Array cookie: ac Intra object redzone: bb ASan internal: fe Left alloca redzone: ca Right alloca redzone: cb Shadow gap: cc ==406079==ABORTING Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Rather than using lw_shared_ptr to keep it around, use do_with. Fixes a use-after-free seen with interrupt_build_process_with_resharding_half_to_max_test in debug mode: ==406079==ERROR: AddressSanitizer: heap-use-after-free on address 0x603001320ac0 at pc 0x00000777a062 bp 0x7ffdeb77b010 sp 0x7ffdeb77b000 READ of size 8 at 0x603001320ac0 thread T0 #0 0x777a061 in _M_next /usr/include/c++/9/bits/hashtable_policy.h:283 scylladb#1 0x779ead7 in _M_incr /usr/include/c++/9/bits/hashtable_policy.h:299 scylladb#2 0x777bbac in operator++ /usr/include/c++/9/bits/hashtable_policy.h:362 scylladb#3 0x77f8384 in run_and_dispose /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future-util.hh:568 scylladb#4 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091 scylladb#5 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506 scylladb#6 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666 scylladb#7 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199 scylladb#8 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115 scylladb#9 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486 scylladb#10 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2) scylladb#11 0x42b69d in _start (/local/home/bhalevy/.dtest/dtest-q85idok7/test/node1/bin/scylla+0x42b69d) 0x603001320ac0 is located 0 bytes inside of 32-byte region [0x603001320ac0,0x603001320ae0) freed by thread T0 here: #0 0x7fb86520996f in operator delete(void*) (/lib64/libasan.so.5+0x11096f) scylladb#1 0x77d1889 in deallocate /usr/include/c++/9/ext/new_allocator.h:128 scylladb#2 0x77bcbb2 in deallocate /usr/include/c++/9/bits/alloc_traits.h:470 scylladb#3 0x77bccc5 in _M_deallocate_node_ptr /usr/include/c++/9/bits/hashtable_policy.h:2113 scylladb#4 0x779dc59 in _M_deallocate_node /usr/include/c++/9/bits/hashtable_policy.h:2103 scylladb#5 0x77bc8b6 in _M_deallocate_nodes /usr/include/c++/9/bits/hashtable_policy.h:2124 scylladb#6 0x779d18f in clear /usr/include/c++/9/bits/hashtable.h:2028 scylladb#7 0x7779b8e in ~_Hashtable /usr/include/c++/9/bits/hashtable.h:1352 scylladb#8 0x774cfed in ~unordered_map /usr/include/c++/9/bits/unordered_map.h:102 scylladb#9 0x774d288 in ~shared_ptr_no_esft /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:157 scylladb#10 0x774d2f1 in dispose /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:213 scylladb#11 0x76f9642 in ~lw_shared_ptr /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:300 scylladb#12 0x76f9642 in ~<lambda> /local/home/bhalevy/dev/scylla/distributed_loader.cc:743 scylladb#13 0x771e0a7 in ~<lambda> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1167 scylladb#14 0x777c64c in destroy /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:133 scylladb#15 0x586059 in seastar::noncopyable_function<seastar::future<> ()>::~noncopyable_function() /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:165 scylladb#16 0x5867da in seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}::~future_state() /local/home/bha levy/dev/scylla/seastar/include/seastar/core/future.hh:1192 scylladb#17 0xab704d in seastar::continuation<seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}>::~continu ation() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:507 scylladb#18 0xab7216 in seastar::continuation<seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}>::run_and_ dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:512 scylladb#19 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091 scylladb#20 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506 scylladb#21 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666 scylladb#22 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199 scylladb#23 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115 scylladb#24 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486 scylladb#25 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2) previously allocated by thread T0 here: #0 0x7fb8652089d7 in operator new(unsigned long) (/lib64/libasan.so.5+0x10f9d7) scylladb#1 0x77d1796 in allocate /usr/include/c++/9/ext/new_allocator.h:114 scylladb#2 0x77bcab3 in allocate /usr/include/c++/9/bits/alloc_traits.h:444 scylladb#3 0x779d936 in _M_allocate_node<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable_policy.h:2081 scylladb#4 0x777a2ed in _M_emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable.h:1638 scylladb#5 0x774d707 in emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable.h:749 scylladb#6 0x771ce3e in emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/unordered_map.h:388 scylladb#7 0x76f8280 in operator() /local/home/bhalevy/dev/scylla/distributed_loader.cc:732 scylladb#8 0x774dc5d in _M_invoke /usr/include/c++/9/bits/std_function.h:286 scylladb#9 0x6cde67b in std::function<seastar::future<> (std::filesystem::__cxx11::path, seastar::directory_entry)>::operator()(std::filesystem::__cxx11::path, seastar::directory_entry) const /usr/include/c++/9/bits/std_function.h:690 scylladb#10 0x6cd345d in operator() /local/home/bhalevy/dev/scylla/lister.cc:30 scylladb#11 0x6cd96e0 in apply /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36 scylladb#12 0x6cd9854 in apply<lister::visit(seastar::directory_entry)::<lambda(seastar::directory_entry)>&, seastar::directory_entry&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:44 scylladb#13 0x6cd9953 in apply<lister::visit(seastar::directory_entry)::<lambda(seastar::directory_entry)>&, seastar::directory_entry&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1650 scylladb#14 0x6cd6ba9 in operator() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1168 scylladb#15 0x6cdbac0 in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:101 scylladb#16 0x6ce6178 in seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>::operator()(seastar::directory_entry&&) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:184 scylladb#17 0x6ce3033 in seastar::apply_helper<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, std::tuple<seastar::directory_entry>&&, std::integer_sequence<unsigned long, 0ul> >::apply(seastar::noncopyable_function<seastar::future<> (seastar::directory_e ntry&&)>&&, std::tuple<seastar::directory_entry>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36 scylladb#18 0x6ce3130 in auto seastar::apply<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::directory_entry>(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&, std::tuple<seastar::directory_entry>&&) /local/home/bha levy/dev/scylla/seastar/include/seastar/core/apply.hh:44 scylladb#19 0x6ce322f in seastar::future<> seastar::futurize<seastar::future<> >::apply<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::directory_entry>(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&, std::tuple<s eastar::directory_entry>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1650 scylladb#20 0x6cdf88d in seastar::future<seastar::directory_entry>::then_impl<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<seastar::directory_entry>&&)scylladb#1}::operator()(seastar::future_state<seastar::directory_entry>) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1196 scylladb#21 0x6cea2e0 in seastar::continuation<seastar::future<seastar::directory_entry>::then_impl<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&)::{lamb da()scylladb#1}::operator()() const::{lambda(seastar::future_state<seastar::directory_entry>&&)scylladb#1}, seastar::directory_entry>::run_and_dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:511 scylladb#22 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091 scylladb#23 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506 scylladb#24 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666 scylladb#25 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199 scylladb#26 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115 scylladb#27 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486 scylladb#28 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2) SUMMARY: AddressSanitizer: heap-use-after-free /usr/include/c++/9/bits/hashtable_policy.h:283 in _M_next Shadow bytes around the buggy address: 0x0c068025c100: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c068025c110: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa 0x0c068025c120: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd 0x0c068025c130: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c068025c140: fd fd fd fa fa fa 00 00 00 fa fa fa fd fd fd fa =>0x0c068025c150: fa fa fd fd fd fa fa fa[fd]fd fd fd fa fa fd fd 0x0c068025c160: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c068025c170: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa 0x0c068025c180: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd 0x0c068025c190: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c068025c1a0: fd fd fd fd fa fa fd fd fd fd fa fa 00 00 00 fa Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa Freed heap region: fd Stack left redzone: f1 Stack mid redzone: f2 Stack right redzone: f3 Stack after return: f5 Stack use after scope: f8 Global redzone: f9 Global init order: f6 Poisoned by user: f7 Container overflow: fc Array cookie: ac Intra object redzone: bb ASan internal: fe Left alloca redzone: ca Right alloca redzone: cb Shadow gap: cc ==406079==ABORTING Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Rather than using lw_shared_ptr to keep it around, use do_with. Fixes a use-after-free seen with interrupt_build_process_with_resharding_half_to_max_test in debug mode: ==406079==ERROR: AddressSanitizer: heap-use-after-free on address 0x603001320ac0 at pc 0x00000777a062 bp 0x7ffdeb77b010 sp 0x7ffdeb77b000 READ of size 8 at 0x603001320ac0 thread T0 #0 0x777a061 in _M_next /usr/include/c++/9/bits/hashtable_policy.h:283 scylladb#1 0x779ead7 in _M_incr /usr/include/c++/9/bits/hashtable_policy.h:299 scylladb#2 0x777bbac in operator++ /usr/include/c++/9/bits/hashtable_policy.h:362 scylladb#3 0x77f8384 in run_and_dispose /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future-util.hh:568 scylladb#4 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091 scylladb#5 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506 scylladb#6 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666 scylladb#7 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199 scylladb#8 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115 scylladb#9 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486 scylladb#10 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2) scylladb#11 0x42b69d in _start (/local/home/bhalevy/.dtest/dtest-q85idok7/test/node1/bin/scylla+0x42b69d) 0x603001320ac0 is located 0 bytes inside of 32-byte region [0x603001320ac0,0x603001320ae0) freed by thread T0 here: #0 0x7fb86520996f in operator delete(void*) (/lib64/libasan.so.5+0x11096f) scylladb#1 0x77d1889 in deallocate /usr/include/c++/9/ext/new_allocator.h:128 scylladb#2 0x77bcbb2 in deallocate /usr/include/c++/9/bits/alloc_traits.h:470 scylladb#3 0x77bccc5 in _M_deallocate_node_ptr /usr/include/c++/9/bits/hashtable_policy.h:2113 scylladb#4 0x779dc59 in _M_deallocate_node /usr/include/c++/9/bits/hashtable_policy.h:2103 scylladb#5 0x77bc8b6 in _M_deallocate_nodes /usr/include/c++/9/bits/hashtable_policy.h:2124 scylladb#6 0x779d18f in clear /usr/include/c++/9/bits/hashtable.h:2028 scylladb#7 0x7779b8e in ~_Hashtable /usr/include/c++/9/bits/hashtable.h:1352 scylladb#8 0x774cfed in ~unordered_map /usr/include/c++/9/bits/unordered_map.h:102 scylladb#9 0x774d288 in ~shared_ptr_no_esft /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:157 scylladb#10 0x774d2f1 in dispose /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:213 scylladb#11 0x76f9642 in ~lw_shared_ptr /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:300 scylladb#12 0x76f9642 in ~<lambda> /local/home/bhalevy/dev/scylla/distributed_loader.cc:743 scylladb#13 0x771e0a7 in ~<lambda> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1167 scylladb#14 0x777c64c in destroy /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:133 scylladb#15 0x586059 in seastar::noncopyable_function<seastar::future<> ()>::~noncopyable_function() /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:165 scylladb#16 0x5867da in seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}::~future_state() /local/home/bha levy/dev/scylla/seastar/include/seastar/core/future.hh:1192 scylladb#17 0xab704d in seastar::continuation<seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}>::~continu ation() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:507 scylladb#18 0xab7216 in seastar::continuation<seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}>::run_and_ dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:512 scylladb#19 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091 scylladb#20 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506 scylladb#21 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666 scylladb#22 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199 scylladb#23 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115 scylladb#24 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486 scylladb#25 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2) previously allocated by thread T0 here: #0 0x7fb8652089d7 in operator new(unsigned long) (/lib64/libasan.so.5+0x10f9d7) scylladb#1 0x77d1796 in allocate /usr/include/c++/9/ext/new_allocator.h:114 scylladb#2 0x77bcab3 in allocate /usr/include/c++/9/bits/alloc_traits.h:444 scylladb#3 0x779d936 in _M_allocate_node<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable_policy.h:2081 scylladb#4 0x777a2ed in _M_emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable.h:1638 scylladb#5 0x774d707 in emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable.h:749 scylladb#6 0x771ce3e in emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/unordered_map.h:388 scylladb#7 0x76f8280 in operator() /local/home/bhalevy/dev/scylla/distributed_loader.cc:732 scylladb#8 0x774dc5d in _M_invoke /usr/include/c++/9/bits/std_function.h:286 scylladb#9 0x6cde67b in std::function<seastar::future<> (std::filesystem::__cxx11::path, seastar::directory_entry)>::operator()(std::filesystem::__cxx11::path, seastar::directory_entry) const /usr/include/c++/9/bits/std_function.h:690 scylladb#10 0x6cd345d in operator() /local/home/bhalevy/dev/scylla/lister.cc:30 scylladb#11 0x6cd96e0 in apply /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36 scylladb#12 0x6cd9854 in apply<lister::visit(seastar::directory_entry)::<lambda(seastar::directory_entry)>&, seastar::directory_entry&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:44 scylladb#13 0x6cd9953 in apply<lister::visit(seastar::directory_entry)::<lambda(seastar::directory_entry)>&, seastar::directory_entry&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1650 scylladb#14 0x6cd6ba9 in operator() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1168 scylladb#15 0x6cdbac0 in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:101 scylladb#16 0x6ce6178 in seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>::operator()(seastar::directory_entry&&) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:184 scylladb#17 0x6ce3033 in seastar::apply_helper<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, std::tuple<seastar::directory_entry>&&, std::integer_sequence<unsigned long, 0ul> >::apply(seastar::noncopyable_function<seastar::future<> (seastar::directory_e ntry&&)>&&, std::tuple<seastar::directory_entry>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36 scylladb#18 0x6ce3130 in auto seastar::apply<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::directory_entry>(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&, std::tuple<seastar::directory_entry>&&) /local/home/bha levy/dev/scylla/seastar/include/seastar/core/apply.hh:44 scylladb#19 0x6ce322f in seastar::future<> seastar::futurize<seastar::future<> >::apply<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::directory_entry>(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&, std::tuple<s eastar::directory_entry>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1650 scylladb#20 0x6cdf88d in seastar::future<seastar::directory_entry>::then_impl<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<seastar::directory_entry>&&)scylladb#1}::operator()(seastar::future_state<seastar::directory_entry>) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1196 scylladb#21 0x6cea2e0 in seastar::continuation<seastar::future<seastar::directory_entry>::then_impl<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&)::{lamb da()scylladb#1}::operator()() const::{lambda(seastar::future_state<seastar::directory_entry>&&)scylladb#1}, seastar::directory_entry>::run_and_dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:511 scylladb#22 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091 scylladb#23 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506 scylladb#24 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666 scylladb#25 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199 scylladb#26 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115 scylladb#27 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486 scylladb#28 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2) SUMMARY: AddressSanitizer: heap-use-after-free /usr/include/c++/9/bits/hashtable_policy.h:283 in _M_next Shadow bytes around the buggy address: 0x0c068025c100: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c068025c110: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa 0x0c068025c120: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd 0x0c068025c130: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c068025c140: fd fd fd fa fa fa 00 00 00 fa fa fa fd fd fd fa =>0x0c068025c150: fa fa fd fd fd fa fa fa[fd]fd fd fd fa fa fd fd 0x0c068025c160: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c068025c170: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa 0x0c068025c180: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd 0x0c068025c190: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c068025c1a0: fd fd fd fd fa fa fd fd fd fd fa fa 00 00 00 fa Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa Freed heap region: fd Stack left redzone: f1 Stack mid redzone: f2 Stack right redzone: f3 Stack after return: f5 Stack use after scope: f8 Global redzone: f9 Global init order: f6 Poisoned by user: f7 Container overflow: fc Array cookie: ac Intra object redzone: bb ASan internal: fe Left alloca redzone: ca Right alloca redzone: cb Shadow gap: cc ==406079==ABORTING Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Rather than using lw_shared_ptr to keep it around, use do_with. Fixes a use-after-free seen with interrupt_build_process_with_resharding_half_to_max_test in debug mode: ==406079==ERROR: AddressSanitizer: heap-use-after-free on address 0x603001320ac0 at pc 0x00000777a062 bp 0x7ffdeb77b010 sp 0x7ffdeb77b000 READ of size 8 at 0x603001320ac0 thread T0 #0 0x777a061 in _M_next /usr/include/c++/9/bits/hashtable_policy.h:283 scylladb#1 0x779ead7 in _M_incr /usr/include/c++/9/bits/hashtable_policy.h:299 scylladb#2 0x777bbac in operator++ /usr/include/c++/9/bits/hashtable_policy.h:362 scylladb#3 0x77f8384 in run_and_dispose /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future-util.hh:568 scylladb#4 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091 scylladb#5 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506 scylladb#6 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666 scylladb#7 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199 scylladb#8 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115 scylladb#9 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486 scylladb#10 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2) scylladb#11 0x42b69d in _start (/local/home/bhalevy/.dtest/dtest-q85idok7/test/node1/bin/scylla+0x42b69d) 0x603001320ac0 is located 0 bytes inside of 32-byte region [0x603001320ac0,0x603001320ae0) freed by thread T0 here: #0 0x7fb86520996f in operator delete(void*) (/lib64/libasan.so.5+0x11096f) scylladb#1 0x77d1889 in deallocate /usr/include/c++/9/ext/new_allocator.h:128 scylladb#2 0x77bcbb2 in deallocate /usr/include/c++/9/bits/alloc_traits.h:470 scylladb#3 0x77bccc5 in _M_deallocate_node_ptr /usr/include/c++/9/bits/hashtable_policy.h:2113 scylladb#4 0x779dc59 in _M_deallocate_node /usr/include/c++/9/bits/hashtable_policy.h:2103 scylladb#5 0x77bc8b6 in _M_deallocate_nodes /usr/include/c++/9/bits/hashtable_policy.h:2124 scylladb#6 0x779d18f in clear /usr/include/c++/9/bits/hashtable.h:2028 scylladb#7 0x7779b8e in ~_Hashtable /usr/include/c++/9/bits/hashtable.h:1352 scylladb#8 0x774cfed in ~unordered_map /usr/include/c++/9/bits/unordered_map.h:102 scylladb#9 0x774d288 in ~shared_ptr_no_esft /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:157 scylladb#10 0x774d2f1 in dispose /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:213 scylladb#11 0x76f9642 in ~lw_shared_ptr /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:300 scylladb#12 0x76f9642 in ~<lambda> /local/home/bhalevy/dev/scylla/distributed_loader.cc:743 scylladb#13 0x771e0a7 in ~<lambda> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1167 scylladb#14 0x777c64c in destroy /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:133 scylladb#15 0x586059 in seastar::noncopyable_function<seastar::future<> ()>::~noncopyable_function() /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:165 scylladb#16 0x5867da in seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}::~future_state() /local/home/bha levy/dev/scylla/seastar/include/seastar/core/future.hh:1192 scylladb#17 0xab704d in seastar::continuation<seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}>::~continu ation() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:507 scylladb#18 0xab7216 in seastar::continuation<seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}>::run_and_ dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:512 scylladb#19 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091 scylladb#20 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506 scylladb#21 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666 scylladb#22 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199 scylladb#23 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115 scylladb#24 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486 scylladb#25 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2) previously allocated by thread T0 here: #0 0x7fb8652089d7 in operator new(unsigned long) (/lib64/libasan.so.5+0x10f9d7) scylladb#1 0x77d1796 in allocate /usr/include/c++/9/ext/new_allocator.h:114 scylladb#2 0x77bcab3 in allocate /usr/include/c++/9/bits/alloc_traits.h:444 scylladb#3 0x779d936 in _M_allocate_node<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable_policy.h:2081 scylladb#4 0x777a2ed in _M_emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable.h:1638 scylladb#5 0x774d707 in emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable.h:749 scylladb#6 0x771ce3e in emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/unordered_map.h:388 scylladb#7 0x76f8280 in operator() /local/home/bhalevy/dev/scylla/distributed_loader.cc:732 scylladb#8 0x774dc5d in _M_invoke /usr/include/c++/9/bits/std_function.h:286 scylladb#9 0x6cde67b in std::function<seastar::future<> (std::filesystem::__cxx11::path, seastar::directory_entry)>::operator()(std::filesystem::__cxx11::path, seastar::directory_entry) const /usr/include/c++/9/bits/std_function.h:690 scylladb#10 0x6cd345d in operator() /local/home/bhalevy/dev/scylla/lister.cc:30 scylladb#11 0x6cd96e0 in apply /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36 scylladb#12 0x6cd9854 in apply<lister::visit(seastar::directory_entry)::<lambda(seastar::directory_entry)>&, seastar::directory_entry&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:44 scylladb#13 0x6cd9953 in apply<lister::visit(seastar::directory_entry)::<lambda(seastar::directory_entry)>&, seastar::directory_entry&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1650 scylladb#14 0x6cd6ba9 in operator() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1168 scylladb#15 0x6cdbac0 in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:101 scylladb#16 0x6ce6178 in seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>::operator()(seastar::directory_entry&&) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:184 scylladb#17 0x6ce3033 in seastar::apply_helper<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, std::tuple<seastar::directory_entry>&&, std::integer_sequence<unsigned long, 0ul> >::apply(seastar::noncopyable_function<seastar::future<> (seastar::directory_e ntry&&)>&&, std::tuple<seastar::directory_entry>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36 scylladb#18 0x6ce3130 in auto seastar::apply<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::directory_entry>(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&, std::tuple<seastar::directory_entry>&&) /local/home/bha levy/dev/scylla/seastar/include/seastar/core/apply.hh:44 scylladb#19 0x6ce322f in seastar::future<> seastar::futurize<seastar::future<> >::apply<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::directory_entry>(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&, std::tuple<s eastar::directory_entry>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1650 scylladb#20 0x6cdf88d in seastar::future<seastar::directory_entry>::then_impl<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<seastar::directory_entry>&&)scylladb#1}::operator()(seastar::future_state<seastar::directory_entry>) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1196 scylladb#21 0x6cea2e0 in seastar::continuation<seastar::future<seastar::directory_entry>::then_impl<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&)::{lamb da()scylladb#1}::operator()() const::{lambda(seastar::future_state<seastar::directory_entry>&&)scylladb#1}, seastar::directory_entry>::run_and_dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:511 scylladb#22 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091 scylladb#23 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506 scylladb#24 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666 scylladb#25 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199 scylladb#26 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115 scylladb#27 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486 scylladb#28 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2) SUMMARY: AddressSanitizer: heap-use-after-free /usr/include/c++/9/bits/hashtable_policy.h:283 in _M_next Shadow bytes around the buggy address: 0x0c068025c100: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c068025c110: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa 0x0c068025c120: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd 0x0c068025c130: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c068025c140: fd fd fd fa fa fa 00 00 00 fa fa fa fd fd fd fa =>0x0c068025c150: fa fa fd fd fd fa fa fa[fd]fd fd fd fa fa fd fd 0x0c068025c160: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c068025c170: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa 0x0c068025c180: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd 0x0c068025c190: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c068025c1a0: fd fd fd fd fa fa fd fd fd fd fa fa 00 00 00 fa Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa Freed heap region: fd Stack left redzone: f1 Stack mid redzone: f2 Stack right redzone: f3 Stack after return: f5 Stack use after scope: f8 Global redzone: f9 Global init order: f6 Poisoned by user: f7 Container overflow: fc Array cookie: ac Intra object redzone: bb ASan internal: fe Left alloca redzone: ca Right alloca redzone: cb Shadow gap: cc ==406079==ABORTING Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Rather than using lw_shared_ptr to keep it around, use do_with. Fixes a use-after-free seen with interrupt_build_process_with_resharding_half_to_max_test in debug mode: ==406079==ERROR: AddressSanitizer: heap-use-after-free on address 0x603001320ac0 at pc 0x00000777a062 bp 0x7ffdeb77b010 sp 0x7ffdeb77b000 READ of size 8 at 0x603001320ac0 thread T0 #0 0x777a061 in _M_next /usr/include/c++/9/bits/hashtable_policy.h:283 scylladb#1 0x779ead7 in _M_incr /usr/include/c++/9/bits/hashtable_policy.h:299 scylladb#2 0x777bbac in operator++ /usr/include/c++/9/bits/hashtable_policy.h:362 scylladb#3 0x77f8384 in run_and_dispose /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future-util.hh:568 scylladb#4 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091 scylladb#5 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506 scylladb#6 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666 scylladb#7 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199 scylladb#8 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115 scylladb#9 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486 scylladb#10 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2) scylladb#11 0x42b69d in _start (/local/home/bhalevy/.dtest/dtest-q85idok7/test/node1/bin/scylla+0x42b69d) 0x603001320ac0 is located 0 bytes inside of 32-byte region [0x603001320ac0,0x603001320ae0) freed by thread T0 here: #0 0x7fb86520996f in operator delete(void*) (/lib64/libasan.so.5+0x11096f) scylladb#1 0x77d1889 in deallocate /usr/include/c++/9/ext/new_allocator.h:128 scylladb#2 0x77bcbb2 in deallocate /usr/include/c++/9/bits/alloc_traits.h:470 scylladb#3 0x77bccc5 in _M_deallocate_node_ptr /usr/include/c++/9/bits/hashtable_policy.h:2113 scylladb#4 0x779dc59 in _M_deallocate_node /usr/include/c++/9/bits/hashtable_policy.h:2103 scylladb#5 0x77bc8b6 in _M_deallocate_nodes /usr/include/c++/9/bits/hashtable_policy.h:2124 scylladb#6 0x779d18f in clear /usr/include/c++/9/bits/hashtable.h:2028 scylladb#7 0x7779b8e in ~_Hashtable /usr/include/c++/9/bits/hashtable.h:1352 scylladb#8 0x774cfed in ~unordered_map /usr/include/c++/9/bits/unordered_map.h:102 scylladb#9 0x774d288 in ~shared_ptr_no_esft /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:157 scylladb#10 0x774d2f1 in dispose /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:213 scylladb#11 0x76f9642 in ~lw_shared_ptr /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:300 scylladb#12 0x76f9642 in ~<lambda> /local/home/bhalevy/dev/scylla/distributed_loader.cc:743 scylladb#13 0x771e0a7 in ~<lambda> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1167 scylladb#14 0x777c64c in destroy /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:133 scylladb#15 0x586059 in seastar::noncopyable_function<seastar::future<> ()>::~noncopyable_function() /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:165 scylladb#16 0x5867da in seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}::~future_state() /local/home/bha levy/dev/scylla/seastar/include/seastar/core/future.hh:1192 scylladb#17 0xab704d in seastar::continuation<seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}>::~continu ation() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:507 scylladb#18 0xab7216 in seastar::continuation<seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}>::run_and_ dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:512 scylladb#19 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091 scylladb#20 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506 scylladb#21 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666 scylladb#22 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199 scylladb#23 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115 scylladb#24 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486 scylladb#25 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2) previously allocated by thread T0 here: #0 0x7fb8652089d7 in operator new(unsigned long) (/lib64/libasan.so.5+0x10f9d7) scylladb#1 0x77d1796 in allocate /usr/include/c++/9/ext/new_allocator.h:114 scylladb#2 0x77bcab3 in allocate /usr/include/c++/9/bits/alloc_traits.h:444 scylladb#3 0x779d936 in _M_allocate_node<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable_policy.h:2081 scylladb#4 0x777a2ed in _M_emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable.h:1638 scylladb#5 0x774d707 in emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable.h:749 scylladb#6 0x771ce3e in emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/unordered_map.h:388 scylladb#7 0x76f8280 in operator() /local/home/bhalevy/dev/scylla/distributed_loader.cc:732 scylladb#8 0x774dc5d in _M_invoke /usr/include/c++/9/bits/std_function.h:286 scylladb#9 0x6cde67b in std::function<seastar::future<> (std::filesystem::__cxx11::path, seastar::directory_entry)>::operator()(std::filesystem::__cxx11::path, seastar::directory_entry) const /usr/include/c++/9/bits/std_function.h:690 scylladb#10 0x6cd345d in operator() /local/home/bhalevy/dev/scylla/lister.cc:30 scylladb#11 0x6cd96e0 in apply /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36 scylladb#12 0x6cd9854 in apply<lister::visit(seastar::directory_entry)::<lambda(seastar::directory_entry)>&, seastar::directory_entry&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:44 scylladb#13 0x6cd9953 in apply<lister::visit(seastar::directory_entry)::<lambda(seastar::directory_entry)>&, seastar::directory_entry&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1650 scylladb#14 0x6cd6ba9 in operator() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1168 scylladb#15 0x6cdbac0 in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:101 scylladb#16 0x6ce6178 in seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>::operator()(seastar::directory_entry&&) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:184 scylladb#17 0x6ce3033 in seastar::apply_helper<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, std::tuple<seastar::directory_entry>&&, std::integer_sequence<unsigned long, 0ul> >::apply(seastar::noncopyable_function<seastar::future<> (seastar::directory_e ntry&&)>&&, std::tuple<seastar::directory_entry>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36 scylladb#18 0x6ce3130 in auto seastar::apply<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::directory_entry>(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&, std::tuple<seastar::directory_entry>&&) /local/home/bha levy/dev/scylla/seastar/include/seastar/core/apply.hh:44 scylladb#19 0x6ce322f in seastar::future<> seastar::futurize<seastar::future<> >::apply<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::directory_entry>(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&, std::tuple<s eastar::directory_entry>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1650 scylladb#20 0x6cdf88d in seastar::future<seastar::directory_entry>::then_impl<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<seastar::directory_entry>&&)scylladb#1}::operator()(seastar::future_state<seastar::directory_entry>) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1196 scylladb#21 0x6cea2e0 in seastar::continuation<seastar::future<seastar::directory_entry>::then_impl<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&)::{lamb da()scylladb#1}::operator()() const::{lambda(seastar::future_state<seastar::directory_entry>&&)scylladb#1}, seastar::directory_entry>::run_and_dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:511 scylladb#22 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091 scylladb#23 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506 scylladb#24 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666 scylladb#25 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199 scylladb#26 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115 scylladb#27 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486 scylladb#28 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2) SUMMARY: AddressSanitizer: heap-use-after-free /usr/include/c++/9/bits/hashtable_policy.h:283 in _M_next Shadow bytes around the buggy address: 0x0c068025c100: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c068025c110: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa 0x0c068025c120: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd 0x0c068025c130: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c068025c140: fd fd fd fa fa fa 00 00 00 fa fa fa fd fd fd fa =>0x0c068025c150: fa fa fd fd fd fa fa fa[fd]fd fd fd fa fa fd fd 0x0c068025c160: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c068025c170: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa 0x0c068025c180: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd 0x0c068025c190: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c068025c1a0: fd fd fd fd fa fa fd fd fd fd fa fa 00 00 00 fa Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa Freed heap region: fd Stack left redzone: f1 Stack mid redzone: f2 Stack right redzone: f3 Stack after return: f5 Stack use after scope: f8 Global redzone: f9 Global init order: f6 Poisoned by user: f7 Container overflow: fc Array cookie: ac Intra object redzone: bb ASan internal: fe Left alloca redzone: ca Right alloca redzone: cb Shadow gap: cc ==406079==ABORTING Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Rather than using lw_shared_ptr to keep it around, use do_with. Fixes a use-after-free seen with interrupt_build_process_with_resharding_half_to_max_test in debug mode: `verifier` was not captured in the scan_dir.then() continuation. We could add a capture in that lambda, but it's cleaner and safer to let do_with hold on to it instead. ==406079==ERROR: AddressSanitizer: heap-use-after-free on address 0x603001320ac0 at pc 0x00000777a062 bp 0x7ffdeb77b010 sp 0x7ffdeb77b000 READ of size 8 at 0x603001320ac0 thread T0 #0 0x777a061 in _M_next /usr/include/c++/9/bits/hashtable_policy.h:283 scylladb#1 0x779ead7 in _M_incr /usr/include/c++/9/bits/hashtable_policy.h:299 scylladb#2 0x777bbac in operator++ /usr/include/c++/9/bits/hashtable_policy.h:362 scylladb#3 0x77f8384 in run_and_dispose /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future-util.hh:568 scylladb#4 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091 scylladb#5 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506 scylladb#6 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666 scylladb#7 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199 scylladb#8 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115 scylladb#9 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486 scylladb#10 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2) scylladb#11 0x42b69d in _start (/local/home/bhalevy/.dtest/dtest-q85idok7/test/node1/bin/scylla+0x42b69d) 0x603001320ac0 is located 0 bytes inside of 32-byte region [0x603001320ac0,0x603001320ae0) freed by thread T0 here: #0 0x7fb86520996f in operator delete(void*) (/lib64/libasan.so.5+0x11096f) scylladb#1 0x77d1889 in deallocate /usr/include/c++/9/ext/new_allocator.h:128 scylladb#2 0x77bcbb2 in deallocate /usr/include/c++/9/bits/alloc_traits.h:470 scylladb#3 0x77bccc5 in _M_deallocate_node_ptr /usr/include/c++/9/bits/hashtable_policy.h:2113 scylladb#4 0x779dc59 in _M_deallocate_node /usr/include/c++/9/bits/hashtable_policy.h:2103 scylladb#5 0x77bc8b6 in _M_deallocate_nodes /usr/include/c++/9/bits/hashtable_policy.h:2124 scylladb#6 0x779d18f in clear /usr/include/c++/9/bits/hashtable.h:2028 scylladb#7 0x7779b8e in ~_Hashtable /usr/include/c++/9/bits/hashtable.h:1352 scylladb#8 0x774cfed in ~unordered_map /usr/include/c++/9/bits/unordered_map.h:102 scylladb#9 0x774d288 in ~shared_ptr_no_esft /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:157 scylladb#10 0x774d2f1 in dispose /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:213 scylladb#11 0x76f9642 in ~lw_shared_ptr /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:300 scylladb#12 0x76f9642 in ~<lambda> /local/home/bhalevy/dev/scylla/distributed_loader.cc:743 scylladb#13 0x771e0a7 in ~<lambda> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1167 scylladb#14 0x777c64c in destroy /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:133 scylladb#15 0x586059 in seastar::noncopyable_function<seastar::future<> ()>::~noncopyable_function() /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:165 scylladb#16 0x5867da in seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}::~future_state() /local/home/bha levy/dev/scylla/seastar/include/seastar/core/future.hh:1192 scylladb#17 0xab704d in seastar::continuation<seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}>::~continu ation() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:507 scylladb#18 0xab7216 in seastar::continuation<seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}>::run_and_ dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:512 scylladb#19 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091 scylladb#20 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506 scylladb#21 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666 scylladb#22 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199 scylladb#23 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115 scylladb#24 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486 scylladb#25 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2) previously allocated by thread T0 here: #0 0x7fb8652089d7 in operator new(unsigned long) (/lib64/libasan.so.5+0x10f9d7) scylladb#1 0x77d1796 in allocate /usr/include/c++/9/ext/new_allocator.h:114 scylladb#2 0x77bcab3 in allocate /usr/include/c++/9/bits/alloc_traits.h:444 scylladb#3 0x779d936 in _M_allocate_node<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable_policy.h:2081 scylladb#4 0x777a2ed in _M_emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable.h:1638 scylladb#5 0x774d707 in emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable.h:749 scylladb#6 0x771ce3e in emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/unordered_map.h:388 scylladb#7 0x76f8280 in operator() /local/home/bhalevy/dev/scylla/distributed_loader.cc:732 scylladb#8 0x774dc5d in _M_invoke /usr/include/c++/9/bits/std_function.h:286 scylladb#9 0x6cde67b in std::function<seastar::future<> (std::filesystem::__cxx11::path, seastar::directory_entry)>::operator()(std::filesystem::__cxx11::path, seastar::directory_entry) const /usr/include/c++/9/bits/std_function.h:690 scylladb#10 0x6cd345d in operator() /local/home/bhalevy/dev/scylla/lister.cc:30 scylladb#11 0x6cd96e0 in apply /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36 scylladb#12 0x6cd9854 in apply<lister::visit(seastar::directory_entry)::<lambda(seastar::directory_entry)>&, seastar::directory_entry&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:44 scylladb#13 0x6cd9953 in apply<lister::visit(seastar::directory_entry)::<lambda(seastar::directory_entry)>&, seastar::directory_entry&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1650 scylladb#14 0x6cd6ba9 in operator() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1168 scylladb#15 0x6cdbac0 in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:101 scylladb#16 0x6ce6178 in seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>::operator()(seastar::directory_entry&&) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:184 scylladb#17 0x6ce3033 in seastar::apply_helper<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, std::tuple<seastar::directory_entry>&&, std::integer_sequence<unsigned long, 0ul> >::apply(seastar::noncopyable_function<seastar::future<> (seastar::directory_e ntry&&)>&&, std::tuple<seastar::directory_entry>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36 scylladb#18 0x6ce3130 in auto seastar::apply<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::directory_entry>(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&, std::tuple<seastar::directory_entry>&&) /local/home/bha levy/dev/scylla/seastar/include/seastar/core/apply.hh:44 scylladb#19 0x6ce322f in seastar::future<> seastar::futurize<seastar::future<> >::apply<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::directory_entry>(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&, std::tuple<s eastar::directory_entry>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1650 scylladb#20 0x6cdf88d in seastar::future<seastar::directory_entry>::then_impl<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<seastar::directory_entry>&&)scylladb#1}::operator()(seastar::future_state<seastar::directory_entry>) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1196 scylladb#21 0x6cea2e0 in seastar::continuation<seastar::future<seastar::directory_entry>::then_impl<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&)::{lamb da()scylladb#1}::operator()() const::{lambda(seastar::future_state<seastar::directory_entry>&&)scylladb#1}, seastar::directory_entry>::run_and_dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:511 scylladb#22 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091 scylladb#23 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506 scylladb#24 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666 scylladb#25 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199 scylladb#26 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115 scylladb#27 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486 scylladb#28 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2) SUMMARY: AddressSanitizer: heap-use-after-free /usr/include/c++/9/bits/hashtable_policy.h:283 in _M_next Shadow bytes around the buggy address: 0x0c068025c100: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c068025c110: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa 0x0c068025c120: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd 0x0c068025c130: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c068025c140: fd fd fd fa fa fa 00 00 00 fa fa fa fd fd fd fa =>0x0c068025c150: fa fa fd fd fd fa fa fa[fd]fd fd fd fa fa fd fd 0x0c068025c160: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c068025c170: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa 0x0c068025c180: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd 0x0c068025c190: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c068025c1a0: fd fd fd fd fa fa fd fd fd fd fa fa 00 00 00 fa Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa Freed heap region: fd Stack left redzone: f1 Stack mid redzone: f2 Stack right redzone: f3 Stack after return: f5 Stack use after scope: f8 Global redzone: f9 Global init order: f6 Poisoned by user: f7 Container overflow: fc Array cookie: ac Intra object redzone: bb ASan internal: fe Left alloca redzone: ca Right alloca redzone: cb Shadow gap: cc ==406079==ABORTING Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Rather than using lw_shared_ptr to keep it around, use do_with. Fixes a use-after-free seen with interrupt_build_process_with_resharding_half_to_max_test in debug mode: `verifier` was not captured in the scan_dir.then() continuation. We could add a capture in that lambda, but it's cleaner and safer to let do_with hold on to it instead. ==406079==ERROR: AddressSanitizer: heap-use-after-free on address 0x603001320ac0 at pc 0x00000777a062 bp 0x7ffdeb77b010 sp 0x7ffdeb77b000 READ of size 8 at 0x603001320ac0 thread T0 #0 0x777a061 in _M_next /usr/include/c++/9/bits/hashtable_policy.h:283 scylladb#1 0x779ead7 in _M_incr /usr/include/c++/9/bits/hashtable_policy.h:299 scylladb#2 0x777bbac in operator++ /usr/include/c++/9/bits/hashtable_policy.h:362 scylladb#3 0x77f8384 in run_and_dispose /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future-util.hh:568 scylladb#4 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091 scylladb#5 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506 scylladb#6 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666 scylladb#7 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199 scylladb#8 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115 scylladb#9 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486 scylladb#10 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2) scylladb#11 0x42b69d in _start (/local/home/bhalevy/.dtest/dtest-q85idok7/test/node1/bin/scylla+0x42b69d) 0x603001320ac0 is located 0 bytes inside of 32-byte region [0x603001320ac0,0x603001320ae0) freed by thread T0 here: #0 0x7fb86520996f in operator delete(void*) (/lib64/libasan.so.5+0x11096f) scylladb#1 0x77d1889 in deallocate /usr/include/c++/9/ext/new_allocator.h:128 scylladb#2 0x77bcbb2 in deallocate /usr/include/c++/9/bits/alloc_traits.h:470 scylladb#3 0x77bccc5 in _M_deallocate_node_ptr /usr/include/c++/9/bits/hashtable_policy.h:2113 scylladb#4 0x779dc59 in _M_deallocate_node /usr/include/c++/9/bits/hashtable_policy.h:2103 scylladb#5 0x77bc8b6 in _M_deallocate_nodes /usr/include/c++/9/bits/hashtable_policy.h:2124 scylladb#6 0x779d18f in clear /usr/include/c++/9/bits/hashtable.h:2028 scylladb#7 0x7779b8e in ~_Hashtable /usr/include/c++/9/bits/hashtable.h:1352 scylladb#8 0x774cfed in ~unordered_map /usr/include/c++/9/bits/unordered_map.h:102 scylladb#9 0x774d288 in ~shared_ptr_no_esft /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:157 scylladb#10 0x774d2f1 in dispose /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:213 scylladb#11 0x76f9642 in ~lw_shared_ptr /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:300 scylladb#12 0x76f9642 in ~<lambda> /local/home/bhalevy/dev/scylla/distributed_loader.cc:743 scylladb#13 0x771e0a7 in ~<lambda> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1167 scylladb#14 0x777c64c in destroy /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:133 scylladb#15 0x586059 in seastar::noncopyable_function<seastar::future<> ()>::~noncopyable_function() /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:165 scylladb#16 0x5867da in seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}::~future_state() /local/home/bha levy/dev/scylla/seastar/include/seastar/core/future.hh:1192 scylladb#17 0xab704d in seastar::continuation<seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}>::~continu ation() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:507 scylladb#18 0xab7216 in seastar::continuation<seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}>::run_and_ dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:512 scylladb#19 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091 scylladb#20 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506 scylladb#21 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666 scylladb#22 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199 scylladb#23 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115 scylladb#24 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486 scylladb#25 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2) previously allocated by thread T0 here: #0 0x7fb8652089d7 in operator new(unsigned long) (/lib64/libasan.so.5+0x10f9d7) scylladb#1 0x77d1796 in allocate /usr/include/c++/9/ext/new_allocator.h:114 scylladb#2 0x77bcab3 in allocate /usr/include/c++/9/bits/alloc_traits.h:444 scylladb#3 0x779d936 in _M_allocate_node<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable_policy.h:2081 scylladb#4 0x777a2ed in _M_emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable.h:1638 scylladb#5 0x774d707 in emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable.h:749 scylladb#6 0x771ce3e in emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/unordered_map.h:388 scylladb#7 0x76f8280 in operator() /local/home/bhalevy/dev/scylla/distributed_loader.cc:732 scylladb#8 0x774dc5d in _M_invoke /usr/include/c++/9/bits/std_function.h:286 scylladb#9 0x6cde67b in std::function<seastar::future<> (std::filesystem::__cxx11::path, seastar::directory_entry)>::operator()(std::filesystem::__cxx11::path, seastar::directory_entry) const /usr/include/c++/9/bits/std_function.h:690 scylladb#10 0x6cd345d in operator() /local/home/bhalevy/dev/scylla/lister.cc:30 scylladb#11 0x6cd96e0 in apply /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36 scylladb#12 0x6cd9854 in apply<lister::visit(seastar::directory_entry)::<lambda(seastar::directory_entry)>&, seastar::directory_entry&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:44 scylladb#13 0x6cd9953 in apply<lister::visit(seastar::directory_entry)::<lambda(seastar::directory_entry)>&, seastar::directory_entry&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1650 scylladb#14 0x6cd6ba9 in operator() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1168 scylladb#15 0x6cdbac0 in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:101 scylladb#16 0x6ce6178 in seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>::operator()(seastar::directory_entry&&) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:184 scylladb#17 0x6ce3033 in seastar::apply_helper<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, std::tuple<seastar::directory_entry>&&, std::integer_sequence<unsigned long, 0ul> >::apply(seastar::noncopyable_function<seastar::future<> (seastar::directory_e ntry&&)>&&, std::tuple<seastar::directory_entry>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36 scylladb#18 0x6ce3130 in auto seastar::apply<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::directory_entry>(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&, std::tuple<seastar::directory_entry>&&) /local/home/bha levy/dev/scylla/seastar/include/seastar/core/apply.hh:44 scylladb#19 0x6ce322f in seastar::future<> seastar::futurize<seastar::future<> >::apply<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::directory_entry>(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&, std::tuple<s eastar::directory_entry>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1650 scylladb#20 0x6cdf88d in seastar::future<seastar::directory_entry>::then_impl<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<seastar::directory_entry>&&)scylladb#1}::operator()(seastar::future_state<seastar::directory_entry>) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1196 scylladb#21 0x6cea2e0 in seastar::continuation<seastar::future<seastar::directory_entry>::then_impl<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&)::{lamb da()scylladb#1}::operator()() const::{lambda(seastar::future_state<seastar::directory_entry>&&)scylladb#1}, seastar::directory_entry>::run_and_dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:511 scylladb#22 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091 scylladb#23 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506 scylladb#24 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666 scylladb#25 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199 scylladb#26 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115 scylladb#27 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486 scylladb#28 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2) SUMMARY: AddressSanitizer: heap-use-after-free /usr/include/c++/9/bits/hashtable_policy.h:283 in _M_next Shadow bytes around the buggy address: 0x0c068025c100: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c068025c110: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa 0x0c068025c120: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd 0x0c068025c130: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c068025c140: fd fd fd fa fa fa 00 00 00 fa fa fa fd fd fd fa =>0x0c068025c150: fa fa fd fd fd fa fa fa[fd]fd fd fd fa fa fd fd 0x0c068025c160: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c068025c170: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa 0x0c068025c180: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd 0x0c068025c190: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c068025c1a0: fd fd fd fd fa fa fd fd fd fd fa fa 00 00 00 fa Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa Freed heap region: fd Stack left redzone: f1 Stack mid redzone: f2 Stack right redzone: f3 Stack after return: f5 Stack use after scope: f8 Global redzone: f9 Global init order: f6 Poisoned by user: f7 Container overflow: fc Array cookie: ac Intra object redzone: bb ASan internal: fe Left alloca redzone: ca Right alloca redzone: cb Shadow gap: cc ==406079==ABORTING Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Rather than using lw_shared_ptr to keep it around, use do_with. Fixes a use-after-free seen with interrupt_build_process_with_resharding_half_to_max_test in debug mode: `verifier` was not captured in the scan_dir.then() continuation. We could add a capture in that lambda, but it's cleaner and safer to let do_with hold on to it instead. ==406079==ERROR: AddressSanitizer: heap-use-after-free on address 0x603001320ac0 at pc 0x00000777a062 bp 0x7ffdeb77b010 sp 0x7ffdeb77b000 READ of size 8 at 0x603001320ac0 thread T0 #0 0x777a061 in _M_next /usr/include/c++/9/bits/hashtable_policy.h:283 scylladb#1 0x779ead7 in _M_incr /usr/include/c++/9/bits/hashtable_policy.h:299 scylladb#2 0x777bbac in operator++ /usr/include/c++/9/bits/hashtable_policy.h:362 scylladb#3 0x77f8384 in run_and_dispose /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future-util.hh:568 scylladb#4 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091 scylladb#5 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506 scylladb#6 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666 scylladb#7 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199 scylladb#8 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115 scylladb#9 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486 scylladb#10 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2) scylladb#11 0x42b69d in _start (/local/home/bhalevy/.dtest/dtest-q85idok7/test/node1/bin/scylla+0x42b69d) 0x603001320ac0 is located 0 bytes inside of 32-byte region [0x603001320ac0,0x603001320ae0) freed by thread T0 here: #0 0x7fb86520996f in operator delete(void*) (/lib64/libasan.so.5+0x11096f) scylladb#1 0x77d1889 in deallocate /usr/include/c++/9/ext/new_allocator.h:128 scylladb#2 0x77bcbb2 in deallocate /usr/include/c++/9/bits/alloc_traits.h:470 scylladb#3 0x77bccc5 in _M_deallocate_node_ptr /usr/include/c++/9/bits/hashtable_policy.h:2113 scylladb#4 0x779dc59 in _M_deallocate_node /usr/include/c++/9/bits/hashtable_policy.h:2103 scylladb#5 0x77bc8b6 in _M_deallocate_nodes /usr/include/c++/9/bits/hashtable_policy.h:2124 scylladb#6 0x779d18f in clear /usr/include/c++/9/bits/hashtable.h:2028 scylladb#7 0x7779b8e in ~_Hashtable /usr/include/c++/9/bits/hashtable.h:1352 scylladb#8 0x774cfed in ~unordered_map /usr/include/c++/9/bits/unordered_map.h:102 scylladb#9 0x774d288 in ~shared_ptr_no_esft /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:157 scylladb#10 0x774d2f1 in dispose /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:213 scylladb#11 0x76f9642 in ~lw_shared_ptr /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:300 scylladb#12 0x76f9642 in ~<lambda> /local/home/bhalevy/dev/scylla/distributed_loader.cc:743 scylladb#13 0x771e0a7 in ~<lambda> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1167 scylladb#14 0x777c64c in destroy /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:133 scylladb#15 0x586059 in seastar::noncopyable_function<seastar::future<> ()>::~noncopyable_function() /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:165 scylladb#16 0x5867da in seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}::~future_state() /local/home/bha levy/dev/scylla/seastar/include/seastar/core/future.hh:1192 scylladb#17 0xab704d in seastar::continuation<seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}>::~continu ation() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:507 scylladb#18 0xab7216 in seastar::continuation<seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}>::run_and_ dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:512 scylladb#19 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091 scylladb#20 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506 scylladb#21 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666 scylladb#22 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199 scylladb#23 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115 scylladb#24 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486 scylladb#25 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2) previously allocated by thread T0 here: #0 0x7fb8652089d7 in operator new(unsigned long) (/lib64/libasan.so.5+0x10f9d7) scylladb#1 0x77d1796 in allocate /usr/include/c++/9/ext/new_allocator.h:114 scylladb#2 0x77bcab3 in allocate /usr/include/c++/9/bits/alloc_traits.h:444 scylladb#3 0x779d936 in _M_allocate_node<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable_policy.h:2081 scylladb#4 0x777a2ed in _M_emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable.h:1638 scylladb#5 0x774d707 in emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable.h:749 scylladb#6 0x771ce3e in emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/unordered_map.h:388 scylladb#7 0x76f8280 in operator() /local/home/bhalevy/dev/scylla/distributed_loader.cc:732 scylladb#8 0x774dc5d in _M_invoke /usr/include/c++/9/bits/std_function.h:286 scylladb#9 0x6cde67b in std::function<seastar::future<> (std::filesystem::__cxx11::path, seastar::directory_entry)>::operator()(std::filesystem::__cxx11::path, seastar::directory_entry) const /usr/include/c++/9/bits/std_function.h:690 scylladb#10 0x6cd345d in operator() /local/home/bhalevy/dev/scylla/lister.cc:30 scylladb#11 0x6cd96e0 in apply /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36 scylladb#12 0x6cd9854 in apply<lister::visit(seastar::directory_entry)::<lambda(seastar::directory_entry)>&, seastar::directory_entry&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:44 scylladb#13 0x6cd9953 in apply<lister::visit(seastar::directory_entry)::<lambda(seastar::directory_entry)>&, seastar::directory_entry&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1650 scylladb#14 0x6cd6ba9 in operator() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1168 scylladb#15 0x6cdbac0 in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:101 scylladb#16 0x6ce6178 in seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>::operator()(seastar::directory_entry&&) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:184 scylladb#17 0x6ce3033 in seastar::apply_helper<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, std::tuple<seastar::directory_entry>&&, std::integer_sequence<unsigned long, 0ul> >::apply(seastar::noncopyable_function<seastar::future<> (seastar::directory_e ntry&&)>&&, std::tuple<seastar::directory_entry>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36 scylladb#18 0x6ce3130 in auto seastar::apply<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::directory_entry>(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&, std::tuple<seastar::directory_entry>&&) /local/home/bha levy/dev/scylla/seastar/include/seastar/core/apply.hh:44 scylladb#19 0x6ce322f in seastar::future<> seastar::futurize<seastar::future<> >::apply<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::directory_entry>(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&, std::tuple<s eastar::directory_entry>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1650 scylladb#20 0x6cdf88d in seastar::future<seastar::directory_entry>::then_impl<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<seastar::directory_entry>&&)scylladb#1}::operator()(seastar::future_state<seastar::directory_entry>) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1196 scylladb#21 0x6cea2e0 in seastar::continuation<seastar::future<seastar::directory_entry>::then_impl<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&)::{lamb da()scylladb#1}::operator()() const::{lambda(seastar::future_state<seastar::directory_entry>&&)scylladb#1}, seastar::directory_entry>::run_and_dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:511 scylladb#22 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091 scylladb#23 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506 scylladb#24 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666 scylladb#25 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199 scylladb#26 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115 scylladb#27 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486 scylladb#28 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2) SUMMARY: AddressSanitizer: heap-use-after-free /usr/include/c++/9/bits/hashtable_policy.h:283 in _M_next Shadow bytes around the buggy address: 0x0c068025c100: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c068025c110: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa 0x0c068025c120: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd 0x0c068025c130: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c068025c140: fd fd fd fa fa fa 00 00 00 fa fa fa fd fd fd fa =>0x0c068025c150: fa fa fd fd fd fa fa fa[fd]fd fd fd fa fa fd fd 0x0c068025c160: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c068025c170: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa 0x0c068025c180: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd 0x0c068025c190: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c068025c1a0: fd fd fd fd fa fa fd fd fd fd fa fa 00 00 00 fa Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa Freed heap region: fd Stack left redzone: f1 Stack mid redzone: f2 Stack right redzone: f3 Stack after return: f5 Stack use after scope: f8 Global redzone: f9 Global init order: f6 Poisoned by user: f7 Container overflow: fc Array cookie: ac Intra object redzone: bb ASan internal: fe Left alloca redzone: ca Right alloca redzone: cb Shadow gap: cc ==406079==ABORTING Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
…y_clustering_slicing test/lib/mutation_source_test.cc:1086:63: runtime error: reference binding to null pointer of type 'const struct column_definition' mutation_partition.cc:1212:22: runtime error: member access within null pointer of type 'const struct column_definition' AddressSanitizer:DEADLYSIGNAL ================================================================= ==250886==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000050 (pc 0x00000ccc8820 bp 0x7f2dcb0a6a20 sp 0x7f2dcb0a6890 T1) ==250886==The signal is caused by a READ memory access. ==250886==Hint: address points to the zero page. #0 0xccc8820 in row::apply_monotonically(column_definition const&, atomic_cell_or_collection&&, seastar::optimized_optional<cell_hash>) /local/home/bhalevy/dev/scylla/mutation_partition.cc:1212 scylladb#1 0xccc8661 in row::apply(column_definition const&, atomic_cell_or_collection&&, seastar::optimized_optional<cell_hash>) /local/home/bhalevy/dev/scylla/mutation_partition.cc:1180 scylladb#2 0xc872a18 in mutation::set_clustered_cell(clustering_key_prefix const&, column_definition const&, atomic_cell_or_collection&&) /local/home/bhalevy/dev/scylla/mutation.cc:77 scylladb#3 0x132288ab in test_compaction_strategy_composite_key_clustering_slicing test/lib/mutation_source_test.cc:1094 scylladb#4 0x1322b9ef in test_all_compaction_strategies_clustering_slicing test/lib/mutation_source_test.cc:1134 scylladb#5 0x1324e25a in run_mutation_reader_tests(std::function<mutation_source (seastar::lw_shared_ptr<schema const>, std::vector<mutation, std::allocator<mutation> > const&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >)>) test/lib/mutation_source_test.cc:1495 scylladb#6 0x132513bb in run_mutation_source_tests(std::function<mutation_source (seastar::lw_shared_ptr<schema const>, std::vector<mutation, std::allocator<mutation> > const&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >)>) test/lib/mutation_source_test.cc:1556 scylladb#7 0xbcbf8c1 in test_mutation_source test/boost/sstable_conforms_to_mutation_source_test.cc:45 scylladb#8 0xbcc019d in operator() test/boost/sstable_conforms_to_mutation_source_test.cc:62 scylladb#9 0xbcc6cfd in __invoke_impl<void, test_sstable_conforms_to_mutation_source::run_test_case()::<lambda()> > /usr/include/c++/10/bits/invoke.h:60 scylladb#10 0xbcc5ccf in __invoke<test_sstable_conforms_to_mutation_source::run_test_case()::<lambda()> > /usr/include/c++/10/bits/invoke.h:95 scylladb#11 0xbcc3f91 in __apply_impl<test_sstable_conforms_to_mutation_source::run_test_case()::<lambda()>, std::tuple<> > /usr/include/c++/10/tuple:1723 scylladb#12 0xbcc3ff7 in apply<test_sstable_conforms_to_mutation_source::run_test_case()::<lambda()>, std::tuple<> > /usr/include/c++/10/tuple:1734 scylladb#13 0xbcc407a in apply<test_sstable_conforms_to_mutation_source::run_test_case()::<lambda()> > /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1978 scylladb#14 0xbcc1840 in operator() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/thread.hh:259 scylladb#15 0xbcc7e10 in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:101 scylladb#16 0xbfadb8f in seastar::noncopyable_function<void ()>::operator()() const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:184 scylladb#17 0x142ef646 in seastar::thread_context::main() /local/home/bhalevy/dev/scylla/seastar/src/core/thread.cc:297 AddressSanitizer can not provide additional info. SUMMARY: AddressSanitizer: SEGV /local/home/bhalevy/dev/scylla/mutation_partition.cc:1212 in row::apply_monotonically(column_definition const&, atomic_cell_or_collection&&, seastar::optimized_optional<cell_hash>) Thread T1 created by T0 here: #0 0x7f2ddba5abe5 in __interceptor_pthread_create (/lib64/libasan.so.6+0x57be5) scylladb#1 0x1425bc6b in seastar::posix_thread::posix_thread(seastar::posix_thread::attr, std::function<void ()>) /local/home/bhalevy/dev/scylla/seastar/src/core/posix.cc:95 scylladb#2 0x1425b84f in seastar::posix_thread::posix_thread(std::function<void ()>) /local/home/bhalevy/dev/scylla/seastar/src/core/posix.cc:65 scylladb#3 0x14a67480 in make_unique<seastar::posix_thread, seastar::testing::test_runner::start(int, char**)::<lambda()> > /usr/include/c++/10/bits/unique_ptr.h:961 scylladb#4 0x14a63e8a in seastar::testing::test_runner::start(int, char**) /local/home/bhalevy/dev/scylla/seastar/src/testing/test_runner.cc:108 scylladb#5 0x14a58b52 in init_unit_test_suite /local/home/bhalevy/dev/scylla/seastar/src/testing/entry_point.cc:47 scylladb#6 0x7f2dda3d2cec (/lib64/libboost_unit_test_framework.so.1.69.0+0x2ccec) ==250886==ABORTING Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
… token_metadata_ptr Fixes use-after-free seen with putget_with_reloaded_certificates_test: ``` ==215==ERROR: AddressSanitizer: heap-use-after-free on address 0x603000a8b180 at pc 0x000012eb5a83 bp 0x7ffd2c16d4c0 sp 0x7ffd2c16d4b0 READ of size 8 at 0x603000a8b180 thread T0 #0 0x12eb5a82 in std::__uniq_ptr_impl<locator::token_metadata_impl, std::default_delete<locator::token_metadata_impl> >::_M_ptr() const /usr/include/c++/10/bits/unique_ptr.h:173 scylladb#1 0x12ea230d in std::unique_ptr<locator::token_metadata_impl, std::default_delete<locator::token_metadata_impl> >::get() const /usr/include/c++/10/bits/unique_ptr.h:422 scylladb#2 0x12e8d3e8 in std::unique_ptr<locator::token_metadata_impl, std::default_delete<locator::token_metadata_impl> >::operator->() const /usr/include/c++/10/bits/unique_ptr.h:416 scylladb#3 0x12e5d0a2 in locator::token_metadata::ring_range(std::optional<interval_bound<dht::ring_position> > const&, bool) const locator/token_metadata.cc:1712 scylladb#4 0x112d0126 in service::query_ranges_to_vnodes_generator::process_one_range(unsigned long, std::vector<nonwrapping_interval<dht::ring_position>, std::allocator<nonwrapping_interval<dht::ring_position> > >&) service/storage_proxy.cc:4658 scylladb#5 0x112cf3c5 in service::query_ranges_to_vnodes_generator::operator()(unsigned long) service/storage_proxy.cc:4616 scylladb#6 0x112b2261 in service::storage_proxy::query_partition_key_range_concurrent(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >, std::vector<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >, std::allocator<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&, seastar::lw_shared_ptr<query::read_command>, db::consistency_level, service::query_ranges_to_vnodes_generator&&, int, tracing::trace_state_ptr, unsigned long, unsigned int, std::unordered_map<nonwrapping_interval<dht::token>, std::vector<utils::UUID, std::allocator<utils::UUID> >, std::hash<nonwrapping_interval<dht::token> >, std::equal_to<nonwrapping_interval<dht::token> >, std::allocator<std::pair<nonwrapping_interval<dht::token> const, std::vector<utils::UUID, std::allocator<utils::UUID> > > > >, service_permit) service/storage_proxy.cc:4023 scylladb#7 0x112b094e in operator() service/storage_proxy.cc:4160 scylladb#8 0x1139c8bb in invoke<service::storage_proxy::query_partition_key_range_concurrent(seastar::lowres_clock::time_point, std::vector<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >&&, seastar::lw_shared_ptr<query::read_command>, db::consistency_level, service::query_ranges_to_vnodes_generator&&, int, tracing::trace_state_ptr, uint64_t, uint32_t, service::replicas_per_token_range, service_permit)::<lambda(seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:2088 scylladb#9 0x1136625b in futurize_invoke<service::storage_proxy::query_partition_key_range_concurrent(seastar::lowres_clock::time_point, std::vector<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >&&, seastar::lw_shared_ptr<query::read_command>, db::consistency_level, service::query_ranges_to_vnodes_generator&&, int, tracing::trace_state_ptr, uint64_t, uint32_t, service::replicas_per_token_range, service_permit)::<lambda(seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:2119 scylladb#10 0x11366372 in operator()<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1480 scylladb#11 0x1139cc3b in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:145 scylladb#12 0x116f4944 in seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>::operator()(seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:201 scylladb#13 0x116b3397 in seastar::future<service::query_partition_key_range_concurrent_result> std::__invoke_impl<seastar::future<service::query_partition_key_range_concurrent_result>, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >(std::__invoke_other, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&) /usr/include/c++/10/bits/invoke.h:60 scylladb#14 0x1165c3a6 in std::__invoke_result<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::type std::__invoke<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&) /usr/include/c++/10/bits/invoke.h:96 scylladb#15 0x115e6542 in decltype(auto) std::__apply_impl<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >, 0ul>(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >&&, std::integer_sequence<unsigned long, 0ul>) /usr/include/c++/10/tuple:1724 scylladb#16 0x115e6663 in decltype(auto) std::apply<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >&&) /usr/include/c++/10/tuple:1736 scylladb#17 0x115e63f9 in seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}::operator()(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&) const::{lambda()scylladb#1}::operator()() const /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1530 scylladb#18 0x1165c4b9 in void seastar::futurize<seastar::future<service::query_partition_key_range_concurrent_result> >::satisfy_with_result_of<seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}::operator()(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&) const::{lambda()scylladb#1}>(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:2073 scylladb#19 0x115e61f5 in seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}::operator()(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1528 scylladb#20 0x1176e9cc in seastar::continuation<seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::run_and_dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:746 scylladb#21 0x16a9a455 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2196 scylladb#22 0x16a9e691 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2575 scylladb#23 0x16aa390e in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2730 scylladb#24 0x168ae4f7 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:207 scylladb#25 0x168ac541 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115 scylladb#26 0xd6cd3c4 in main /local/home/bhalevy/dev/scylla/main.cc:504 scylladb#27 0x7f8d905d8041 in __libc_start_main (/local/home/bhalevy/dev/scylla/build/debug/dynamic_libs/libc.so.6+0x27041) scylladb#28 0xd67c9ed in _start (/local/home/bhalevy/.dtest/dtest-o0qoqmkr/test/node3/bin/scylla+0xd67c9ed) 0x603000a8b180 is located 16 bytes inside of 24-byte region [0x603000a8b170,0x603000a8b188) freed by thread T0 here: #0 0x7f8d92a190cf in operator delete(void*, unsigned long) (/local/home/bhalevy/dev/scylla/build/debug/dynamic_libs/libasan.so.6+0xb30cf) scylladb#1 0xd7ebe54 in seastar::internal::lw_shared_ptr_accessors_no_esft<locator::token_metadata>::dispose(seastar::lw_shared_ptr_counter_base*) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:213 scylladb#2 0x112b155d in seastar::lw_shared_ptr<locator::token_metadata const>::~lw_shared_ptr() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:300 scylladb#3 0x112b155d in ~<lambda> service/storage_proxy.cc:4137 scylladb#4 0x1132e92d in ~<lambda> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1479 scylladb#5 0x1139cc91 in destroy /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:148 scylladb#6 0x11565673 in seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>::~noncopyable_function() /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:181 scylladb#7 0x1176e783 in seastar::continuation<seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::~continuation() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:729 scylladb#8 0x1176ea06 in seastar::continuation<seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::run_and_dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:750 scylladb#9 0x16a9a455 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2196 scylladb#10 0x16a9e691 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2575 scylladb#11 0x16aa390e in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2730 scylladb#12 0x168ae4f7 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:207 scylladb#13 0x168ac541 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115 scylladb#14 0xd6cd3c4 in main /local/home/bhalevy/dev/scylla/main.cc:504 scylladb#15 0x7f8d905d8041 in __libc_start_main (/local/home/bhalevy/dev/scylla/build/debug/dynamic_libs/libc.so.6+0x27041) previously allocated by thread T0 here: #0 0x7f8d92a18067 in operator new(unsigned long) (/local/home/bhalevy/dev/scylla/build/debug/dynamic_libs/libasan.so.6+0xb2067) scylladb#1 0x13cf7132 in seastar::lw_shared_ptr<locator::token_metadata> seastar::lw_shared_ptr<locator::token_metadata>::make<locator::token_metadata>(locator::token_metadata&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:266 scylladb#2 0x13cc3bfa in seastar::lw_shared_ptr<locator::token_metadata> seastar::make_lw_shared<locator::token_metadata>(locator::token_metadata&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:422 scylladb#3 0x13ca3007 in seastar::lw_shared_ptr<locator::token_metadata> locator::make_token_metadata_ptr<locator::token_metadata>(locator::token_metadata) locator/token_metadata.hh:338 scylladb#4 0x13c9bdd4 in locator::shared_token_metadata::clone() const locator/token_metadata.hh:358 scylladb#5 0x13c9c18a in service::storage_service::get_mutable_token_metadata_ptr() service/storage_service.hh:184 scylladb#6 0x13a5a445 in service::storage_service::handle_state_normal(gms::inet_address) service/storage_service.cc:1129 scylladb#7 0x13a6371c in service::storage_service::on_change(gms::inet_address, gms::application_state, gms::versioned_value const&) service/storage_service.cc:1421 scylladb#8 0x12a86269 in operator() gms/gossiper.cc:1639 scylladb#9 0x12ad3eea in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:145 scylladb#10 0x12be2aff in seastar::noncopyable_function<void (seastar::shared_ptr<gms::i_endpoint_state_change_subscriber>)>::operator()(seastar::shared_ptr<gms::i_endpoint_state_change_subscriber>) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:201 scylladb#11 0x12bb8e98 in atomic_vector<seastar::shared_ptr<gms::i_endpoint_state_change_subscriber> >::for_each(seastar::noncopyable_function<void (seastar::shared_ptr<gms::i_endpoint_state_change_subscriber>)>) utils/atomic_vector.hh:62 scylladb#12 0x12a8662b in gms::gossiper::do_on_change_notifications(gms::inet_address, gms::application_state const&, gms::versioned_value const&) gms/gossiper.cc:1638 scylladb#13 0x12a9387c in operator() gms/gossiper.cc:1978 scylladb#14 0x12b49b20 in __invoke_impl<void, gms::gossiper::add_local_application_state(std::__cxx11::list<std::pair<gms::application_state, gms::versioned_value> >)::<lambda(gms::gossiper&)> mutable::<lambda()> > /usr/include/c++/10/bits/invoke.h:60 scylladb#15 0x12b21fd6 in __invoke<gms::gossiper::add_local_application_state(std::__cxx11::list<std::pair<gms::application_state, gms::versioned_value> >)::<lambda(gms::gossiper&)> mutable::<lambda()> > /usr/include/c++/10/bits/invoke.h:95 scylladb#16 0x12b02865 in __apply_impl<gms::gossiper::add_local_application_state(std::__cxx11::list<std::pair<gms::application_state, gms::versioned_value> >)::<lambda(gms::gossiper&)> mutable::<lambda()>, std::tuple<> > /usr/include/c++/10/tuple:1723 scylladb#17 0x12b028d8 in apply<gms::gossiper::add_local_application_state(std::__cxx11::list<std::pair<gms::application_state, gms::versioned_value> >)::<lambda(gms::gossiper&)> mutable::<lambda()>, std::tuple<> > /usr/include/c++/10/tuple:1734 scylladb#18 0x12b02967 in apply<gms::gossiper::add_local_application_state(std::__cxx11::list<std::pair<gms::application_state, gms::versioned_value> >)::<lambda(gms::gossiper&)> mutable::<lambda()> > /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:2052 scylladb#19 0x12ad866a in operator() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/thread.hh:258 scylladb#20 0x12b609c2 in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:116 scylladb#21 0xdfabb5f in seastar::noncopyable_function<void ()>::operator()() const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:201 scylladb#22 0x16e21bb4 in seastar::thread_context::main() /local/home/bhalevy/dev/scylla/seastar/src/core/thread.cc:297 scylladb#23 0x16e2190f in seastar::thread_context::s_main(int, int) /local/home/bhalevy/dev/scylla/seastar/src/core/thread.cc:275 scylladb#24 0x7f8d9060322f (/local/home/bhalevy/dev/scylla/build/debug/dynamic_libs/libc.so.6+0x5222f) ``` Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
… token_metadata_ptr Fixes use-after-free seen with putget_with_reloaded_certificates_test: ``` ==215==ERROR: AddressSanitizer: heap-use-after-free on address 0x603000a8b180 at pc 0x000012eb5a83 bp 0x7ffd2c16d4c0 sp 0x7ffd2c16d4b0 READ of size 8 at 0x603000a8b180 thread T0 #0 0x12eb5a82 in std::__uniq_ptr_impl<locator::token_metadata_impl, std::default_delete<locator::token_metadata_impl> >::_M_ptr() const /usr/include/c++/10/bits/unique_ptr.h:173 scylladb#1 0x12ea230d in std::unique_ptr<locator::token_metadata_impl, std::default_delete<locator::token_metadata_impl> >::get() const /usr/include/c++/10/bits/unique_ptr.h:422 scylladb#2 0x12e8d3e8 in std::unique_ptr<locator::token_metadata_impl, std::default_delete<locator::token_metadata_impl> >::operator->() const /usr/include/c++/10/bits/unique_ptr.h:416 scylladb#3 0x12e5d0a2 in locator::token_metadata::ring_range(std::optional<interval_bound<dht::ring_position> > const&, bool) const locator/token_metadata.cc:1712 scylladb#4 0x112d0126 in service::query_ranges_to_vnodes_generator::process_one_range(unsigned long, std::vector<nonwrapping_interval<dht::ring_position>, std::allocator<nonwrapping_interval<dht::ring_position> > >&) service/storage_proxy.cc:4658 scylladb#5 0x112cf3c5 in service::query_ranges_to_vnodes_generator::operator()(unsigned long) service/storage_proxy.cc:4616 scylladb#6 0x112b2261 in service::storage_proxy::query_partition_key_range_concurrent(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >, std::vector<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >, std::allocator<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&, seastar::lw_shared_ptr<query::read_command>, db::consistency_level, service::query_ranges_to_vnodes_generator&&, int, tracing::trace_state_ptr, unsigned long, unsigned int, std::unordered_map<nonwrapping_interval<dht::token>, std::vector<utils::UUID, std::allocator<utils::UUID> >, std::hash<nonwrapping_interval<dht::token> >, std::equal_to<nonwrapping_interval<dht::token> >, std::allocator<std::pair<nonwrapping_interval<dht::token> const, std::vector<utils::UUID, std::allocator<utils::UUID> > > > >, service_permit) service/storage_proxy.cc:4023 scylladb#7 0x112b094e in operator() service/storage_proxy.cc:4160 scylladb#8 0x1139c8bb in invoke<service::storage_proxy::query_partition_key_range_concurrent(seastar::lowres_clock::time_point, std::vector<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >&&, seastar::lw_shared_ptr<query::read_command>, db::consistency_level, service::query_ranges_to_vnodes_generator&&, int, tracing::trace_state_ptr, uint64_t, uint32_t, service::replicas_per_token_range, service_permit)::<lambda(seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:2088 scylladb#9 0x1136625b in futurize_invoke<service::storage_proxy::query_partition_key_range_concurrent(seastar::lowres_clock::time_point, std::vector<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >&&, seastar::lw_shared_ptr<query::read_command>, db::consistency_level, service::query_ranges_to_vnodes_generator&&, int, tracing::trace_state_ptr, uint64_t, uint32_t, service::replicas_per_token_range, service_permit)::<lambda(seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:2119 scylladb#10 0x11366372 in operator()<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1480 scylladb#11 0x1139cc3b in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:145 scylladb#12 0x116f4944 in seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>::operator()(seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:201 scylladb#13 0x116b3397 in seastar::future<service::query_partition_key_range_concurrent_result> std::__invoke_impl<seastar::future<service::query_partition_key_range_concurrent_result>, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >(std::__invoke_other, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&) /usr/include/c++/10/bits/invoke.h:60 scylladb#14 0x1165c3a6 in std::__invoke_result<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::type std::__invoke<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&) /usr/include/c++/10/bits/invoke.h:96 scylladb#15 0x115e6542 in decltype(auto) std::__apply_impl<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >, 0ul>(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >&&, std::integer_sequence<unsigned long, 0ul>) /usr/include/c++/10/tuple:1724 scylladb#16 0x115e6663 in decltype(auto) std::apply<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >&&) /usr/include/c++/10/tuple:1736 scylladb#17 0x115e63f9 in seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}::operator()(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&) const::{lambda()scylladb#1}::operator()() const /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1530 scylladb#18 0x1165c4b9 in void seastar::futurize<seastar::future<service::query_partition_key_range_concurrent_result> >::satisfy_with_result_of<seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}::operator()(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&) const::{lambda()scylladb#1}>(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:2073 scylladb#19 0x115e61f5 in seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}::operator()(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1528 scylladb#20 0x1176e9cc in seastar::continuation<seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::run_and_dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:746 scylladb#21 0x16a9a455 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2196 scylladb#22 0x16a9e691 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2575 scylladb#23 0x16aa390e in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2730 scylladb#24 0x168ae4f7 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:207 scylladb#25 0x168ac541 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115 scylladb#26 0xd6cd3c4 in main /local/home/bhalevy/dev/scylla/main.cc:504 scylladb#27 0x7f8d905d8041 in __libc_start_main (/local/home/bhalevy/dev/scylla/build/debug/dynamic_libs/libc.so.6+0x27041) scylladb#28 0xd67c9ed in _start (/local/home/bhalevy/.dtest/dtest-o0qoqmkr/test/node3/bin/scylla+0xd67c9ed) 0x603000a8b180 is located 16 bytes inside of 24-byte region [0x603000a8b170,0x603000a8b188) freed by thread T0 here: #0 0x7f8d92a190cf in operator delete(void*, unsigned long) (/local/home/bhalevy/dev/scylla/build/debug/dynamic_libs/libasan.so.6+0xb30cf) scylladb#1 0xd7ebe54 in seastar::internal::lw_shared_ptr_accessors_no_esft<locator::token_metadata>::dispose(seastar::lw_shared_ptr_counter_base*) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:213 scylladb#2 0x112b155d in seastar::lw_shared_ptr<locator::token_metadata const>::~lw_shared_ptr() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:300 scylladb#3 0x112b155d in ~<lambda> service/storage_proxy.cc:4137 scylladb#4 0x1132e92d in ~<lambda> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1479 scylladb#5 0x1139cc91 in destroy /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:148 scylladb#6 0x11565673 in seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>::~noncopyable_function() /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:181 scylladb#7 0x1176e783 in seastar::continuation<seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::~continuation() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:729 scylladb#8 0x1176ea06 in seastar::continuation<seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::run_and_dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:750 scylladb#9 0x16a9a455 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2196 scylladb#10 0x16a9e691 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2575 scylladb#11 0x16aa390e in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2730 scylladb#12 0x168ae4f7 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:207 scylladb#13 0x168ac541 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115 scylladb#14 0xd6cd3c4 in main /local/home/bhalevy/dev/scylla/main.cc:504 scylladb#15 0x7f8d905d8041 in __libc_start_main (/local/home/bhalevy/dev/scylla/build/debug/dynamic_libs/libc.so.6+0x27041) previously allocated by thread T0 here: #0 0x7f8d92a18067 in operator new(unsigned long) (/local/home/bhalevy/dev/scylla/build/debug/dynamic_libs/libasan.so.6+0xb2067) scylladb#1 0x13cf7132 in seastar::lw_shared_ptr<locator::token_metadata> seastar::lw_shared_ptr<locator::token_metadata>::make<locator::token_metadata>(locator::token_metadata&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:266 scylladb#2 0x13cc3bfa in seastar::lw_shared_ptr<locator::token_metadata> seastar::make_lw_shared<locator::token_metadata>(locator::token_metadata&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:422 scylladb#3 0x13ca3007 in seastar::lw_shared_ptr<locator::token_metadata> locator::make_token_metadata_ptr<locator::token_metadata>(locator::token_metadata) locator/token_metadata.hh:338 scylladb#4 0x13c9bdd4 in locator::shared_token_metadata::clone() const locator/token_metadata.hh:358 scylladb#5 0x13c9c18a in service::storage_service::get_mutable_token_metadata_ptr() service/storage_service.hh:184 scylladb#6 0x13a5a445 in service::storage_service::handle_state_normal(gms::inet_address) service/storage_service.cc:1129 scylladb#7 0x13a6371c in service::storage_service::on_change(gms::inet_address, gms::application_state, gms::versioned_value const&) service/storage_service.cc:1421 scylladb#8 0x12a86269 in operator() gms/gossiper.cc:1639 scylladb#9 0x12ad3eea in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:145 scylladb#10 0x12be2aff in seastar::noncopyable_function<void (seastar::shared_ptr<gms::i_endpoint_state_change_subscriber>)>::operator()(seastar::shared_ptr<gms::i_endpoint_state_change_subscriber>) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:201 scylladb#11 0x12bb8e98 in atomic_vector<seastar::shared_ptr<gms::i_endpoint_state_change_subscriber> >::for_each(seastar::noncopyable_function<void (seastar::shared_ptr<gms::i_endpoint_state_change_subscriber>)>) utils/atomic_vector.hh:62 scylladb#12 0x12a8662b in gms::gossiper::do_on_change_notifications(gms::inet_address, gms::application_state const&, gms::versioned_value const&) gms/gossiper.cc:1638 scylladb#13 0x12a9387c in operator() gms/gossiper.cc:1978 scylladb#14 0x12b49b20 in __invoke_impl<void, gms::gossiper::add_local_application_state(std::__cxx11::list<std::pair<gms::application_state, gms::versioned_value> >)::<lambda(gms::gossiper&)> mutable::<lambda()> > /usr/include/c++/10/bits/invoke.h:60 scylladb#15 0x12b21fd6 in __invoke<gms::gossiper::add_local_application_state(std::__cxx11::list<std::pair<gms::application_state, gms::versioned_value> >)::<lambda(gms::gossiper&)> mutable::<lambda()> > /usr/include/c++/10/bits/invoke.h:95 scylladb#16 0x12b02865 in __apply_impl<gms::gossiper::add_local_application_state(std::__cxx11::list<std::pair<gms::application_state, gms::versioned_value> >)::<lambda(gms::gossiper&)> mutable::<lambda()>, std::tuple<> > /usr/include/c++/10/tuple:1723 scylladb#17 0x12b028d8 in apply<gms::gossiper::add_local_application_state(std::__cxx11::list<std::pair<gms::application_state, gms::versioned_value> >)::<lambda(gms::gossiper&)> mutable::<lambda()>, std::tuple<> > /usr/include/c++/10/tuple:1734 scylladb#18 0x12b02967 in apply<gms::gossiper::add_local_application_state(std::__cxx11::list<std::pair<gms::application_state, gms::versioned_value> >)::<lambda(gms::gossiper&)> mutable::<lambda()> > /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:2052 scylladb#19 0x12ad866a in operator() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/thread.hh:258 scylladb#20 0x12b609c2 in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:116 scylladb#21 0xdfabb5f in seastar::noncopyable_function<void ()>::operator()() const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:201 scylladb#22 0x16e21bb4 in seastar::thread_context::main() /local/home/bhalevy/dev/scylla/seastar/src/core/thread.cc:297 scylladb#23 0x16e2190f in seastar::thread_context::s_main(int, int) /local/home/bhalevy/dev/scylla/seastar/src/core/thread.cc:275 scylladb#24 0x7f8d9060322f (/local/home/bhalevy/dev/scylla/build/debug/dynamic_libs/libc.so.6+0x5222f) ``` Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
… token_metadata_ptr Fixes use-after-free seen with putget_with_reloaded_certificates_test: ``` ==215==ERROR: AddressSanitizer: heap-use-after-free on address 0x603000a8b180 at pc 0x000012eb5a83 bp 0x7ffd2c16d4c0 sp 0x7ffd2c16d4b0 READ of size 8 at 0x603000a8b180 thread T0 #0 0x12eb5a82 in std::__uniq_ptr_impl<locator::token_metadata_impl, std::default_delete<locator::token_metadata_impl> >::_M_ptr() const /usr/include/c++/10/bits/unique_ptr.h:173 scylladb#1 0x12ea230d in std::unique_ptr<locator::token_metadata_impl, std::default_delete<locator::token_metadata_impl> >::get() const /usr/include/c++/10/bits/unique_ptr.h:422 scylladb#2 0x12e8d3e8 in std::unique_ptr<locator::token_metadata_impl, std::default_delete<locator::token_metadata_impl> >::operator->() const /usr/include/c++/10/bits/unique_ptr.h:416 scylladb#3 0x12e5d0a2 in locator::token_metadata::ring_range(std::optional<interval_bound<dht::ring_position> > const&, bool) const locator/token_metadata.cc:1712 scylladb#4 0x112d0126 in service::query_ranges_to_vnodes_generator::process_one_range(unsigned long, std::vector<nonwrapping_interval<dht::ring_position>, std::allocator<nonwrapping_interval<dht::ring_position> > >&) service/storage_proxy.cc:4658 scylladb#5 0x112cf3c5 in service::query_ranges_to_vnodes_generator::operator()(unsigned long) service/storage_proxy.cc:4616 scylladb#6 0x112b2261 in service::storage_proxy::query_partition_key_range_concurrent(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >, std::vector<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >, std::allocator<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&, seastar::lw_shared_ptr<query::read_command>, db::consistency_level, service::query_ranges_to_vnodes_generator&&, int, tracing::trace_state_ptr, unsigned long, unsigned int, std::unordered_map<nonwrapping_interval<dht::token>, std::vector<utils::UUID, std::allocator<utils::UUID> >, std::hash<nonwrapping_interval<dht::token> >, std::equal_to<nonwrapping_interval<dht::token> >, std::allocator<std::pair<nonwrapping_interval<dht::token> const, std::vector<utils::UUID, std::allocator<utils::UUID> > > > >, service_permit) service/storage_proxy.cc:4023 scylladb#7 0x112b094e in operator() service/storage_proxy.cc:4160 scylladb#8 0x1139c8bb in invoke<service::storage_proxy::query_partition_key_range_concurrent(seastar::lowres_clock::time_point, std::vector<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >&&, seastar::lw_shared_ptr<query::read_command>, db::consistency_level, service::query_ranges_to_vnodes_generator&&, int, tracing::trace_state_ptr, uint64_t, uint32_t, service::replicas_per_token_range, service_permit)::<lambda(seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:2088 scylladb#9 0x1136625b in futurize_invoke<service::storage_proxy::query_partition_key_range_concurrent(seastar::lowres_clock::time_point, std::vector<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >&&, seastar::lw_shared_ptr<query::read_command>, db::consistency_level, service::query_ranges_to_vnodes_generator&&, int, tracing::trace_state_ptr, uint64_t, uint32_t, service::replicas_per_token_range, service_permit)::<lambda(seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:2119 scylladb#10 0x11366372 in operator()<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1480 scylladb#11 0x1139cc3b in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:145 scylladb#12 0x116f4944 in seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>::operator()(seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:201 scylladb#13 0x116b3397 in seastar::future<service::query_partition_key_range_concurrent_result> std::__invoke_impl<seastar::future<service::query_partition_key_range_concurrent_result>, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >(std::__invoke_other, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&) /usr/include/c++/10/bits/invoke.h:60 scylladb#14 0x1165c3a6 in std::__invoke_result<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::type std::__invoke<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&) /usr/include/c++/10/bits/invoke.h:96 scylladb#15 0x115e6542 in decltype(auto) std::__apply_impl<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >, 0ul>(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >&&, std::integer_sequence<unsigned long, 0ul>) /usr/include/c++/10/tuple:1724 scylladb#16 0x115e6663 in decltype(auto) std::apply<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >&&) /usr/include/c++/10/tuple:1736 scylladb#17 0x115e63f9 in seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}::operator()(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&) const::{lambda()scylladb#1}::operator()() const /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1530 scylladb#18 0x1165c4b9 in void seastar::futurize<seastar::future<service::query_partition_key_range_concurrent_result> >::satisfy_with_result_of<seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}::operator()(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&) const::{lambda()scylladb#1}>(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:2073 scylladb#19 0x115e61f5 in seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}::operator()(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1528 scylladb#20 0x1176e9cc in seastar::continuation<seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::run_and_dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:746 scylladb#21 0x16a9a455 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2196 scylladb#22 0x16a9e691 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2575 scylladb#23 0x16aa390e in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2730 scylladb#24 0x168ae4f7 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:207 scylladb#25 0x168ac541 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115 scylladb#26 0xd6cd3c4 in main /local/home/bhalevy/dev/scylla/main.cc:504 scylladb#27 0x7f8d905d8041 in __libc_start_main (/local/home/bhalevy/dev/scylla/build/debug/dynamic_libs/libc.so.6+0x27041) scylladb#28 0xd67c9ed in _start (/local/home/bhalevy/.dtest/dtest-o0qoqmkr/test/node3/bin/scylla+0xd67c9ed) 0x603000a8b180 is located 16 bytes inside of 24-byte region [0x603000a8b170,0x603000a8b188) freed by thread T0 here: #0 0x7f8d92a190cf in operator delete(void*, unsigned long) (/local/home/bhalevy/dev/scylla/build/debug/dynamic_libs/libasan.so.6+0xb30cf) scylladb#1 0xd7ebe54 in seastar::internal::lw_shared_ptr_accessors_no_esft<locator::token_metadata>::dispose(seastar::lw_shared_ptr_counter_base*) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:213 scylladb#2 0x112b155d in seastar::lw_shared_ptr<locator::token_metadata const>::~lw_shared_ptr() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:300 scylladb#3 0x112b155d in ~<lambda> service/storage_proxy.cc:4137 scylladb#4 0x1132e92d in ~<lambda> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1479 scylladb#5 0x1139cc91 in destroy /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:148 scylladb#6 0x11565673 in seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>::~noncopyable_function() /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:181 scylladb#7 0x1176e783 in seastar::continuation<seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::~continuation() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:729 scylladb#8 0x1176ea06 in seastar::continuation<seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::run_and_dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:750 scylladb#9 0x16a9a455 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2196 scylladb#10 0x16a9e691 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2575 scylladb#11 0x16aa390e in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2730 scylladb#12 0x168ae4f7 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:207 scylladb#13 0x168ac541 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115 scylladb#14 0xd6cd3c4 in main /local/home/bhalevy/dev/scylla/main.cc:504 scylladb#15 0x7f8d905d8041 in __libc_start_main (/local/home/bhalevy/dev/scylla/build/debug/dynamic_libs/libc.so.6+0x27041) previously allocated by thread T0 here: #0 0x7f8d92a18067 in operator new(unsigned long) (/local/home/bhalevy/dev/scylla/build/debug/dynamic_libs/libasan.so.6+0xb2067) scylladb#1 0x13cf7132 in seastar::lw_shared_ptr<locator::token_metadata> seastar::lw_shared_ptr<locator::token_metadata>::make<locator::token_metadata>(locator::token_metadata&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:266 scylladb#2 0x13cc3bfa in seastar::lw_shared_ptr<locator::token_metadata> seastar::make_lw_shared<locator::token_metadata>(locator::token_metadata&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:422 scylladb#3 0x13ca3007 in seastar::lw_shared_ptr<locator::token_metadata> locator::make_token_metadata_ptr<locator::token_metadata>(locator::token_metadata) locator/token_metadata.hh:338 scylladb#4 0x13c9bdd4 in locator::shared_token_metadata::clone() const locator/token_metadata.hh:358 scylladb#5 0x13c9c18a in service::storage_service::get_mutable_token_metadata_ptr() service/storage_service.hh:184 scylladb#6 0x13a5a445 in service::storage_service::handle_state_normal(gms::inet_address) service/storage_service.cc:1129 scylladb#7 0x13a6371c in service::storage_service::on_change(gms::inet_address, gms::application_state, gms::versioned_value const&) service/storage_service.cc:1421 scylladb#8 0x12a86269 in operator() gms/gossiper.cc:1639 scylladb#9 0x12ad3eea in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:145 scylladb#10 0x12be2aff in seastar::noncopyable_function<void (seastar::shared_ptr<gms::i_endpoint_state_change_subscriber>)>::operator()(seastar::shared_ptr<gms::i_endpoint_state_change_subscriber>) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:201 scylladb#11 0x12bb8e98 in atomic_vector<seastar::shared_ptr<gms::i_endpoint_state_change_subscriber> >::for_each(seastar::noncopyable_function<void (seastar::shared_ptr<gms::i_endpoint_state_change_subscriber>)>) utils/atomic_vector.hh:62 scylladb#12 0x12a8662b in gms::gossiper::do_on_change_notifications(gms::inet_address, gms::application_state const&, gms::versioned_value const&) gms/gossiper.cc:1638 scylladb#13 0x12a9387c in operator() gms/gossiper.cc:1978 scylladb#14 0x12b49b20 in __invoke_impl<void, gms::gossiper::add_local_application_state(std::__cxx11::list<std::pair<gms::application_state, gms::versioned_value> >)::<lambda(gms::gossiper&)> mutable::<lambda()> > /usr/include/c++/10/bits/invoke.h:60 scylladb#15 0x12b21fd6 in __invoke<gms::gossiper::add_local_application_state(std::__cxx11::list<std::pair<gms::application_state, gms::versioned_value> >)::<lambda(gms::gossiper&)> mutable::<lambda()> > /usr/include/c++/10/bits/invoke.h:95 scylladb#16 0x12b02865 in __apply_impl<gms::gossiper::add_local_application_state(std::__cxx11::list<std::pair<gms::application_state, gms::versioned_value> >)::<lambda(gms::gossiper&)> mutable::<lambda()>, std::tuple<> > /usr/include/c++/10/tuple:1723 scylladb#17 0x12b028d8 in apply<gms::gossiper::add_local_application_state(std::__cxx11::list<std::pair<gms::application_state, gms::versioned_value> >)::<lambda(gms::gossiper&)> mutable::<lambda()>, std::tuple<> > /usr/include/c++/10/tuple:1734 scylladb#18 0x12b02967 in apply<gms::gossiper::add_local_application_state(std::__cxx11::list<std::pair<gms::application_state, gms::versioned_value> >)::<lambda(gms::gossiper&)> mutable::<lambda()> > /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:2052 scylladb#19 0x12ad866a in operator() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/thread.hh:258 scylladb#20 0x12b609c2 in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:116 scylladb#21 0xdfabb5f in seastar::noncopyable_function<void ()>::operator()() const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:201 scylladb#22 0x16e21bb4 in seastar::thread_context::main() /local/home/bhalevy/dev/scylla/seastar/src/core/thread.cc:297 scylladb#23 0x16e2190f in seastar::thread_context::s_main(int, int) /local/home/bhalevy/dev/scylla/seastar/src/core/thread.cc:275 scylladb#24 0x7f8d9060322f (/local/home/bhalevy/dev/scylla/build/debug/dynamic_libs/libc.so.6+0x5222f) ``` Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
… token_metadata_ptr Fixes use-after-free seen with putget_with_reloaded_certificates_test: ``` ==215==ERROR: AddressSanitizer: heap-use-after-free on address 0x603000a8b180 at pc 0x000012eb5a83 bp 0x7ffd2c16d4c0 sp 0x7ffd2c16d4b0 READ of size 8 at 0x603000a8b180 thread T0 #0 0x12eb5a82 in std::__uniq_ptr_impl<locator::token_metadata_impl, std::default_delete<locator::token_metadata_impl> >::_M_ptr() const /usr/include/c++/10/bits/unique_ptr.h:173 scylladb#1 0x12ea230d in std::unique_ptr<locator::token_metadata_impl, std::default_delete<locator::token_metadata_impl> >::get() const /usr/include/c++/10/bits/unique_ptr.h:422 scylladb#2 0x12e8d3e8 in std::unique_ptr<locator::token_metadata_impl, std::default_delete<locator::token_metadata_impl> >::operator->() const /usr/include/c++/10/bits/unique_ptr.h:416 scylladb#3 0x12e5d0a2 in locator::token_metadata::ring_range(std::optional<interval_bound<dht::ring_position> > const&, bool) const locator/token_metadata.cc:1712 scylladb#4 0x112d0126 in service::query_ranges_to_vnodes_generator::process_one_range(unsigned long, std::vector<nonwrapping_interval<dht::ring_position>, std::allocator<nonwrapping_interval<dht::ring_position> > >&) service/storage_proxy.cc:4658 scylladb#5 0x112cf3c5 in service::query_ranges_to_vnodes_generator::operator()(unsigned long) service/storage_proxy.cc:4616 scylladb#6 0x112b2261 in service::storage_proxy::query_partition_key_range_concurrent(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >, std::vector<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >, std::allocator<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&, seastar::lw_shared_ptr<query::read_command>, db::consistency_level, service::query_ranges_to_vnodes_generator&&, int, tracing::trace_state_ptr, unsigned long, unsigned int, std::unordered_map<nonwrapping_interval<dht::token>, std::vector<utils::UUID, std::allocator<utils::UUID> >, std::hash<nonwrapping_interval<dht::token> >, std::equal_to<nonwrapping_interval<dht::token> >, std::allocator<std::pair<nonwrapping_interval<dht::token> const, std::vector<utils::UUID, std::allocator<utils::UUID> > > > >, service_permit) service/storage_proxy.cc:4023 scylladb#7 0x112b094e in operator() service/storage_proxy.cc:4160 scylladb#8 0x1139c8bb in invoke<service::storage_proxy::query_partition_key_range_concurrent(seastar::lowres_clock::time_point, std::vector<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >&&, seastar::lw_shared_ptr<query::read_command>, db::consistency_level, service::query_ranges_to_vnodes_generator&&, int, tracing::trace_state_ptr, uint64_t, uint32_t, service::replicas_per_token_range, service_permit)::<lambda(seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:2088 scylladb#9 0x1136625b in futurize_invoke<service::storage_proxy::query_partition_key_range_concurrent(seastar::lowres_clock::time_point, std::vector<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >&&, seastar::lw_shared_ptr<query::read_command>, db::consistency_level, service::query_ranges_to_vnodes_generator&&, int, tracing::trace_state_ptr, uint64_t, uint32_t, service::replicas_per_token_range, service_permit)::<lambda(seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:2119 scylladb#10 0x11366372 in operator()<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1480 scylladb#11 0x1139cc3b in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:145 scylladb#12 0x116f4944 in seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>::operator()(seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:201 scylladb#13 0x116b3397 in seastar::future<service::query_partition_key_range_concurrent_result> std::__invoke_impl<seastar::future<service::query_partition_key_range_concurrent_result>, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >(std::__invoke_other, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&) /usr/include/c++/10/bits/invoke.h:60 scylladb#14 0x1165c3a6 in std::__invoke_result<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::type std::__invoke<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&) /usr/include/c++/10/bits/invoke.h:96 scylladb#15 0x115e6542 in decltype(auto) std::__apply_impl<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >, 0ul>(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >&&, std::integer_sequence<unsigned long, 0ul>) /usr/include/c++/10/tuple:1724 scylladb#16 0x115e6663 in decltype(auto) std::apply<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >&&) /usr/include/c++/10/tuple:1736 scylladb#17 0x115e63f9 in seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}::operator()(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&) const::{lambda()scylladb#1}::operator()() const /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1530 scylladb#18 0x1165c4b9 in void seastar::futurize<seastar::future<service::query_partition_key_range_concurrent_result> >::satisfy_with_result_of<seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}::operator()(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&) const::{lambda()scylladb#1}>(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:2073 scylladb#19 0x115e61f5 in seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}::operator()(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1528 scylladb#20 0x1176e9cc in seastar::continuation<seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::run_and_dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:746 scylladb#21 0x16a9a455 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2196 scylladb#22 0x16a9e691 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2575 scylladb#23 0x16aa390e in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2730 scylladb#24 0x168ae4f7 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:207 scylladb#25 0x168ac541 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115 scylladb#26 0xd6cd3c4 in main /local/home/bhalevy/dev/scylla/main.cc:504 scylladb#27 0x7f8d905d8041 in __libc_start_main (/local/home/bhalevy/dev/scylla/build/debug/dynamic_libs/libc.so.6+0x27041) scylladb#28 0xd67c9ed in _start (/local/home/bhalevy/.dtest/dtest-o0qoqmkr/test/node3/bin/scylla+0xd67c9ed) 0x603000a8b180 is located 16 bytes inside of 24-byte region [0x603000a8b170,0x603000a8b188) freed by thread T0 here: #0 0x7f8d92a190cf in operator delete(void*, unsigned long) (/local/home/bhalevy/dev/scylla/build/debug/dynamic_libs/libasan.so.6+0xb30cf) scylladb#1 0xd7ebe54 in seastar::internal::lw_shared_ptr_accessors_no_esft<locator::token_metadata>::dispose(seastar::lw_shared_ptr_counter_base*) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:213 scylladb#2 0x112b155d in seastar::lw_shared_ptr<locator::token_metadata const>::~lw_shared_ptr() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:300 scylladb#3 0x112b155d in ~<lambda> service/storage_proxy.cc:4137 scylladb#4 0x1132e92d in ~<lambda> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1479 scylladb#5 0x1139cc91 in destroy /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:148 scylladb#6 0x11565673 in seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>::~noncopyable_function() /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:181 scylladb#7 0x1176e783 in seastar::continuation<seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::~continuation() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:729 scylladb#8 0x1176ea06 in seastar::continuation<seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::run_and_dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:750 scylladb#9 0x16a9a455 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2196 scylladb#10 0x16a9e691 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2575 scylladb#11 0x16aa390e in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2730 scylladb#12 0x168ae4f7 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:207 scylladb#13 0x168ac541 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115 scylladb#14 0xd6cd3c4 in main /local/home/bhalevy/dev/scylla/main.cc:504 scylladb#15 0x7f8d905d8041 in __libc_start_main (/local/home/bhalevy/dev/scylla/build/debug/dynamic_libs/libc.so.6+0x27041) previously allocated by thread T0 here: #0 0x7f8d92a18067 in operator new(unsigned long) (/local/home/bhalevy/dev/scylla/build/debug/dynamic_libs/libasan.so.6+0xb2067) scylladb#1 0x13cf7132 in seastar::lw_shared_ptr<locator::token_metadata> seastar::lw_shared_ptr<locator::token_metadata>::make<locator::token_metadata>(locator::token_metadata&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:266 scylladb#2 0x13cc3bfa in seastar::lw_shared_ptr<locator::token_metadata> seastar::make_lw_shared<locator::token_metadata>(locator::token_metadata&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:422 scylladb#3 0x13ca3007 in seastar::lw_shared_ptr<locator::token_metadata> locator::make_token_metadata_ptr<locator::token_metadata>(locator::token_metadata) locator/token_metadata.hh:338 scylladb#4 0x13c9bdd4 in locator::shared_token_metadata::clone() const locator/token_metadata.hh:358 scylladb#5 0x13c9c18a in service::storage_service::get_mutable_token_metadata_ptr() service/storage_service.hh:184 scylladb#6 0x13a5a445 in service::storage_service::handle_state_normal(gms::inet_address) service/storage_service.cc:1129 scylladb#7 0x13a6371c in service::storage_service::on_change(gms::inet_address, gms::application_state, gms::versioned_value const&) service/storage_service.cc:1421 scylladb#8 0x12a86269 in operator() gms/gossiper.cc:1639 scylladb#9 0x12ad3eea in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:145 scylladb#10 0x12be2aff in seastar::noncopyable_function<void (seastar::shared_ptr<gms::i_endpoint_state_change_subscriber>)>::operator()(seastar::shared_ptr<gms::i_endpoint_state_change_subscriber>) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:201 scylladb#11 0x12bb8e98 in atomic_vector<seastar::shared_ptr<gms::i_endpoint_state_change_subscriber> >::for_each(seastar::noncopyable_function<void (seastar::shared_ptr<gms::i_endpoint_state_change_subscriber>)>) utils/atomic_vector.hh:62 scylladb#12 0x12a8662b in gms::gossiper::do_on_change_notifications(gms::inet_address, gms::application_state const&, gms::versioned_value const&) gms/gossiper.cc:1638 scylladb#13 0x12a9387c in operator() gms/gossiper.cc:1978 scylladb#14 0x12b49b20 in __invoke_impl<void, gms::gossiper::add_local_application_state(std::__cxx11::list<std::pair<gms::application_state, gms::versioned_value> >)::<lambda(gms::gossiper&)> mutable::<lambda()> > /usr/include/c++/10/bits/invoke.h:60 scylladb#15 0x12b21fd6 in __invoke<gms::gossiper::add_local_application_state(std::__cxx11::list<std::pair<gms::application_state, gms::versioned_value> >)::<lambda(gms::gossiper&)> mutable::<lambda()> > /usr/include/c++/10/bits/invoke.h:95 scylladb#16 0x12b02865 in __apply_impl<gms::gossiper::add_local_application_state(std::__cxx11::list<std::pair<gms::application_state, gms::versioned_value> >)::<lambda(gms::gossiper&)> mutable::<lambda()>, std::tuple<> > /usr/include/c++/10/tuple:1723 scylladb#17 0x12b028d8 in apply<gms::gossiper::add_local_application_state(std::__cxx11::list<std::pair<gms::application_state, gms::versioned_value> >)::<lambda(gms::gossiper&)> mutable::<lambda()>, std::tuple<> > /usr/include/c++/10/tuple:1734 scylladb#18 0x12b02967 in apply<gms::gossiper::add_local_application_state(std::__cxx11::list<std::pair<gms::application_state, gms::versioned_value> >)::<lambda(gms::gossiper&)> mutable::<lambda()> > /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:2052 scylladb#19 0x12ad866a in operator() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/thread.hh:258 scylladb#20 0x12b609c2 in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:116 scylladb#21 0xdfabb5f in seastar::noncopyable_function<void ()>::operator()() const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:201 scylladb#22 0x16e21bb4 in seastar::thread_context::main() /local/home/bhalevy/dev/scylla/seastar/src/core/thread.cc:297 scylladb#23 0x16e2190f in seastar::thread_context::s_main(int, int) /local/home/bhalevy/dev/scylla/seastar/src/core/thread.cc:275 scylladb#24 0x7f8d9060322f (/local/home/bhalevy/dev/scylla/build/debug/dynamic_libs/libc.so.6+0x5222f) ``` Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
" Current storage of cells in a row is a union of vector and set. The vector holds 5 cell_and_hash's inline, up to 32 ones in the external storage and then it's switched to std::set. Once switched, the whole union becomes the waste of space, as it's size is sizeof(vector head) + 5 * sizeof(cell and hash) = 90+ bytes and only 3 pointers from it are used (std::set header). Also the overhead to keep cell_and_hash as a set entry is more then the size of the structure itself. Column ids are 32-bit integers that most likely come sequentialy. For this kind of a search key a radix tree (with some care for non-sequential cases) can be beneficial. This set introduces a compact radix tree, that uses 7-bit sub values from the search key to index on each node and compacts the nodes themselves for better memory usage. Then the row::_storage is replaced with the new tree. The most notable result is the memory footprint decrease, for wide rows down to 2x times. The performance of micro-benchmarks is a bit lower for small rows and (!) higer for longer (8+ cells). The numbers are in patch #12 (spoiler: they are better than for v2) v3: - trimmed size of radix down to 7 bits - simplified the nodes layouts, now there are 2 of them (was 4) - enhanced perf_mutation to test N-cells schema - added AVX intra-nodes search for medium-sized nodes - added .clone_from() method that helped to improve perf_mutation - minor - changed functions not to return values via refs-arguments - fixed nested classes to properly use language constructors - renamed index_to to key_t to distinguish from node_index_t - improved recurring variadic templates not to use sentinel argument - use standard concepts v2: - fixed potential mis-compilation due to strict-aliasing violation - added oracle test (radix tree is compared with std::map) - added radix to perf_collection - cosmetic changes (concepts, comments, names) A note on item 1 from v2 changelog. The nodes are no longer packed perfectly, each has grown 3 bytes. But it turned out that when used as cells container most of this growth drowned in lsa alignments. next todo: - aarch64 version of 16-keys node search tests: unit(dev), unit(debug for radix*), pref(dev) " * 'br-radix-tree-for-cells-3' of https://github.com/xemul/scylla: test/memory_footpring: Print radix tree node sizes row: Remove old storages row: Prepare row::equal for switch row: Prepare row::difference for switch row: Introduce radix tree storage type row-equal: Re-declare the cells_equal lambda test: Add tests for radix tree utils: Compact radix tree array-search: Add helpers to search for a byte in array test/perf_collection: Add callback to check the speed of clone test/perf_mutation: Add option to run with more than 1 columns test/perf_mutation: Prepare to have several regular columns test/perf_mutation: Use builder to build schema
This exception is ignored by design, but if it's left unhandled, it generates `Exceptional future ignored` warnings, like the following. Also, ignore f2 if f1 failed since we return early in this case. ``` [shard 5] seastar - Exceptional future ignored: seastar::named_semaphore_timed_out (Semaphore timed out: _read_concurrency_sem), backtrace: 0x431689e 0x4316d40 0x43170e8 0x3f35486 0x218d14a 0x3f8002f 0x3f81217 0x3f9f868 0x3f4b76a /opt/scylladb/libreloc/libpthread.so.0+0x93f8 /opt/scylladb/libreloc/libc.so.6+0x101902#012 N7seastar12continuationINS_8internal22promise_base_with_typeISt7variantIJN5utils4UUIDEN7service5paxos7promiseEEEEEZZZZNS7_11paxos_state7prepareEN7tracing15trace_state_ptrENS_13lw_shared_ptrIK6schemaEERKN5query12read_commandERK13partition_keyS5_bNSI_16digest_algorithmENSt6chrono10time_pointINS_12lowres_clockENSQ_8durationIlSt5ratioILl1ELl1000EEEEEEENK3$_0clEvENUlvE_clEvENKUlSB_E_clESB_EUlT_E_ZNS_6futureISt5tupleIJNS13_IvEENS13_IS14_IJNSE_INSI_6resultEEE17cache_temperatureEEEEEEE14then_impl_nrvoIS12_NS13_IS9_EEEET0_OS11_EUlOSA_RS12_ONS_12future_stateIS1B_EEE_S1B_EE#012 seastar::continuation<seastar::internal::promise_base_with_type<std::variant<utils::UUID, service::paxos::promise> >, seastar::future<std::variant<utils::UUID, service::paxos::promise> >::finally_body<seastar::with_semaphore<seastar::semaphore_default_exception_factory, seastar::lowres_clock, service::paxos::paxos_state::prepare(tracing::trace_state_ptr, seastar::lw_shared_ptr<schema const>, query::read_command const&, partition_key const&, utils::UUID, bool, query::digest_algorithm, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::$_0::operator()() const::{lambda()#1}>(seastar::basic_semaphore<seastar::semaphore_default_exception_factory, seastar::lowres_clock>&, unsigned long, seastar::lowres_clock::duration, std::result_of&&)::{lambda(seastar::basic_semaphore)#1}::operator()<seastar::semaphore_units<seastar::semaphore_default_exception_factory, seastar::lowres_clock> >(seastar::basic_semaphore)::{lambda()#1}, false>, seastar::future<std::variant<utils::UUID, service::paxos::promise> >::then_wrapped_nrvo<seastar::future<std::variant<utils::UUID, service::paxos::promise> >, seastar::semaphore_units<seastar::semaphore_default_exception_factory, seastar::lowres_clock> >(seastar::future<std::variant<utils::UUID, service::paxos::promise> >&&)::{lambda(seastar::internal::promise_base_with_type<std::variant<utils::UUID, service::paxos::promise> >&&, seastar::semaphore_units<seastar::semaphore_default_exception_factory, seastar::lowres_clock>&, seastar::future_state<std::variant<utils::UUID, service::paxos::promise> >&&)#1}, std::variant<utils::UUID, service::paxos::promise> >#12 seastar::continuation<seastar::internal::promise_base_with_type<std::variant<utils::UUID, service::paxos::promise> >, seastar::future<std::variant<utils::UUID, service::paxos::promise> >::finally_body<service::paxos::paxos_state::key_lock_map::with_locked_key<service::paxos::paxos_state::prepare(tracing::trace_state_ptr, seastar::lw_shared_ptr<schema const>, query::read_command const&, partition_key const&, utils::UUID, bool, query::digest_algorithm, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::$_0::operator()() const::{lambda()#1}>(dht::token const&, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >, std::result_of)::{lambda()#1}, false>, seastar::future<std::variant<utils::UUID, service::paxos::promise> >::then_wrapped_nrvo<seastar::future<std::variant<utils::UUID, service::paxos::promise> >, {lambda()#1}>({lambda()#1}&&)::{lambda(seastar::internal::promise_base_with_type<std::variant<utils::UUID, service::paxos::promise> >&&, {lambda()#1}&, seastar::future_state<std::variant<utils::UUID, service::paxos::promise> >&&)#1}, std::variant<utils::UUID, service::paxos::promise> >#12 seastar::continuation<seastar::internal::promise_base_with_type<std::variant<utils::UUID, service::paxos::promise> >, seastar::future<std::variant<utils::UUID, service::paxos::promise> >::finally_body<service::paxos::paxos_state::prepare(tracing::trace_state_ptr, seastar::lw_shared_ptr<schema const>, query::read_command const&, partition_key const&, utils::UUID, bool, query::digest_algorithm, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::$_0::operator()() const::{lambda()#2}, false>, seastar::future<std::variant<utils::UUID, service::paxos::promise> >::then_wrapped_nrvo<seastar::future<std::variant<utils::UUID, service::paxos::promise> >, service::paxos::paxos_state::prepare(tracing::trace_state_ptr, seastar::lw_shared_ptr<schema const>, query::read_command const&, partition_key const&, utils::UUID, bool, query::digest_algorithm, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::$_0::operator()() const::{lambda()#2}>(service::paxos::paxos_state::prepare(tracing::trace_state_ptr, seastar::lw_shared_ptr<schema const>, query::read_command const&, partition_key const&, utils::UUID, bool, query::digest_algorithm, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::$_0::operator()() const::{lambda()#2}&&)::{lambda(seastar::internal::promise_base_with_type<std::variant<utils::UUID, service::paxos::promise> >&&, service::paxos::paxos_state::prepare(tracing::trace_state_ptr, seastar::lw_shared_ptr<schema const>, query::read_command const&, partition_key const&, utils::UUID, bool, query::digest_algorithm, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::$_0::operator()() const::{lambda()#2}&, seastar::future_state<std::variant<utils::UUID, service::paxos::promise> >&&)#1}, std::variant<utils::UUID, service::paxos::promise> >#12 seastar::continuation<seastar::internal::promise_base_with_type<seastar::foreign_ptr<std::unique_ptr<std::variant<utils::UUID, service::paxos::promise>, std::default_delete<std::variant<utils::UUID, service::paxos::promise> > > > >, service::storage_proxy::init_messaging_service()::$_51::operator()(seastar::rpc::client_info const&, seastar::rpc::opt_time_point, query::read_command, partition_key, utils::UUID, bool, query::digest_algorithm, std::optional<tracing::trace_info>) const::{lambda(seastar::lw_shared_ptr<schema const>)#1}::operator()(seastar::lw_shared_ptr<schema const>)::{lambda()#1}::operator()() const::{lambda(std::variant<utils::UUID, service::paxos::promise>)#1}, seastar::future<std::variant<utils::UUID, service::paxos::promise> >::then_impl_nrvo<{lambda()#1}, {lambda()#1}<seastar::foreign_ptr<std::unique_ptr<std::variant<utils::UUID, service::paxos::promise>, std::default_delete<std::variant<utils::UUID, service::paxos::promise> > > > > >({lambda()#1}&&)::{lambda(seastar::internal::promise_base_with_type<seastar::foreign_ptr<std::unique_ptr<std::variant<utils::UUID, service::paxos::promise>, std::default_delete<std::variant<utils::UUID, service::paxos::promise> > > > >&&, {lambda()#1}&, seastar::future_state<std::variant<utils::UUID, service::paxos::promise> >&&)#1}, std::variant<utils::UUID, service::paxos::promise> >#12 seastar::continuation<seastar::internal::promise_base_with_type<seastar::foreign_ptr<std::unique_ptr<std::variant<utils::UUID, service::paxos::promise>, std::default_delete<std::variant<utils::UUID, service::paxos::promise> > > > >, seastar::future<seastar::foreign_ptr<std::unique_ptr<std::variant<utils::UUID, service::paxos::promise>, std::default_delete<std::variant<utils::UUID, service::paxos::promise> > > > >::finally_body<seastar::smp::submit_to<service::storage_proxy::init_messaging_service()::$_51::operator()(seastar::rpc::client_info const&, seastar::rpc::opt_time_point, query::read_command, partition_key, utils::UUID, bool, query::digest_algorithm, std::optional<tracing::trace_info>) const::{lambda(seastar::lw_shared_ptr<schema const>)#1}::operator()(seastar::lw_shared_ptr<schema const>)::{lambda()#1}>(unsigned int, se ``` Refs #7779 Refs #9331 Signed-off-by: Benny Halevy <bhalevy@scylladb.com> Message-Id: <20210919053007.13960-1-bhalevy@scylladb.com>
…olodovnikov Since the code is using LWT, we need to handle bounce to shard requests. Switch to using `execute_direct` which returns a `result_message` object and handle `msg->move_to_shard` explicitly. Quite hackish, but sufficient for hackathon purposes. Also, introduce a simple test to verify that these basic operations work correctly. Closes scylladb#12 * github.com:avikivity/scylla: db: shared_sstables: add a test for distributed sstable reference counting routines system_distributed_keyspace: handle bounce_to_shard requests in `shared_sstables*` methods
We need it to be like origin:
INFO 12:27:51 Enqueuing flush of compactions_in_progress: 164 (0%) on-heap, 0 (0%) off-heap
INFO 12:27:51 Writing Memtable-compactions_in_progress@1174094719(8 serialized bytes, 1 ops, 0%/0% of on/off-heap limit)
INFO 12:27:51 Completed flushing /data/var/lib/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-ka-4-Data.db (42 bytes) for commitlog position ReplayPosition(segmentId=1436271859142, position=8145773)
INFO 12:27:51 Compacting [SSTableReader(path='/data/var/lib/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-ka-2-Data.db'), SSTableReader(path='/data/var/lib/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-ka-1-Data.db'), SSTableReader(path='/data/var/lib/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-ka-3-Data.db'), SSTableReader(path='/data/var/lib/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-ka-4-Data.db')]
INFO 12:27:51 Compacted 4 sstables to [/data/var/lib/cassandra/data/keyspace1/standard1-34b16ba024a311e588eab980f3a13a97/keyspace1-standard1-ka-11,]. 1,028,147,528 bytes to 281,000,000 (~27% of original) in 61,605ms = 4.350012MB/s. 3,658,888 total partitions merged to 1,000,000. Partition merge counts were {3:341112, 4:658888, }
INFO 12:27:51 Compacted 4 sstables to [/data/var/lib/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-ka-5,]. 427 bytes to 42 (~9% of original) in 13ms = 0.003081MB/s. 4 total partitions merged to 1. Partition merge counts were {2:2, }
The text was updated successfully, but these errors were encountered: