Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Missing initialization of system.schema_columnfamilies.key_validator field #7

Closed
slivne opened this issue Jul 5, 2015 · 0 comments
Closed

Comments

@slivne
Copy link
Contributor

slivne commented Jul 5, 2015

JavaDriver requires the field to exist

Applying this hack - allows the driver to boot.

shlomi@localhost~/urchin (master)$ git diff
diff --git a/db/legacy_schema_tables.cc b/db/legacy_schema_tables.cc
index 463210c..b0d90a2 100644
--- a/db/legacy_schema_tables.cc
+++ b/db/legacy_schema_tables.cc
@@ -1024,6 +1024,7 @@ std::vector<const char*> ALL { KEYSPACES, COLUMNFAMILIES, COLUMNS, TRIGGERS, USE
adder.add("compaction_strategy_class", table.compactionStrategyClass.getName());
adder.add("compaction_strategy_options", json(table.compactionStrategyOptions));
#endif

  •    m.set_clustered_cell(ckey, "key_validator", sstring("org.apache.cassandra.db.marshal.UTF8Type"), timestamp);
     const auto& compression_options = table->get_compressor_params();
     m.set_clustered_cell(ckey, "compression_parameters", json::to_json(compression_options.get_options()), timestamp);
     m.set_clustered_cell(ckey, "default_time_to_live", table->default_time_to_live().count(), timestamp)
    
tgrabiec added a commit that referenced this issue Sep 10, 2015
Also fixes scylladb/seastar#54

==5658==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x6250006b7848 at pc 0x1413e02 bp 0x7fff7cd7f1e0 sp 0x7fff7cd7f1d8
WRITE of size 8 at 0x6250006b7848 thread T0
    #0 0x1413e01 in unsigned long* std::__copy_move<false, false, std::random_access_iterator_tag>::__copy_m<std::_Deque_iterator<unsigned long, unsigned long&, unsigned long*>, unsigned long*>(std::_Deque_iterator<unsigned long, unsigned long&, unsigned long*>, std::_Deque_iterator<unsigned long, unsigned long&, unsigned long*>, unsigned long*) /usr/include/c++/4.9/bits/stl_algobase.h:336
    #1 0x1413c59 in unsigned long* std::__copy_move_a<false, std::_Deque_iterator<unsigned long, unsigned long&, unsigned long*>, unsigned long*>(std::_Deque_iterator<unsigned long, unsigned long&, unsigned long*>, std::_Deque_iterator<unsigned long, unsigned long&, unsigned long*>, unsigned long*) /usr/include/c++/4.9/bits/stl_algobase.h:396
    #2 0x1413aea in unsigned long* std::__copy_move_a2<false, std::_Deque_iterator<unsigned long, unsigned long&, unsigned long*>, unsigned long*>(std::_Deque_iterator<unsigned long, unsigned long&, unsigned long*>, std::_Deque_iterator<unsigned long, unsigned long&, unsigned long*>, unsigned long*) /usr/include/c++/4.9/bits/stl_algobase.h:434
    #3 0x14138df in unsigned long* std::copy<std::_Deque_iterator<unsigned long, unsigned long&, unsigned long*>, unsigned long*>(std::_Deque_iterator<unsigned long, unsigned long&, unsigned long*>, std::_Deque_iterator<unsigned long, unsigned long&, unsigned long*>, unsigned long*) /usr/include/c++/4.9/bits/stl_algobase.h:466
    #4 0x1413545 in unsigned long* std::__copy_n<std::_Deque_iterator<unsigned long, unsigned long&, unsigned long*>, unsigned long, unsigned long*>(std::_Deque_iterator<unsigned long, unsigned long&, unsigned long*>, unsigned long, unsigned long*, std::random_access_iterator_tag) /usr/include/c++/4.9/bits/stl_algo.h:779
    #5 0x1412d44 in unsigned long* std::copy_n<std::_Deque_iterator<unsigned long, unsigned long&, unsigned long*>, unsigned long, unsigned long*>(std::_Deque_iterator<unsigned long, unsigned long&, unsigned long*>, unsigned long, unsigned long*) /usr/include/c++/4.9/bits/stl_algo.h:804
    #6 0x14112b3 in unsigned long large_bitset::load<std::_Deque_iterator<unsigned long, unsigned long&, unsigned long*> >(std::_Deque_iterator<unsigned long, unsigned long&, unsigned long*>, std::_Deque_iterator<unsigned long, unsigned long&, unsigned long*>, unsigned long) utils/large_bitset.hh:81
    #7 0x13fcfc9 in _ZZZN8sstables7sstable11read_filterEvENKUlRT_E_clINS_6filterEEEDaS2_ENKUlvE_clEv (/home/tgrabiec/src/urchin/build/debug/scylla+0x13fcfc9)
    #8 0x1400a50 in apply /home/tgrabiec/src/urchin/seastar/core/apply.hh:34
    #9 0x1400afb in apply<sstables::sstable::read_filter()::<lambda(auto:25&)> [with auto:25 = sstables::filter]::<lambda()> > /home/tgrabiec/src/urchin/seastar/core/apply.hh:42
    #10 0x1400bb2 in apply<sstables::sstable::read_filter()::<lambda(auto:25&)> [with auto:25 = sstables::filter]::<lambda()> > /home/tgrabiec/src/urchin/seastar/core/future.hh:1062
    #11 0x140f1b7 in _ZZN6futureIIEE4thenIZZN8sstables7sstable11read_filterEvENKUlRT_E_clINS2_6filterEEEDaS5_EUlvE_S0_EET0_OT_ENUlOS4_E_clI12future_stateIIEEEEDaSC_ (/home/tgrabiec/src/urchin/build/debug/scylla+0x140f1b7)
    #12 0x140f350 in run /home/tgrabiec/src/urchin/seastar/core/future.hh:359
    #13 0x426e2c in reactor::run_tasks(circular_buffer<std::unique_ptr<task, std::default_delete<task> >, std::allocator<std::unique_ptr<task, std::default_delete<task> > > >&, unsigned long) core/reactor.cc:1093
    #14 0x429cb1 in reactor::run() core/reactor.cc:1190
    #15 0x72bc69 in app_template::run_deprecated(int, char**, std::function<void ()>&&) core/app-template.cc:122
    #16 0xa119bc in main /home/tgrabiec/src/urchin/main.cc:279
    #17 0x7ffc1b6beec4 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21ec4)
    #18 0x412558 (/home/tgrabiec/src/urchin/build/debug/scylla+0x412558)

0x6250006b7848 is located 0 bytes to the right of 8008-byte region [0x6250006b5900,0x6250006b7848)
allocated by thread T0 here:
    #0 0x7ffc1cf6c7df in __interceptor_malloc (/usr/lib/x86_64-linux-gnu/libasan.so.1+0x547df)
    #1 0x7ffc204eef17 in operator new(unsigned long) (/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0x8df17)
    #2 0xfa5d4f in large_bitset::large_bitset(unsigned long) utils/large_bitset.cc:15
    #3 0x13fcec6 in _ZZZN8sstables7sstable11read_filterEvENKUlRT_E_clINS_6filterEEEDaS2_ENKUlvE_clEv (/home/tgrabiec/src/urchin/build/debug/scylla+0x13fcec6)
    #4 0x1400a50 in apply /home/tgrabiec/src/urchin/seastar/core/apply.hh:34
    #5 0x1400afb in apply<sstables::sstable::read_filter()::<lambda(auto:25&)> [with auto:25 = sstables::filter]::<lambda()> > /home/tgrabiec/src/urchin/seastar/core/apply.hh:42
    #6 0x1400bb2 in apply<sstables::sstable::read_filter()::<lambda(auto:25&)> [with auto:25 = sstables::filter]::<lambda()> > /home/tgrabiec/src/urchin/seastar/core/future.hh:1062
    #7 0x140f1b7 in _ZZN6futureIIEE4thenIZZN8sstables7sstable11read_filterEvENKUlRT_E_clINS2_6filterEEEDaS5_EUlvE_S0_EET0_OT_ENUlOS4_E_clI12future_stateIIEEEEDaSC_ (/home/tgrabiec/src/urchin/build/debug/scylla+0x140f1b7)
    #8 0x140f350 in run /home/tgrabiec/src/urchin/seastar/core/future.hh:359
    #9 0x426e2c in reactor::run_tasks(circular_buffer<std::unique_ptr<task, std::default_delete<task> >, std::allocator<std::unique_ptr<task, std::default_delete<task> > > >&, unsigned long) core/reactor.cc:1093
    #10 0x429cb1 in reactor::run() core/reactor.cc:1190
    #11 0x72bc69 in app_template::run_deprecated(int, char**, std::function<void ()>&&) core/app-template.cc:122
    #12 0xa119bc in main /home/tgrabiec/src/urchin/main.cc:279
    #13 0x7ffc1b6beec4 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21ec4)

SUMMARY: AddressSanitizer: heap-buffer-overflow /usr/include/c++/4.9/bits/stl_algobase.h:336 unsigned long* std::__copy_move<false, false, std::random_access_iterator_tag>::__copy_m<std::_Deque_iterator<unsigned long, unsigned long&, unsigned long*>, unsigned long*>(std::_Deque_iterator<unsigned long, unsigned long&, unsigned long*>, std::_Deque_iterator<unsigned long, unsigned long&, unsigned long*>, unsigned long*)
Shadow bytes around the buggy address:
  0x0c4a800ceeb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c4a800ceec0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c4a800ceed0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c4a800ceee0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c4a800ceef0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x0c4a800cef00: 00 00 00 00 00 00 00 00 00[fa]fa fa fa fa fa fa
  0x0c4a800cef10: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c4a800cef20: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c4a800cef30: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c4a800cef40: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c4a800cef50: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Heap right redzone:      fb
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack partial redzone:   f4
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Contiguous container OOB:fc
  ASan internal:           fe
==5658==ABORTING
penberg pushed a commit that referenced this issue Jan 25, 2016
Fixes the heap-use-after-free error in build/debug/tests/auth_test

==1415==ERROR: AddressSanitizer: heap-use-after-free on address
0x62200032cfa8 at pc 0x00000350701d bp 0x7fec96df8d40 sp
0x7fec96df8d30
READ of size 8 at 0x62200032cfa8 thread T1
    #0 0x350701c in
_ZZN3gms8gossiper3runEvENKUlOT_E0_clI6futureIJEEEEDaS2_
(/home/penberg/scylla/build/debug/tests/auth_test_g+0x350701c)
    #1 0x35795b1 in apply<gms::gossiper::run()::<lambda(auto:40&&)>,
future<> > /home/penberg/scylla/seastar/core/future.hh:1203
    #2 0x369103d in
_ZZN6futureIJEE12then_wrappedIZN3gms8gossiper3runEvEUlOT_E0_S0_EET0_S5_ENUlS5_E_clI12future_stateIJEEEEDaS5_
(/home/penberg/scylla/build/debug/tests/auth_test_g+0x369103d)
    #3 0x369182a in run /home/penberg/scylla/seastar/core/future.hh:399
    #4 0x435f24 in
reactor::run_tasks(circular_buffer<std::unique_ptr<task,
std::default_delete<task> >, std::allocator<std::unique_ptr<task,
std::default_delete<task> > > >&) core/reactor.cc:1368
    #5 0x43a44f in reactor::run() core/reactor.cc:1672
    #6 0x952e4b in app_template::run_deprecated(int, char**,
std::function<void ()>&&) core/app-template.cc:123
    #7 0x58dc79d in test_runner::start(int,
char**)::{lambda()#1}::operator()()
(/home/penberg/scylla/build/debug/tests/auth_test_g+0x58dc79d)
    #8 0x58e6cd6 in _M_invoke /usr/include/c++/5.3.1/functional:1871
    #9 0x688639 in std::function<void ()>::operator()() const
/usr/include/c++/5.3.1/functional:2271
    #10 0x8d939c in posix_thread::start_routine(void*) core/posix.cc:51
    #11 0x7feca02a4609 in start_thread (/lib64/libpthread.so.0+0x7609)
    #12 0x7fec9ffdea4c in clone (/lib64/libc.so.6+0x102a4c)

0x62200032cfa8 is located 5800 bytes inside of 5808-byte region
[0x62200032b900,0x62200032cfb0)
freed by thread T1 here:
    #0 0x7feca4f76472 in operator delete(void*, unsigned long)
(/lib64/libasan.so.2+0x9a472)
    #1 0x3740772 in gms::gossiper::~gossiper()
(/home/penberg/scylla/build/debug/tests/auth_test_g+0x3740772)
    #2 0x2588ba1 in shared_ptr<gms::gossiper>::~shared_ptr()
seastar/core/shared_ptr.hh:389
    #3 0x4fc908c in
seastar::sharded<gms::gossiper>::stop()::{lambda(unsigned
int)#1}::operator()(unsigned
int)::{lambda()#1}::operator()()::{lambda()#1}::~stop()
(/home/penberg/scylla/build/debug/tests/auth_test_g+0x4fc908c)
    #4 0x4ff722a in future<>
future<>::then<seastar::sharded<gms::gossiper>::stop()::{lambda(unsigned
int)#1}::operator()(unsigned
int)::{lambda()#1}::operator()()::{lambda()#1}, future<>
>(seastar::sharded<gms::gossiper>::stop()::{lambda(unsigned
int)#1}::operator()(unsigned
int)::{lambda()#1}::operator()()::{lambda()#1}&&)::{lambda(seastar::sharded<gms::gossiper>::stop()::{lambda(unsigned
int)#1}::operator()(unsigned
int)::{lambda()#1}::operator()()::{lambda()#1})#1}::~then()
(/home/penberg/scylla/build/debug/tests/auth_test_g+0x4ff722a)
    #5 0x509a28c in continuation<future<>
future<>::then<seastar::sharded<gms::gossiper>::stop()::{lambda(unsigned
int)#1}::operator()(unsigned
int)::{lambda()#1}::operator()()::{lambda()#1}, future<>
>(seastar::sharded<gms::gossiper>::stop()::{lambda(unsigned
int)#1}::operator()(unsigned
int)::{lambda()#1}::operator()()::{lambda()#1}&&)::{lambda(seastar::sharded<gms::gossiper>::stop()::{lambda(unsigned
int)#1}::operator()(unsigned
int)::{lambda()#1}::operator()()::{lambda()#1})#1}>::~continuation()
seastar/core/future.hh:395
    #6 0x509a40d in continuation<future<>
Message-Id: <f8f1c92c1eb88687ab0534f5e7874d53050a5b93.1453446350.git.asias@scylladb.com>
penberg pushed a commit that referenced this issue Mar 11, 2016
Fix bootstrap_test.py:TestBootstrap.failed_bootstap_wiped_node_can_join_test

Logs on node 1:
 INFO  2016-03-11 15:53:43,287 [shard 0] gossip - FatClient 127.0.0.2 has been silent for 30000ms, removing from gossip
 INFO  2016-03-11 15:53:43,287 [shard 0] stream_session - stream_manager: Close all stream_session with peer = 127.0.0.2 in on_remove
 WARN  2016-03-11 15:53:43,498 [shard 0] stream_session - [Stream #4e411ba0-e75e-11e5-81f8-000000000000] stream_transfer_task: Fail to send STREAM_MUTATION_DONE to 127.0.0.2:0: std::runtime_error ([Stream #4e411ba0-e75e-11e5-81f8-000000000000] GOT STREAM_ MUTATION_DONE 127.0.0.1: Can not find stream_manager)

 terminate called without an active exception

Backtrace on node 1:

 #0  0x00007fb74723da98 in raise () from /lib64/libc.so.6
 #1  0x00007fb74723f69a in abort () from /lib64/libc.so.6
 #2  0x00007fb74ab84aed in __gnu_cxx::__verbose_terminate_handler() () from /lib64/libstdc++.so.6
 #3  0x00007fb74ab82936 in ?? () from /lib64/libstdc++.so.6
 #4  0x00007fb74ab82981 in std::terminate() () from /lib64/libstdc++.so.6
 #5  0x00007fb74ab82be9 in __cxa_rethrow () from /lib64/libstdc++.so.6
 #6  0x0000000000f3521e in streaming::stream_transfer_task::<lambda()>::<lambda(auto:44)>::operator()<std::__exception_ptr::exception_ptr> (ep=..., __closure=0x7ffce74d8630) at streaming/stream_transfer_task.cc:169
 #7  do_void_futurize_apply<const streaming::stream_transfer_task::start()::<lambda()>::<lambda(auto:44)>&, std::__exception_ptr::exception_ptr> (func=...) at /home/asias/src/cloudius-systems/scylla/seastar/core/future.hh:1142
 #8  futurize<void>::apply<const streaming::stream_transfer_task::start()::<lambda()>::<lambda(auto:44)>&, std::__exception_ptr::exception_ptr> (func=...) at /home/asias/src/cloudius-systems/scylla/seastar/core/future.hh:1190
 #9  future<>::<lambda(auto:7&&)>::operator()<future<> > ( fut=fut@entry=<unknown type in /home/asias/src/cloudius-systems/scylla/build/release/scylla, CU 0xec84d00, DIE 0xee2561d>, __closure=__closure@entry=0x7ffce74d8630) at /home/asias/src/cloudius-systems/scylla/seastar/core/future.hh:1014

Message-Id: <1457684884-4776-2-git-send-email-asias@scylladb.com>
bhalevy added a commit to bhalevy/scylla that referenced this issue Mar 10, 2020
Rather than using lw_shared_ptr to keep it around, use do_with.
Fixes a use-after-free seen with interrupt_build_process_with_resharding_half_to_max_test
in debug mode:

==406079==ERROR: AddressSanitizer: heap-use-after-free on address 0x603001320ac0 at pc 0x00000777a062 bp 0x7ffdeb77b010 sp 0x7ffdeb77b000
READ of size 8 at 0x603001320ac0 thread T0
    #0 0x777a061 in _M_next /usr/include/c++/9/bits/hashtable_policy.h:283
    scylladb#1 0x779ead7 in _M_incr /usr/include/c++/9/bits/hashtable_policy.h:299
    scylladb#2 0x777bbac in operator++ /usr/include/c++/9/bits/hashtable_policy.h:362
    scylladb#3 0x77f8384 in run_and_dispose /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future-util.hh:568
    scylladb#4 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091
    scylladb#5 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506
    scylladb#6 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666
    scylladb#7 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199
    scylladb#8 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115
    scylladb#9 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486
    scylladb#10 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2)
    scylladb#11 0x42b69d in _start (/local/home/bhalevy/.dtest/dtest-q85idok7/test/node1/bin/scylla+0x42b69d)

0x603001320ac0 is located 0 bytes inside of 32-byte region [0x603001320ac0,0x603001320ae0)
freed by thread T0 here:
    #0 0x7fb86520996f in operator delete(void*) (/lib64/libasan.so.5+0x11096f)
    scylladb#1 0x77d1889 in deallocate /usr/include/c++/9/ext/new_allocator.h:128
    scylladb#2 0x77bcbb2 in deallocate /usr/include/c++/9/bits/alloc_traits.h:470
    scylladb#3 0x77bccc5 in _M_deallocate_node_ptr /usr/include/c++/9/bits/hashtable_policy.h:2113
    scylladb#4 0x779dc59 in _M_deallocate_node /usr/include/c++/9/bits/hashtable_policy.h:2103
    scylladb#5 0x77bc8b6 in _M_deallocate_nodes /usr/include/c++/9/bits/hashtable_policy.h:2124
    scylladb#6 0x779d18f in clear /usr/include/c++/9/bits/hashtable.h:2028
    scylladb#7 0x7779b8e in ~_Hashtable /usr/include/c++/9/bits/hashtable.h:1352
    scylladb#8 0x774cfed in ~unordered_map /usr/include/c++/9/bits/unordered_map.h:102
    scylladb#9 0x774d288 in ~shared_ptr_no_esft /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:157
    scylladb#10 0x774d2f1 in dispose /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:213
    scylladb#11 0x76f9642 in ~lw_shared_ptr /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:300
    scylladb#12 0x76f9642 in ~<lambda> /local/home/bhalevy/dev/scylla/distributed_loader.cc:743
    scylladb#13 0x771e0a7 in ~<lambda> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1167
    scylladb#14 0x777c64c in destroy /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:133
    scylladb#15 0x586059 in seastar::noncopyable_function<seastar::future<> ()>::~noncopyable_function() /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:165
    scylladb#16 0x5867da in seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}::~future_state() /local/home/bha
levy/dev/scylla/seastar/include/seastar/core/future.hh:1192
    scylladb#17 0xab704d in seastar::continuation<seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}>::~continu
ation() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:507
    scylladb#18 0xab7216 in seastar::continuation<seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}>::run_and_
dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:512
    scylladb#19 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091
    scylladb#20 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506
    scylladb#21 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666
    scylladb#22 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199
    scylladb#23 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115
    scylladb#24 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486
    scylladb#25 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2)

previously allocated by thread T0 here:
    #0 0x7fb8652089d7 in operator new(unsigned long) (/lib64/libasan.so.5+0x10f9d7)
    scylladb#1 0x77d1796 in allocate /usr/include/c++/9/ext/new_allocator.h:114
    scylladb#2 0x77bcab3 in allocate /usr/include/c++/9/bits/alloc_traits.h:444
    scylladb#3 0x779d936 in _M_allocate_node<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable_policy.h:2081
    scylladb#4 0x777a2ed in _M_emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable.h:1638
    scylladb#5 0x774d707 in emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable.h:749
    scylladb#6 0x771ce3e in emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/unordered_map.h:388
    scylladb#7 0x76f8280 in operator() /local/home/bhalevy/dev/scylla/distributed_loader.cc:732
    scylladb#8 0x774dc5d in _M_invoke /usr/include/c++/9/bits/std_function.h:286
    scylladb#9 0x6cde67b in std::function<seastar::future<> (std::filesystem::__cxx11::path, seastar::directory_entry)>::operator()(std::filesystem::__cxx11::path, seastar::directory_entry) const /usr/include/c++/9/bits/std_function.h:690
    scylladb#10 0x6cd345d in operator() /local/home/bhalevy/dev/scylla/lister.cc:30
    scylladb#11 0x6cd96e0 in apply /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36
    scylladb#12 0x6cd9854 in apply<lister::visit(seastar::directory_entry)::<lambda(seastar::directory_entry)>&, seastar::directory_entry&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:44
    scylladb#13 0x6cd9953 in apply<lister::visit(seastar::directory_entry)::<lambda(seastar::directory_entry)>&, seastar::directory_entry&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1650
    scylladb#14 0x6cd6ba9 in operator() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1168
    scylladb#15 0x6cdbac0 in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:101
    scylladb#16 0x6ce6178 in seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>::operator()(seastar::directory_entry&&) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:184
    scylladb#17 0x6ce3033 in seastar::apply_helper<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, std::tuple<seastar::directory_entry>&&, std::integer_sequence<unsigned long, 0ul> >::apply(seastar::noncopyable_function<seastar::future<> (seastar::directory_e
ntry&&)>&&, std::tuple<seastar::directory_entry>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36
    scylladb#18 0x6ce3130 in auto seastar::apply<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::directory_entry>(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&, std::tuple<seastar::directory_entry>&&) /local/home/bha
levy/dev/scylla/seastar/include/seastar/core/apply.hh:44
    scylladb#19 0x6ce322f in seastar::future<> seastar::futurize<seastar::future<> >::apply<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::directory_entry>(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&, std::tuple<s
eastar::directory_entry>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1650
    scylladb#20 0x6cdf88d in seastar::future<seastar::directory_entry>::then_impl<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&)::{lambda()scylladb#1}::operator()()
const::{lambda(seastar::future_state<seastar::directory_entry>&&)scylladb#1}::operator()(seastar::future_state<seastar::directory_entry>) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1196
    scylladb#21 0x6cea2e0 in seastar::continuation<seastar::future<seastar::directory_entry>::then_impl<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&)::{lamb
da()scylladb#1}::operator()() const::{lambda(seastar::future_state<seastar::directory_entry>&&)scylladb#1}, seastar::directory_entry>::run_and_dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:511
    scylladb#22 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091
    scylladb#23 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506
    scylladb#24 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666
    scylladb#25 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199
    scylladb#26 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115
    scylladb#27 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486
    scylladb#28 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2)

SUMMARY: AddressSanitizer: heap-use-after-free /usr/include/c++/9/bits/hashtable_policy.h:283 in _M_next
Shadow bytes around the buggy address:
  0x0c068025c100: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c068025c110: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa
  0x0c068025c120: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd
  0x0c068025c130: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c068025c140: fd fd fd fa fa fa 00 00 00 fa fa fa fd fd fd fa
=>0x0c068025c150: fa fa fd fd fd fa fa fa[fd]fd fd fd fa fa fd fd
  0x0c068025c160: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c068025c170: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa
  0x0c068025c180: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd
  0x0c068025c190: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c068025c1a0: fd fd fd fd fa fa fd fd fd fd fa fa 00 00 00 fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==406079==ABORTING

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
bhalevy added a commit to bhalevy/scylla that referenced this issue Mar 11, 2020
Rather than using lw_shared_ptr to keep it around, use do_with.
Fixes a use-after-free seen with interrupt_build_process_with_resharding_half_to_max_test
in debug mode:

==406079==ERROR: AddressSanitizer: heap-use-after-free on address 0x603001320ac0 at pc 0x00000777a062 bp 0x7ffdeb77b010 sp 0x7ffdeb77b000
READ of size 8 at 0x603001320ac0 thread T0
    #0 0x777a061 in _M_next /usr/include/c++/9/bits/hashtable_policy.h:283
    scylladb#1 0x779ead7 in _M_incr /usr/include/c++/9/bits/hashtable_policy.h:299
    scylladb#2 0x777bbac in operator++ /usr/include/c++/9/bits/hashtable_policy.h:362
    scylladb#3 0x77f8384 in run_and_dispose /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future-util.hh:568
    scylladb#4 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091
    scylladb#5 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506
    scylladb#6 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666
    scylladb#7 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199
    scylladb#8 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115
    scylladb#9 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486
    scylladb#10 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2)
    scylladb#11 0x42b69d in _start (/local/home/bhalevy/.dtest/dtest-q85idok7/test/node1/bin/scylla+0x42b69d)

0x603001320ac0 is located 0 bytes inside of 32-byte region [0x603001320ac0,0x603001320ae0)
freed by thread T0 here:
    #0 0x7fb86520996f in operator delete(void*) (/lib64/libasan.so.5+0x11096f)
    scylladb#1 0x77d1889 in deallocate /usr/include/c++/9/ext/new_allocator.h:128
    scylladb#2 0x77bcbb2 in deallocate /usr/include/c++/9/bits/alloc_traits.h:470
    scylladb#3 0x77bccc5 in _M_deallocate_node_ptr /usr/include/c++/9/bits/hashtable_policy.h:2113
    scylladb#4 0x779dc59 in _M_deallocate_node /usr/include/c++/9/bits/hashtable_policy.h:2103
    scylladb#5 0x77bc8b6 in _M_deallocate_nodes /usr/include/c++/9/bits/hashtable_policy.h:2124
    scylladb#6 0x779d18f in clear /usr/include/c++/9/bits/hashtable.h:2028
    scylladb#7 0x7779b8e in ~_Hashtable /usr/include/c++/9/bits/hashtable.h:1352
    scylladb#8 0x774cfed in ~unordered_map /usr/include/c++/9/bits/unordered_map.h:102
    scylladb#9 0x774d288 in ~shared_ptr_no_esft /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:157
    scylladb#10 0x774d2f1 in dispose /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:213
    scylladb#11 0x76f9642 in ~lw_shared_ptr /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:300
    scylladb#12 0x76f9642 in ~<lambda> /local/home/bhalevy/dev/scylla/distributed_loader.cc:743
    scylladb#13 0x771e0a7 in ~<lambda> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1167
    scylladb#14 0x777c64c in destroy /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:133
    scylladb#15 0x586059 in seastar::noncopyable_function<seastar::future<> ()>::~noncopyable_function() /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:165
    scylladb#16 0x5867da in seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}::~future_state() /local/home/bha
levy/dev/scylla/seastar/include/seastar/core/future.hh:1192
    scylladb#17 0xab704d in seastar::continuation<seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}>::~continu
ation() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:507
    scylladb#18 0xab7216 in seastar::continuation<seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}>::run_and_
dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:512
    scylladb#19 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091
    scylladb#20 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506
    scylladb#21 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666
    scylladb#22 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199
    scylladb#23 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115
    scylladb#24 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486
    scylladb#25 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2)

previously allocated by thread T0 here:
    #0 0x7fb8652089d7 in operator new(unsigned long) (/lib64/libasan.so.5+0x10f9d7)
    scylladb#1 0x77d1796 in allocate /usr/include/c++/9/ext/new_allocator.h:114
    scylladb#2 0x77bcab3 in allocate /usr/include/c++/9/bits/alloc_traits.h:444
    scylladb#3 0x779d936 in _M_allocate_node<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable_policy.h:2081
    scylladb#4 0x777a2ed in _M_emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable.h:1638
    scylladb#5 0x774d707 in emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable.h:749
    scylladb#6 0x771ce3e in emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/unordered_map.h:388
    scylladb#7 0x76f8280 in operator() /local/home/bhalevy/dev/scylla/distributed_loader.cc:732
    scylladb#8 0x774dc5d in _M_invoke /usr/include/c++/9/bits/std_function.h:286
    scylladb#9 0x6cde67b in std::function<seastar::future<> (std::filesystem::__cxx11::path, seastar::directory_entry)>::operator()(std::filesystem::__cxx11::path, seastar::directory_entry) const /usr/include/c++/9/bits/std_function.h:690
    scylladb#10 0x6cd345d in operator() /local/home/bhalevy/dev/scylla/lister.cc:30
    scylladb#11 0x6cd96e0 in apply /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36
    scylladb#12 0x6cd9854 in apply<lister::visit(seastar::directory_entry)::<lambda(seastar::directory_entry)>&, seastar::directory_entry&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:44
    scylladb#13 0x6cd9953 in apply<lister::visit(seastar::directory_entry)::<lambda(seastar::directory_entry)>&, seastar::directory_entry&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1650
    scylladb#14 0x6cd6ba9 in operator() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1168
    scylladb#15 0x6cdbac0 in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:101
    scylladb#16 0x6ce6178 in seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>::operator()(seastar::directory_entry&&) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:184
    scylladb#17 0x6ce3033 in seastar::apply_helper<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, std::tuple<seastar::directory_entry>&&, std::integer_sequence<unsigned long, 0ul> >::apply(seastar::noncopyable_function<seastar::future<> (seastar::directory_e
ntry&&)>&&, std::tuple<seastar::directory_entry>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36
    scylladb#18 0x6ce3130 in auto seastar::apply<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::directory_entry>(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&, std::tuple<seastar::directory_entry>&&) /local/home/bha
levy/dev/scylla/seastar/include/seastar/core/apply.hh:44
    scylladb#19 0x6ce322f in seastar::future<> seastar::futurize<seastar::future<> >::apply<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::directory_entry>(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&, std::tuple<s
eastar::directory_entry>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1650
    scylladb#20 0x6cdf88d in seastar::future<seastar::directory_entry>::then_impl<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&)::{lambda()scylladb#1}::operator()()
const::{lambda(seastar::future_state<seastar::directory_entry>&&)scylladb#1}::operator()(seastar::future_state<seastar::directory_entry>) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1196
    scylladb#21 0x6cea2e0 in seastar::continuation<seastar::future<seastar::directory_entry>::then_impl<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&)::{lamb
da()scylladb#1}::operator()() const::{lambda(seastar::future_state<seastar::directory_entry>&&)scylladb#1}, seastar::directory_entry>::run_and_dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:511
    scylladb#22 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091
    scylladb#23 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506
    scylladb#24 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666
    scylladb#25 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199
    scylladb#26 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115
    scylladb#27 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486
    scylladb#28 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2)

SUMMARY: AddressSanitizer: heap-use-after-free /usr/include/c++/9/bits/hashtable_policy.h:283 in _M_next
Shadow bytes around the buggy address:
  0x0c068025c100: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c068025c110: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa
  0x0c068025c120: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd
  0x0c068025c130: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c068025c140: fd fd fd fa fa fa 00 00 00 fa fa fa fd fd fd fa
=>0x0c068025c150: fa fa fd fd fd fa fa fa[fd]fd fd fd fa fa fd fd
  0x0c068025c160: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c068025c170: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa
  0x0c068025c180: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd
  0x0c068025c190: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c068025c1a0: fd fd fd fd fa fa fd fd fd fd fa fa 00 00 00 fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==406079==ABORTING

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
bhalevy added a commit to bhalevy/scylla that referenced this issue Apr 13, 2020
Rather than using lw_shared_ptr to keep it around, use do_with.
Fixes a use-after-free seen with interrupt_build_process_with_resharding_half_to_max_test
in debug mode:

==406079==ERROR: AddressSanitizer: heap-use-after-free on address 0x603001320ac0 at pc 0x00000777a062 bp 0x7ffdeb77b010 sp 0x7ffdeb77b000
READ of size 8 at 0x603001320ac0 thread T0
    #0 0x777a061 in _M_next /usr/include/c++/9/bits/hashtable_policy.h:283
    scylladb#1 0x779ead7 in _M_incr /usr/include/c++/9/bits/hashtable_policy.h:299
    scylladb#2 0x777bbac in operator++ /usr/include/c++/9/bits/hashtable_policy.h:362
    scylladb#3 0x77f8384 in run_and_dispose /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future-util.hh:568
    scylladb#4 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091
    scylladb#5 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506
    scylladb#6 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666
    scylladb#7 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199
    scylladb#8 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115
    scylladb#9 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486
    scylladb#10 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2)
    scylladb#11 0x42b69d in _start (/local/home/bhalevy/.dtest/dtest-q85idok7/test/node1/bin/scylla+0x42b69d)

0x603001320ac0 is located 0 bytes inside of 32-byte region [0x603001320ac0,0x603001320ae0)
freed by thread T0 here:
    #0 0x7fb86520996f in operator delete(void*) (/lib64/libasan.so.5+0x11096f)
    scylladb#1 0x77d1889 in deallocate /usr/include/c++/9/ext/new_allocator.h:128
    scylladb#2 0x77bcbb2 in deallocate /usr/include/c++/9/bits/alloc_traits.h:470
    scylladb#3 0x77bccc5 in _M_deallocate_node_ptr /usr/include/c++/9/bits/hashtable_policy.h:2113
    scylladb#4 0x779dc59 in _M_deallocate_node /usr/include/c++/9/bits/hashtable_policy.h:2103
    scylladb#5 0x77bc8b6 in _M_deallocate_nodes /usr/include/c++/9/bits/hashtable_policy.h:2124
    scylladb#6 0x779d18f in clear /usr/include/c++/9/bits/hashtable.h:2028
    scylladb#7 0x7779b8e in ~_Hashtable /usr/include/c++/9/bits/hashtable.h:1352
    scylladb#8 0x774cfed in ~unordered_map /usr/include/c++/9/bits/unordered_map.h:102
    scylladb#9 0x774d288 in ~shared_ptr_no_esft /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:157
    scylladb#10 0x774d2f1 in dispose /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:213
    scylladb#11 0x76f9642 in ~lw_shared_ptr /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:300
    scylladb#12 0x76f9642 in ~<lambda> /local/home/bhalevy/dev/scylla/distributed_loader.cc:743
    scylladb#13 0x771e0a7 in ~<lambda> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1167
    scylladb#14 0x777c64c in destroy /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:133
    scylladb#15 0x586059 in seastar::noncopyable_function<seastar::future<> ()>::~noncopyable_function() /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:165
    scylladb#16 0x5867da in seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}::~future_state() /local/home/bha
levy/dev/scylla/seastar/include/seastar/core/future.hh:1192
    scylladb#17 0xab704d in seastar::continuation<seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}>::~continu
ation() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:507
    scylladb#18 0xab7216 in seastar::continuation<seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}>::run_and_
dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:512
    scylladb#19 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091
    scylladb#20 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506
    scylladb#21 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666
    scylladb#22 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199
    scylladb#23 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115
    scylladb#24 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486
    scylladb#25 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2)

previously allocated by thread T0 here:
    #0 0x7fb8652089d7 in operator new(unsigned long) (/lib64/libasan.so.5+0x10f9d7)
    scylladb#1 0x77d1796 in allocate /usr/include/c++/9/ext/new_allocator.h:114
    scylladb#2 0x77bcab3 in allocate /usr/include/c++/9/bits/alloc_traits.h:444
    scylladb#3 0x779d936 in _M_allocate_node<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable_policy.h:2081
    scylladb#4 0x777a2ed in _M_emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable.h:1638
    scylladb#5 0x774d707 in emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable.h:749
    scylladb#6 0x771ce3e in emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/unordered_map.h:388
    scylladb#7 0x76f8280 in operator() /local/home/bhalevy/dev/scylla/distributed_loader.cc:732
    scylladb#8 0x774dc5d in _M_invoke /usr/include/c++/9/bits/std_function.h:286
    scylladb#9 0x6cde67b in std::function<seastar::future<> (std::filesystem::__cxx11::path, seastar::directory_entry)>::operator()(std::filesystem::__cxx11::path, seastar::directory_entry) const /usr/include/c++/9/bits/std_function.h:690
    scylladb#10 0x6cd345d in operator() /local/home/bhalevy/dev/scylla/lister.cc:30
    scylladb#11 0x6cd96e0 in apply /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36
    scylladb#12 0x6cd9854 in apply<lister::visit(seastar::directory_entry)::<lambda(seastar::directory_entry)>&, seastar::directory_entry&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:44
    scylladb#13 0x6cd9953 in apply<lister::visit(seastar::directory_entry)::<lambda(seastar::directory_entry)>&, seastar::directory_entry&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1650
    scylladb#14 0x6cd6ba9 in operator() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1168
    scylladb#15 0x6cdbac0 in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:101
    scylladb#16 0x6ce6178 in seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>::operator()(seastar::directory_entry&&) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:184
    scylladb#17 0x6ce3033 in seastar::apply_helper<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, std::tuple<seastar::directory_entry>&&, std::integer_sequence<unsigned long, 0ul> >::apply(seastar::noncopyable_function<seastar::future<> (seastar::directory_e
ntry&&)>&&, std::tuple<seastar::directory_entry>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36
    scylladb#18 0x6ce3130 in auto seastar::apply<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::directory_entry>(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&, std::tuple<seastar::directory_entry>&&) /local/home/bha
levy/dev/scylla/seastar/include/seastar/core/apply.hh:44
    scylladb#19 0x6ce322f in seastar::future<> seastar::futurize<seastar::future<> >::apply<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::directory_entry>(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&, std::tuple<s
eastar::directory_entry>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1650
    scylladb#20 0x6cdf88d in seastar::future<seastar::directory_entry>::then_impl<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&)::{lambda()scylladb#1}::operator()()
const::{lambda(seastar::future_state<seastar::directory_entry>&&)scylladb#1}::operator()(seastar::future_state<seastar::directory_entry>) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1196
    scylladb#21 0x6cea2e0 in seastar::continuation<seastar::future<seastar::directory_entry>::then_impl<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&)::{lamb
da()scylladb#1}::operator()() const::{lambda(seastar::future_state<seastar::directory_entry>&&)scylladb#1}, seastar::directory_entry>::run_and_dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:511
    scylladb#22 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091
    scylladb#23 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506
    scylladb#24 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666
    scylladb#25 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199
    scylladb#26 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115
    scylladb#27 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486
    scylladb#28 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2)

SUMMARY: AddressSanitizer: heap-use-after-free /usr/include/c++/9/bits/hashtable_policy.h:283 in _M_next
Shadow bytes around the buggy address:
  0x0c068025c100: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c068025c110: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa
  0x0c068025c120: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd
  0x0c068025c130: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c068025c140: fd fd fd fa fa fa 00 00 00 fa fa fa fd fd fd fa
=>0x0c068025c150: fa fa fd fd fd fa fa fa[fd]fd fd fd fa fa fd fd
  0x0c068025c160: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c068025c170: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa
  0x0c068025c180: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd
  0x0c068025c190: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c068025c1a0: fd fd fd fd fa fa fd fd fd fd fa fa 00 00 00 fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==406079==ABORTING

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
bhalevy added a commit to bhalevy/scylla that referenced this issue Apr 17, 2020
Rather than using lw_shared_ptr to keep it around, use do_with.
Fixes a use-after-free seen with interrupt_build_process_with_resharding_half_to_max_test
in debug mode:

==406079==ERROR: AddressSanitizer: heap-use-after-free on address 0x603001320ac0 at pc 0x00000777a062 bp 0x7ffdeb77b010 sp 0x7ffdeb77b000
READ of size 8 at 0x603001320ac0 thread T0
    #0 0x777a061 in _M_next /usr/include/c++/9/bits/hashtable_policy.h:283
    scylladb#1 0x779ead7 in _M_incr /usr/include/c++/9/bits/hashtable_policy.h:299
    scylladb#2 0x777bbac in operator++ /usr/include/c++/9/bits/hashtable_policy.h:362
    scylladb#3 0x77f8384 in run_and_dispose /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future-util.hh:568
    scylladb#4 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091
    scylladb#5 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506
    scylladb#6 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666
    scylladb#7 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199
    scylladb#8 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115
    scylladb#9 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486
    scylladb#10 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2)
    scylladb#11 0x42b69d in _start (/local/home/bhalevy/.dtest/dtest-q85idok7/test/node1/bin/scylla+0x42b69d)

0x603001320ac0 is located 0 bytes inside of 32-byte region [0x603001320ac0,0x603001320ae0)
freed by thread T0 here:
    #0 0x7fb86520996f in operator delete(void*) (/lib64/libasan.so.5+0x11096f)
    scylladb#1 0x77d1889 in deallocate /usr/include/c++/9/ext/new_allocator.h:128
    scylladb#2 0x77bcbb2 in deallocate /usr/include/c++/9/bits/alloc_traits.h:470
    scylladb#3 0x77bccc5 in _M_deallocate_node_ptr /usr/include/c++/9/bits/hashtable_policy.h:2113
    scylladb#4 0x779dc59 in _M_deallocate_node /usr/include/c++/9/bits/hashtable_policy.h:2103
    scylladb#5 0x77bc8b6 in _M_deallocate_nodes /usr/include/c++/9/bits/hashtable_policy.h:2124
    scylladb#6 0x779d18f in clear /usr/include/c++/9/bits/hashtable.h:2028
    scylladb#7 0x7779b8e in ~_Hashtable /usr/include/c++/9/bits/hashtable.h:1352
    scylladb#8 0x774cfed in ~unordered_map /usr/include/c++/9/bits/unordered_map.h:102
    scylladb#9 0x774d288 in ~shared_ptr_no_esft /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:157
    scylladb#10 0x774d2f1 in dispose /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:213
    scylladb#11 0x76f9642 in ~lw_shared_ptr /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:300
    scylladb#12 0x76f9642 in ~<lambda> /local/home/bhalevy/dev/scylla/distributed_loader.cc:743
    scylladb#13 0x771e0a7 in ~<lambda> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1167
    scylladb#14 0x777c64c in destroy /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:133
    scylladb#15 0x586059 in seastar::noncopyable_function<seastar::future<> ()>::~noncopyable_function() /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:165
    scylladb#16 0x5867da in seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}::~future_state() /local/home/bha
levy/dev/scylla/seastar/include/seastar/core/future.hh:1192
    scylladb#17 0xab704d in seastar::continuation<seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}>::~continu
ation() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:507
    scylladb#18 0xab7216 in seastar::continuation<seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}>::run_and_
dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:512
    scylladb#19 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091
    scylladb#20 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506
    scylladb#21 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666
    scylladb#22 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199
    scylladb#23 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115
    scylladb#24 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486
    scylladb#25 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2)

previously allocated by thread T0 here:
    #0 0x7fb8652089d7 in operator new(unsigned long) (/lib64/libasan.so.5+0x10f9d7)
    scylladb#1 0x77d1796 in allocate /usr/include/c++/9/ext/new_allocator.h:114
    scylladb#2 0x77bcab3 in allocate /usr/include/c++/9/bits/alloc_traits.h:444
    scylladb#3 0x779d936 in _M_allocate_node<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable_policy.h:2081
    scylladb#4 0x777a2ed in _M_emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable.h:1638
    scylladb#5 0x774d707 in emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable.h:749
    scylladb#6 0x771ce3e in emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/unordered_map.h:388
    scylladb#7 0x76f8280 in operator() /local/home/bhalevy/dev/scylla/distributed_loader.cc:732
    scylladb#8 0x774dc5d in _M_invoke /usr/include/c++/9/bits/std_function.h:286
    scylladb#9 0x6cde67b in std::function<seastar::future<> (std::filesystem::__cxx11::path, seastar::directory_entry)>::operator()(std::filesystem::__cxx11::path, seastar::directory_entry) const /usr/include/c++/9/bits/std_function.h:690
    scylladb#10 0x6cd345d in operator() /local/home/bhalevy/dev/scylla/lister.cc:30
    scylladb#11 0x6cd96e0 in apply /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36
    scylladb#12 0x6cd9854 in apply<lister::visit(seastar::directory_entry)::<lambda(seastar::directory_entry)>&, seastar::directory_entry&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:44
    scylladb#13 0x6cd9953 in apply<lister::visit(seastar::directory_entry)::<lambda(seastar::directory_entry)>&, seastar::directory_entry&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1650
    scylladb#14 0x6cd6ba9 in operator() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1168
    scylladb#15 0x6cdbac0 in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:101
    scylladb#16 0x6ce6178 in seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>::operator()(seastar::directory_entry&&) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:184
    scylladb#17 0x6ce3033 in seastar::apply_helper<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, std::tuple<seastar::directory_entry>&&, std::integer_sequence<unsigned long, 0ul> >::apply(seastar::noncopyable_function<seastar::future<> (seastar::directory_e
ntry&&)>&&, std::tuple<seastar::directory_entry>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36
    scylladb#18 0x6ce3130 in auto seastar::apply<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::directory_entry>(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&, std::tuple<seastar::directory_entry>&&) /local/home/bha
levy/dev/scylla/seastar/include/seastar/core/apply.hh:44
    scylladb#19 0x6ce322f in seastar::future<> seastar::futurize<seastar::future<> >::apply<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::directory_entry>(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&, std::tuple<s
eastar::directory_entry>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1650
    scylladb#20 0x6cdf88d in seastar::future<seastar::directory_entry>::then_impl<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&)::{lambda()scylladb#1}::operator()()
const::{lambda(seastar::future_state<seastar::directory_entry>&&)scylladb#1}::operator()(seastar::future_state<seastar::directory_entry>) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1196
    scylladb#21 0x6cea2e0 in seastar::continuation<seastar::future<seastar::directory_entry>::then_impl<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&)::{lamb
da()scylladb#1}::operator()() const::{lambda(seastar::future_state<seastar::directory_entry>&&)scylladb#1}, seastar::directory_entry>::run_and_dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:511
    scylladb#22 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091
    scylladb#23 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506
    scylladb#24 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666
    scylladb#25 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199
    scylladb#26 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115
    scylladb#27 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486
    scylladb#28 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2)

SUMMARY: AddressSanitizer: heap-use-after-free /usr/include/c++/9/bits/hashtable_policy.h:283 in _M_next
Shadow bytes around the buggy address:
  0x0c068025c100: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c068025c110: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa
  0x0c068025c120: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd
  0x0c068025c130: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c068025c140: fd fd fd fa fa fa 00 00 00 fa fa fa fd fd fd fa
=>0x0c068025c150: fa fa fd fd fd fa fa fa[fd]fd fd fd fa fa fd fd
  0x0c068025c160: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c068025c170: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa
  0x0c068025c180: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd
  0x0c068025c190: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c068025c1a0: fd fd fd fd fa fa fd fd fd fd fa fa 00 00 00 fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==406079==ABORTING

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
bhalevy added a commit to bhalevy/scylla that referenced this issue Apr 29, 2020
Rather than using lw_shared_ptr to keep it around, use do_with.
Fixes a use-after-free seen with interrupt_build_process_with_resharding_half_to_max_test
in debug mode:

==406079==ERROR: AddressSanitizer: heap-use-after-free on address 0x603001320ac0 at pc 0x00000777a062 bp 0x7ffdeb77b010 sp 0x7ffdeb77b000
READ of size 8 at 0x603001320ac0 thread T0
    #0 0x777a061 in _M_next /usr/include/c++/9/bits/hashtable_policy.h:283
    scylladb#1 0x779ead7 in _M_incr /usr/include/c++/9/bits/hashtable_policy.h:299
    scylladb#2 0x777bbac in operator++ /usr/include/c++/9/bits/hashtable_policy.h:362
    scylladb#3 0x77f8384 in run_and_dispose /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future-util.hh:568
    scylladb#4 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091
    scylladb#5 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506
    scylladb#6 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666
    scylladb#7 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199
    scylladb#8 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115
    scylladb#9 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486
    scylladb#10 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2)
    scylladb#11 0x42b69d in _start (/local/home/bhalevy/.dtest/dtest-q85idok7/test/node1/bin/scylla+0x42b69d)

0x603001320ac0 is located 0 bytes inside of 32-byte region [0x603001320ac0,0x603001320ae0)
freed by thread T0 here:
    #0 0x7fb86520996f in operator delete(void*) (/lib64/libasan.so.5+0x11096f)
    scylladb#1 0x77d1889 in deallocate /usr/include/c++/9/ext/new_allocator.h:128
    scylladb#2 0x77bcbb2 in deallocate /usr/include/c++/9/bits/alloc_traits.h:470
    scylladb#3 0x77bccc5 in _M_deallocate_node_ptr /usr/include/c++/9/bits/hashtable_policy.h:2113
    scylladb#4 0x779dc59 in _M_deallocate_node /usr/include/c++/9/bits/hashtable_policy.h:2103
    scylladb#5 0x77bc8b6 in _M_deallocate_nodes /usr/include/c++/9/bits/hashtable_policy.h:2124
    scylladb#6 0x779d18f in clear /usr/include/c++/9/bits/hashtable.h:2028
    scylladb#7 0x7779b8e in ~_Hashtable /usr/include/c++/9/bits/hashtable.h:1352
    scylladb#8 0x774cfed in ~unordered_map /usr/include/c++/9/bits/unordered_map.h:102
    scylladb#9 0x774d288 in ~shared_ptr_no_esft /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:157
    scylladb#10 0x774d2f1 in dispose /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:213
    scylladb#11 0x76f9642 in ~lw_shared_ptr /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:300
    scylladb#12 0x76f9642 in ~<lambda> /local/home/bhalevy/dev/scylla/distributed_loader.cc:743
    scylladb#13 0x771e0a7 in ~<lambda> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1167
    scylladb#14 0x777c64c in destroy /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:133
    scylladb#15 0x586059 in seastar::noncopyable_function<seastar::future<> ()>::~noncopyable_function() /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:165
    scylladb#16 0x5867da in seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}::~future_state() /local/home/bha
levy/dev/scylla/seastar/include/seastar/core/future.hh:1192
    scylladb#17 0xab704d in seastar::continuation<seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}>::~continu
ation() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:507
    scylladb#18 0xab7216 in seastar::continuation<seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}>::run_and_
dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:512
    scylladb#19 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091
    scylladb#20 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506
    scylladb#21 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666
    scylladb#22 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199
    scylladb#23 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115
    scylladb#24 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486
    scylladb#25 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2)

previously allocated by thread T0 here:
    #0 0x7fb8652089d7 in operator new(unsigned long) (/lib64/libasan.so.5+0x10f9d7)
    scylladb#1 0x77d1796 in allocate /usr/include/c++/9/ext/new_allocator.h:114
    scylladb#2 0x77bcab3 in allocate /usr/include/c++/9/bits/alloc_traits.h:444
    scylladb#3 0x779d936 in _M_allocate_node<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable_policy.h:2081
    scylladb#4 0x777a2ed in _M_emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable.h:1638
    scylladb#5 0x774d707 in emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable.h:749
    scylladb#6 0x771ce3e in emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/unordered_map.h:388
    scylladb#7 0x76f8280 in operator() /local/home/bhalevy/dev/scylla/distributed_loader.cc:732
    scylladb#8 0x774dc5d in _M_invoke /usr/include/c++/9/bits/std_function.h:286
    scylladb#9 0x6cde67b in std::function<seastar::future<> (std::filesystem::__cxx11::path, seastar::directory_entry)>::operator()(std::filesystem::__cxx11::path, seastar::directory_entry) const /usr/include/c++/9/bits/std_function.h:690
    scylladb#10 0x6cd345d in operator() /local/home/bhalevy/dev/scylla/lister.cc:30
    scylladb#11 0x6cd96e0 in apply /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36
    scylladb#12 0x6cd9854 in apply<lister::visit(seastar::directory_entry)::<lambda(seastar::directory_entry)>&, seastar::directory_entry&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:44
    scylladb#13 0x6cd9953 in apply<lister::visit(seastar::directory_entry)::<lambda(seastar::directory_entry)>&, seastar::directory_entry&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1650
    scylladb#14 0x6cd6ba9 in operator() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1168
    scylladb#15 0x6cdbac0 in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:101
    scylladb#16 0x6ce6178 in seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>::operator()(seastar::directory_entry&&) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:184
    scylladb#17 0x6ce3033 in seastar::apply_helper<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, std::tuple<seastar::directory_entry>&&, std::integer_sequence<unsigned long, 0ul> >::apply(seastar::noncopyable_function<seastar::future<> (seastar::directory_e
ntry&&)>&&, std::tuple<seastar::directory_entry>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36
    scylladb#18 0x6ce3130 in auto seastar::apply<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::directory_entry>(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&, std::tuple<seastar::directory_entry>&&) /local/home/bha
levy/dev/scylla/seastar/include/seastar/core/apply.hh:44
    scylladb#19 0x6ce322f in seastar::future<> seastar::futurize<seastar::future<> >::apply<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::directory_entry>(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&, std::tuple<s
eastar::directory_entry>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1650
    scylladb#20 0x6cdf88d in seastar::future<seastar::directory_entry>::then_impl<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&)::{lambda()scylladb#1}::operator()()
const::{lambda(seastar::future_state<seastar::directory_entry>&&)scylladb#1}::operator()(seastar::future_state<seastar::directory_entry>) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1196
    scylladb#21 0x6cea2e0 in seastar::continuation<seastar::future<seastar::directory_entry>::then_impl<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&)::{lamb
da()scylladb#1}::operator()() const::{lambda(seastar::future_state<seastar::directory_entry>&&)scylladb#1}, seastar::directory_entry>::run_and_dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:511
    scylladb#22 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091
    scylladb#23 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506
    scylladb#24 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666
    scylladb#25 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199
    scylladb#26 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115
    scylladb#27 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486
    scylladb#28 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2)

SUMMARY: AddressSanitizer: heap-use-after-free /usr/include/c++/9/bits/hashtable_policy.h:283 in _M_next
Shadow bytes around the buggy address:
  0x0c068025c100: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c068025c110: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa
  0x0c068025c120: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd
  0x0c068025c130: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c068025c140: fd fd fd fa fa fa 00 00 00 fa fa fa fd fd fd fa
=>0x0c068025c150: fa fa fd fd fd fa fa fa[fd]fd fd fd fa fa fd fd
  0x0c068025c160: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c068025c170: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa
  0x0c068025c180: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd
  0x0c068025c190: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c068025c1a0: fd fd fd fd fa fa fd fd fd fd fa fa 00 00 00 fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==406079==ABORTING

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
bhalevy added a commit to bhalevy/scylla that referenced this issue Apr 30, 2020
Rather than using lw_shared_ptr to keep it around, use do_with.
Fixes a use-after-free seen with interrupt_build_process_with_resharding_half_to_max_test
in debug mode:

==406079==ERROR: AddressSanitizer: heap-use-after-free on address 0x603001320ac0 at pc 0x00000777a062 bp 0x7ffdeb77b010 sp 0x7ffdeb77b000
READ of size 8 at 0x603001320ac0 thread T0
    #0 0x777a061 in _M_next /usr/include/c++/9/bits/hashtable_policy.h:283
    scylladb#1 0x779ead7 in _M_incr /usr/include/c++/9/bits/hashtable_policy.h:299
    scylladb#2 0x777bbac in operator++ /usr/include/c++/9/bits/hashtable_policy.h:362
    scylladb#3 0x77f8384 in run_and_dispose /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future-util.hh:568
    scylladb#4 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091
    scylladb#5 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506
    scylladb#6 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666
    scylladb#7 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199
    scylladb#8 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115
    scylladb#9 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486
    scylladb#10 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2)
    scylladb#11 0x42b69d in _start (/local/home/bhalevy/.dtest/dtest-q85idok7/test/node1/bin/scylla+0x42b69d)

0x603001320ac0 is located 0 bytes inside of 32-byte region [0x603001320ac0,0x603001320ae0)
freed by thread T0 here:
    #0 0x7fb86520996f in operator delete(void*) (/lib64/libasan.so.5+0x11096f)
    scylladb#1 0x77d1889 in deallocate /usr/include/c++/9/ext/new_allocator.h:128
    scylladb#2 0x77bcbb2 in deallocate /usr/include/c++/9/bits/alloc_traits.h:470
    scylladb#3 0x77bccc5 in _M_deallocate_node_ptr /usr/include/c++/9/bits/hashtable_policy.h:2113
    scylladb#4 0x779dc59 in _M_deallocate_node /usr/include/c++/9/bits/hashtable_policy.h:2103
    scylladb#5 0x77bc8b6 in _M_deallocate_nodes /usr/include/c++/9/bits/hashtable_policy.h:2124
    scylladb#6 0x779d18f in clear /usr/include/c++/9/bits/hashtable.h:2028
    scylladb#7 0x7779b8e in ~_Hashtable /usr/include/c++/9/bits/hashtable.h:1352
    scylladb#8 0x774cfed in ~unordered_map /usr/include/c++/9/bits/unordered_map.h:102
    scylladb#9 0x774d288 in ~shared_ptr_no_esft /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:157
    scylladb#10 0x774d2f1 in dispose /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:213
    scylladb#11 0x76f9642 in ~lw_shared_ptr /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:300
    scylladb#12 0x76f9642 in ~<lambda> /local/home/bhalevy/dev/scylla/distributed_loader.cc:743
    scylladb#13 0x771e0a7 in ~<lambda> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1167
    scylladb#14 0x777c64c in destroy /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:133
    scylladb#15 0x586059 in seastar::noncopyable_function<seastar::future<> ()>::~noncopyable_function() /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:165
    scylladb#16 0x5867da in seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}::~future_state() /local/home/bha
levy/dev/scylla/seastar/include/seastar/core/future.hh:1192
    scylladb#17 0xab704d in seastar::continuation<seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}>::~continu
ation() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:507
    scylladb#18 0xab7216 in seastar::continuation<seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}>::run_and_
dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:512
    scylladb#19 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091
    scylladb#20 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506
    scylladb#21 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666
    scylladb#22 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199
    scylladb#23 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115
    scylladb#24 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486
    scylladb#25 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2)

previously allocated by thread T0 here:
    #0 0x7fb8652089d7 in operator new(unsigned long) (/lib64/libasan.so.5+0x10f9d7)
    scylladb#1 0x77d1796 in allocate /usr/include/c++/9/ext/new_allocator.h:114
    scylladb#2 0x77bcab3 in allocate /usr/include/c++/9/bits/alloc_traits.h:444
    scylladb#3 0x779d936 in _M_allocate_node<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable_policy.h:2081
    scylladb#4 0x777a2ed in _M_emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable.h:1638
    scylladb#5 0x774d707 in emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable.h:749
    scylladb#6 0x771ce3e in emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/unordered_map.h:388
    scylladb#7 0x76f8280 in operator() /local/home/bhalevy/dev/scylla/distributed_loader.cc:732
    scylladb#8 0x774dc5d in _M_invoke /usr/include/c++/9/bits/std_function.h:286
    scylladb#9 0x6cde67b in std::function<seastar::future<> (std::filesystem::__cxx11::path, seastar::directory_entry)>::operator()(std::filesystem::__cxx11::path, seastar::directory_entry) const /usr/include/c++/9/bits/std_function.h:690
    scylladb#10 0x6cd345d in operator() /local/home/bhalevy/dev/scylla/lister.cc:30
    scylladb#11 0x6cd96e0 in apply /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36
    scylladb#12 0x6cd9854 in apply<lister::visit(seastar::directory_entry)::<lambda(seastar::directory_entry)>&, seastar::directory_entry&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:44
    scylladb#13 0x6cd9953 in apply<lister::visit(seastar::directory_entry)::<lambda(seastar::directory_entry)>&, seastar::directory_entry&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1650
    scylladb#14 0x6cd6ba9 in operator() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1168
    scylladb#15 0x6cdbac0 in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:101
    scylladb#16 0x6ce6178 in seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>::operator()(seastar::directory_entry&&) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:184
    scylladb#17 0x6ce3033 in seastar::apply_helper<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, std::tuple<seastar::directory_entry>&&, std::integer_sequence<unsigned long, 0ul> >::apply(seastar::noncopyable_function<seastar::future<> (seastar::directory_e
ntry&&)>&&, std::tuple<seastar::directory_entry>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36
    scylladb#18 0x6ce3130 in auto seastar::apply<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::directory_entry>(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&, std::tuple<seastar::directory_entry>&&) /local/home/bha
levy/dev/scylla/seastar/include/seastar/core/apply.hh:44
    scylladb#19 0x6ce322f in seastar::future<> seastar::futurize<seastar::future<> >::apply<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::directory_entry>(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&, std::tuple<s
eastar::directory_entry>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1650
    scylladb#20 0x6cdf88d in seastar::future<seastar::directory_entry>::then_impl<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&)::{lambda()scylladb#1}::operator()()
const::{lambda(seastar::future_state<seastar::directory_entry>&&)scylladb#1}::operator()(seastar::future_state<seastar::directory_entry>) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1196
    scylladb#21 0x6cea2e0 in seastar::continuation<seastar::future<seastar::directory_entry>::then_impl<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&)::{lamb
da()scylladb#1}::operator()() const::{lambda(seastar::future_state<seastar::directory_entry>&&)scylladb#1}, seastar::directory_entry>::run_and_dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:511
    scylladb#22 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091
    scylladb#23 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506
    scylladb#24 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666
    scylladb#25 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199
    scylladb#26 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115
    scylladb#27 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486
    scylladb#28 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2)

SUMMARY: AddressSanitizer: heap-use-after-free /usr/include/c++/9/bits/hashtable_policy.h:283 in _M_next
Shadow bytes around the buggy address:
  0x0c068025c100: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c068025c110: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa
  0x0c068025c120: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd
  0x0c068025c130: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c068025c140: fd fd fd fa fa fa 00 00 00 fa fa fa fd fd fd fa
=>0x0c068025c150: fa fa fd fd fd fa fa fa[fd]fd fd fd fa fa fd fd
  0x0c068025c160: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c068025c170: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa
  0x0c068025c180: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd
  0x0c068025c190: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c068025c1a0: fd fd fd fd fa fa fd fd fd fd fa fa 00 00 00 fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==406079==ABORTING

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
bhalevy added a commit to bhalevy/scylla that referenced this issue May 5, 2020
Rather than using lw_shared_ptr to keep it around, use do_with.
Fixes a use-after-free seen with interrupt_build_process_with_resharding_half_to_max_test
in debug mode:

==406079==ERROR: AddressSanitizer: heap-use-after-free on address 0x603001320ac0 at pc 0x00000777a062 bp 0x7ffdeb77b010 sp 0x7ffdeb77b000
READ of size 8 at 0x603001320ac0 thread T0
    #0 0x777a061 in _M_next /usr/include/c++/9/bits/hashtable_policy.h:283
    scylladb#1 0x779ead7 in _M_incr /usr/include/c++/9/bits/hashtable_policy.h:299
    scylladb#2 0x777bbac in operator++ /usr/include/c++/9/bits/hashtable_policy.h:362
    scylladb#3 0x77f8384 in run_and_dispose /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future-util.hh:568
    scylladb#4 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091
    scylladb#5 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506
    scylladb#6 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666
    scylladb#7 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199
    scylladb#8 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115
    scylladb#9 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486
    scylladb#10 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2)
    scylladb#11 0x42b69d in _start (/local/home/bhalevy/.dtest/dtest-q85idok7/test/node1/bin/scylla+0x42b69d)

0x603001320ac0 is located 0 bytes inside of 32-byte region [0x603001320ac0,0x603001320ae0)
freed by thread T0 here:
    #0 0x7fb86520996f in operator delete(void*) (/lib64/libasan.so.5+0x11096f)
    scylladb#1 0x77d1889 in deallocate /usr/include/c++/9/ext/new_allocator.h:128
    scylladb#2 0x77bcbb2 in deallocate /usr/include/c++/9/bits/alloc_traits.h:470
    scylladb#3 0x77bccc5 in _M_deallocate_node_ptr /usr/include/c++/9/bits/hashtable_policy.h:2113
    scylladb#4 0x779dc59 in _M_deallocate_node /usr/include/c++/9/bits/hashtable_policy.h:2103
    scylladb#5 0x77bc8b6 in _M_deallocate_nodes /usr/include/c++/9/bits/hashtable_policy.h:2124
    scylladb#6 0x779d18f in clear /usr/include/c++/9/bits/hashtable.h:2028
    scylladb#7 0x7779b8e in ~_Hashtable /usr/include/c++/9/bits/hashtable.h:1352
    scylladb#8 0x774cfed in ~unordered_map /usr/include/c++/9/bits/unordered_map.h:102
    scylladb#9 0x774d288 in ~shared_ptr_no_esft /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:157
    scylladb#10 0x774d2f1 in dispose /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:213
    scylladb#11 0x76f9642 in ~lw_shared_ptr /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:300
    scylladb#12 0x76f9642 in ~<lambda> /local/home/bhalevy/dev/scylla/distributed_loader.cc:743
    scylladb#13 0x771e0a7 in ~<lambda> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1167
    scylladb#14 0x777c64c in destroy /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:133
    scylladb#15 0x586059 in seastar::noncopyable_function<seastar::future<> ()>::~noncopyable_function() /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:165
    scylladb#16 0x5867da in seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}::~future_state() /local/home/bha
levy/dev/scylla/seastar/include/seastar/core/future.hh:1192
    scylladb#17 0xab704d in seastar::continuation<seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}>::~continu
ation() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:507
    scylladb#18 0xab7216 in seastar::continuation<seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}>::run_and_
dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:512
    scylladb#19 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091
    scylladb#20 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506
    scylladb#21 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666
    scylladb#22 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199
    scylladb#23 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115
    scylladb#24 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486
    scylladb#25 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2)

previously allocated by thread T0 here:
    #0 0x7fb8652089d7 in operator new(unsigned long) (/lib64/libasan.so.5+0x10f9d7)
    scylladb#1 0x77d1796 in allocate /usr/include/c++/9/ext/new_allocator.h:114
    scylladb#2 0x77bcab3 in allocate /usr/include/c++/9/bits/alloc_traits.h:444
    scylladb#3 0x779d936 in _M_allocate_node<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable_policy.h:2081
    scylladb#4 0x777a2ed in _M_emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable.h:1638
    scylladb#5 0x774d707 in emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable.h:749
    scylladb#6 0x771ce3e in emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/unordered_map.h:388
    scylladb#7 0x76f8280 in operator() /local/home/bhalevy/dev/scylla/distributed_loader.cc:732
    scylladb#8 0x774dc5d in _M_invoke /usr/include/c++/9/bits/std_function.h:286
    scylladb#9 0x6cde67b in std::function<seastar::future<> (std::filesystem::__cxx11::path, seastar::directory_entry)>::operator()(std::filesystem::__cxx11::path, seastar::directory_entry) const /usr/include/c++/9/bits/std_function.h:690
    scylladb#10 0x6cd345d in operator() /local/home/bhalevy/dev/scylla/lister.cc:30
    scylladb#11 0x6cd96e0 in apply /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36
    scylladb#12 0x6cd9854 in apply<lister::visit(seastar::directory_entry)::<lambda(seastar::directory_entry)>&, seastar::directory_entry&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:44
    scylladb#13 0x6cd9953 in apply<lister::visit(seastar::directory_entry)::<lambda(seastar::directory_entry)>&, seastar::directory_entry&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1650
    scylladb#14 0x6cd6ba9 in operator() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1168
    scylladb#15 0x6cdbac0 in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:101
    scylladb#16 0x6ce6178 in seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>::operator()(seastar::directory_entry&&) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:184
    scylladb#17 0x6ce3033 in seastar::apply_helper<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, std::tuple<seastar::directory_entry>&&, std::integer_sequence<unsigned long, 0ul> >::apply(seastar::noncopyable_function<seastar::future<> (seastar::directory_e
ntry&&)>&&, std::tuple<seastar::directory_entry>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36
    scylladb#18 0x6ce3130 in auto seastar::apply<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::directory_entry>(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&, std::tuple<seastar::directory_entry>&&) /local/home/bha
levy/dev/scylla/seastar/include/seastar/core/apply.hh:44
    scylladb#19 0x6ce322f in seastar::future<> seastar::futurize<seastar::future<> >::apply<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::directory_entry>(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&, std::tuple<s
eastar::directory_entry>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1650
    scylladb#20 0x6cdf88d in seastar::future<seastar::directory_entry>::then_impl<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&)::{lambda()scylladb#1}::operator()()
const::{lambda(seastar::future_state<seastar::directory_entry>&&)scylladb#1}::operator()(seastar::future_state<seastar::directory_entry>) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1196
    scylladb#21 0x6cea2e0 in seastar::continuation<seastar::future<seastar::directory_entry>::then_impl<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&)::{lamb
da()scylladb#1}::operator()() const::{lambda(seastar::future_state<seastar::directory_entry>&&)scylladb#1}, seastar::directory_entry>::run_and_dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:511
    scylladb#22 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091
    scylladb#23 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506
    scylladb#24 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666
    scylladb#25 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199
    scylladb#26 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115
    scylladb#27 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486
    scylladb#28 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2)

SUMMARY: AddressSanitizer: heap-use-after-free /usr/include/c++/9/bits/hashtable_policy.h:283 in _M_next
Shadow bytes around the buggy address:
  0x0c068025c100: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c068025c110: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa
  0x0c068025c120: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd
  0x0c068025c130: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c068025c140: fd fd fd fa fa fa 00 00 00 fa fa fa fd fd fd fa
=>0x0c068025c150: fa fa fd fd fd fa fa fa[fd]fd fd fd fa fa fd fd
  0x0c068025c160: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c068025c170: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa
  0x0c068025c180: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd
  0x0c068025c190: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c068025c1a0: fd fd fd fd fa fa fd fd fd fd fa fa 00 00 00 fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==406079==ABORTING

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
bhalevy added a commit to bhalevy/scylla that referenced this issue May 6, 2020
Rather than using lw_shared_ptr to keep it around, use do_with.

Fixes a use-after-free seen with interrupt_build_process_with_resharding_half_to_max_test
in debug mode:

`verifier` was not captured in the scan_dir.then() continuation.
We could add a capture in that lambda, but it's cleaner and safer to let do_with hold on to it instead.

==406079==ERROR: AddressSanitizer: heap-use-after-free on address 0x603001320ac0 at pc 0x00000777a062 bp 0x7ffdeb77b010 sp 0x7ffdeb77b000
READ of size 8 at 0x603001320ac0 thread T0
    #0 0x777a061 in _M_next /usr/include/c++/9/bits/hashtable_policy.h:283
    scylladb#1 0x779ead7 in _M_incr /usr/include/c++/9/bits/hashtable_policy.h:299
    scylladb#2 0x777bbac in operator++ /usr/include/c++/9/bits/hashtable_policy.h:362
    scylladb#3 0x77f8384 in run_and_dispose /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future-util.hh:568
    scylladb#4 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091
    scylladb#5 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506
    scylladb#6 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666
    scylladb#7 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199
    scylladb#8 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115
    scylladb#9 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486
    scylladb#10 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2)
    scylladb#11 0x42b69d in _start (/local/home/bhalevy/.dtest/dtest-q85idok7/test/node1/bin/scylla+0x42b69d)

0x603001320ac0 is located 0 bytes inside of 32-byte region [0x603001320ac0,0x603001320ae0)
freed by thread T0 here:
    #0 0x7fb86520996f in operator delete(void*) (/lib64/libasan.so.5+0x11096f)
    scylladb#1 0x77d1889 in deallocate /usr/include/c++/9/ext/new_allocator.h:128
    scylladb#2 0x77bcbb2 in deallocate /usr/include/c++/9/bits/alloc_traits.h:470
    scylladb#3 0x77bccc5 in _M_deallocate_node_ptr /usr/include/c++/9/bits/hashtable_policy.h:2113
    scylladb#4 0x779dc59 in _M_deallocate_node /usr/include/c++/9/bits/hashtable_policy.h:2103
    scylladb#5 0x77bc8b6 in _M_deallocate_nodes /usr/include/c++/9/bits/hashtable_policy.h:2124
    scylladb#6 0x779d18f in clear /usr/include/c++/9/bits/hashtable.h:2028
    scylladb#7 0x7779b8e in ~_Hashtable /usr/include/c++/9/bits/hashtable.h:1352
    scylladb#8 0x774cfed in ~unordered_map /usr/include/c++/9/bits/unordered_map.h:102
    scylladb#9 0x774d288 in ~shared_ptr_no_esft /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:157
    scylladb#10 0x774d2f1 in dispose /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:213
    scylladb#11 0x76f9642 in ~lw_shared_ptr /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:300
    scylladb#12 0x76f9642 in ~<lambda> /local/home/bhalevy/dev/scylla/distributed_loader.cc:743
    scylladb#13 0x771e0a7 in ~<lambda> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1167
    scylladb#14 0x777c64c in destroy /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:133
    scylladb#15 0x586059 in seastar::noncopyable_function<seastar::future<> ()>::~noncopyable_function() /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:165
    scylladb#16 0x5867da in seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}::~future_state() /local/home/bha
levy/dev/scylla/seastar/include/seastar/core/future.hh:1192
    scylladb#17 0xab704d in seastar::continuation<seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}>::~continu
ation() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:507
    scylladb#18 0xab7216 in seastar::continuation<seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}>::run_and_
dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:512
    scylladb#19 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091
    scylladb#20 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506
    scylladb#21 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666
    scylladb#22 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199
    scylladb#23 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115
    scylladb#24 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486
    scylladb#25 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2)

previously allocated by thread T0 here:
    #0 0x7fb8652089d7 in operator new(unsigned long) (/lib64/libasan.so.5+0x10f9d7)
    scylladb#1 0x77d1796 in allocate /usr/include/c++/9/ext/new_allocator.h:114
    scylladb#2 0x77bcab3 in allocate /usr/include/c++/9/bits/alloc_traits.h:444
    scylladb#3 0x779d936 in _M_allocate_node<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable_policy.h:2081
    scylladb#4 0x777a2ed in _M_emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable.h:1638
    scylladb#5 0x774d707 in emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable.h:749
    scylladb#6 0x771ce3e in emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/unordered_map.h:388
    scylladb#7 0x76f8280 in operator() /local/home/bhalevy/dev/scylla/distributed_loader.cc:732
    scylladb#8 0x774dc5d in _M_invoke /usr/include/c++/9/bits/std_function.h:286
    scylladb#9 0x6cde67b in std::function<seastar::future<> (std::filesystem::__cxx11::path, seastar::directory_entry)>::operator()(std::filesystem::__cxx11::path, seastar::directory_entry) const /usr/include/c++/9/bits/std_function.h:690
    scylladb#10 0x6cd345d in operator() /local/home/bhalevy/dev/scylla/lister.cc:30
    scylladb#11 0x6cd96e0 in apply /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36
    scylladb#12 0x6cd9854 in apply<lister::visit(seastar::directory_entry)::<lambda(seastar::directory_entry)>&, seastar::directory_entry&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:44
    scylladb#13 0x6cd9953 in apply<lister::visit(seastar::directory_entry)::<lambda(seastar::directory_entry)>&, seastar::directory_entry&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1650
    scylladb#14 0x6cd6ba9 in operator() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1168
    scylladb#15 0x6cdbac0 in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:101
    scylladb#16 0x6ce6178 in seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>::operator()(seastar::directory_entry&&) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:184
    scylladb#17 0x6ce3033 in seastar::apply_helper<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, std::tuple<seastar::directory_entry>&&, std::integer_sequence<unsigned long, 0ul> >::apply(seastar::noncopyable_function<seastar::future<> (seastar::directory_e
ntry&&)>&&, std::tuple<seastar::directory_entry>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36
    scylladb#18 0x6ce3130 in auto seastar::apply<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::directory_entry>(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&, std::tuple<seastar::directory_entry>&&) /local/home/bha
levy/dev/scylla/seastar/include/seastar/core/apply.hh:44
    scylladb#19 0x6ce322f in seastar::future<> seastar::futurize<seastar::future<> >::apply<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::directory_entry>(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&, std::tuple<s
eastar::directory_entry>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1650
    scylladb#20 0x6cdf88d in seastar::future<seastar::directory_entry>::then_impl<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&)::{lambda()scylladb#1}::operator()()
const::{lambda(seastar::future_state<seastar::directory_entry>&&)scylladb#1}::operator()(seastar::future_state<seastar::directory_entry>) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1196
    scylladb#21 0x6cea2e0 in seastar::continuation<seastar::future<seastar::directory_entry>::then_impl<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&)::{lamb
da()scylladb#1}::operator()() const::{lambda(seastar::future_state<seastar::directory_entry>&&)scylladb#1}, seastar::directory_entry>::run_and_dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:511
    scylladb#22 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091
    scylladb#23 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506
    scylladb#24 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666
    scylladb#25 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199
    scylladb#26 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115
    scylladb#27 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486
    scylladb#28 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2)

SUMMARY: AddressSanitizer: heap-use-after-free /usr/include/c++/9/bits/hashtable_policy.h:283 in _M_next
Shadow bytes around the buggy address:
  0x0c068025c100: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c068025c110: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa
  0x0c068025c120: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd
  0x0c068025c130: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c068025c140: fd fd fd fa fa fa 00 00 00 fa fa fa fd fd fd fa
=>0x0c068025c150: fa fa fd fd fd fa fa fa[fd]fd fd fd fa fa fd fd
  0x0c068025c160: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c068025c170: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa
  0x0c068025c180: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd
  0x0c068025c190: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c068025c1a0: fd fd fd fd fa fa fd fd fd fd fa fa 00 00 00 fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==406079==ABORTING

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
bhalevy added a commit to bhalevy/scylla that referenced this issue May 11, 2020
Rather than using lw_shared_ptr to keep it around, use do_with.

Fixes a use-after-free seen with interrupt_build_process_with_resharding_half_to_max_test
in debug mode:

`verifier` was not captured in the scan_dir.then() continuation.
We could add a capture in that lambda, but it's cleaner and safer to let do_with hold on to it instead.

==406079==ERROR: AddressSanitizer: heap-use-after-free on address 0x603001320ac0 at pc 0x00000777a062 bp 0x7ffdeb77b010 sp 0x7ffdeb77b000
READ of size 8 at 0x603001320ac0 thread T0
    #0 0x777a061 in _M_next /usr/include/c++/9/bits/hashtable_policy.h:283
    scylladb#1 0x779ead7 in _M_incr /usr/include/c++/9/bits/hashtable_policy.h:299
    scylladb#2 0x777bbac in operator++ /usr/include/c++/9/bits/hashtable_policy.h:362
    scylladb#3 0x77f8384 in run_and_dispose /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future-util.hh:568
    scylladb#4 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091
    scylladb#5 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506
    scylladb#6 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666
    scylladb#7 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199
    scylladb#8 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115
    scylladb#9 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486
    scylladb#10 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2)
    scylladb#11 0x42b69d in _start (/local/home/bhalevy/.dtest/dtest-q85idok7/test/node1/bin/scylla+0x42b69d)

0x603001320ac0 is located 0 bytes inside of 32-byte region [0x603001320ac0,0x603001320ae0)
freed by thread T0 here:
    #0 0x7fb86520996f in operator delete(void*) (/lib64/libasan.so.5+0x11096f)
    scylladb#1 0x77d1889 in deallocate /usr/include/c++/9/ext/new_allocator.h:128
    scylladb#2 0x77bcbb2 in deallocate /usr/include/c++/9/bits/alloc_traits.h:470
    scylladb#3 0x77bccc5 in _M_deallocate_node_ptr /usr/include/c++/9/bits/hashtable_policy.h:2113
    scylladb#4 0x779dc59 in _M_deallocate_node /usr/include/c++/9/bits/hashtable_policy.h:2103
    scylladb#5 0x77bc8b6 in _M_deallocate_nodes /usr/include/c++/9/bits/hashtable_policy.h:2124
    scylladb#6 0x779d18f in clear /usr/include/c++/9/bits/hashtable.h:2028
    scylladb#7 0x7779b8e in ~_Hashtable /usr/include/c++/9/bits/hashtable.h:1352
    scylladb#8 0x774cfed in ~unordered_map /usr/include/c++/9/bits/unordered_map.h:102
    scylladb#9 0x774d288 in ~shared_ptr_no_esft /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:157
    scylladb#10 0x774d2f1 in dispose /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:213
    scylladb#11 0x76f9642 in ~lw_shared_ptr /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:300
    scylladb#12 0x76f9642 in ~<lambda> /local/home/bhalevy/dev/scylla/distributed_loader.cc:743
    scylladb#13 0x771e0a7 in ~<lambda> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1167
    scylladb#14 0x777c64c in destroy /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:133
    scylladb#15 0x586059 in seastar::noncopyable_function<seastar::future<> ()>::~noncopyable_function() /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:165
    scylladb#16 0x5867da in seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}::~future_state() /local/home/bha
levy/dev/scylla/seastar/include/seastar/core/future.hh:1192
    scylladb#17 0xab704d in seastar::continuation<seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}>::~continu
ation() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:507
    scylladb#18 0xab7216 in seastar::continuation<seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}>::run_and_
dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:512
    scylladb#19 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091
    scylladb#20 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506
    scylladb#21 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666
    scylladb#22 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199
    scylladb#23 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115
    scylladb#24 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486
    scylladb#25 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2)

previously allocated by thread T0 here:
    #0 0x7fb8652089d7 in operator new(unsigned long) (/lib64/libasan.so.5+0x10f9d7)
    scylladb#1 0x77d1796 in allocate /usr/include/c++/9/ext/new_allocator.h:114
    scylladb#2 0x77bcab3 in allocate /usr/include/c++/9/bits/alloc_traits.h:444
    scylladb#3 0x779d936 in _M_allocate_node<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable_policy.h:2081
    scylladb#4 0x777a2ed in _M_emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable.h:1638
    scylladb#5 0x774d707 in emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable.h:749
    scylladb#6 0x771ce3e in emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/unordered_map.h:388
    scylladb#7 0x76f8280 in operator() /local/home/bhalevy/dev/scylla/distributed_loader.cc:732
    scylladb#8 0x774dc5d in _M_invoke /usr/include/c++/9/bits/std_function.h:286
    scylladb#9 0x6cde67b in std::function<seastar::future<> (std::filesystem::__cxx11::path, seastar::directory_entry)>::operator()(std::filesystem::__cxx11::path, seastar::directory_entry) const /usr/include/c++/9/bits/std_function.h:690
    scylladb#10 0x6cd345d in operator() /local/home/bhalevy/dev/scylla/lister.cc:30
    scylladb#11 0x6cd96e0 in apply /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36
    scylladb#12 0x6cd9854 in apply<lister::visit(seastar::directory_entry)::<lambda(seastar::directory_entry)>&, seastar::directory_entry&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:44
    scylladb#13 0x6cd9953 in apply<lister::visit(seastar::directory_entry)::<lambda(seastar::directory_entry)>&, seastar::directory_entry&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1650
    scylladb#14 0x6cd6ba9 in operator() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1168
    scylladb#15 0x6cdbac0 in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:101
    scylladb#16 0x6ce6178 in seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>::operator()(seastar::directory_entry&&) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:184
    scylladb#17 0x6ce3033 in seastar::apply_helper<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, std::tuple<seastar::directory_entry>&&, std::integer_sequence<unsigned long, 0ul> >::apply(seastar::noncopyable_function<seastar::future<> (seastar::directory_e
ntry&&)>&&, std::tuple<seastar::directory_entry>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36
    scylladb#18 0x6ce3130 in auto seastar::apply<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::directory_entry>(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&, std::tuple<seastar::directory_entry>&&) /local/home/bha
levy/dev/scylla/seastar/include/seastar/core/apply.hh:44
    scylladb#19 0x6ce322f in seastar::future<> seastar::futurize<seastar::future<> >::apply<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::directory_entry>(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&, std::tuple<s
eastar::directory_entry>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1650
    scylladb#20 0x6cdf88d in seastar::future<seastar::directory_entry>::then_impl<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&)::{lambda()scylladb#1}::operator()()
const::{lambda(seastar::future_state<seastar::directory_entry>&&)scylladb#1}::operator()(seastar::future_state<seastar::directory_entry>) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1196
    scylladb#21 0x6cea2e0 in seastar::continuation<seastar::future<seastar::directory_entry>::then_impl<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&)::{lamb
da()scylladb#1}::operator()() const::{lambda(seastar::future_state<seastar::directory_entry>&&)scylladb#1}, seastar::directory_entry>::run_and_dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:511
    scylladb#22 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091
    scylladb#23 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506
    scylladb#24 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666
    scylladb#25 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199
    scylladb#26 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115
    scylladb#27 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486
    scylladb#28 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2)

SUMMARY: AddressSanitizer: heap-use-after-free /usr/include/c++/9/bits/hashtable_policy.h:283 in _M_next
Shadow bytes around the buggy address:
  0x0c068025c100: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c068025c110: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa
  0x0c068025c120: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd
  0x0c068025c130: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c068025c140: fd fd fd fa fa fa 00 00 00 fa fa fa fd fd fd fa
=>0x0c068025c150: fa fa fd fd fd fa fa fa[fd]fd fd fd fa fa fd fd
  0x0c068025c160: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c068025c170: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa
  0x0c068025c180: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd
  0x0c068025c190: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c068025c1a0: fd fd fd fd fa fa fd fd fd fd fa fa 00 00 00 fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==406079==ABORTING

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
bhalevy added a commit to bhalevy/scylla that referenced this issue May 13, 2020
Rather than using lw_shared_ptr to keep it around, use do_with.

Fixes a use-after-free seen with interrupt_build_process_with_resharding_half_to_max_test
in debug mode:

`verifier` was not captured in the scan_dir.then() continuation.
We could add a capture in that lambda, but it's cleaner and safer to let do_with hold on to it instead.

==406079==ERROR: AddressSanitizer: heap-use-after-free on address 0x603001320ac0 at pc 0x00000777a062 bp 0x7ffdeb77b010 sp 0x7ffdeb77b000
READ of size 8 at 0x603001320ac0 thread T0
    #0 0x777a061 in _M_next /usr/include/c++/9/bits/hashtable_policy.h:283
    scylladb#1 0x779ead7 in _M_incr /usr/include/c++/9/bits/hashtable_policy.h:299
    scylladb#2 0x777bbac in operator++ /usr/include/c++/9/bits/hashtable_policy.h:362
    scylladb#3 0x77f8384 in run_and_dispose /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future-util.hh:568
    scylladb#4 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091
    scylladb#5 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506
    scylladb#6 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666
    scylladb#7 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199
    scylladb#8 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115
    scylladb#9 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486
    scylladb#10 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2)
    scylladb#11 0x42b69d in _start (/local/home/bhalevy/.dtest/dtest-q85idok7/test/node1/bin/scylla+0x42b69d)

0x603001320ac0 is located 0 bytes inside of 32-byte region [0x603001320ac0,0x603001320ae0)
freed by thread T0 here:
    #0 0x7fb86520996f in operator delete(void*) (/lib64/libasan.so.5+0x11096f)
    scylladb#1 0x77d1889 in deallocate /usr/include/c++/9/ext/new_allocator.h:128
    scylladb#2 0x77bcbb2 in deallocate /usr/include/c++/9/bits/alloc_traits.h:470
    scylladb#3 0x77bccc5 in _M_deallocate_node_ptr /usr/include/c++/9/bits/hashtable_policy.h:2113
    scylladb#4 0x779dc59 in _M_deallocate_node /usr/include/c++/9/bits/hashtable_policy.h:2103
    scylladb#5 0x77bc8b6 in _M_deallocate_nodes /usr/include/c++/9/bits/hashtable_policy.h:2124
    scylladb#6 0x779d18f in clear /usr/include/c++/9/bits/hashtable.h:2028
    scylladb#7 0x7779b8e in ~_Hashtable /usr/include/c++/9/bits/hashtable.h:1352
    scylladb#8 0x774cfed in ~unordered_map /usr/include/c++/9/bits/unordered_map.h:102
    scylladb#9 0x774d288 in ~shared_ptr_no_esft /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:157
    scylladb#10 0x774d2f1 in dispose /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:213
    scylladb#11 0x76f9642 in ~lw_shared_ptr /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:300
    scylladb#12 0x76f9642 in ~<lambda> /local/home/bhalevy/dev/scylla/distributed_loader.cc:743
    scylladb#13 0x771e0a7 in ~<lambda> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1167
    scylladb#14 0x777c64c in destroy /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:133
    scylladb#15 0x586059 in seastar::noncopyable_function<seastar::future<> ()>::~noncopyable_function() /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:165
    scylladb#16 0x5867da in seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}::~future_state() /local/home/bha
levy/dev/scylla/seastar/include/seastar/core/future.hh:1192
    scylladb#17 0xab704d in seastar::continuation<seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}>::~continu
ation() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:507
    scylladb#18 0xab7216 in seastar::continuation<seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}>::run_and_
dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:512
    scylladb#19 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091
    scylladb#20 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506
    scylladb#21 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666
    scylladb#22 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199
    scylladb#23 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115
    scylladb#24 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486
    scylladb#25 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2)

previously allocated by thread T0 here:
    #0 0x7fb8652089d7 in operator new(unsigned long) (/lib64/libasan.so.5+0x10f9d7)
    scylladb#1 0x77d1796 in allocate /usr/include/c++/9/ext/new_allocator.h:114
    scylladb#2 0x77bcab3 in allocate /usr/include/c++/9/bits/alloc_traits.h:444
    scylladb#3 0x779d936 in _M_allocate_node<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable_policy.h:2081
    scylladb#4 0x777a2ed in _M_emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable.h:1638
    scylladb#5 0x774d707 in emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable.h:749
    scylladb#6 0x771ce3e in emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/unordered_map.h:388
    scylladb#7 0x76f8280 in operator() /local/home/bhalevy/dev/scylla/distributed_loader.cc:732
    scylladb#8 0x774dc5d in _M_invoke /usr/include/c++/9/bits/std_function.h:286
    scylladb#9 0x6cde67b in std::function<seastar::future<> (std::filesystem::__cxx11::path, seastar::directory_entry)>::operator()(std::filesystem::__cxx11::path, seastar::directory_entry) const /usr/include/c++/9/bits/std_function.h:690
    scylladb#10 0x6cd345d in operator() /local/home/bhalevy/dev/scylla/lister.cc:30
    scylladb#11 0x6cd96e0 in apply /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36
    scylladb#12 0x6cd9854 in apply<lister::visit(seastar::directory_entry)::<lambda(seastar::directory_entry)>&, seastar::directory_entry&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:44
    scylladb#13 0x6cd9953 in apply<lister::visit(seastar::directory_entry)::<lambda(seastar::directory_entry)>&, seastar::directory_entry&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1650
    scylladb#14 0x6cd6ba9 in operator() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1168
    scylladb#15 0x6cdbac0 in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:101
    scylladb#16 0x6ce6178 in seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>::operator()(seastar::directory_entry&&) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:184
    scylladb#17 0x6ce3033 in seastar::apply_helper<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, std::tuple<seastar::directory_entry>&&, std::integer_sequence<unsigned long, 0ul> >::apply(seastar::noncopyable_function<seastar::future<> (seastar::directory_e
ntry&&)>&&, std::tuple<seastar::directory_entry>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36
    scylladb#18 0x6ce3130 in auto seastar::apply<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::directory_entry>(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&, std::tuple<seastar::directory_entry>&&) /local/home/bha
levy/dev/scylla/seastar/include/seastar/core/apply.hh:44
    scylladb#19 0x6ce322f in seastar::future<> seastar::futurize<seastar::future<> >::apply<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::directory_entry>(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&, std::tuple<s
eastar::directory_entry>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1650
    scylladb#20 0x6cdf88d in seastar::future<seastar::directory_entry>::then_impl<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&)::{lambda()scylladb#1}::operator()()
const::{lambda(seastar::future_state<seastar::directory_entry>&&)scylladb#1}::operator()(seastar::future_state<seastar::directory_entry>) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1196
    scylladb#21 0x6cea2e0 in seastar::continuation<seastar::future<seastar::directory_entry>::then_impl<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&)::{lamb
da()scylladb#1}::operator()() const::{lambda(seastar::future_state<seastar::directory_entry>&&)scylladb#1}, seastar::directory_entry>::run_and_dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:511
    scylladb#22 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091
    scylladb#23 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506
    scylladb#24 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666
    scylladb#25 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199
    scylladb#26 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115
    scylladb#27 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486
    scylladb#28 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2)

SUMMARY: AddressSanitizer: heap-use-after-free /usr/include/c++/9/bits/hashtable_policy.h:283 in _M_next
Shadow bytes around the buggy address:
  0x0c068025c100: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c068025c110: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa
  0x0c068025c120: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd
  0x0c068025c130: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c068025c140: fd fd fd fa fa fa 00 00 00 fa fa fa fd fd fd fa
=>0x0c068025c150: fa fa fd fd fd fa fa fa[fd]fd fd fd fa fa fd fd
  0x0c068025c160: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c068025c170: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa
  0x0c068025c180: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd
  0x0c068025c190: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c068025c1a0: fd fd fd fd fa fa fd fd fd fd fa fa 00 00 00 fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==406079==ABORTING

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
avikivity pushed a commit that referenced this issue May 18, 2020
In a recent next failure I got the following backtrace

#3  0x00007efd71251a66 in __GI___assert_fail (assertion=assertion@entry=0x2d0c00 "this->_con->get()->sink_closed()", file=file@entry=0x32c9d0 "./seastar/include/seastar/rpc/rpc_impl.hh", line=line@entry=795,
    function=function@entry=0x270360 "seastar::rpc::sink_impl<Serializer, Out>::~sink_impl() [with Serializer = netw::serializer; Out = {repair_row_on_wire_with_cmd}]") at assert.c:101
#4  0x0000000001f5d2c3 in seastar::rpc::sink_impl<netw::serializer, repair_row_on_wire_with_cmd>::~sink_impl (this=<optimized out>, __in_chrg=<optimized out>) at ./seastar/include/seastar/core/future.hh:312
#5  0x0000000001f5d2f4 in seastar::shared_ptr_count_for<seastar::rpc::sink_impl<netw::serializer, repair_row_on_wire_with_cmd> >::~shared_ptr_count_for (this=0x60100075b680, __in_chrg=<optimized out>)
    at ./seastar/include/seastar/core/shared_ptr.hh:463
#6  seastar::shared_ptr_count_for<seastar::rpc::sink_impl<netw::serializer, repair_row_on_wire_with_cmd> >::~shared_ptr_count_for (this=0x60100075b680, __in_chrg=<optimized out>) at ./seastar/include/seastar/core/shared_ptr.hh:463
#7  0x000000000240f2e6 in seastar::shared_ptr<seastar::rpc::sink<repair_row_on_wire_with_cmd>::impl>::~shared_ptr (this=0x601003118590, __in_chrg=<optimized out>) at ./seastar/include/seastar/core/future.hh:427
#8  seastar::rpc::sink<repair_row_on_wire_with_cmd>::~sink (this=0x601003118590, __in_chrg=<optimized out>) at ./seastar/include/seastar/rpc/rpc_types.hh:270
#9  <lambda(auto:134&)>::<lambda(const seastar::rpc::client_info&, uint64_t, seastar::rpc::source<repair_hash_with_cmd>)>::<lambda(std::__exception_ptr::exception_ptr)>::~<lambda> (this=0x601003118570, __in_chrg=<optimized out>)
    at repair/row_level.cc:2059

This patch changes a few functions to use finally to make sure the sink
is always closed.

Signed-off-by: Rafael Ávila de Espíndola <espindola@scylladb.com>
Message-Id: <20200515202803.60020-1-espindola@scylladb.com>
bhalevy added a commit to bhalevy/scylla that referenced this issue Jul 20, 2020
…y_clustering_slicing

test/lib/mutation_source_test.cc:1086:63: runtime error: reference binding to null pointer of type 'const struct column_definition'
mutation_partition.cc:1212:22: runtime error: member access within null pointer of type 'const struct column_definition'
AddressSanitizer:DEADLYSIGNAL
=================================================================
==250886==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000050 (pc 0x00000ccc8820 bp 0x7f2dcb0a6a20 sp 0x7f2dcb0a6890 T1)
==250886==The signal is caused by a READ memory access.
==250886==Hint: address points to the zero page.

    #0 0xccc8820 in row::apply_monotonically(column_definition const&, atomic_cell_or_collection&&, seastar::optimized_optional<cell_hash>) /local/home/bhalevy/dev/scylla/mutation_partition.cc:1212
    scylladb#1 0xccc8661 in row::apply(column_definition const&, atomic_cell_or_collection&&, seastar::optimized_optional<cell_hash>) /local/home/bhalevy/dev/scylla/mutation_partition.cc:1180
    scylladb#2 0xc872a18 in mutation::set_clustered_cell(clustering_key_prefix const&, column_definition const&, atomic_cell_or_collection&&) /local/home/bhalevy/dev/scylla/mutation.cc:77
    scylladb#3 0x132288ab in test_compaction_strategy_composite_key_clustering_slicing test/lib/mutation_source_test.cc:1094
    scylladb#4 0x1322b9ef in test_all_compaction_strategies_clustering_slicing test/lib/mutation_source_test.cc:1134
    scylladb#5 0x1324e25a in run_mutation_reader_tests(std::function<mutation_source (seastar::lw_shared_ptr<schema const>, std::vector<mutation, std::allocator<mutation> > const&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >)>) test/lib/mutation_source_test.cc:1495
    scylladb#6 0x132513bb in run_mutation_source_tests(std::function<mutation_source (seastar::lw_shared_ptr<schema const>, std::vector<mutation, std::allocator<mutation> > const&, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >)>) test/lib/mutation_source_test.cc:1556
    scylladb#7 0xbcbf8c1 in test_mutation_source test/boost/sstable_conforms_to_mutation_source_test.cc:45
    scylladb#8 0xbcc019d in operator() test/boost/sstable_conforms_to_mutation_source_test.cc:62
    scylladb#9 0xbcc6cfd in __invoke_impl<void, test_sstable_conforms_to_mutation_source::run_test_case()::<lambda()> > /usr/include/c++/10/bits/invoke.h:60
    scylladb#10 0xbcc5ccf in __invoke<test_sstable_conforms_to_mutation_source::run_test_case()::<lambda()> > /usr/include/c++/10/bits/invoke.h:95
    scylladb#11 0xbcc3f91 in __apply_impl<test_sstable_conforms_to_mutation_source::run_test_case()::<lambda()>, std::tuple<> > /usr/include/c++/10/tuple:1723
    scylladb#12 0xbcc3ff7 in apply<test_sstable_conforms_to_mutation_source::run_test_case()::<lambda()>, std::tuple<> > /usr/include/c++/10/tuple:1734
    scylladb#13 0xbcc407a in apply<test_sstable_conforms_to_mutation_source::run_test_case()::<lambda()> > /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1978
    scylladb#14 0xbcc1840 in operator() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/thread.hh:259
    scylladb#15 0xbcc7e10 in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:101
    scylladb#16 0xbfadb8f in seastar::noncopyable_function<void ()>::operator()() const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:184
    scylladb#17 0x142ef646 in seastar::thread_context::main() /local/home/bhalevy/dev/scylla/seastar/src/core/thread.cc:297

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV /local/home/bhalevy/dev/scylla/mutation_partition.cc:1212 in row::apply_monotonically(column_definition const&, atomic_cell_or_collection&&, seastar::optimized_optional<cell_hash>)
Thread T1 created by T0 here:
    #0 0x7f2ddba5abe5 in __interceptor_pthread_create (/lib64/libasan.so.6+0x57be5)
    scylladb#1 0x1425bc6b in seastar::posix_thread::posix_thread(seastar::posix_thread::attr, std::function<void ()>) /local/home/bhalevy/dev/scylla/seastar/src/core/posix.cc:95
    scylladb#2 0x1425b84f in seastar::posix_thread::posix_thread(std::function<void ()>) /local/home/bhalevy/dev/scylla/seastar/src/core/posix.cc:65
    scylladb#3 0x14a67480 in make_unique<seastar::posix_thread, seastar::testing::test_runner::start(int, char**)::<lambda()> > /usr/include/c++/10/bits/unique_ptr.h:961
    scylladb#4 0x14a63e8a in seastar::testing::test_runner::start(int, char**) /local/home/bhalevy/dev/scylla/seastar/src/testing/test_runner.cc:108
    scylladb#5 0x14a58b52 in init_unit_test_suite /local/home/bhalevy/dev/scylla/seastar/src/testing/entry_point.cc:47
    scylladb#6 0x7f2dda3d2cec  (/lib64/libboost_unit_test_framework.so.1.69.0+0x2ccec)

==250886==ABORTING

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
bhalevy added a commit to bhalevy/scylla that referenced this issue Oct 4, 2020
… token_metadata_ptr

Fixes use-after-free seen with putget_with_reloaded_certificates_test:
```
==215==ERROR: AddressSanitizer: heap-use-after-free on address 0x603000a8b180 at pc 0x000012eb5a83 bp 0x7ffd2c16d4c0 sp 0x7ffd2c16d4b0
READ of size 8 at 0x603000a8b180 thread T0
    #0 0x12eb5a82 in std::__uniq_ptr_impl<locator::token_metadata_impl, std::default_delete<locator::token_metadata_impl> >::_M_ptr() const /usr/include/c++/10/bits/unique_ptr.h:173
    scylladb#1 0x12ea230d in std::unique_ptr<locator::token_metadata_impl, std::default_delete<locator::token_metadata_impl> >::get() const /usr/include/c++/10/bits/unique_ptr.h:422
    scylladb#2 0x12e8d3e8 in std::unique_ptr<locator::token_metadata_impl, std::default_delete<locator::token_metadata_impl> >::operator->() const /usr/include/c++/10/bits/unique_ptr.h:416
    scylladb#3 0x12e5d0a2 in locator::token_metadata::ring_range(std::optional<interval_bound<dht::ring_position> > const&, bool) const locator/token_metadata.cc:1712
    scylladb#4 0x112d0126 in service::query_ranges_to_vnodes_generator::process_one_range(unsigned long, std::vector<nonwrapping_interval<dht::ring_position>, std::allocator<nonwrapping_interval<dht::ring_position> > >&) service/storage_proxy.cc:4658
    scylladb#5 0x112cf3c5 in service::query_ranges_to_vnodes_generator::operator()(unsigned long) service/storage_proxy.cc:4616
    scylladb#6 0x112b2261 in service::storage_proxy::query_partition_key_range_concurrent(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >, std::vector<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >, std::allocator<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&, seastar::lw_shared_ptr<query::read_command>, db::consistency_level, service::query_ranges_to_vnodes_generator&&, int, tracing::trace_state_ptr, unsigned long, unsigned int, std::unordered_map<nonwrapping_interval<dht::token>, std::vector<utils::UUID, std::allocator<utils::UUID> >, std::hash<nonwrapping_interval<dht::token> >, std::equal_to<nonwrapping_interval<dht::token> >, std::allocator<std::pair<nonwrapping_interval<dht::token> const, std::vector<utils::UUID, std::allocator<utils::UUID> > > > >, service_permit) service/storage_proxy.cc:4023
    scylladb#7 0x112b094e in operator() service/storage_proxy.cc:4160
    scylladb#8 0x1139c8bb in invoke<service::storage_proxy::query_partition_key_range_concurrent(seastar::lowres_clock::time_point, std::vector<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >&&, seastar::lw_shared_ptr<query::read_command>, db::consistency_level, service::query_ranges_to_vnodes_generator&&, int, tracing::trace_state_ptr, uint64_t, uint32_t, service::replicas_per_token_range, service_permit)::<lambda(seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:2088
    scylladb#9 0x1136625b in futurize_invoke<service::storage_proxy::query_partition_key_range_concurrent(seastar::lowres_clock::time_point, std::vector<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >&&, seastar::lw_shared_ptr<query::read_command>, db::consistency_level, service::query_ranges_to_vnodes_generator&&, int, tracing::trace_state_ptr, uint64_t, uint32_t, service::replicas_per_token_range, service_permit)::<lambda(seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:2119
    scylladb#10 0x11366372 in operator()<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1480
    scylladb#11 0x1139cc3b in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:145
    scylladb#12 0x116f4944 in seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>::operator()(seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:201
    scylladb#13 0x116b3397 in seastar::future<service::query_partition_key_range_concurrent_result> std::__invoke_impl<seastar::future<service::query_partition_key_range_concurrent_result>, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >(std::__invoke_other, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&) /usr/include/c++/10/bits/invoke.h:60
    scylladb#14 0x1165c3a6 in std::__invoke_result<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::type std::__invoke<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&) /usr/include/c++/10/bits/invoke.h:96
    scylladb#15 0x115e6542 in decltype(auto) std::__apply_impl<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >, 0ul>(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >&&, std::integer_sequence<unsigned long, 0ul>) /usr/include/c++/10/tuple:1724
    scylladb#16 0x115e6663 in decltype(auto) std::apply<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >&&) /usr/include/c++/10/tuple:1736
    scylladb#17 0x115e63f9 in seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}::operator()(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&) const::{lambda()scylladb#1}::operator()() const /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1530
    scylladb#18 0x1165c4b9 in void seastar::futurize<seastar::future<service::query_partition_key_range_concurrent_result> >::satisfy_with_result_of<seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}::operator()(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&) const::{lambda()scylladb#1}>(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:2073
    scylladb#19 0x115e61f5 in seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}::operator()(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1528
    scylladb#20 0x1176e9cc in seastar::continuation<seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::run_and_dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:746
    scylladb#21 0x16a9a455 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2196
    scylladb#22 0x16a9e691 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2575
    scylladb#23 0x16aa390e in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2730
    scylladb#24 0x168ae4f7 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:207
    scylladb#25 0x168ac541 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115
    scylladb#26 0xd6cd3c4 in main /local/home/bhalevy/dev/scylla/main.cc:504
    scylladb#27 0x7f8d905d8041 in __libc_start_main (/local/home/bhalevy/dev/scylla/build/debug/dynamic_libs/libc.so.6+0x27041)
    scylladb#28 0xd67c9ed in _start (/local/home/bhalevy/.dtest/dtest-o0qoqmkr/test/node3/bin/scylla+0xd67c9ed)

0x603000a8b180 is located 16 bytes inside of 24-byte region [0x603000a8b170,0x603000a8b188)
freed by thread T0 here:
    #0 0x7f8d92a190cf in operator delete(void*, unsigned long) (/local/home/bhalevy/dev/scylla/build/debug/dynamic_libs/libasan.so.6+0xb30cf)
    scylladb#1 0xd7ebe54 in seastar::internal::lw_shared_ptr_accessors_no_esft<locator::token_metadata>::dispose(seastar::lw_shared_ptr_counter_base*) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:213
    scylladb#2 0x112b155d in seastar::lw_shared_ptr<locator::token_metadata const>::~lw_shared_ptr() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:300
    scylladb#3 0x112b155d in ~<lambda> service/storage_proxy.cc:4137
    scylladb#4 0x1132e92d in ~<lambda> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1479
    scylladb#5 0x1139cc91 in destroy /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:148
    scylladb#6 0x11565673 in seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>::~noncopyable_function() /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:181
    scylladb#7 0x1176e783 in seastar::continuation<seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::~continuation() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:729
    scylladb#8 0x1176ea06 in seastar::continuation<seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::run_and_dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:750
    scylladb#9 0x16a9a455 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2196
    scylladb#10 0x16a9e691 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2575
    scylladb#11 0x16aa390e in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2730
    scylladb#12 0x168ae4f7 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:207
    scylladb#13 0x168ac541 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115
    scylladb#14 0xd6cd3c4 in main /local/home/bhalevy/dev/scylla/main.cc:504
    scylladb#15 0x7f8d905d8041 in __libc_start_main (/local/home/bhalevy/dev/scylla/build/debug/dynamic_libs/libc.so.6+0x27041)

previously allocated by thread T0 here:
    #0 0x7f8d92a18067 in operator new(unsigned long) (/local/home/bhalevy/dev/scylla/build/debug/dynamic_libs/libasan.so.6+0xb2067)
    scylladb#1 0x13cf7132 in seastar::lw_shared_ptr<locator::token_metadata> seastar::lw_shared_ptr<locator::token_metadata>::make<locator::token_metadata>(locator::token_metadata&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:266
    scylladb#2 0x13cc3bfa in seastar::lw_shared_ptr<locator::token_metadata> seastar::make_lw_shared<locator::token_metadata>(locator::token_metadata&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:422
    scylladb#3 0x13ca3007 in seastar::lw_shared_ptr<locator::token_metadata> locator::make_token_metadata_ptr<locator::token_metadata>(locator::token_metadata) locator/token_metadata.hh:338
    scylladb#4 0x13c9bdd4 in locator::shared_token_metadata::clone() const locator/token_metadata.hh:358
    scylladb#5 0x13c9c18a in service::storage_service::get_mutable_token_metadata_ptr() service/storage_service.hh:184
    scylladb#6 0x13a5a445 in service::storage_service::handle_state_normal(gms::inet_address) service/storage_service.cc:1129
    scylladb#7 0x13a6371c in service::storage_service::on_change(gms::inet_address, gms::application_state, gms::versioned_value const&) service/storage_service.cc:1421
    scylladb#8 0x12a86269 in operator() gms/gossiper.cc:1639
    scylladb#9 0x12ad3eea in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:145
    scylladb#10 0x12be2aff in seastar::noncopyable_function<void (seastar::shared_ptr<gms::i_endpoint_state_change_subscriber>)>::operator()(seastar::shared_ptr<gms::i_endpoint_state_change_subscriber>) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:201
    scylladb#11 0x12bb8e98 in atomic_vector<seastar::shared_ptr<gms::i_endpoint_state_change_subscriber> >::for_each(seastar::noncopyable_function<void (seastar::shared_ptr<gms::i_endpoint_state_change_subscriber>)>) utils/atomic_vector.hh:62
    scylladb#12 0x12a8662b in gms::gossiper::do_on_change_notifications(gms::inet_address, gms::application_state const&, gms::versioned_value const&) gms/gossiper.cc:1638
    scylladb#13 0x12a9387c in operator() gms/gossiper.cc:1978
    scylladb#14 0x12b49b20 in __invoke_impl<void, gms::gossiper::add_local_application_state(std::__cxx11::list<std::pair<gms::application_state, gms::versioned_value> >)::<lambda(gms::gossiper&)> mutable::<lambda()> > /usr/include/c++/10/bits/invoke.h:60
    scylladb#15 0x12b21fd6 in __invoke<gms::gossiper::add_local_application_state(std::__cxx11::list<std::pair<gms::application_state, gms::versioned_value> >)::<lambda(gms::gossiper&)> mutable::<lambda()> > /usr/include/c++/10/bits/invoke.h:95
    scylladb#16 0x12b02865 in __apply_impl<gms::gossiper::add_local_application_state(std::__cxx11::list<std::pair<gms::application_state, gms::versioned_value> >)::<lambda(gms::gossiper&)> mutable::<lambda()>, std::tuple<> > /usr/include/c++/10/tuple:1723
    scylladb#17 0x12b028d8 in apply<gms::gossiper::add_local_application_state(std::__cxx11::list<std::pair<gms::application_state, gms::versioned_value> >)::<lambda(gms::gossiper&)> mutable::<lambda()>, std::tuple<> > /usr/include/c++/10/tuple:1734
    scylladb#18 0x12b02967 in apply<gms::gossiper::add_local_application_state(std::__cxx11::list<std::pair<gms::application_state, gms::versioned_value> >)::<lambda(gms::gossiper&)> mutable::<lambda()> > /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:2052
    scylladb#19 0x12ad866a in operator() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/thread.hh:258
    scylladb#20 0x12b609c2 in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:116
    scylladb#21 0xdfabb5f in seastar::noncopyable_function<void ()>::operator()() const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:201
    scylladb#22 0x16e21bb4 in seastar::thread_context::main() /local/home/bhalevy/dev/scylla/seastar/src/core/thread.cc:297
    scylladb#23 0x16e2190f in seastar::thread_context::s_main(int, int) /local/home/bhalevy/dev/scylla/seastar/src/core/thread.cc:275
    scylladb#24 0x7f8d9060322f  (/local/home/bhalevy/dev/scylla/build/debug/dynamic_libs/libc.so.6+0x5222f)
```

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
bhalevy added a commit to bhalevy/scylla that referenced this issue Oct 4, 2020
… token_metadata_ptr

Fixes use-after-free seen with putget_with_reloaded_certificates_test:
```
==215==ERROR: AddressSanitizer: heap-use-after-free on address 0x603000a8b180 at pc 0x000012eb5a83 bp 0x7ffd2c16d4c0 sp 0x7ffd2c16d4b0
READ of size 8 at 0x603000a8b180 thread T0
    #0 0x12eb5a82 in std::__uniq_ptr_impl<locator::token_metadata_impl, std::default_delete<locator::token_metadata_impl> >::_M_ptr() const /usr/include/c++/10/bits/unique_ptr.h:173
    scylladb#1 0x12ea230d in std::unique_ptr<locator::token_metadata_impl, std::default_delete<locator::token_metadata_impl> >::get() const /usr/include/c++/10/bits/unique_ptr.h:422
    scylladb#2 0x12e8d3e8 in std::unique_ptr<locator::token_metadata_impl, std::default_delete<locator::token_metadata_impl> >::operator->() const /usr/include/c++/10/bits/unique_ptr.h:416
    scylladb#3 0x12e5d0a2 in locator::token_metadata::ring_range(std::optional<interval_bound<dht::ring_position> > const&, bool) const locator/token_metadata.cc:1712
    scylladb#4 0x112d0126 in service::query_ranges_to_vnodes_generator::process_one_range(unsigned long, std::vector<nonwrapping_interval<dht::ring_position>, std::allocator<nonwrapping_interval<dht::ring_position> > >&) service/storage_proxy.cc:4658
    scylladb#5 0x112cf3c5 in service::query_ranges_to_vnodes_generator::operator()(unsigned long) service/storage_proxy.cc:4616
    scylladb#6 0x112b2261 in service::storage_proxy::query_partition_key_range_concurrent(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >, std::vector<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >, std::allocator<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&, seastar::lw_shared_ptr<query::read_command>, db::consistency_level, service::query_ranges_to_vnodes_generator&&, int, tracing::trace_state_ptr, unsigned long, unsigned int, std::unordered_map<nonwrapping_interval<dht::token>, std::vector<utils::UUID, std::allocator<utils::UUID> >, std::hash<nonwrapping_interval<dht::token> >, std::equal_to<nonwrapping_interval<dht::token> >, std::allocator<std::pair<nonwrapping_interval<dht::token> const, std::vector<utils::UUID, std::allocator<utils::UUID> > > > >, service_permit) service/storage_proxy.cc:4023
    scylladb#7 0x112b094e in operator() service/storage_proxy.cc:4160
    scylladb#8 0x1139c8bb in invoke<service::storage_proxy::query_partition_key_range_concurrent(seastar::lowres_clock::time_point, std::vector<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >&&, seastar::lw_shared_ptr<query::read_command>, db::consistency_level, service::query_ranges_to_vnodes_generator&&, int, tracing::trace_state_ptr, uint64_t, uint32_t, service::replicas_per_token_range, service_permit)::<lambda(seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:2088
    scylladb#9 0x1136625b in futurize_invoke<service::storage_proxy::query_partition_key_range_concurrent(seastar::lowres_clock::time_point, std::vector<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >&&, seastar::lw_shared_ptr<query::read_command>, db::consistency_level, service::query_ranges_to_vnodes_generator&&, int, tracing::trace_state_ptr, uint64_t, uint32_t, service::replicas_per_token_range, service_permit)::<lambda(seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:2119
    scylladb#10 0x11366372 in operator()<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1480
    scylladb#11 0x1139cc3b in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:145
    scylladb#12 0x116f4944 in seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>::operator()(seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:201
    scylladb#13 0x116b3397 in seastar::future<service::query_partition_key_range_concurrent_result> std::__invoke_impl<seastar::future<service::query_partition_key_range_concurrent_result>, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >(std::__invoke_other, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&) /usr/include/c++/10/bits/invoke.h:60
    scylladb#14 0x1165c3a6 in std::__invoke_result<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::type std::__invoke<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&) /usr/include/c++/10/bits/invoke.h:96
    scylladb#15 0x115e6542 in decltype(auto) std::__apply_impl<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >, 0ul>(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >&&, std::integer_sequence<unsigned long, 0ul>) /usr/include/c++/10/tuple:1724
    scylladb#16 0x115e6663 in decltype(auto) std::apply<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >&&) /usr/include/c++/10/tuple:1736
    scylladb#17 0x115e63f9 in seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}::operator()(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&) const::{lambda()scylladb#1}::operator()() const /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1530
    scylladb#18 0x1165c4b9 in void seastar::futurize<seastar::future<service::query_partition_key_range_concurrent_result> >::satisfy_with_result_of<seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}::operator()(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&) const::{lambda()scylladb#1}>(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:2073
    scylladb#19 0x115e61f5 in seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}::operator()(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1528
    scylladb#20 0x1176e9cc in seastar::continuation<seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::run_and_dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:746
    scylladb#21 0x16a9a455 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2196
    scylladb#22 0x16a9e691 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2575
    scylladb#23 0x16aa390e in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2730
    scylladb#24 0x168ae4f7 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:207
    scylladb#25 0x168ac541 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115
    scylladb#26 0xd6cd3c4 in main /local/home/bhalevy/dev/scylla/main.cc:504
    scylladb#27 0x7f8d905d8041 in __libc_start_main (/local/home/bhalevy/dev/scylla/build/debug/dynamic_libs/libc.so.6+0x27041)
    scylladb#28 0xd67c9ed in _start (/local/home/bhalevy/.dtest/dtest-o0qoqmkr/test/node3/bin/scylla+0xd67c9ed)

0x603000a8b180 is located 16 bytes inside of 24-byte region [0x603000a8b170,0x603000a8b188)
freed by thread T0 here:
    #0 0x7f8d92a190cf in operator delete(void*, unsigned long) (/local/home/bhalevy/dev/scylla/build/debug/dynamic_libs/libasan.so.6+0xb30cf)
    scylladb#1 0xd7ebe54 in seastar::internal::lw_shared_ptr_accessors_no_esft<locator::token_metadata>::dispose(seastar::lw_shared_ptr_counter_base*) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:213
    scylladb#2 0x112b155d in seastar::lw_shared_ptr<locator::token_metadata const>::~lw_shared_ptr() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:300
    scylladb#3 0x112b155d in ~<lambda> service/storage_proxy.cc:4137
    scylladb#4 0x1132e92d in ~<lambda> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1479
    scylladb#5 0x1139cc91 in destroy /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:148
    scylladb#6 0x11565673 in seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>::~noncopyable_function() /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:181
    scylladb#7 0x1176e783 in seastar::continuation<seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::~continuation() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:729
    scylladb#8 0x1176ea06 in seastar::continuation<seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::run_and_dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:750
    scylladb#9 0x16a9a455 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2196
    scylladb#10 0x16a9e691 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2575
    scylladb#11 0x16aa390e in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2730
    scylladb#12 0x168ae4f7 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:207
    scylladb#13 0x168ac541 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115
    scylladb#14 0xd6cd3c4 in main /local/home/bhalevy/dev/scylla/main.cc:504
    scylladb#15 0x7f8d905d8041 in __libc_start_main (/local/home/bhalevy/dev/scylla/build/debug/dynamic_libs/libc.so.6+0x27041)

previously allocated by thread T0 here:
    #0 0x7f8d92a18067 in operator new(unsigned long) (/local/home/bhalevy/dev/scylla/build/debug/dynamic_libs/libasan.so.6+0xb2067)
    scylladb#1 0x13cf7132 in seastar::lw_shared_ptr<locator::token_metadata> seastar::lw_shared_ptr<locator::token_metadata>::make<locator::token_metadata>(locator::token_metadata&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:266
    scylladb#2 0x13cc3bfa in seastar::lw_shared_ptr<locator::token_metadata> seastar::make_lw_shared<locator::token_metadata>(locator::token_metadata&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:422
    scylladb#3 0x13ca3007 in seastar::lw_shared_ptr<locator::token_metadata> locator::make_token_metadata_ptr<locator::token_metadata>(locator::token_metadata) locator/token_metadata.hh:338
    scylladb#4 0x13c9bdd4 in locator::shared_token_metadata::clone() const locator/token_metadata.hh:358
    scylladb#5 0x13c9c18a in service::storage_service::get_mutable_token_metadata_ptr() service/storage_service.hh:184
    scylladb#6 0x13a5a445 in service::storage_service::handle_state_normal(gms::inet_address) service/storage_service.cc:1129
    scylladb#7 0x13a6371c in service::storage_service::on_change(gms::inet_address, gms::application_state, gms::versioned_value const&) service/storage_service.cc:1421
    scylladb#8 0x12a86269 in operator() gms/gossiper.cc:1639
    scylladb#9 0x12ad3eea in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:145
    scylladb#10 0x12be2aff in seastar::noncopyable_function<void (seastar::shared_ptr<gms::i_endpoint_state_change_subscriber>)>::operator()(seastar::shared_ptr<gms::i_endpoint_state_change_subscriber>) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:201
    scylladb#11 0x12bb8e98 in atomic_vector<seastar::shared_ptr<gms::i_endpoint_state_change_subscriber> >::for_each(seastar::noncopyable_function<void (seastar::shared_ptr<gms::i_endpoint_state_change_subscriber>)>) utils/atomic_vector.hh:62
    scylladb#12 0x12a8662b in gms::gossiper::do_on_change_notifications(gms::inet_address, gms::application_state const&, gms::versioned_value const&) gms/gossiper.cc:1638
    scylladb#13 0x12a9387c in operator() gms/gossiper.cc:1978
    scylladb#14 0x12b49b20 in __invoke_impl<void, gms::gossiper::add_local_application_state(std::__cxx11::list<std::pair<gms::application_state, gms::versioned_value> >)::<lambda(gms::gossiper&)> mutable::<lambda()> > /usr/include/c++/10/bits/invoke.h:60
    scylladb#15 0x12b21fd6 in __invoke<gms::gossiper::add_local_application_state(std::__cxx11::list<std::pair<gms::application_state, gms::versioned_value> >)::<lambda(gms::gossiper&)> mutable::<lambda()> > /usr/include/c++/10/bits/invoke.h:95
    scylladb#16 0x12b02865 in __apply_impl<gms::gossiper::add_local_application_state(std::__cxx11::list<std::pair<gms::application_state, gms::versioned_value> >)::<lambda(gms::gossiper&)> mutable::<lambda()>, std::tuple<> > /usr/include/c++/10/tuple:1723
    scylladb#17 0x12b028d8 in apply<gms::gossiper::add_local_application_state(std::__cxx11::list<std::pair<gms::application_state, gms::versioned_value> >)::<lambda(gms::gossiper&)> mutable::<lambda()>, std::tuple<> > /usr/include/c++/10/tuple:1734
    scylladb#18 0x12b02967 in apply<gms::gossiper::add_local_application_state(std::__cxx11::list<std::pair<gms::application_state, gms::versioned_value> >)::<lambda(gms::gossiper&)> mutable::<lambda()> > /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:2052
    scylladb#19 0x12ad866a in operator() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/thread.hh:258
    scylladb#20 0x12b609c2 in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:116
    scylladb#21 0xdfabb5f in seastar::noncopyable_function<void ()>::operator()() const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:201
    scylladb#22 0x16e21bb4 in seastar::thread_context::main() /local/home/bhalevy/dev/scylla/seastar/src/core/thread.cc:297
    scylladb#23 0x16e2190f in seastar::thread_context::s_main(int, int) /local/home/bhalevy/dev/scylla/seastar/src/core/thread.cc:275
    scylladb#24 0x7f8d9060322f  (/local/home/bhalevy/dev/scylla/build/debug/dynamic_libs/libc.so.6+0x5222f)
```

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
bhalevy added a commit to bhalevy/scylla that referenced this issue Oct 22, 2020
… token_metadata_ptr

Fixes use-after-free seen with putget_with_reloaded_certificates_test:
```
==215==ERROR: AddressSanitizer: heap-use-after-free on address 0x603000a8b180 at pc 0x000012eb5a83 bp 0x7ffd2c16d4c0 sp 0x7ffd2c16d4b0
READ of size 8 at 0x603000a8b180 thread T0
    #0 0x12eb5a82 in std::__uniq_ptr_impl<locator::token_metadata_impl, std::default_delete<locator::token_metadata_impl> >::_M_ptr() const /usr/include/c++/10/bits/unique_ptr.h:173
    scylladb#1 0x12ea230d in std::unique_ptr<locator::token_metadata_impl, std::default_delete<locator::token_metadata_impl> >::get() const /usr/include/c++/10/bits/unique_ptr.h:422
    scylladb#2 0x12e8d3e8 in std::unique_ptr<locator::token_metadata_impl, std::default_delete<locator::token_metadata_impl> >::operator->() const /usr/include/c++/10/bits/unique_ptr.h:416
    scylladb#3 0x12e5d0a2 in locator::token_metadata::ring_range(std::optional<interval_bound<dht::ring_position> > const&, bool) const locator/token_metadata.cc:1712
    scylladb#4 0x112d0126 in service::query_ranges_to_vnodes_generator::process_one_range(unsigned long, std::vector<nonwrapping_interval<dht::ring_position>, std::allocator<nonwrapping_interval<dht::ring_position> > >&) service/storage_proxy.cc:4658
    scylladb#5 0x112cf3c5 in service::query_ranges_to_vnodes_generator::operator()(unsigned long) service/storage_proxy.cc:4616
    scylladb#6 0x112b2261 in service::storage_proxy::query_partition_key_range_concurrent(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >, std::vector<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >, std::allocator<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&, seastar::lw_shared_ptr<query::read_command>, db::consistency_level, service::query_ranges_to_vnodes_generator&&, int, tracing::trace_state_ptr, unsigned long, unsigned int, std::unordered_map<nonwrapping_interval<dht::token>, std::vector<utils::UUID, std::allocator<utils::UUID> >, std::hash<nonwrapping_interval<dht::token> >, std::equal_to<nonwrapping_interval<dht::token> >, std::allocator<std::pair<nonwrapping_interval<dht::token> const, std::vector<utils::UUID, std::allocator<utils::UUID> > > > >, service_permit) service/storage_proxy.cc:4023
    scylladb#7 0x112b094e in operator() service/storage_proxy.cc:4160
    scylladb#8 0x1139c8bb in invoke<service::storage_proxy::query_partition_key_range_concurrent(seastar::lowres_clock::time_point, std::vector<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >&&, seastar::lw_shared_ptr<query::read_command>, db::consistency_level, service::query_ranges_to_vnodes_generator&&, int, tracing::trace_state_ptr, uint64_t, uint32_t, service::replicas_per_token_range, service_permit)::<lambda(seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:2088
    scylladb#9 0x1136625b in futurize_invoke<service::storage_proxy::query_partition_key_range_concurrent(seastar::lowres_clock::time_point, std::vector<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >&&, seastar::lw_shared_ptr<query::read_command>, db::consistency_level, service::query_ranges_to_vnodes_generator&&, int, tracing::trace_state_ptr, uint64_t, uint32_t, service::replicas_per_token_range, service_permit)::<lambda(seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:2119
    scylladb#10 0x11366372 in operator()<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1480
    scylladb#11 0x1139cc3b in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:145
    scylladb#12 0x116f4944 in seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>::operator()(seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:201
    scylladb#13 0x116b3397 in seastar::future<service::query_partition_key_range_concurrent_result> std::__invoke_impl<seastar::future<service::query_partition_key_range_concurrent_result>, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >(std::__invoke_other, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&) /usr/include/c++/10/bits/invoke.h:60
    scylladb#14 0x1165c3a6 in std::__invoke_result<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::type std::__invoke<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&) /usr/include/c++/10/bits/invoke.h:96
    scylladb#15 0x115e6542 in decltype(auto) std::__apply_impl<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >, 0ul>(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >&&, std::integer_sequence<unsigned long, 0ul>) /usr/include/c++/10/tuple:1724
    scylladb#16 0x115e6663 in decltype(auto) std::apply<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >&&) /usr/include/c++/10/tuple:1736
    scylladb#17 0x115e63f9 in seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}::operator()(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&) const::{lambda()scylladb#1}::operator()() const /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1530
    scylladb#18 0x1165c4b9 in void seastar::futurize<seastar::future<service::query_partition_key_range_concurrent_result> >::satisfy_with_result_of<seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}::operator()(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&) const::{lambda()scylladb#1}>(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:2073
    scylladb#19 0x115e61f5 in seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}::operator()(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1528
    scylladb#20 0x1176e9cc in seastar::continuation<seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::run_and_dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:746
    scylladb#21 0x16a9a455 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2196
    scylladb#22 0x16a9e691 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2575
    scylladb#23 0x16aa390e in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2730
    scylladb#24 0x168ae4f7 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:207
    scylladb#25 0x168ac541 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115
    scylladb#26 0xd6cd3c4 in main /local/home/bhalevy/dev/scylla/main.cc:504
    scylladb#27 0x7f8d905d8041 in __libc_start_main (/local/home/bhalevy/dev/scylla/build/debug/dynamic_libs/libc.so.6+0x27041)
    scylladb#28 0xd67c9ed in _start (/local/home/bhalevy/.dtest/dtest-o0qoqmkr/test/node3/bin/scylla+0xd67c9ed)

0x603000a8b180 is located 16 bytes inside of 24-byte region [0x603000a8b170,0x603000a8b188)
freed by thread T0 here:
    #0 0x7f8d92a190cf in operator delete(void*, unsigned long) (/local/home/bhalevy/dev/scylla/build/debug/dynamic_libs/libasan.so.6+0xb30cf)
    scylladb#1 0xd7ebe54 in seastar::internal::lw_shared_ptr_accessors_no_esft<locator::token_metadata>::dispose(seastar::lw_shared_ptr_counter_base*) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:213
    scylladb#2 0x112b155d in seastar::lw_shared_ptr<locator::token_metadata const>::~lw_shared_ptr() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:300
    scylladb#3 0x112b155d in ~<lambda> service/storage_proxy.cc:4137
    scylladb#4 0x1132e92d in ~<lambda> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1479
    scylladb#5 0x1139cc91 in destroy /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:148
    scylladb#6 0x11565673 in seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>::~noncopyable_function() /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:181
    scylladb#7 0x1176e783 in seastar::continuation<seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::~continuation() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:729
    scylladb#8 0x1176ea06 in seastar::continuation<seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::run_and_dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:750
    scylladb#9 0x16a9a455 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2196
    scylladb#10 0x16a9e691 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2575
    scylladb#11 0x16aa390e in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2730
    scylladb#12 0x168ae4f7 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:207
    scylladb#13 0x168ac541 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115
    scylladb#14 0xd6cd3c4 in main /local/home/bhalevy/dev/scylla/main.cc:504
    scylladb#15 0x7f8d905d8041 in __libc_start_main (/local/home/bhalevy/dev/scylla/build/debug/dynamic_libs/libc.so.6+0x27041)

previously allocated by thread T0 here:
    #0 0x7f8d92a18067 in operator new(unsigned long) (/local/home/bhalevy/dev/scylla/build/debug/dynamic_libs/libasan.so.6+0xb2067)
    scylladb#1 0x13cf7132 in seastar::lw_shared_ptr<locator::token_metadata> seastar::lw_shared_ptr<locator::token_metadata>::make<locator::token_metadata>(locator::token_metadata&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:266
    scylladb#2 0x13cc3bfa in seastar::lw_shared_ptr<locator::token_metadata> seastar::make_lw_shared<locator::token_metadata>(locator::token_metadata&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:422
    scylladb#3 0x13ca3007 in seastar::lw_shared_ptr<locator::token_metadata> locator::make_token_metadata_ptr<locator::token_metadata>(locator::token_metadata) locator/token_metadata.hh:338
    scylladb#4 0x13c9bdd4 in locator::shared_token_metadata::clone() const locator/token_metadata.hh:358
    scylladb#5 0x13c9c18a in service::storage_service::get_mutable_token_metadata_ptr() service/storage_service.hh:184
    scylladb#6 0x13a5a445 in service::storage_service::handle_state_normal(gms::inet_address) service/storage_service.cc:1129
    scylladb#7 0x13a6371c in service::storage_service::on_change(gms::inet_address, gms::application_state, gms::versioned_value const&) service/storage_service.cc:1421
    scylladb#8 0x12a86269 in operator() gms/gossiper.cc:1639
    scylladb#9 0x12ad3eea in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:145
    scylladb#10 0x12be2aff in seastar::noncopyable_function<void (seastar::shared_ptr<gms::i_endpoint_state_change_subscriber>)>::operator()(seastar::shared_ptr<gms::i_endpoint_state_change_subscriber>) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:201
    scylladb#11 0x12bb8e98 in atomic_vector<seastar::shared_ptr<gms::i_endpoint_state_change_subscriber> >::for_each(seastar::noncopyable_function<void (seastar::shared_ptr<gms::i_endpoint_state_change_subscriber>)>) utils/atomic_vector.hh:62
    scylladb#12 0x12a8662b in gms::gossiper::do_on_change_notifications(gms::inet_address, gms::application_state const&, gms::versioned_value const&) gms/gossiper.cc:1638
    scylladb#13 0x12a9387c in operator() gms/gossiper.cc:1978
    scylladb#14 0x12b49b20 in __invoke_impl<void, gms::gossiper::add_local_application_state(std::__cxx11::list<std::pair<gms::application_state, gms::versioned_value> >)::<lambda(gms::gossiper&)> mutable::<lambda()> > /usr/include/c++/10/bits/invoke.h:60
    scylladb#15 0x12b21fd6 in __invoke<gms::gossiper::add_local_application_state(std::__cxx11::list<std::pair<gms::application_state, gms::versioned_value> >)::<lambda(gms::gossiper&)> mutable::<lambda()> > /usr/include/c++/10/bits/invoke.h:95
    scylladb#16 0x12b02865 in __apply_impl<gms::gossiper::add_local_application_state(std::__cxx11::list<std::pair<gms::application_state, gms::versioned_value> >)::<lambda(gms::gossiper&)> mutable::<lambda()>, std::tuple<> > /usr/include/c++/10/tuple:1723
    scylladb#17 0x12b028d8 in apply<gms::gossiper::add_local_application_state(std::__cxx11::list<std::pair<gms::application_state, gms::versioned_value> >)::<lambda(gms::gossiper&)> mutable::<lambda()>, std::tuple<> > /usr/include/c++/10/tuple:1734
    scylladb#18 0x12b02967 in apply<gms::gossiper::add_local_application_state(std::__cxx11::list<std::pair<gms::application_state, gms::versioned_value> >)::<lambda(gms::gossiper&)> mutable::<lambda()> > /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:2052
    scylladb#19 0x12ad866a in operator() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/thread.hh:258
    scylladb#20 0x12b609c2 in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:116
    scylladb#21 0xdfabb5f in seastar::noncopyable_function<void ()>::operator()() const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:201
    scylladb#22 0x16e21bb4 in seastar::thread_context::main() /local/home/bhalevy/dev/scylla/seastar/src/core/thread.cc:297
    scylladb#23 0x16e2190f in seastar::thread_context::s_main(int, int) /local/home/bhalevy/dev/scylla/seastar/src/core/thread.cc:275
    scylladb#24 0x7f8d9060322f  (/local/home/bhalevy/dev/scylla/build/debug/dynamic_libs/libc.so.6+0x5222f)
```

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
bhalevy added a commit to bhalevy/scylla that referenced this issue Nov 11, 2020
… token_metadata_ptr

Fixes use-after-free seen with putget_with_reloaded_certificates_test:
```
==215==ERROR: AddressSanitizer: heap-use-after-free on address 0x603000a8b180 at pc 0x000012eb5a83 bp 0x7ffd2c16d4c0 sp 0x7ffd2c16d4b0
READ of size 8 at 0x603000a8b180 thread T0
    #0 0x12eb5a82 in std::__uniq_ptr_impl<locator::token_metadata_impl, std::default_delete<locator::token_metadata_impl> >::_M_ptr() const /usr/include/c++/10/bits/unique_ptr.h:173
    scylladb#1 0x12ea230d in std::unique_ptr<locator::token_metadata_impl, std::default_delete<locator::token_metadata_impl> >::get() const /usr/include/c++/10/bits/unique_ptr.h:422
    scylladb#2 0x12e8d3e8 in std::unique_ptr<locator::token_metadata_impl, std::default_delete<locator::token_metadata_impl> >::operator->() const /usr/include/c++/10/bits/unique_ptr.h:416
    scylladb#3 0x12e5d0a2 in locator::token_metadata::ring_range(std::optional<interval_bound<dht::ring_position> > const&, bool) const locator/token_metadata.cc:1712
    scylladb#4 0x112d0126 in service::query_ranges_to_vnodes_generator::process_one_range(unsigned long, std::vector<nonwrapping_interval<dht::ring_position>, std::allocator<nonwrapping_interval<dht::ring_position> > >&) service/storage_proxy.cc:4658
    scylladb#5 0x112cf3c5 in service::query_ranges_to_vnodes_generator::operator()(unsigned long) service/storage_proxy.cc:4616
    scylladb#6 0x112b2261 in service::storage_proxy::query_partition_key_range_concurrent(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >, std::vector<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >, std::allocator<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&, seastar::lw_shared_ptr<query::read_command>, db::consistency_level, service::query_ranges_to_vnodes_generator&&, int, tracing::trace_state_ptr, unsigned long, unsigned int, std::unordered_map<nonwrapping_interval<dht::token>, std::vector<utils::UUID, std::allocator<utils::UUID> >, std::hash<nonwrapping_interval<dht::token> >, std::equal_to<nonwrapping_interval<dht::token> >, std::allocator<std::pair<nonwrapping_interval<dht::token> const, std::vector<utils::UUID, std::allocator<utils::UUID> > > > >, service_permit) service/storage_proxy.cc:4023
    scylladb#7 0x112b094e in operator() service/storage_proxy.cc:4160
    scylladb#8 0x1139c8bb in invoke<service::storage_proxy::query_partition_key_range_concurrent(seastar::lowres_clock::time_point, std::vector<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >&&, seastar::lw_shared_ptr<query::read_command>, db::consistency_level, service::query_ranges_to_vnodes_generator&&, int, tracing::trace_state_ptr, uint64_t, uint32_t, service::replicas_per_token_range, service_permit)::<lambda(seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:2088
    scylladb#9 0x1136625b in futurize_invoke<service::storage_proxy::query_partition_key_range_concurrent(seastar::lowres_clock::time_point, std::vector<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >&&, seastar::lw_shared_ptr<query::read_command>, db::consistency_level, service::query_ranges_to_vnodes_generator&&, int, tracing::trace_state_ptr, uint64_t, uint32_t, service::replicas_per_token_range, service_permit)::<lambda(seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:2119
    scylladb#10 0x11366372 in operator()<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1480
    scylladb#11 0x1139cc3b in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:145
    scylladb#12 0x116f4944 in seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>::operator()(seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:201
    scylladb#13 0x116b3397 in seastar::future<service::query_partition_key_range_concurrent_result> std::__invoke_impl<seastar::future<service::query_partition_key_range_concurrent_result>, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >(std::__invoke_other, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&) /usr/include/c++/10/bits/invoke.h:60
    scylladb#14 0x1165c3a6 in std::__invoke_result<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::type std::__invoke<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&) /usr/include/c++/10/bits/invoke.h:96
    scylladb#15 0x115e6542 in decltype(auto) std::__apply_impl<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >, 0ul>(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >&&, std::integer_sequence<unsigned long, 0ul>) /usr/include/c++/10/tuple:1724
    scylladb#16 0x115e6663 in decltype(auto) std::apply<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >&&) /usr/include/c++/10/tuple:1736
    scylladb#17 0x115e63f9 in seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}::operator()(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&) const::{lambda()scylladb#1}::operator()() const /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1530
    scylladb#18 0x1165c4b9 in void seastar::futurize<seastar::future<service::query_partition_key_range_concurrent_result> >::satisfy_with_result_of<seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}::operator()(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&) const::{lambda()scylladb#1}>(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:2073
    scylladb#19 0x115e61f5 in seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}::operator()(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1528
    scylladb#20 0x1176e9cc in seastar::continuation<seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::run_and_dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:746
    scylladb#21 0x16a9a455 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2196
    scylladb#22 0x16a9e691 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2575
    scylladb#23 0x16aa390e in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2730
    scylladb#24 0x168ae4f7 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:207
    scylladb#25 0x168ac541 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115
    scylladb#26 0xd6cd3c4 in main /local/home/bhalevy/dev/scylla/main.cc:504
    scylladb#27 0x7f8d905d8041 in __libc_start_main (/local/home/bhalevy/dev/scylla/build/debug/dynamic_libs/libc.so.6+0x27041)
    scylladb#28 0xd67c9ed in _start (/local/home/bhalevy/.dtest/dtest-o0qoqmkr/test/node3/bin/scylla+0xd67c9ed)

0x603000a8b180 is located 16 bytes inside of 24-byte region [0x603000a8b170,0x603000a8b188)
freed by thread T0 here:
    #0 0x7f8d92a190cf in operator delete(void*, unsigned long) (/local/home/bhalevy/dev/scylla/build/debug/dynamic_libs/libasan.so.6+0xb30cf)
    scylladb#1 0xd7ebe54 in seastar::internal::lw_shared_ptr_accessors_no_esft<locator::token_metadata>::dispose(seastar::lw_shared_ptr_counter_base*) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:213
    scylladb#2 0x112b155d in seastar::lw_shared_ptr<locator::token_metadata const>::~lw_shared_ptr() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:300
    scylladb#3 0x112b155d in ~<lambda> service/storage_proxy.cc:4137
    scylladb#4 0x1132e92d in ~<lambda> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1479
    scylladb#5 0x1139cc91 in destroy /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:148
    scylladb#6 0x11565673 in seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>::~noncopyable_function() /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:181
    scylladb#7 0x1176e783 in seastar::continuation<seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::~continuation() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:729
    scylladb#8 0x1176ea06 in seastar::continuation<seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::run_and_dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:750
    scylladb#9 0x16a9a455 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2196
    scylladb#10 0x16a9e691 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2575
    scylladb#11 0x16aa390e in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2730
    scylladb#12 0x168ae4f7 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:207
    scylladb#13 0x168ac541 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115
    scylladb#14 0xd6cd3c4 in main /local/home/bhalevy/dev/scylla/main.cc:504
    scylladb#15 0x7f8d905d8041 in __libc_start_main (/local/home/bhalevy/dev/scylla/build/debug/dynamic_libs/libc.so.6+0x27041)

previously allocated by thread T0 here:
    #0 0x7f8d92a18067 in operator new(unsigned long) (/local/home/bhalevy/dev/scylla/build/debug/dynamic_libs/libasan.so.6+0xb2067)
    scylladb#1 0x13cf7132 in seastar::lw_shared_ptr<locator::token_metadata> seastar::lw_shared_ptr<locator::token_metadata>::make<locator::token_metadata>(locator::token_metadata&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:266
    scylladb#2 0x13cc3bfa in seastar::lw_shared_ptr<locator::token_metadata> seastar::make_lw_shared<locator::token_metadata>(locator::token_metadata&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:422
    scylladb#3 0x13ca3007 in seastar::lw_shared_ptr<locator::token_metadata> locator::make_token_metadata_ptr<locator::token_metadata>(locator::token_metadata) locator/token_metadata.hh:338
    scylladb#4 0x13c9bdd4 in locator::shared_token_metadata::clone() const locator/token_metadata.hh:358
    scylladb#5 0x13c9c18a in service::storage_service::get_mutable_token_metadata_ptr() service/storage_service.hh:184
    scylladb#6 0x13a5a445 in service::storage_service::handle_state_normal(gms::inet_address) service/storage_service.cc:1129
    scylladb#7 0x13a6371c in service::storage_service::on_change(gms::inet_address, gms::application_state, gms::versioned_value const&) service/storage_service.cc:1421
    scylladb#8 0x12a86269 in operator() gms/gossiper.cc:1639
    scylladb#9 0x12ad3eea in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:145
    scylladb#10 0x12be2aff in seastar::noncopyable_function<void (seastar::shared_ptr<gms::i_endpoint_state_change_subscriber>)>::operator()(seastar::shared_ptr<gms::i_endpoint_state_change_subscriber>) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:201
    scylladb#11 0x12bb8e98 in atomic_vector<seastar::shared_ptr<gms::i_endpoint_state_change_subscriber> >::for_each(seastar::noncopyable_function<void (seastar::shared_ptr<gms::i_endpoint_state_change_subscriber>)>) utils/atomic_vector.hh:62
    scylladb#12 0x12a8662b in gms::gossiper::do_on_change_notifications(gms::inet_address, gms::application_state const&, gms::versioned_value const&) gms/gossiper.cc:1638
    scylladb#13 0x12a9387c in operator() gms/gossiper.cc:1978
    scylladb#14 0x12b49b20 in __invoke_impl<void, gms::gossiper::add_local_application_state(std::__cxx11::list<std::pair<gms::application_state, gms::versioned_value> >)::<lambda(gms::gossiper&)> mutable::<lambda()> > /usr/include/c++/10/bits/invoke.h:60
    scylladb#15 0x12b21fd6 in __invoke<gms::gossiper::add_local_application_state(std::__cxx11::list<std::pair<gms::application_state, gms::versioned_value> >)::<lambda(gms::gossiper&)> mutable::<lambda()> > /usr/include/c++/10/bits/invoke.h:95
    scylladb#16 0x12b02865 in __apply_impl<gms::gossiper::add_local_application_state(std::__cxx11::list<std::pair<gms::application_state, gms::versioned_value> >)::<lambda(gms::gossiper&)> mutable::<lambda()>, std::tuple<> > /usr/include/c++/10/tuple:1723
    scylladb#17 0x12b028d8 in apply<gms::gossiper::add_local_application_state(std::__cxx11::list<std::pair<gms::application_state, gms::versioned_value> >)::<lambda(gms::gossiper&)> mutable::<lambda()>, std::tuple<> > /usr/include/c++/10/tuple:1734
    scylladb#18 0x12b02967 in apply<gms::gossiper::add_local_application_state(std::__cxx11::list<std::pair<gms::application_state, gms::versioned_value> >)::<lambda(gms::gossiper&)> mutable::<lambda()> > /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:2052
    scylladb#19 0x12ad866a in operator() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/thread.hh:258
    scylladb#20 0x12b609c2 in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:116
    scylladb#21 0xdfabb5f in seastar::noncopyable_function<void ()>::operator()() const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:201
    scylladb#22 0x16e21bb4 in seastar::thread_context::main() /local/home/bhalevy/dev/scylla/seastar/src/core/thread.cc:297
    scylladb#23 0x16e2190f in seastar::thread_context::s_main(int, int) /local/home/bhalevy/dev/scylla/seastar/src/core/thread.cc:275
    scylladb#24 0x7f8d9060322f  (/local/home/bhalevy/dev/scylla/build/debug/dynamic_libs/libc.so.6+0x5222f)
```

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
tgrabiec pushed a commit to tgrabiec/scylla that referenced this issue Oct 25, 2021
… node ' from Pavel Solodovnikov

Also, a couple of code cleanups for CQL syntax in other shared_sstables methods.

Expose a `system_distributed_keyspace` sharded service in the `cql_test_env` to be able to write tests using `system_distributed*` keyspaces.

Closes scylladb#7

* github.com:avikivity/scylla:
  db: shared_sstables: add method to return the list of shared sstables owned by a node
  db: system_distributed_keyspace: shared_sstables: CQL syntax cleanup
  test: add system_distributed_keyspace to cql test environment
avikivity added a commit that referenced this issue Nov 3, 2022
The main theme of this patchset is improving `scylla-fiber`, with some assorted unrelated improvement tagging along.
In lieu of explicit support for mapping up continuation chains in memory from seastar (there is one but it uses function calls), scylla fiber uses a quite crude method to do this: it scans task objects for outbound references to other task objects to find waiters tasks and scans inbound references from other tasks to find waited-on tasks. This works well for most objects, but there are some problematic ones:
* `seastar::thread_context`: the waited-on task (`seastar::(anonymous namespace)::thread_wake_task`) is allocated on the thread's stack which is not in the object itself. Scylla fiber now scans the stack bottom-up to find this task.
* `seastar::smp_message_queue::async_work_item`: the waited on task lives on another shard. Scylla fiber now digs out the remote shard from the work item and continues the search on the remote shard.
* `seastar::when_all_state`: the waited on task is a member in the same object tripping loop detection and terminating the search. Seastar fiber now uses the `_continuation` member explicitely to look for the next links.

Other minor improvements were also done, like including the shard of the task in the printout.
Example demonstrating all the new additions:
```
(gdb) scylla fiber 0x000060002d650200
Stopping because loop is detected: task 0x000061c00385fb60 was seen before.
[shard 28] #-13 (task*) 0x000061c00385fba0 0x00000000003b5b00 vtable for seastar::internal::when_all_state_component<seastar::future<void> > + 16
[shard 28] #-12 (task*) 0x000061c00385fb60 0x0000000000417010 vtable for seastar::internal::when_all_state<seastar::internal::identity_futures_tuple<seastar::future<void>, seastar::future<void> >, seastar::future<void>, seastar::future<void> > + 16
[shard 28] #-11 (task*) 0x000061c009f16420 0x0000000000419830 _ZTVN7seastar12continuationINS_8internal22promise_base_with_typeIvEEZNS_6futureISt5tupleIJNS4_IvEES6_EEE14discard_resultEvEUlDpOT_E_ZNS8_14then_impl_nrvoISC_S6_EET0_OT_EUlOS3_RSC_ONS_12future_stateIS7_EEE_S7_EE + 16
[shard 28] #-10 (task*) 0x000061c0098e9e00 0x0000000000447440 vtable for seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::smp_message_queue::async_work_item<seastar::sharded<cql_transport::cql_server>::stop()::{lambda(unsigned int)#1}::operator()(unsigned int)::{lambda()#1}>::run_and_dispose()::{lambda(auto:1)#1}, seastar::future<void>::then_wrapped_nrvo<void, seastar::smp_message_queue::async_work_item<seastar::sharded<cql_transport::cql_server>::stop()::{lambda(unsigned int)#1}::operator()(unsigned int)::{lambda()#1}> >(seastar::smp_message_queue::async_work_item<seastar::sharded<cql_transport::cql_server>::stop()::{lambda(unsigned int)#1}::operator()(unsigned int)::{lambda()#1}>&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::smp_message_queue::async_work_item<seastar::sharded<cql_transport::cql_server>::stop()::{lambda(unsigned int)#1}::operator()(unsigned int)::{lambda()#1}>&, seastar::future_state<seastar::internal::monostate>&&)#1}, void> + 16
[shard  0] #-9 (task*) 0x000060000858dcd0 0x0000000000449d68 vtable for seastar::smp_message_queue::async_work_item<seastar::sharded<cql_transport::cql_server>::stop()::{lambda(unsigned int)#1}::operator()(unsigned int)::{lambda()#1}> + 16
[shard  0] #-8 (task*) 0x0000600050c39f60 0x00000000007abe98 vtable for seastar::parallel_for_each_state + 16
[shard  0] #-7 (task*) 0x000060000a59c1c0 0x0000000000449f60 vtable for seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::sharded<cql_transport::cql_server>::stop()::{lambda(seastar::future<void>)#2}, seastar::future<void>::then_wrapped_nrvo<seastar::future<void>, {lambda(seastar::future<void>)#2}>({lambda(seastar::future<void>)#2}&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, {lambda(seastar::future<void>)#2}&, seastar::future_state<seastar::internal::monostate>&&)#1}, void> + 16
[shard  0] #-6 (task*) 0x000060000a59c400 0x0000000000449ea0 vtable for seastar::continuation<seastar::internal::promise_base_with_type<void>, cql_transport::controller::do_stop_server()::{lambda(std::unique_ptr<seastar::sharded<cql_transport::cql_server>, std::default_delete<seastar::sharded<cql_transport::cql_server> > >&)#1}::operator()(std::unique_ptr<seastar::sharded<cql_transport::cql_server>, std::default_delete<seastar::sharded<cql_transport::cql_server> > >&) const::{lambda()#1}::operator()() const::{lambda()#1}, seastar::future<void>::then_impl_nrvo<{lambda()#1}, {lambda()#1}>({lambda()#1}&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, {lambda()#1}&, seastar::future_state<seastar::internal::monostate>&&)#1}, void> + 16
[shard  0] #-5 (task*) 0x0000600009d86cc0 0x0000000000449c00 vtable for seastar::internal::do_with_state<std::tuple<std::unique_ptr<seastar::sharded<cql_transport::cql_server>, std::default_delete<seastar::sharded<cql_transport::cql_server> > > >, seastar::future<void> > + 16
[shard  0] #-4 (task*) 0x00006000019ffe20 0x00000000007ab368 vtable for seastar::(anonymous namespace)::thread_wake_task + 16
[shard  0] #-3 (task*) 0x00006000085ad080 0x0000000000809e18 vtable for seastar::thread_context + 16
[shard  0] #-2 (task*) 0x0000600009c04100 0x00000000006067f8 _ZTVN7seastar12continuationINS_8internal22promise_base_with_typeIvEEZNS_5asyncIZZN7service15storage_service5drainEvENKUlRS6_E_clES7_EUlvE_JEEENS_8futurizeINSt9result_ofIFNSt5decayIT_E4typeEDpNSC_IT0_E4typeEEE4typeEE4typeENS_17thread_attributesEOSD_DpOSG_EUlvE0_ZNS_6futureIvE14then_impl_nrvoIST_SV_EET0_SQ_EUlOS3_RST_ONS_12future_stateINS1_9monostateEEEE_vEE + 16
[shard  0] #-1 (task*) 0x000060000a59c080 0x0000000000606ae8 _ZTVN7seastar12continuationINS_8internal22promise_base_with_typeIvEENS_6futureIvE12finally_bodyIZNS_5asyncIZZN7service15storage_service5drainEvENKUlRS9_E_clESA_EUlvE_JEEENS_8futurizeINSt9result_ofIFNSt5decayIT_E4typeEDpNSF_IT0_E4typeEEE4typeEE4typeENS_17thread_attributesEOSG_DpOSJ_EUlvE1_Lb0EEEZNS5_17then_wrapped_nrvoIS5_SX_EENSD_ISG_E4typeEOT0_EUlOS3_RSX_ONS_12future_stateINS1_9monostateEEEE_vEE + 16
[shard  0] #0  (task*) 0x000060002d650200 0x0000000000606378 vtable for seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::future<void>::finally_body<service::storage_service::run_with_api_lock<service::storage_service::drain()::{lambda(service::storage_service&)#1}>(seastar::basic_sstring<char, unsigned int, 15u, true>, service::storage_service::drain()::{lambda(service::storage_service&)#1}&&)::{lambda(service::storage_service&)#1}::operator()(service::storage_service&)::{lambda()#1}, false>, seastar::future<void>::then_wrapped_nrvo<seastar::future<void>, {lambda(service::storage_service&)#1}>({lambda(service::storage_service&)#1}&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, {lambda(service::storage_service&)#1}&, seastar::future_state<seastar::internal::monostate>&&)#1}, void> + 16
[shard  0] #1  (task*) 0x000060000bc40540 0x0000000000606d48 _ZTVN7seastar12continuationINS_8internal22promise_base_with_typeIvEENS_6futureIvE12finally_bodyIZNS_3smp9submit_toIZNS_7shardedIN7service15storage_serviceEE9invoke_onIZNSB_17run_with_api_lockIZNSB_5drainEvEUlRSB_E_EEDaNS_13basic_sstringIcjLj15ELb1EEEOT_EUlSF_E_JES5_EET1_jNS_21smp_submit_to_optionsESK_DpOT0_EUlvE_EENS_8futurizeINSt9result_ofIFSJ_vEE4typeEE4typeEjSN_SK_EUlvE_Lb0EEEZNS5_17then_wrapped_nrvoIS5_S10_EENSS_ISJ_E4typeEOT0_EUlOS3_RS10_ONS_12future_stateINS1_9monostateEEEE_vEE + 16
[shard  0] #2  (task*) 0x000060000332afc0 0x00000000006cb1c8 vtable for seastar::continuation<seastar::internal::promise_base_with_type<seastar::json::json_return_type>, api::set_storage_service(api::http_context&, seastar::httpd::routes&)::{lambda(std::unique_ptr<seastar::httpd::request, std::default_delete<seastar::httpd::request> >)#38}::operator()(std::unique_ptr<seastar::httpd::request, std::default_delete<seastar::httpd::request> >) const::{lambda()#1}, seastar::future<void>::then_impl_nrvo<{lambda(std::unique_ptr<seastar::httpd::request, std::default_delete<seastar::httpd::request> >)#38}, {lambda()#1}<seastar::json::json_return_type> >({lambda(std::unique_ptr<seastar::httpd::request, std::default_delete<seastar::httpd::request> >)#38}&&)::{lambda(seastar::internal::promise_base_with_type<seastar::json::json_return_type>&&, {lambda(std::unique_ptr<seastar::httpd::request, std::default_delete<seastar::httpd::request> >)#38}&, seastar::future_state<seastar::internal::monostate>&&)#1}, void> + 16
[shard  0] #3  (task*) 0x000060000a1af700 0x0000000000812208 vtable for seastar::continuation<seastar::internal::promise_base_with_type<std::unique_ptr<seastar::httpd::reply, std::default_delete<seastar::httpd::reply> > >, seastar::httpd::function_handler::function_handler(std::function<seastar::future<seastar::json::json_return_type> (std::unique_ptr<seastar::httpd::request, std::default_delete<seastar::httpd::request> >)> const&)::{lambda(std::unique_ptr<seastar::httpd::request, std::default_delete<seastar::httpd::request> >, std::unique_ptr<seastar::httpd::reply, std::default_delete<seastar::httpd::reply> >)#1}::operator()(std::unique_ptr<seastar::httpd::request, std::default_delete<seastar::httpd::request> >, std::unique_ptr<seastar::httpd::reply, std::default_delete<seastar::httpd::reply> >) const::{lambda(seastar::json::json_return_type&&)#1}, seastar::future<seastar::json::json_return_type>::then_impl_nrvo<seastar::json::json_return_type&&, seastar::future<std::unique_ptr<seastar::httpd::reply, std::default_delete<seastar::httpd::reply> > > >(seastar::json::json_return_type&&)::{lambda(seastar::internal::promise_base_with_type<std::unique_ptr<seastar::httpd::reply, std::default_delete<seastar::httpd::reply> > >&&, seastar::json::json_return_type&, seastar::future_state<seastar::json::json_return_type>&&)#1}, seastar::json::json_return_type> + 16
[shard  0] #4  (task*) 0x0000600009d86440 0x0000000000812228 vtable for seastar::continuation<seastar::internal::promise_base_with_type<std::unique_ptr<seastar::httpd::reply, std::default_delete<seastar::httpd::reply> > >, seastar::httpd::function_handler::handle(seastar::basic_sstring<char, unsigned int, 15u, true> const&, std::unique_ptr<seastar::httpd::request, std::default_delete<seastar::httpd::request> >, std::unique_ptr<seastar::httpd::reply, std::default_delete<seastar::httpd::reply> >)::{lambda(std::unique_ptr<seastar::httpd::reply, std::default_delete<seastar::httpd::reply> >)#1}, seastar::future<std::unique_ptr<seastar::httpd::reply, std::default_delete<seastar::httpd::reply> > >::then_impl_nrvo<{lambda(std::unique_ptr<seastar::httpd::reply, std::default_delete<seastar::httpd::reply> >)#1}, seastar::future>({lambda(std::unique_ptr<seastar::httpd::reply, std::default_delete<seastar::httpd::reply> >)#1}&&)::{lambda(seastar::internal::promise_base_with_type<std::unique_ptr<seastar::httpd::reply, std::default_delete<seastar::httpd::reply> > >&&, {lambda(std::unique_ptr<seastar::httpd::reply, std::default_delete<seastar::httpd::reply> >)#1}&, seastar::future_state<std::unique_ptr<seastar::httpd::reply, std::default_delete<seastar::httpd::reply> > >&&)#1}, std::unique_ptr<seastar::httpd::reply, std::default_delete<seastar::httpd::reply> > > + 16
[shard  0] #5  (task*) 0x0000600009dba0c0 0x0000000000812f48 vtable for seastar::continuation<seastar::internal::promise_base_with_type<std::unique_ptr<seastar::httpd::reply, std::default_delete<seastar::httpd::reply> > >, seastar::future<std::unique_ptr<seastar::httpd::reply, std::default_delete<seastar::httpd::reply> > >::handle_exception<std::function<std::unique_ptr<seastar::httpd::reply, std::default_delete<seastar::httpd::reply> > (std::__exception_ptr::exception_ptr)>&>(std::function<std::unique_ptr<seastar::httpd::reply, std::default_delete<seastar::httpd::reply> > (std::__exception_ptr::exception_ptr)>&)::{lambda(auto:1&&)#1}, seastar::future<std::unique_ptr<seastar::httpd::reply, std::default_delete<seastar::httpd::reply> > >::then_wrapped_nrvo<seastar::future<std::unique_ptr<seastar::httpd::reply, std::default_delete<seastar::httpd::reply> > >, {lambda(auto:1&&)#1}>({lambda(auto:1&&)#1}&&)::{lambda(seastar::internal::promise_base_with_type<std::unique_ptr<seastar::httpd::reply, std::default_delete<seastar::httpd::reply> > >&&, {lambda(auto:1&&)#1}&, seastar::future_state<std::unique_ptr<seastar::httpd::reply, std::default_delete<seastar::httpd::reply> > >&&)#1}, std::unique_ptr<seastar::httpd::reply, std::default_delete<seastar::httpd::reply> > > + 16
[shard  0] #6  (task*) 0x0000600026783ae0 0x00000000008118b0 vtable for seastar::continuation<seastar::internal::promise_base_with_type<bool>, seastar::httpd::connection::generate_reply(std::unique_ptr<seastar::httpd::request, std::default_delete<seastar::httpd::request> >)::{lambda(std::unique_ptr<seastar::httpd::reply, std::default_delete<seastar::httpd::reply> >)#1}, seastar::future<std::unique_ptr<seastar::httpd::reply, std::default_delete<seastar::httpd::reply> > >::then_impl_nrvo<{lambda(std::unique_ptr<seastar::httpd::reply, std::default_delete<seastar::httpd::reply> >)#1}, seastar::httpd::connection::generate_reply(std::unique_ptr<seastar::httpd::request, std::default_delete<seastar::httpd::request> >)::{lambda(std::unique_ptr<seastar::httpd::reply, std::default_delete<seastar::httpd::reply> >)#1}<bool> >({lambda(std::unique_ptr<seastar::httpd::reply, std::default_delete<seastar::httpd::reply> >)#1}&&)::{lambda(seastar::internal::promise_base_with_type<bool>&&, {lambda(std::unique_ptr<seastar::httpd::reply, std::default_delete<seastar::httpd::reply> >)#1}&, seastar::future_state<std::unique_ptr<seastar::httpd::reply, std::default_delete<seastar::httpd::reply> > >&&)#1}, std::unique_ptr<seastar::httpd::reply, std::default_delete<seastar::httpd::reply> > > + 16
[shard  0] #7  (task*) 0x000060000a4089c0 0x0000000000811790 vtable for seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::httpd::connection::read_one()::{lambda()#1}::operator()()::{lambda(std::unique_ptr<seastar::httpd::request, std::default_delete<std::unique_ptr> >)#2}::operator()(std::default_delete<std::unique_ptr>) const::{lambda(std::default_delete<std::unique_ptr>)#1}::operator()(std::default_delete<std::unique_ptr>) const::{lambda(bool)#2}, seastar::future<bool>::then_impl_nrvo<{lambda(std::unique_ptr<seastar::httpd::request, std::default_delete<std::unique_ptr> >)#2}, {lambda(std::default_delete<std::unique_ptr>)#1}<void> >({lambda(std::unique_ptr<seastar::httpd::request, std::default_delete<std::unique_ptr> >)#2}&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, {lambda(std::unique_ptr<seastar::httpd::request, std::default_delete<std::unique_ptr> >)#2}&, seastar::future_state<bool>&&)#1}, bool> + 16
[shard  0] #8  (task*) 0x000060000a5b16e0 0x0000000000811430 vtable for seastar::internal::do_until_state<seastar::httpd::connection::read()::{lambda()#1}, seastar::httpd::connection::read()::{lambda()#2}> + 16
[shard  0] #9  (task*) 0x000060000aec1080 0x00000000008116d0 vtable for seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::httpd::connection::read()::{lambda(seastar::future<void>)#3}, seastar::future<void>::then_wrapped_nrvo<seastar::future<void>, {lambda(seastar::future<void>)#3}>({lambda(seastar::future<void>)#3}&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, {lambda(seastar::future<void>)#3}&, seastar::future_state<seastar::internal::monostate>&&)#1}, void> + 16
[shard  0] #10 (task*) 0x000060000b7d2900 0x0000000000811950 vtable for seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::future<void>::finally_body<seastar::httpd::connection::read()::{lambda()#4}, true>, seastar::future<void>::then_wrapped_nrvo<seastar::future<void>, seastar::httpd::connection::read()::{lambda()#4}>(seastar::httpd::connection::read()::{lambda()#4}&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::httpd::connection::read()::{lambda()#4}&, seastar::future_state<seastar::internal::monostate>&&)#1}, void> + 16

Found no further pointers to task objects.
If you think there should be more, run `scylla fiber 0x000060002d650200 --verbose` to learn more.
Note that continuation across user-created seastar::promise<> objects are not detected by scylla-fiber.
```

Closes #11822

* github.com:scylladb/scylladb:
  scylla-gdb.py: collection_element: add support for boost::intrusive::list
  scylla-gdb.py: optional_printer: eliminate infinite loop
  scylla-gdb.py: scylla-fiber: add note about user-instantiated promise objects
  scylla-gdb.py: scylla-fiber: reject self-references when probing pointers
  scylla-gdb.py: scylla-fiber: add starting task to known tasks
  scylla-gdb.py: scylla-fiber: add support for walking over when_all
  scylla-gdb.py: add when_all_state to task type whitelist
  scylla-gdb.py: scylla-fiber: also print shard of tasks
  scylla-gdb.py: scylla-fiber: unify task printing
  scylla-gdb.py: scylla fiber: add support for walking over shards
  scylla-gdb.py: scylla fiber: add support for walking over seastar threads
  scylla-gdb.py: scylla-ptr: keep current thread context
  scylla-gdb.py: improve scylla column_families
  scylla-gdb.py: scylla_sstables.filename(): fix generation formatting
  scylla-gdb.py: improve schema_ptr
  scylla-gdb.py: scylla memory: restore compatibility with <= 5.1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant