Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Result of token() is showed by cqlsh as integer, whereas on urchin it's showed as bytes #20

Closed
tgrabiec opened this issue Jul 13, 2015 · 1 comment
Closed

Result of token() is showed by cqlsh as integer, whereas on urchin it's showed as bytes #20

tgrabiec opened this issue Jul 13, 2015 · 1 comment

Comments

@tgrabiec
Copy link
Contributor 

cqlsh> create table cf (k blob, v int, primary key (k));
cqlsh> insert into cf (k, v) values (0x01, 0);

On Origin:

cqlsh> select k, token(k) from cf;

 k    | system.token(k)
------+---------------------
 0x01 | 8849112093580131862

On urchin:

cqlsh> select k, token(k) from cf;

 k    | system.token(k)
------+--------------------
 0x02 | 0x635e0e9e37361ef2
@slivne slivne mentioned this issue Jul 9, 2015
Closed
@slivne slivne mentioned this issue Jul 30, 2015
Closed
@slivne slivne mentioned this issue Aug 10, 2015
Closed
@avikivity
Copy link
Member

Fixed by 14ad301

@slivne slivne mentioned this issue Sep 7, 2015
Closed
@gleb-cloudius gleb-cloudius mentioned this issue Sep 8, 2015
Closed
@vladzcloudius vladzcloudius mentioned this issue Sep 17, 2015
Closed
@gleb-cloudius gleb-cloudius mentioned this issue Sep 29, 2015
Closed
@avikivity avikivity mentioned this issue Nov 5, 2015
Closed
penberg pushed a commit that referenced this issue Dec 3, 2015
@asias @penberg
storage_service: Fix debug build
dcb9b44 
Start non-seed with debug build I saw:

==9844==WARNING: ASan is ignoring requested __asan_handle_no_return:
stack top: 0x7ffdabd73000; bottom 0x7fe309218000; size: 0x001aa2b5b000 (114398965760)
False positive error reports may follow
For details see http://code.google.com/p/address-sanitizer/issues/detail?id=189
DEBUG [shard 0] storage_service - Starting shadow gossip round to check for endpoint collision
=================================================================
==9844==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7fe309219ad0 at pc 0x00000495a88e bp 0x7fe309219960 sp 0x7fe309219950
WRITE of size 8 at 0x7fe309219ad0 thread T0
    #0 0x495a88d in _Head_base<seastar::async(Func&&, Args&& ...)
       [with Func = service::storage_service::check_for_endpoint_collision()::<lambda()>; Args = {};
       futurize_t<typename std::result_of<typename std::decay<_Tp>::type(std::decay_t<Args>...)>::type> = future<>]::work*>
       /usr/include/c++/5.1.1/tuple:115
    #1 0x495a993 in _Tuple_impl<seastar::async(Func&&, Args&& ...)
       [with Func = service::storage_service::check_for_endpoint_collision()::<lambda()>; Args = {};
       futurize_t<typename std::result_of<typename std::decay<_Tp>::type(std::decay_t<Args>...)>::type> = future<>]::work*,
       std::default_delete<seastar::async(Func&&, Args&& ...) [with Func = service::storage_service::check_for_endpoint_collision()::<lambda()>;
       Args = {}; futurize_t<typename std::result_of<typename std::decay<_Tp>::type(std::decay_t<Args>...)>::type> = future<>]::work>, void>
       /usr/include/c++/5.1.1/tuple:213
    #2 0x495aa73 in tuple<seastar::async(Func&&, Args&& ...)
       [with Func = service::storage_service::check_for_endpoint_collision()::<lambda()>; Args = {};
       futurize_t<typename std::result_of<typename std::decay<_Tp>::type(std::decay_t<Args>...)>::type> = future<>]::work*,
       std::default_delete<seastar::async(Func&&, Args&& ...)
       [with Func = service::storage_service::check_for_endpoint_collision()::<lambda()>; Args = {};
       futurize_t<typename std::result_of<typename std::decay<_Tp>::type(std::decay_t<Args>...)>::type> = future<>]::work>, void>
       /usr/include/c++/5.1.1/tuple:613
    #3 0x495ab82 in unique_ptr /usr/include/c++/5.1.1/bits/unique_ptr.h:206
    ...
    #16 0x4d44c8e in _M_invoke /usr/include/c++/5.1.1/functional:1871
    #17 0x5d2fb7 in std::function<void ()>::operator()() const /usr/include/c++/5.1.1/functional:2271
    #18 0x8a1e70 in seastar::thread_context::main() core/thread.cc:139
    #19 0x8a1d89 in seastar::thread_context::s_main(unsigned int, unsigned int) core/thread.cc:130
    #20 0x7fe311b6cf0f  (/lib64/libc.so.6+0x48f0f)

I'm not sure why this patch helps. Perhaps the exception makes ASAN unhappy.

Anyway, this patch makes the debug build work again.

Fixes #613.
@nirmaayan nirmaayan mentioned this issue Aug 1, 2016
Closed
@slivne slivne mentioned this issue Aug 8, 2016
Closed
@vladzcloudius vladzcloudius mentioned this issue Sep 19, 2016
Closed
@gleb-cloudius gleb-cloudius mentioned this issue Sep 20, 2016
Closed
This was referenced Sep 26, 2016
Closed
Closed
@GpStore GpStore mentioned this issue Nov 10, 2016
Closed
@roydahan roydahan mentioned this issue Jan 18, 2017
Closed
@slivne slivne mentioned this issue Jul 3, 2017
Closed
@frank8989 frank8989 mentioned this issue Oct 26, 2017
Closed
@glommer glommer mentioned this issue Nov 14, 2017
Closed
@Leroy8508 Leroy8508 mentioned this issue Jan 11, 2018
Closed
@frank8989 frank8989 mentioned this issue Mar 13, 2018
Closed
@bhalevy bhalevy mentioned this issue Jan 20, 2019
Closed
@tgrabiec tgrabiec mentioned this issue Sep 24, 2019
Closed
1 task
bhalevy added a commit to bhalevy/scylla that referenced this issue Mar 10, 2020
@bhalevy
distributed_loader::do_populate_column_family: do_with verifier
f9c18b1 
Rather than using lw_shared_ptr to keep it around, use do_with.
Fixes a use-after-free seen with interrupt_build_process_with_resharding_half_to_max_test
in debug mode:

==406079==ERROR: AddressSanitizer: heap-use-after-free on address 0x603001320ac0 at pc 0x00000777a062 bp 0x7ffdeb77b010 sp 0x7ffdeb77b000
READ of size 8 at 0x603001320ac0 thread T0
    #0 0x777a061 in _M_next /usr/include/c++/9/bits/hashtable_policy.h:283
    scylladb#1 0x779ead7 in _M_incr /usr/include/c++/9/bits/hashtable_policy.h:299
    scylladb#2 0x777bbac in operator++ /usr/include/c++/9/bits/hashtable_policy.h:362
    scylladb#3 0x77f8384 in run_and_dispose /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future-util.hh:568
    scylladb#4 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091
    scylladb#5 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506
    scylladb#6 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666
    scylladb#7 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199
    scylladb#8 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115
    scylladb#9 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486
    scylladb#10 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2)
    scylladb#11 0x42b69d in _start (/local/home/bhalevy/.dtest/dtest-q85idok7/test/node1/bin/scylla+0x42b69d)

0x603001320ac0 is located 0 bytes inside of 32-byte region [0x603001320ac0,0x603001320ae0)
freed by thread T0 here:
    #0 0x7fb86520996f in operator delete(void*) (/lib64/libasan.so.5+0x11096f)
    scylladb#1 0x77d1889 in deallocate /usr/include/c++/9/ext/new_allocator.h:128
    scylladb#2 0x77bcbb2 in deallocate /usr/include/c++/9/bits/alloc_traits.h:470
    scylladb#3 0x77bccc5 in _M_deallocate_node_ptr /usr/include/c++/9/bits/hashtable_policy.h:2113
    scylladb#4 0x779dc59 in _M_deallocate_node /usr/include/c++/9/bits/hashtable_policy.h:2103
    scylladb#5 0x77bc8b6 in _M_deallocate_nodes /usr/include/c++/9/bits/hashtable_policy.h:2124
    scylladb#6 0x779d18f in clear /usr/include/c++/9/bits/hashtable.h:2028
    scylladb#7 0x7779b8e in ~_Hashtable /usr/include/c++/9/bits/hashtable.h:1352
    scylladb#8 0x774cfed in ~unordered_map /usr/include/c++/9/bits/unordered_map.h:102
    scylladb#9 0x774d288 in ~shared_ptr_no_esft /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:157
    scylladb#10 0x774d2f1 in dispose /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:213
    scylladb#11 0x76f9642 in ~lw_shared_ptr /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:300
    scylladb#12 0x76f9642 in ~<lambda> /local/home/bhalevy/dev/scylla/distributed_loader.cc:743
    scylladb#13 0x771e0a7 in ~<lambda> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1167
    scylladb#14 0x777c64c in destroy /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:133
    scylladb#15 0x586059 in seastar::noncopyable_function<seastar::future<> ()>::~noncopyable_function() /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:165
    scylladb#16 0x5867da in seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}::~future_state() /local/home/bha
levy/dev/scylla/seastar/include/seastar/core/future.hh:1192
    scylladb#17 0xab704d in seastar::continuation<seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}>::~continu
ation() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:507
    scylladb#18 0xab7216 in seastar::continuation<seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}>::run_and_
dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:512
    scylladb#19 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091
    scylladb#20 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506
    scylladb#21 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666
    scylladb#22 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199
    scylladb#23 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115
    scylladb#24 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486
    scylladb#25 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2)

previously allocated by thread T0 here:
    #0 0x7fb8652089d7 in operator new(unsigned long) (/lib64/libasan.so.5+0x10f9d7)
    scylladb#1 0x77d1796 in allocate /usr/include/c++/9/ext/new_allocator.h:114
    scylladb#2 0x77bcab3 in allocate /usr/include/c++/9/bits/alloc_traits.h:444
    scylladb#3 0x779d936 in _M_allocate_node<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable_policy.h:2081
    scylladb#4 0x777a2ed in _M_emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable.h:1638
    scylladb#5 0x774d707 in emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable.h:749
    scylladb#6 0x771ce3e in emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/unordered_map.h:388
    scylladb#7 0x76f8280 in operator() /local/home/bhalevy/dev/scylla/distributed_loader.cc:732
    scylladb#8 0x774dc5d in _M_invoke /usr/include/c++/9/bits/std_function.h:286
    scylladb#9 0x6cde67b in std::function<seastar::future<> (std::filesystem::__cxx11::path, seastar::directory_entry)>::operator()(std::filesystem::__cxx11::path, seastar::directory_entry) const /usr/include/c++/9/bits/std_function.h:690
    scylladb#10 0x6cd345d in operator() /local/home/bhalevy/dev/scylla/lister.cc:30
    scylladb#11 0x6cd96e0 in apply /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36
    scylladb#12 0x6cd9854 in apply<lister::visit(seastar::directory_entry)::<lambda(seastar::directory_entry)>&, seastar::directory_entry&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:44
    scylladb#13 0x6cd9953 in apply<lister::visit(seastar::directory_entry)::<lambda(seastar::directory_entry)>&, seastar::directory_entry&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1650
    scylladb#14 0x6cd6ba9 in operator() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1168
    scylladb#15 0x6cdbac0 in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:101
    scylladb#16 0x6ce6178 in seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>::operator()(seastar::directory_entry&&) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:184
    scylladb#17 0x6ce3033 in seastar::apply_helper<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, std::tuple<seastar::directory_entry>&&, std::integer_sequence<unsigned long, 0ul> >::apply(seastar::noncopyable_function<seastar::future<> (seastar::directory_e
ntry&&)>&&, std::tuple<seastar::directory_entry>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36
    scylladb#18 0x6ce3130 in auto seastar::apply<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::directory_entry>(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&, std::tuple<seastar::directory_entry>&&) /local/home/bha
levy/dev/scylla/seastar/include/seastar/core/apply.hh:44
    scylladb#19 0x6ce322f in seastar::future<> seastar::futurize<seastar::future<> >::apply<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::directory_entry>(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&, std::tuple<s
eastar::directory_entry>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1650
    scylladb#20 0x6cdf88d in seastar::future<seastar::directory_entry>::then_impl<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&)::{lambda()scylladb#1}::operator()()
const::{lambda(seastar::future_state<seastar::directory_entry>&&)scylladb#1}::operator()(seastar::future_state<seastar::directory_entry>) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1196
    scylladb#21 0x6cea2e0 in seastar::continuation<seastar::future<seastar::directory_entry>::then_impl<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&)::{lamb
da()scylladb#1}::operator()() const::{lambda(seastar::future_state<seastar::directory_entry>&&)scylladb#1}, seastar::directory_entry>::run_and_dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:511
    scylladb#22 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091
    scylladb#23 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506
    scylladb#24 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666
    scylladb#25 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199
    scylladb#26 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115
    scylladb#27 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486
    scylladb#28 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2)

SUMMARY: AddressSanitizer: heap-use-after-free /usr/include/c++/9/bits/hashtable_policy.h:283 in _M_next
Shadow bytes around the buggy address:
  0x0c068025c100: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c068025c110: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa
  0x0c068025c120: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd
  0x0c068025c130: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c068025c140: fd fd fd fa fa fa 00 00 00 fa fa fa fd fd fd fa
=>0x0c068025c150: fa fa fd fd fd fa fa fa[fd]fd fd fd fa fa fd fd
  0x0c068025c160: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c068025c170: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa
  0x0c068025c180: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd
  0x0c068025c190: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c068025c1a0: fd fd fd fd fa fa fd fd fd fd fa fa 00 00 00 fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==406079==ABORTING

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
bhalevy added a commit to bhalevy/scylla that referenced this issue Mar 11, 2020
@bhalevy
distributed_loader::do_populate_column_family: do_with verifier
7eb4e44 
Rather than using lw_shared_ptr to keep it around, use do_with.
Fixes a use-after-free seen with interrupt_build_process_with_resharding_half_to_max_test
in debug mode:

==406079==ERROR: AddressSanitizer: heap-use-after-free on address 0x603001320ac0 at pc 0x00000777a062 bp 0x7ffdeb77b010 sp 0x7ffdeb77b000
READ of size 8 at 0x603001320ac0 thread T0
    #0 0x777a061 in _M_next /usr/include/c++/9/bits/hashtable_policy.h:283
    scylladb#1 0x779ead7 in _M_incr /usr/include/c++/9/bits/hashtable_policy.h:299
    scylladb#2 0x777bbac in operator++ /usr/include/c++/9/bits/hashtable_policy.h:362
    scylladb#3 0x77f8384 in run_and_dispose /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future-util.hh:568
    scylladb#4 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091
    scylladb#5 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506
    scylladb#6 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666
    scylladb#7 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199
    scylladb#8 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115
    scylladb#9 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486
    scylladb#10 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2)
    scylladb#11 0x42b69d in _start (/local/home/bhalevy/.dtest/dtest-q85idok7/test/node1/bin/scylla+0x42b69d)

0x603001320ac0 is located 0 bytes inside of 32-byte region [0x603001320ac0,0x603001320ae0)
freed by thread T0 here:
    #0 0x7fb86520996f in operator delete(void*) (/lib64/libasan.so.5+0x11096f)
    scylladb#1 0x77d1889 in deallocate /usr/include/c++/9/ext/new_allocator.h:128
    scylladb#2 0x77bcbb2 in deallocate /usr/include/c++/9/bits/alloc_traits.h:470
    scylladb#3 0x77bccc5 in _M_deallocate_node_ptr /usr/include/c++/9/bits/hashtable_policy.h:2113
    scylladb#4 0x779dc59 in _M_deallocate_node /usr/include/c++/9/bits/hashtable_policy.h:2103
    scylladb#5 0x77bc8b6 in _M_deallocate_nodes /usr/include/c++/9/bits/hashtable_policy.h:2124
    scylladb#6 0x779d18f in clear /usr/include/c++/9/bits/hashtable.h:2028
    scylladb#7 0x7779b8e in ~_Hashtable /usr/include/c++/9/bits/hashtable.h:1352
    scylladb#8 0x774cfed in ~unordered_map /usr/include/c++/9/bits/unordered_map.h:102
    scylladb#9 0x774d288 in ~shared_ptr_no_esft /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:157
    scylladb#10 0x774d2f1 in dispose /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:213
    scylladb#11 0x76f9642 in ~lw_shared_ptr /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:300
    scylladb#12 0x76f9642 in ~<lambda> /local/home/bhalevy/dev/scylla/distributed_loader.cc:743
    scylladb#13 0x771e0a7 in ~<lambda> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1167
    scylladb#14 0x777c64c in destroy /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:133
    scylladb#15 0x586059 in seastar::noncopyable_function<seastar::future<> ()>::~noncopyable_function() /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:165
    scylladb#16 0x5867da in seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}::~future_state() /local/home/bha
levy/dev/scylla/seastar/include/seastar/core/future.hh:1192
    scylladb#17 0xab704d in seastar::continuation<seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}>::~continu
ation() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:507
    scylladb#18 0xab7216 in seastar::continuation<seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}>::run_and_
dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:512
    scylladb#19 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091
    scylladb#20 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506
    scylladb#21 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666
    scylladb#22 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199
    scylladb#23 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115
    scylladb#24 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486
    scylladb#25 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2)

previously allocated by thread T0 here:
    #0 0x7fb8652089d7 in operator new(unsigned long) (/lib64/libasan.so.5+0x10f9d7)
    scylladb#1 0x77d1796 in allocate /usr/include/c++/9/ext/new_allocator.h:114
    scylladb#2 0x77bcab3 in allocate /usr/include/c++/9/bits/alloc_traits.h:444
    scylladb#3 0x779d936 in _M_allocate_node<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable_policy.h:2081
    scylladb#4 0x777a2ed in _M_emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable.h:1638
    scylladb#5 0x774d707 in emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable.h:749
    scylladb#6 0x771ce3e in emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/unordered_map.h:388
    scylladb#7 0x76f8280 in operator() /local/home/bhalevy/dev/scylla/distributed_loader.cc:732
    scylladb#8 0x774dc5d in _M_invoke /usr/include/c++/9/bits/std_function.h:286
    scylladb#9 0x6cde67b in std::function<seastar::future<> (std::filesystem::__cxx11::path, seastar::directory_entry)>::operator()(std::filesystem::__cxx11::path, seastar::directory_entry) const /usr/include/c++/9/bits/std_function.h:690
    scylladb#10 0x6cd345d in operator() /local/home/bhalevy/dev/scylla/lister.cc:30
    scylladb#11 0x6cd96e0 in apply /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36
    scylladb#12 0x6cd9854 in apply<lister::visit(seastar::directory_entry)::<lambda(seastar::directory_entry)>&, seastar::directory_entry&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:44
    scylladb#13 0x6cd9953 in apply<lister::visit(seastar::directory_entry)::<lambda(seastar::directory_entry)>&, seastar::directory_entry&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1650
    scylladb#14 0x6cd6ba9 in operator() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1168
    scylladb#15 0x6cdbac0 in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:101
    scylladb#16 0x6ce6178 in seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>::operator()(seastar::directory_entry&&) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:184
    scylladb#17 0x6ce3033 in seastar::apply_helper<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, std::tuple<seastar::directory_entry>&&, std::integer_sequence<unsigned long, 0ul> >::apply(seastar::noncopyable_function<seastar::future<> (seastar::directory_e
ntry&&)>&&, std::tuple<seastar::directory_entry>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36
    scylladb#18 0x6ce3130 in auto seastar::apply<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::directory_entry>(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&, std::tuple<seastar::directory_entry>&&) /local/home/bha
levy/dev/scylla/seastar/include/seastar/core/apply.hh:44
    scylladb#19 0x6ce322f in seastar::future<> seastar::futurize<seastar::future<> >::apply<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::directory_entry>(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&, std::tuple<s
eastar::directory_entry>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1650
    scylladb#20 0x6cdf88d in seastar::future<seastar::directory_entry>::then_impl<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&)::{lambda()scylladb#1}::operator()()
const::{lambda(seastar::future_state<seastar::directory_entry>&&)scylladb#1}::operator()(seastar::future_state<seastar::directory_entry>) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1196
    scylladb#21 0x6cea2e0 in seastar::continuation<seastar::future<seastar::directory_entry>::then_impl<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&)::{lamb
da()scylladb#1}::operator()() const::{lambda(seastar::future_state<seastar::directory_entry>&&)scylladb#1}, seastar::directory_entry>::run_and_dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:511
    scylladb#22 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091
    scylladb#23 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506
    scylladb#24 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666
    scylladb#25 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199
    scylladb#26 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115
    scylladb#27 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486
    scylladb#28 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2)

SUMMARY: AddressSanitizer: heap-use-after-free /usr/include/c++/9/bits/hashtable_policy.h:283 in _M_next
Shadow bytes around the buggy address:
  0x0c068025c100: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c068025c110: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa
  0x0c068025c120: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd
  0x0c068025c130: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c068025c140: fd fd fd fa fa fa 00 00 00 fa fa fa fd fd fd fa
=>0x0c068025c150: fa fa fd fd fd fa fa fa[fd]fd fd fd fa fa fd fd
  0x0c068025c160: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c068025c170: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa
  0x0c068025c180: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd
  0x0c068025c190: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c068025c1a0: fd fd fd fd fa fa fd fd fd fd fa fa 00 00 00 fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==406079==ABORTING

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
bhalevy added a commit to bhalevy/scylla that referenced this issue Apr 13, 2020
@bhalevy
distributed_loader::do_populate_column_family: do_with verifier
7f7d4b6 
Rather than using lw_shared_ptr to keep it around, use do_with.
Fixes a use-after-free seen with interrupt_build_process_with_resharding_half_to_max_test
in debug mode:

==406079==ERROR: AddressSanitizer: heap-use-after-free on address 0x603001320ac0 at pc 0x00000777a062 bp 0x7ffdeb77b010 sp 0x7ffdeb77b000
READ of size 8 at 0x603001320ac0 thread T0
    #0 0x777a061 in _M_next /usr/include/c++/9/bits/hashtable_policy.h:283
    scylladb#1 0x779ead7 in _M_incr /usr/include/c++/9/bits/hashtable_policy.h:299
    scylladb#2 0x777bbac in operator++ /usr/include/c++/9/bits/hashtable_policy.h:362
    scylladb#3 0x77f8384 in run_and_dispose /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future-util.hh:568
    scylladb#4 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091
    scylladb#5 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506
    scylladb#6 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666
    scylladb#7 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199
    scylladb#8 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115
    scylladb#9 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486
    scylladb#10 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2)
    scylladb#11 0x42b69d in _start (/local/home/bhalevy/.dtest/dtest-q85idok7/test/node1/bin/scylla+0x42b69d)

0x603001320ac0 is located 0 bytes inside of 32-byte region [0x603001320ac0,0x603001320ae0)
freed by thread T0 here:
    #0 0x7fb86520996f in operator delete(void*) (/lib64/libasan.so.5+0x11096f)
    scylladb#1 0x77d1889 in deallocate /usr/include/c++/9/ext/new_allocator.h:128
    scylladb#2 0x77bcbb2 in deallocate /usr/include/c++/9/bits/alloc_traits.h:470
    scylladb#3 0x77bccc5 in _M_deallocate_node_ptr /usr/include/c++/9/bits/hashtable_policy.h:2113
    scylladb#4 0x779dc59 in _M_deallocate_node /usr/include/c++/9/bits/hashtable_policy.h:2103
    scylladb#5 0x77bc8b6 in _M_deallocate_nodes /usr/include/c++/9/bits/hashtable_policy.h:2124
    scylladb#6 0x779d18f in clear /usr/include/c++/9/bits/hashtable.h:2028
    scylladb#7 0x7779b8e in ~_Hashtable /usr/include/c++/9/bits/hashtable.h:1352
    scylladb#8 0x774cfed in ~unordered_map /usr/include/c++/9/bits/unordered_map.h:102
    scylladb#9 0x774d288 in ~shared_ptr_no_esft /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:157
    scylladb#10 0x774d2f1 in dispose /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:213
    scylladb#11 0x76f9642 in ~lw_shared_ptr /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:300
    scylladb#12 0x76f9642 in ~<lambda> /local/home/bhalevy/dev/scylla/distributed_loader.cc:743
    scylladb#13 0x771e0a7 in ~<lambda> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1167
    scylladb#14 0x777c64c in destroy /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:133
    scylladb#15 0x586059 in seastar::noncopyable_function<seastar::future<> ()>::~noncopyable_function() /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:165
    scylladb#16 0x5867da in seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}::~future_state() /local/home/bha
levy/dev/scylla/seastar/include/seastar/core/future.hh:1192
    scylladb#17 0xab704d in seastar::continuation<seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}>::~continu
ation() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:507
    scylladb#18 0xab7216 in seastar::continuation<seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}>::run_and_
dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:512
    scylladb#19 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091
    scylladb#20 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506
    scylladb#21 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666
    scylladb#22 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199
    scylladb#23 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115
    scylladb#24 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486
    scylladb#25 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2)

previously allocated by thread T0 here:
    #0 0x7fb8652089d7 in operator new(unsigned long) (/lib64/libasan.so.5+0x10f9d7)
    scylladb#1 0x77d1796 in allocate /usr/include/c++/9/ext/new_allocator.h:114
    scylladb#2 0x77bcab3 in allocate /usr/include/c++/9/bits/alloc_traits.h:444
    scylladb#3 0x779d936 in _M_allocate_node<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable_policy.h:2081
    scylladb#4 0x777a2ed in _M_emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable.h:1638
    scylladb#5 0x774d707 in emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable.h:749
    scylladb#6 0x771ce3e in emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/unordered_map.h:388
    scylladb#7 0x76f8280 in operator() /local/home/bhalevy/dev/scylla/distributed_loader.cc:732
    scylladb#8 0x774dc5d in _M_invoke /usr/include/c++/9/bits/std_function.h:286
    scylladb#9 0x6cde67b in std::function<seastar::future<> (std::filesystem::__cxx11::path, seastar::directory_entry)>::operator()(std::filesystem::__cxx11::path, seastar::directory_entry) const /usr/include/c++/9/bits/std_function.h:690
    scylladb#10 0x6cd345d in operator() /local/home/bhalevy/dev/scylla/lister.cc:30
    scylladb#11 0x6cd96e0 in apply /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36
    scylladb#12 0x6cd9854 in apply<lister::visit(seastar::directory_entry)::<lambda(seastar::directory_entry)>&, seastar::directory_entry&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:44
    scylladb#13 0x6cd9953 in apply<lister::visit(seastar::directory_entry)::<lambda(seastar::directory_entry)>&, seastar::directory_entry&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1650
    scylladb#14 0x6cd6ba9 in operator() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1168
    scylladb#15 0x6cdbac0 in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:101
    scylladb#16 0x6ce6178 in seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>::operator()(seastar::directory_entry&&) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:184
    scylladb#17 0x6ce3033 in seastar::apply_helper<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, std::tuple<seastar::directory_entry>&&, std::integer_sequence<unsigned long, 0ul> >::apply(seastar::noncopyable_function<seastar::future<> (seastar::directory_e
ntry&&)>&&, std::tuple<seastar::directory_entry>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36
    scylladb#18 0x6ce3130 in auto seastar::apply<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::directory_entry>(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&, std::tuple<seastar::directory_entry>&&) /local/home/bha
levy/dev/scylla/seastar/include/seastar/core/apply.hh:44
    scylladb#19 0x6ce322f in seastar::future<> seastar::futurize<seastar::future<> >::apply<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::directory_entry>(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&, std::tuple<s
eastar::directory_entry>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1650
    scylladb#20 0x6cdf88d in seastar::future<seastar::directory_entry>::then_impl<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&)::{lambda()scylladb#1}::operator()()
const::{lambda(seastar::future_state<seastar::directory_entry>&&)scylladb#1}::operator()(seastar::future_state<seastar::directory_entry>) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1196
    scylladb#21 0x6cea2e0 in seastar::continuation<seastar::future<seastar::directory_entry>::then_impl<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&)::{lamb
da()scylladb#1}::operator()() const::{lambda(seastar::future_state<seastar::directory_entry>&&)scylladb#1}, seastar::directory_entry>::run_and_dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:511
    scylladb#22 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091
    scylladb#23 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506
    scylladb#24 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666
    scylladb#25 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199
    scylladb#26 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115
    scylladb#27 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486
    scylladb#28 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2)

SUMMARY: AddressSanitizer: heap-use-after-free /usr/include/c++/9/bits/hashtable_policy.h:283 in _M_next
Shadow bytes around the buggy address:
  0x0c068025c100: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c068025c110: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa
  0x0c068025c120: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd
  0x0c068025c130: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c068025c140: fd fd fd fa fa fa 00 00 00 fa fa fa fd fd fd fa
=>0x0c068025c150: fa fa fd fd fd fa fa fa[fd]fd fd fd fa fa fd fd
  0x0c068025c160: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c068025c170: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa
  0x0c068025c180: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd
  0x0c068025c190: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c068025c1a0: fd fd fd fd fa fa fd fd fd fd fa fa 00 00 00 fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==406079==ABORTING

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
bhalevy added a commit to bhalevy/scylla that referenced this issue Apr 17, 2020
@bhalevy
distributed_loader::do_populate_column_family: do_with verifier
2d255d0 
Rather than using lw_shared_ptr to keep it around, use do_with.
Fixes a use-after-free seen with interrupt_build_process_with_resharding_half_to_max_test
in debug mode:

==406079==ERROR: AddressSanitizer: heap-use-after-free on address 0x603001320ac0 at pc 0x00000777a062 bp 0x7ffdeb77b010 sp 0x7ffdeb77b000
READ of size 8 at 0x603001320ac0 thread T0
    #0 0x777a061 in _M_next /usr/include/c++/9/bits/hashtable_policy.h:283
    scylladb#1 0x779ead7 in _M_incr /usr/include/c++/9/bits/hashtable_policy.h:299
    scylladb#2 0x777bbac in operator++ /usr/include/c++/9/bits/hashtable_policy.h:362
    scylladb#3 0x77f8384 in run_and_dispose /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future-util.hh:568
    scylladb#4 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091
    scylladb#5 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506
    scylladb#6 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666
    scylladb#7 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199
    scylladb#8 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115
    scylladb#9 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486
    scylladb#10 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2)
    scylladb#11 0x42b69d in _start (/local/home/bhalevy/.dtest/dtest-q85idok7/test/node1/bin/scylla+0x42b69d)

0x603001320ac0 is located 0 bytes inside of 32-byte region [0x603001320ac0,0x603001320ae0)
freed by thread T0 here:
    #0 0x7fb86520996f in operator delete(void*) (/lib64/libasan.so.5+0x11096f)
    scylladb#1 0x77d1889 in deallocate /usr/include/c++/9/ext/new_allocator.h:128
    scylladb#2 0x77bcbb2 in deallocate /usr/include/c++/9/bits/alloc_traits.h:470
    scylladb#3 0x77bccc5 in _M_deallocate_node_ptr /usr/include/c++/9/bits/hashtable_policy.h:2113
    scylladb#4 0x779dc59 in _M_deallocate_node /usr/include/c++/9/bits/hashtable_policy.h:2103
    scylladb#5 0x77bc8b6 in _M_deallocate_nodes /usr/include/c++/9/bits/hashtable_policy.h:2124
    scylladb#6 0x779d18f in clear /usr/include/c++/9/bits/hashtable.h:2028
    scylladb#7 0x7779b8e in ~_Hashtable /usr/include/c++/9/bits/hashtable.h:1352
    scylladb#8 0x774cfed in ~unordered_map /usr/include/c++/9/bits/unordered_map.h:102
    scylladb#9 0x774d288 in ~shared_ptr_no_esft /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:157
    scylladb#10 0x774d2f1 in dispose /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:213
    scylladb#11 0x76f9642 in ~lw_shared_ptr /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:300
    scylladb#12 0x76f9642 in ~<lambda> /local/home/bhalevy/dev/scylla/distributed_loader.cc:743
    scylladb#13 0x771e0a7 in ~<lambda> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1167
    scylladb#14 0x777c64c in destroy /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:133
    scylladb#15 0x586059 in seastar::noncopyable_function<seastar::future<> ()>::~noncopyable_function() /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:165
    scylladb#16 0x5867da in seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}::~future_state() /local/home/bha
levy/dev/scylla/seastar/include/seastar/core/future.hh:1192
    scylladb#17 0xab704d in seastar::continuation<seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}>::~continu
ation() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:507
    scylladb#18 0xab7216 in seastar::continuation<seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}>::run_and_
dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:512
    scylladb#19 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091
    scylladb#20 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506
    scylladb#21 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666
    scylladb#22 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199
    scylladb#23 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115
    scylladb#24 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486
    scylladb#25 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2)

previously allocated by thread T0 here:
    #0 0x7fb8652089d7 in operator new(unsigned long) (/lib64/libasan.so.5+0x10f9d7)
    scylladb#1 0x77d1796 in allocate /usr/include/c++/9/ext/new_allocator.h:114
    scylladb#2 0x77bcab3 in allocate /usr/include/c++/9/bits/alloc_traits.h:444
    scylladb#3 0x779d936 in _M_allocate_node<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable_policy.h:2081
    scylladb#4 0x777a2ed in _M_emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable.h:1638
    scylladb#5 0x774d707 in emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable.h:749
    scylladb#6 0x771ce3e in emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/unordered_map.h:388
    scylladb#7 0x76f8280 in operator() /local/home/bhalevy/dev/scylla/distributed_loader.cc:732
    scylladb#8 0x774dc5d in _M_invoke /usr/include/c++/9/bits/std_function.h:286
    scylladb#9 0x6cde67b in std::function<seastar::future<> (std::filesystem::__cxx11::path, seastar::directory_entry)>::operator()(std::filesystem::__cxx11::path, seastar::directory_entry) const /usr/include/c++/9/bits/std_function.h:690
    scylladb#10 0x6cd345d in operator() /local/home/bhalevy/dev/scylla/lister.cc:30
    scylladb#11 0x6cd96e0 in apply /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36
    scylladb#12 0x6cd9854 in apply<lister::visit(seastar::directory_entry)::<lambda(seastar::directory_entry)>&, seastar::directory_entry&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:44
    scylladb#13 0x6cd9953 in apply<lister::visit(seastar::directory_entry)::<lambda(seastar::directory_entry)>&, seastar::directory_entry&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1650
    scylladb#14 0x6cd6ba9 in operator() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1168
    scylladb#15 0x6cdbac0 in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:101
    scylladb#16 0x6ce6178 in seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>::operator()(seastar::directory_entry&&) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:184
    scylladb#17 0x6ce3033 in seastar::apply_helper<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, std::tuple<seastar::directory_entry>&&, std::integer_sequence<unsigned long, 0ul> >::apply(seastar::noncopyable_function<seastar::future<> (seastar::directory_e
ntry&&)>&&, std::tuple<seastar::directory_entry>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36
    scylladb#18 0x6ce3130 in auto seastar::apply<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::directory_entry>(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&, std::tuple<seastar::directory_entry>&&) /local/home/bha
levy/dev/scylla/seastar/include/seastar/core/apply.hh:44
    scylladb#19 0x6ce322f in seastar::future<> seastar::futurize<seastar::future<> >::apply<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::directory_entry>(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&, std::tuple<s
eastar::directory_entry>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1650
    scylladb#20 0x6cdf88d in seastar::future<seastar::directory_entry>::then_impl<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&)::{lambda()scylladb#1}::operator()()
const::{lambda(seastar::future_state<seastar::directory_entry>&&)scylladb#1}::operator()(seastar::future_state<seastar::directory_entry>) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1196
    scylladb#21 0x6cea2e0 in seastar::continuation<seastar::future<seastar::directory_entry>::then_impl<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&)::{lamb
da()scylladb#1}::operator()() const::{lambda(seastar::future_state<seastar::directory_entry>&&)scylladb#1}, seastar::directory_entry>::run_and_dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:511
    scylladb#22 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091
    scylladb#23 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506
    scylladb#24 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666
    scylladb#25 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199
    scylladb#26 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115
    scylladb#27 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486
    scylladb#28 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2)

SUMMARY: AddressSanitizer: heap-use-after-free /usr/include/c++/9/bits/hashtable_policy.h:283 in _M_next
Shadow bytes around the buggy address:
  0x0c068025c100: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c068025c110: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa
  0x0c068025c120: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd
  0x0c068025c130: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c068025c140: fd fd fd fa fa fa 00 00 00 fa fa fa fd fd fd fa
=>0x0c068025c150: fa fa fd fd fd fa fa fa[fd]fd fd fd fa fa fd fd
  0x0c068025c160: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c068025c170: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa
  0x0c068025c180: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd
  0x0c068025c190: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c068025c1a0: fd fd fd fd fa fa fd fd fd fd fa fa 00 00 00 fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==406079==ABORTING

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
avikivity added a commit to avikivity/scylladb that referenced this issue Apr 28, 2020
@avikivity
Merge "Branch 2019.1 ics" from Benny
cb30777 
"
This PR includes backports from enterprise for implementing ICS.

Fixes scylladb#708

Tests:
  unit(release)
  network_topology_strategy_test that doesn't pass in enterprise
  incremental_compaction_test(debug)
  dtest(release): compaction_test.py compaction_additional_test.py, with following failures:
  compaction_is_started_on_boot_test fails with ICS scylladb#20

The following tests failed, but they don't run on branch-2019.1, so they are not considered regressions:
  compaction_test.py:data_size_test fails for all compaction strategies
  compaction_additional_test:TestTimeWindowDataSegregation.test_streaming
"

* 'branch-2019.1-ics' of github.com:scylladb/scylla-enterprise: (69 commits)
  distributed_loader: Release disk space of SSTables deleted by resharding
  sstables/ICS: Fix SSTable resharding
  compaction: dont rely on undefined behavior when making garbage collected writer
  incremental_compaction_test: make it aware that incremental approach is conditional
  sstables: Fix sluggish backlog controller with incremental compaction
  table: Don't account for shared SSTables in compaction backlog tracker
  sstables: Fix major compaction's space amplification with incremental compaction
  tests: Check that partition is not resurrected on compaction failure
  tests: Add sstable compaction test for gc-only mutation compactor consumer
  sstables: Fix Incremental Compaction Efficiency
  compaction_manager: Prevent sstable runs from being partially compacted
  sstables/compaction_manager: Fix logic for filtering out partial sstable runs
  Introduce Garbage Collected Consumer to Mutation Compactor
  row_marker: add garbage_collector
  row_marker: de-inline compact_and_expire()
  row: add garbage_collector
  collection_type_impl::mutation: compact_and_expire() add collector parameter
  Introduce compaction_garbage_collector interface
  types: collection_type: drop tombstone if covered by higher-level one
  tests/sstable_datafile_test: Check cleaned sstable is generated with expected run id
  ...
bhalevy added a commit to bhalevy/scylla that referenced this issue Apr 29, 2020
@bhalevy
distributed_loader::do_populate_column_family: do_with verifier
1cde5e9 
Rather than using lw_shared_ptr to keep it around, use do_with.
Fixes a use-after-free seen with interrupt_build_process_with_resharding_half_to_max_test
in debug mode:

==406079==ERROR: AddressSanitizer: heap-use-after-free on address 0x603001320ac0 at pc 0x00000777a062 bp 0x7ffdeb77b010 sp 0x7ffdeb77b000
READ of size 8 at 0x603001320ac0 thread T0
    #0 0x777a061 in _M_next /usr/include/c++/9/bits/hashtable_policy.h:283
    scylladb#1 0x779ead7 in _M_incr /usr/include/c++/9/bits/hashtable_policy.h:299
    scylladb#2 0x777bbac in operator++ /usr/include/c++/9/bits/hashtable_policy.h:362
    scylladb#3 0x77f8384 in run_and_dispose /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future-util.hh:568
    scylladb#4 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091
    scylladb#5 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506
    scylladb#6 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666
    scylladb#7 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199
    scylladb#8 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115
    scylladb#9 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486
    scylladb#10 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2)
    scylladb#11 0x42b69d in _start (/local/home/bhalevy/.dtest/dtest-q85idok7/test/node1/bin/scylla+0x42b69d)

0x603001320ac0 is located 0 bytes inside of 32-byte region [0x603001320ac0,0x603001320ae0)
freed by thread T0 here:
    #0 0x7fb86520996f in operator delete(void*) (/lib64/libasan.so.5+0x11096f)
    scylladb#1 0x77d1889 in deallocate /usr/include/c++/9/ext/new_allocator.h:128
    scylladb#2 0x77bcbb2 in deallocate /usr/include/c++/9/bits/alloc_traits.h:470
    scylladb#3 0x77bccc5 in _M_deallocate_node_ptr /usr/include/c++/9/bits/hashtable_policy.h:2113
    scylladb#4 0x779dc59 in _M_deallocate_node /usr/include/c++/9/bits/hashtable_policy.h:2103
    scylladb#5 0x77bc8b6 in _M_deallocate_nodes /usr/include/c++/9/bits/hashtable_policy.h:2124
    scylladb#6 0x779d18f in clear /usr/include/c++/9/bits/hashtable.h:2028
    scylladb#7 0x7779b8e in ~_Hashtable /usr/include/c++/9/bits/hashtable.h:1352
    scylladb#8 0x774cfed in ~unordered_map /usr/include/c++/9/bits/unordered_map.h:102
    scylladb#9 0x774d288 in ~shared_ptr_no_esft /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:157
    scylladb#10 0x774d2f1 in dispose /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:213
    scylladb#11 0x76f9642 in ~lw_shared_ptr /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:300
    scylladb#12 0x76f9642 in ~<lambda> /local/home/bhalevy/dev/scylla/distributed_loader.cc:743
    scylladb#13 0x771e0a7 in ~<lambda> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1167
    scylladb#14 0x777c64c in destroy /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:133
    scylladb#15 0x586059 in seastar::noncopyable_function<seastar::future<> ()>::~noncopyable_function() /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:165
    scylladb#16 0x5867da in seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}::~future_state() /local/home/bha
levy/dev/scylla/seastar/include/seastar/core/future.hh:1192
    scylladb#17 0xab704d in seastar::continuation<seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}>::~continu
ation() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:507
    scylladb#18 0xab7216 in seastar::continuation<seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}>::run_and_
dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:512
    scylladb#19 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091
    scylladb#20 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506
    scylladb#21 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666
    scylladb#22 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199
    scylladb#23 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115
    scylladb#24 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486
    scylladb#25 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2)

previously allocated by thread T0 here:
    #0 0x7fb8652089d7 in operator new(unsigned long) (/lib64/libasan.so.5+0x10f9d7)
    scylladb#1 0x77d1796 in allocate /usr/include/c++/9/ext/new_allocator.h:114
    scylladb#2 0x77bcab3 in allocate /usr/include/c++/9/bits/alloc_traits.h:444
    scylladb#3 0x779d936 in _M_allocate_node<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable_policy.h:2081
    scylladb#4 0x777a2ed in _M_emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable.h:1638
    scylladb#5 0x774d707 in emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable.h:749
    scylladb#6 0x771ce3e in emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/unordered_map.h:388
    scylladb#7 0x76f8280 in operator() /local/home/bhalevy/dev/scylla/distributed_loader.cc:732
    scylladb#8 0x774dc5d in _M_invoke /usr/include/c++/9/bits/std_function.h:286
    scylladb#9 0x6cde67b in std::function<seastar::future<> (std::filesystem::__cxx11::path, seastar::directory_entry)>::operator()(std::filesystem::__cxx11::path, seastar::directory_entry) const /usr/include/c++/9/bits/std_function.h:690
    scylladb#10 0x6cd345d in operator() /local/home/bhalevy/dev/scylla/lister.cc:30
    scylladb#11 0x6cd96e0 in apply /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36
    scylladb#12 0x6cd9854 in apply<lister::visit(seastar::directory_entry)::<lambda(seastar::directory_entry)>&, seastar::directory_entry&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:44
    scylladb#13 0x6cd9953 in apply<lister::visit(seastar::directory_entry)::<lambda(seastar::directory_entry)>&, seastar::directory_entry&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1650
    scylladb#14 0x6cd6ba9 in operator() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1168
    scylladb#15 0x6cdbac0 in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:101
    scylladb#16 0x6ce6178 in seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>::operator()(seastar::directory_entry&&) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:184
    scylladb#17 0x6ce3033 in seastar::apply_helper<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, std::tuple<seastar::directory_entry>&&, std::integer_sequence<unsigned long, 0ul> >::apply(seastar::noncopyable_function<seastar::future<> (seastar::directory_e
ntry&&)>&&, std::tuple<seastar::directory_entry>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36
    scylladb#18 0x6ce3130 in auto seastar::apply<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::directory_entry>(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&, std::tuple<seastar::directory_entry>&&) /local/home/bha
levy/dev/scylla/seastar/include/seastar/core/apply.hh:44
    scylladb#19 0x6ce322f in seastar::future<> seastar::futurize<seastar::future<> >::apply<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::directory_entry>(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&, std::tuple<s
eastar::directory_entry>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1650
    scylladb#20 0x6cdf88d in seastar::future<seastar::directory_entry>::then_impl<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&)::{lambda()scylladb#1}::operator()()
const::{lambda(seastar::future_state<seastar::directory_entry>&&)scylladb#1}::operator()(seastar::future_state<seastar::directory_entry>) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1196
    scylladb#21 0x6cea2e0 in seastar::continuation<seastar::future<seastar::directory_entry>::then_impl<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&)::{lamb
da()scylladb#1}::operator()() const::{lambda(seastar::future_state<seastar::directory_entry>&&)scylladb#1}, seastar::directory_entry>::run_and_dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:511
    scylladb#22 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091
    scylladb#23 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506
    scylladb#24 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666
    scylladb#25 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199
    scylladb#26 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115
    scylladb#27 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486
    scylladb#28 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2)

SUMMARY: AddressSanitizer: heap-use-after-free /usr/include/c++/9/bits/hashtable_policy.h:283 in _M_next
Shadow bytes around the buggy address:
  0x0c068025c100: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c068025c110: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa
  0x0c068025c120: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd
  0x0c068025c130: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c068025c140: fd fd fd fa fa fa 00 00 00 fa fa fa fd fd fd fa
=>0x0c068025c150: fa fa fd fd fd fa fa fa[fd]fd fd fd fa fa fd fd
  0x0c068025c160: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c068025c170: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa
  0x0c068025c180: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd
  0x0c068025c190: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c068025c1a0: fd fd fd fd fa fa fd fd fd fd fa fa 00 00 00 fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==406079==ABORTING

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
bhalevy added a commit to bhalevy/scylla that referenced this issue Apr 30, 2020
@bhalevy
distributed_loader::do_populate_column_family: do_with verifier
1f34ff0 
Rather than using lw_shared_ptr to keep it around, use do_with.
Fixes a use-after-free seen with interrupt_build_process_with_resharding_half_to_max_test
in debug mode:

==406079==ERROR: AddressSanitizer: heap-use-after-free on address 0x603001320ac0 at pc 0x00000777a062 bp 0x7ffdeb77b010 sp 0x7ffdeb77b000
READ of size 8 at 0x603001320ac0 thread T0
    #0 0x777a061 in _M_next /usr/include/c++/9/bits/hashtable_policy.h:283
    scylladb#1 0x779ead7 in _M_incr /usr/include/c++/9/bits/hashtable_policy.h:299
    scylladb#2 0x777bbac in operator++ /usr/include/c++/9/bits/hashtable_policy.h:362
    scylladb#3 0x77f8384 in run_and_dispose /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future-util.hh:568
    scylladb#4 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091
    scylladb#5 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506
    scylladb#6 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666
    scylladb#7 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199
    scylladb#8 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115
    scylladb#9 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486
    scylladb#10 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2)
    scylladb#11 0x42b69d in _start (/local/home/bhalevy/.dtest/dtest-q85idok7/test/node1/bin/scylla+0x42b69d)

0x603001320ac0 is located 0 bytes inside of 32-byte region [0x603001320ac0,0x603001320ae0)
freed by thread T0 here:
    #0 0x7fb86520996f in operator delete(void*) (/lib64/libasan.so.5+0x11096f)
    scylladb#1 0x77d1889 in deallocate /usr/include/c++/9/ext/new_allocator.h:128
    scylladb#2 0x77bcbb2 in deallocate /usr/include/c++/9/bits/alloc_traits.h:470
    scylladb#3 0x77bccc5 in _M_deallocate_node_ptr /usr/include/c++/9/bits/hashtable_policy.h:2113
    scylladb#4 0x779dc59 in _M_deallocate_node /usr/include/c++/9/bits/hashtable_policy.h:2103
    scylladb#5 0x77bc8b6 in _M_deallocate_nodes /usr/include/c++/9/bits/hashtable_policy.h:2124
    scylladb#6 0x779d18f in clear /usr/include/c++/9/bits/hashtable.h:2028
    scylladb#7 0x7779b8e in ~_Hashtable /usr/include/c++/9/bits/hashtable.h:1352
    scylladb#8 0x774cfed in ~unordered_map /usr/include/c++/9/bits/unordered_map.h:102
    scylladb#9 0x774d288 in ~shared_ptr_no_esft /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:157
    scylladb#10 0x774d2f1 in dispose /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:213
    scylladb#11 0x76f9642 in ~lw_shared_ptr /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:300
    scylladb#12 0x76f9642 in ~<lambda> /local/home/bhalevy/dev/scylla/distributed_loader.cc:743
    scylladb#13 0x771e0a7 in ~<lambda> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1167
    scylladb#14 0x777c64c in destroy /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:133
    scylladb#15 0x586059 in seastar::noncopyable_function<seastar::future<> ()>::~noncopyable_function() /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:165
    scylladb#16 0x5867da in seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}::~future_state() /local/home/bha
levy/dev/scylla/seastar/include/seastar/core/future.hh:1192
    scylladb#17 0xab704d in seastar::continuation<seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}>::~continu
ation() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:507
    scylladb#18 0xab7216 in seastar::continuation<seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}>::run_and_
dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:512
    scylladb#19 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091
    scylladb#20 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506
    scylladb#21 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666
    scylladb#22 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199
    scylladb#23 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115
    scylladb#24 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486
    scylladb#25 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2)

previously allocated by thread T0 here:
    #0 0x7fb8652089d7 in operator new(unsigned long) (/lib64/libasan.so.5+0x10f9d7)
    scylladb#1 0x77d1796 in allocate /usr/include/c++/9/ext/new_allocator.h:114
    scylladb#2 0x77bcab3 in allocate /usr/include/c++/9/bits/alloc_traits.h:444
    scylladb#3 0x779d936 in _M_allocate_node<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable_policy.h:2081
    scylladb#4 0x777a2ed in _M_emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable.h:1638
    scylladb#5 0x774d707 in emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable.h:749
    scylladb#6 0x771ce3e in emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/unordered_map.h:388
    scylladb#7 0x76f8280 in operator() /local/home/bhalevy/dev/scylla/distributed_loader.cc:732
    scylladb#8 0x774dc5d in _M_invoke /usr/include/c++/9/bits/std_function.h:286
    scylladb#9 0x6cde67b in std::function<seastar::future<> (std::filesystem::__cxx11::path, seastar::directory_entry)>::operator()(std::filesystem::__cxx11::path, seastar::directory_entry) const /usr/include/c++/9/bits/std_function.h:690
    scylladb#10 0x6cd345d in operator() /local/home/bhalevy/dev/scylla/lister.cc:30
    scylladb#11 0x6cd96e0 in apply /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36
    scylladb#12 0x6cd9854 in apply<lister::visit(seastar::directory_entry)::<lambda(seastar::directory_entry)>&, seastar::directory_entry&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:44
    scylladb#13 0x6cd9953 in apply<lister::visit(seastar::directory_entry)::<lambda(seastar::directory_entry)>&, seastar::directory_entry&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1650
    scylladb#14 0x6cd6ba9 in operator() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1168
    scylladb#15 0x6cdbac0 in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:101
    scylladb#16 0x6ce6178 in seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>::operator()(seastar::directory_entry&&) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:184
    scylladb#17 0x6ce3033 in seastar::apply_helper<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, std::tuple<seastar::directory_entry>&&, std::integer_sequence<unsigned long, 0ul> >::apply(seastar::noncopyable_function<seastar::future<> (seastar::directory_e
ntry&&)>&&, std::tuple<seastar::directory_entry>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36
    scylladb#18 0x6ce3130 in auto seastar::apply<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::directory_entry>(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&, std::tuple<seastar::directory_entry>&&) /local/home/bha
levy/dev/scylla/seastar/include/seastar/core/apply.hh:44
    scylladb#19 0x6ce322f in seastar::future<> seastar::futurize<seastar::future<> >::apply<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::directory_entry>(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&, std::tuple<s
eastar::directory_entry>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1650
    scylladb#20 0x6cdf88d in seastar::future<seastar::directory_entry>::then_impl<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&)::{lambda()scylladb#1}::operator()()
const::{lambda(seastar::future_state<seastar::directory_entry>&&)scylladb#1}::operator()(seastar::future_state<seastar::directory_entry>) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1196
    scylladb#21 0x6cea2e0 in seastar::continuation<seastar::future<seastar::directory_entry>::then_impl<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&)::{lamb
da()scylladb#1}::operator()() const::{lambda(seastar::future_state<seastar::directory_entry>&&)scylladb#1}, seastar::directory_entry>::run_and_dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:511
    scylladb#22 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091
    scylladb#23 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506
    scylladb#24 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666
    scylladb#25 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199
    scylladb#26 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115
    scylladb#27 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486
    scylladb#28 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2)

SUMMARY: AddressSanitizer: heap-use-after-free /usr/include/c++/9/bits/hashtable_policy.h:283 in _M_next
Shadow bytes around the buggy address:
  0x0c068025c100: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c068025c110: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa
  0x0c068025c120: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd
  0x0c068025c130: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c068025c140: fd fd fd fa fa fa 00 00 00 fa fa fa fd fd fd fa
=>0x0c068025c150: fa fa fd fd fd fa fa fa[fd]fd fd fd fa fa fd fd
  0x0c068025c160: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c068025c170: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa
  0x0c068025c180: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd
  0x0c068025c190: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c068025c1a0: fd fd fd fd fa fa fd fd fd fd fa fa 00 00 00 fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==406079==ABORTING

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
bhalevy added a commit to bhalevy/scylla that referenced this issue May 5, 2020
@bhalevy
distributed_loader::do_populate_column_family: do_with verifier
7e569bd 
Rather than using lw_shared_ptr to keep it around, use do_with.
Fixes a use-after-free seen with interrupt_build_process_with_resharding_half_to_max_test
in debug mode:

==406079==ERROR: AddressSanitizer: heap-use-after-free on address 0x603001320ac0 at pc 0x00000777a062 bp 0x7ffdeb77b010 sp 0x7ffdeb77b000
READ of size 8 at 0x603001320ac0 thread T0
    #0 0x777a061 in _M_next /usr/include/c++/9/bits/hashtable_policy.h:283
    scylladb#1 0x779ead7 in _M_incr /usr/include/c++/9/bits/hashtable_policy.h:299
    scylladb#2 0x777bbac in operator++ /usr/include/c++/9/bits/hashtable_policy.h:362
    scylladb#3 0x77f8384 in run_and_dispose /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future-util.hh:568
    scylladb#4 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091
    scylladb#5 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506
    scylladb#6 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666
    scylladb#7 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199
    scylladb#8 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115
    scylladb#9 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486
    scylladb#10 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2)
    scylladb#11 0x42b69d in _start (/local/home/bhalevy/.dtest/dtest-q85idok7/test/node1/bin/scylla+0x42b69d)

0x603001320ac0 is located 0 bytes inside of 32-byte region [0x603001320ac0,0x603001320ae0)
freed by thread T0 here:
    #0 0x7fb86520996f in operator delete(void*) (/lib64/libasan.so.5+0x11096f)
    scylladb#1 0x77d1889 in deallocate /usr/include/c++/9/ext/new_allocator.h:128
    scylladb#2 0x77bcbb2 in deallocate /usr/include/c++/9/bits/alloc_traits.h:470
    scylladb#3 0x77bccc5 in _M_deallocate_node_ptr /usr/include/c++/9/bits/hashtable_policy.h:2113
    scylladb#4 0x779dc59 in _M_deallocate_node /usr/include/c++/9/bits/hashtable_policy.h:2103
    scylladb#5 0x77bc8b6 in _M_deallocate_nodes /usr/include/c++/9/bits/hashtable_policy.h:2124
    scylladb#6 0x779d18f in clear /usr/include/c++/9/bits/hashtable.h:2028
    scylladb#7 0x7779b8e in ~_Hashtable /usr/include/c++/9/bits/hashtable.h:1352
    scylladb#8 0x774cfed in ~unordered_map /usr/include/c++/9/bits/unordered_map.h:102
    scylladb#9 0x774d288 in ~shared_ptr_no_esft /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:157
    scylladb#10 0x774d2f1 in dispose /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:213
    scylladb#11 0x76f9642 in ~lw_shared_ptr /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:300
    scylladb#12 0x76f9642 in ~<lambda> /local/home/bhalevy/dev/scylla/distributed_loader.cc:743
    scylladb#13 0x771e0a7 in ~<lambda> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1167
    scylladb#14 0x777c64c in destroy /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:133
    scylladb#15 0x586059 in seastar::noncopyable_function<seastar::future<> ()>::~noncopyable_function() /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:165
    scylladb#16 0x5867da in seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}::~future_state() /local/home/bha
levy/dev/scylla/seastar/include/seastar/core/future.hh:1192
    scylladb#17 0xab704d in seastar::continuation<seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}>::~continu
ation() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:507
    scylladb#18 0xab7216 in seastar::continuation<seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}>::run_and_
dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:512
    scylladb#19 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091
    scylladb#20 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506
    scylladb#21 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666
    scylladb#22 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199
    scylladb#23 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115
    scylladb#24 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486
    scylladb#25 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2)

previously allocated by thread T0 here:
    #0 0x7fb8652089d7 in operator new(unsigned long) (/lib64/libasan.so.5+0x10f9d7)
    scylladb#1 0x77d1796 in allocate /usr/include/c++/9/ext/new_allocator.h:114
    scylladb#2 0x77bcab3 in allocate /usr/include/c++/9/bits/alloc_traits.h:444
    scylladb#3 0x779d936 in _M_allocate_node<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable_policy.h:2081
    scylladb#4 0x777a2ed in _M_emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable.h:1638
    scylladb#5 0x774d707 in emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable.h:749
    scylladb#6 0x771ce3e in emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/unordered_map.h:388
    scylladb#7 0x76f8280 in operator() /local/home/bhalevy/dev/scylla/distributed_loader.cc:732
    scylladb#8 0x774dc5d in _M_invoke /usr/include/c++/9/bits/std_function.h:286
    scylladb#9 0x6cde67b in std::function<seastar::future<> (std::filesystem::__cxx11::path, seastar::directory_entry)>::operator()(std::filesystem::__cxx11::path, seastar::directory_entry) const /usr/include/c++/9/bits/std_function.h:690
    scylladb#10 0x6cd345d in operator() /local/home/bhalevy/dev/scylla/lister.cc:30
    scylladb#11 0x6cd96e0 in apply /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36
    scylladb#12 0x6cd9854 in apply<lister::visit(seastar::directory_entry)::<lambda(seastar::directory_entry)>&, seastar::directory_entry&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:44
    scylladb#13 0x6cd9953 in apply<lister::visit(seastar::directory_entry)::<lambda(seastar::directory_entry)>&, seastar::directory_entry&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1650
    scylladb#14 0x6cd6ba9 in operator() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1168
    scylladb#15 0x6cdbac0 in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:101
    scylladb#16 0x6ce6178 in seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>::operator()(seastar::directory_entry&&) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:184
    scylladb#17 0x6ce3033 in seastar::apply_helper<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, std::tuple<seastar::directory_entry>&&, std::integer_sequence<unsigned long, 0ul> >::apply(seastar::noncopyable_function<seastar::future<> (seastar::directory_e
ntry&&)>&&, std::tuple<seastar::directory_entry>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36
    scylladb#18 0x6ce3130 in auto seastar::apply<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::directory_entry>(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&, std::tuple<seastar::directory_entry>&&) /local/home/bha
levy/dev/scylla/seastar/include/seastar/core/apply.hh:44
    scylladb#19 0x6ce322f in seastar::future<> seastar::futurize<seastar::future<> >::apply<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::directory_entry>(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&, std::tuple<s
eastar::directory_entry>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1650
    scylladb#20 0x6cdf88d in seastar::future<seastar::directory_entry>::then_impl<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&)::{lambda()scylladb#1}::operator()()
const::{lambda(seastar::future_state<seastar::directory_entry>&&)scylladb#1}::operator()(seastar::future_state<seastar::directory_entry>) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1196
    scylladb#21 0x6cea2e0 in seastar::continuation<seastar::future<seastar::directory_entry>::then_impl<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&)::{lamb
da()scylladb#1}::operator()() const::{lambda(seastar::future_state<seastar::directory_entry>&&)scylladb#1}, seastar::directory_entry>::run_and_dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:511
    scylladb#22 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091
    scylladb#23 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506
    scylladb#24 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666
    scylladb#25 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199
    scylladb#26 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115
    scylladb#27 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486
    scylladb#28 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2)

SUMMARY: AddressSanitizer: heap-use-after-free /usr/include/c++/9/bits/hashtable_policy.h:283 in _M_next
Shadow bytes around the buggy address:
  0x0c068025c100: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c068025c110: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa
  0x0c068025c120: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd
  0x0c068025c130: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c068025c140: fd fd fd fa fa fa 00 00 00 fa fa fa fd fd fd fa
=>0x0c068025c150: fa fa fd fd fd fa fa fa[fd]fd fd fd fa fa fd fd
  0x0c068025c160: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c068025c170: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa
  0x0c068025c180: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd
  0x0c068025c190: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c068025c1a0: fd fd fd fd fa fa fd fd fd fd fa fa 00 00 00 fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==406079==ABORTING

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
bhalevy added a commit to bhalevy/scylla that referenced this issue May 6, 2020
@bhalevy
distributed_loader::do_populate_column_family: do_with verifier
6b0d548 
Rather than using lw_shared_ptr to keep it around, use do_with.

Fixes a use-after-free seen with interrupt_build_process_with_resharding_half_to_max_test
in debug mode:

`verifier` was not captured in the scan_dir.then() continuation.
We could add a capture in that lambda, but it's cleaner and safer to let do_with hold on to it instead.

==406079==ERROR: AddressSanitizer: heap-use-after-free on address 0x603001320ac0 at pc 0x00000777a062 bp 0x7ffdeb77b010 sp 0x7ffdeb77b000
READ of size 8 at 0x603001320ac0 thread T0
    #0 0x777a061 in _M_next /usr/include/c++/9/bits/hashtable_policy.h:283
    scylladb#1 0x779ead7 in _M_incr /usr/include/c++/9/bits/hashtable_policy.h:299
    scylladb#2 0x777bbac in operator++ /usr/include/c++/9/bits/hashtable_policy.h:362
    scylladb#3 0x77f8384 in run_and_dispose /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future-util.hh:568
    scylladb#4 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091
    scylladb#5 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506
    scylladb#6 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666
    scylladb#7 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199
    scylladb#8 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115
    scylladb#9 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486
    scylladb#10 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2)
    scylladb#11 0x42b69d in _start (/local/home/bhalevy/.dtest/dtest-q85idok7/test/node1/bin/scylla+0x42b69d)

0x603001320ac0 is located 0 bytes inside of 32-byte region [0x603001320ac0,0x603001320ae0)
freed by thread T0 here:
    #0 0x7fb86520996f in operator delete(void*) (/lib64/libasan.so.5+0x11096f)
    scylladb#1 0x77d1889 in deallocate /usr/include/c++/9/ext/new_allocator.h:128
    scylladb#2 0x77bcbb2 in deallocate /usr/include/c++/9/bits/alloc_traits.h:470
    scylladb#3 0x77bccc5 in _M_deallocate_node_ptr /usr/include/c++/9/bits/hashtable_policy.h:2113
    scylladb#4 0x779dc59 in _M_deallocate_node /usr/include/c++/9/bits/hashtable_policy.h:2103
    scylladb#5 0x77bc8b6 in _M_deallocate_nodes /usr/include/c++/9/bits/hashtable_policy.h:2124
    scylladb#6 0x779d18f in clear /usr/include/c++/9/bits/hashtable.h:2028
    scylladb#7 0x7779b8e in ~_Hashtable /usr/include/c++/9/bits/hashtable.h:1352
    scylladb#8 0x774cfed in ~unordered_map /usr/include/c++/9/bits/unordered_map.h:102
    scylladb#9 0x774d288 in ~shared_ptr_no_esft /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:157
    scylladb#10 0x774d2f1 in dispose /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:213
    scylladb#11 0x76f9642 in ~lw_shared_ptr /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:300
    scylladb#12 0x76f9642 in ~<lambda> /local/home/bhalevy/dev/scylla/distributed_loader.cc:743
    scylladb#13 0x771e0a7 in ~<lambda> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1167
    scylladb#14 0x777c64c in destroy /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:133
    scylladb#15 0x586059 in seastar::noncopyable_function<seastar::future<> ()>::~noncopyable_function() /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:165
    scylladb#16 0x5867da in seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}::~future_state() /local/home/bha
levy/dev/scylla/seastar/include/seastar/core/future.hh:1192
    scylladb#17 0xab704d in seastar::continuation<seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}>::~continu
ation() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:507
    scylladb#18 0xab7216 in seastar::continuation<seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}>::run_and_
dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:512
    scylladb#19 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091
    scylladb#20 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506
    scylladb#21 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666
    scylladb#22 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199
    scylladb#23 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115
    scylladb#24 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486
    scylladb#25 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2)

previously allocated by thread T0 here:
    #0 0x7fb8652089d7 in operator new(unsigned long) (/lib64/libasan.so.5+0x10f9d7)
    scylladb#1 0x77d1796 in allocate /usr/include/c++/9/ext/new_allocator.h:114
    scylladb#2 0x77bcab3 in allocate /usr/include/c++/9/bits/alloc_traits.h:444
    scylladb#3 0x779d936 in _M_allocate_node<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable_policy.h:2081
    scylladb#4 0x777a2ed in _M_emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable.h:1638
    scylladb#5 0x774d707 in emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable.h:749
    scylladb#6 0x771ce3e in emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/unordered_map.h:388
    scylladb#7 0x76f8280 in operator() /local/home/bhalevy/dev/scylla/distributed_loader.cc:732
    scylladb#8 0x774dc5d in _M_invoke /usr/include/c++/9/bits/std_function.h:286
    scylladb#9 0x6cde67b in std::function<seastar::future<> (std::filesystem::__cxx11::path, seastar::directory_entry)>::operator()(std::filesystem::__cxx11::path, seastar::directory_entry) const /usr/include/c++/9/bits/std_function.h:690
    scylladb#10 0x6cd345d in operator() /local/home/bhalevy/dev/scylla/lister.cc:30
    scylladb#11 0x6cd96e0 in apply /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36
    scylladb#12 0x6cd9854 in apply<lister::visit(seastar::directory_entry)::<lambda(seastar::directory_entry)>&, seastar::directory_entry&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:44
    scylladb#13 0x6cd9953 in apply<lister::visit(seastar::directory_entry)::<lambda(seastar::directory_entry)>&, seastar::directory_entry&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1650
    scylladb#14 0x6cd6ba9 in operator() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1168
    scylladb#15 0x6cdbac0 in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:101
    scylladb#16 0x6ce6178 in seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>::operator()(seastar::directory_entry&&) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:184
    scylladb#17 0x6ce3033 in seastar::apply_helper<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, std::tuple<seastar::directory_entry>&&, std::integer_sequence<unsigned long, 0ul> >::apply(seastar::noncopyable_function<seastar::future<> (seastar::directory_e
ntry&&)>&&, std::tuple<seastar::directory_entry>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36
    scylladb#18 0x6ce3130 in auto seastar::apply<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::directory_entry>(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&, std::tuple<seastar::directory_entry>&&) /local/home/bha
levy/dev/scylla/seastar/include/seastar/core/apply.hh:44
    scylladb#19 0x6ce322f in seastar::future<> seastar::futurize<seastar::future<> >::apply<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::directory_entry>(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&, std::tuple<s
eastar::directory_entry>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1650
    scylladb#20 0x6cdf88d in seastar::future<seastar::directory_entry>::then_impl<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&)::{lambda()scylladb#1}::operator()()
const::{lambda(seastar::future_state<seastar::directory_entry>&&)scylladb#1}::operator()(seastar::future_state<seastar::directory_entry>) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1196
    scylladb#21 0x6cea2e0 in seastar::continuation<seastar::future<seastar::directory_entry>::then_impl<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&)::{lamb
da()scylladb#1}::operator()() const::{lambda(seastar::future_state<seastar::directory_entry>&&)scylladb#1}, seastar::directory_entry>::run_and_dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:511
    scylladb#22 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091
    scylladb#23 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506
    scylladb#24 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666
    scylladb#25 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199
    scylladb#26 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115
    scylladb#27 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486
    scylladb#28 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2)

SUMMARY: AddressSanitizer: heap-use-after-free /usr/include/c++/9/bits/hashtable_policy.h:283 in _M_next
Shadow bytes around the buggy address:
  0x0c068025c100: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c068025c110: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa
  0x0c068025c120: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd
  0x0c068025c130: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c068025c140: fd fd fd fa fa fa 00 00 00 fa fa fa fd fd fd fa
=>0x0c068025c150: fa fa fd fd fd fa fa fa[fd]fd fd fd fa fa fd fd
  0x0c068025c160: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c068025c170: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa
  0x0c068025c180: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd
  0x0c068025c190: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c068025c1a0: fd fd fd fd fa fa fd fd fd fd fa fa 00 00 00 fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==406079==ABORTING

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
bhalevy added a commit to bhalevy/scylla that referenced this issue May 11, 2020
@bhalevy
distributed_loader::do_populate_column_family: do_with verifier
1ae3453 
Rather than using lw_shared_ptr to keep it around, use do_with.

Fixes a use-after-free seen with interrupt_build_process_with_resharding_half_to_max_test
in debug mode:

`verifier` was not captured in the scan_dir.then() continuation.
We could add a capture in that lambda, but it's cleaner and safer to let do_with hold on to it instead.

==406079==ERROR: AddressSanitizer: heap-use-after-free on address 0x603001320ac0 at pc 0x00000777a062 bp 0x7ffdeb77b010 sp 0x7ffdeb77b000
READ of size 8 at 0x603001320ac0 thread T0
    #0 0x777a061 in _M_next /usr/include/c++/9/bits/hashtable_policy.h:283
    scylladb#1 0x779ead7 in _M_incr /usr/include/c++/9/bits/hashtable_policy.h:299
    scylladb#2 0x777bbac in operator++ /usr/include/c++/9/bits/hashtable_policy.h:362
    scylladb#3 0x77f8384 in run_and_dispose /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future-util.hh:568
    scylladb#4 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091
    scylladb#5 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506
    scylladb#6 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666
    scylladb#7 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199
    scylladb#8 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115
    scylladb#9 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486
    scylladb#10 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2)
    scylladb#11 0x42b69d in _start (/local/home/bhalevy/.dtest/dtest-q85idok7/test/node1/bin/scylla+0x42b69d)

0x603001320ac0 is located 0 bytes inside of 32-byte region [0x603001320ac0,0x603001320ae0)
freed by thread T0 here:
    #0 0x7fb86520996f in operator delete(void*) (/lib64/libasan.so.5+0x11096f)
    scylladb#1 0x77d1889 in deallocate /usr/include/c++/9/ext/new_allocator.h:128
    scylladb#2 0x77bcbb2 in deallocate /usr/include/c++/9/bits/alloc_traits.h:470
    scylladb#3 0x77bccc5 in _M_deallocate_node_ptr /usr/include/c++/9/bits/hashtable_policy.h:2113
    scylladb#4 0x779dc59 in _M_deallocate_node /usr/include/c++/9/bits/hashtable_policy.h:2103
    scylladb#5 0x77bc8b6 in _M_deallocate_nodes /usr/include/c++/9/bits/hashtable_policy.h:2124
    scylladb#6 0x779d18f in clear /usr/include/c++/9/bits/hashtable.h:2028
    scylladb#7 0x7779b8e in ~_Hashtable /usr/include/c++/9/bits/hashtable.h:1352
    scylladb#8 0x774cfed in ~unordered_map /usr/include/c++/9/bits/unordered_map.h:102
    scylladb#9 0x774d288 in ~shared_ptr_no_esft /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:157
    scylladb#10 0x774d2f1 in dispose /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:213
    scylladb#11 0x76f9642 in ~lw_shared_ptr /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:300
    scylladb#12 0x76f9642 in ~<lambda> /local/home/bhalevy/dev/scylla/distributed_loader.cc:743
    scylladb#13 0x771e0a7 in ~<lambda> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1167
    scylladb#14 0x777c64c in destroy /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:133
    scylladb#15 0x586059 in seastar::noncopyable_function<seastar::future<> ()>::~noncopyable_function() /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:165
    scylladb#16 0x5867da in seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}::~future_state() /local/home/bha
levy/dev/scylla/seastar/include/seastar/core/future.hh:1192
    scylladb#17 0xab704d in seastar::continuation<seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}>::~continu
ation() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:507
    scylladb#18 0xab7216 in seastar::continuation<seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}>::run_and_
dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:512
    scylladb#19 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091
    scylladb#20 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506
    scylladb#21 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666
    scylladb#22 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199
    scylladb#23 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115
    scylladb#24 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486
    scylladb#25 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2)

previously allocated by thread T0 here:
    #0 0x7fb8652089d7 in operator new(unsigned long) (/lib64/libasan.so.5+0x10f9d7)
    scylladb#1 0x77d1796 in allocate /usr/include/c++/9/ext/new_allocator.h:114
    scylladb#2 0x77bcab3 in allocate /usr/include/c++/9/bits/alloc_traits.h:444
    scylladb#3 0x779d936 in _M_allocate_node<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable_policy.h:2081
    scylladb#4 0x777a2ed in _M_emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable.h:1638
    scylladb#5 0x774d707 in emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable.h:749
    scylladb#6 0x771ce3e in emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/unordered_map.h:388
    scylladb#7 0x76f8280 in operator() /local/home/bhalevy/dev/scylla/distributed_loader.cc:732
    scylladb#8 0x774dc5d in _M_invoke /usr/include/c++/9/bits/std_function.h:286
    scylladb#9 0x6cde67b in std::function<seastar::future<> (std::filesystem::__cxx11::path, seastar::directory_entry)>::operator()(std::filesystem::__cxx11::path, seastar::directory_entry) const /usr/include/c++/9/bits/std_function.h:690
    scylladb#10 0x6cd345d in operator() /local/home/bhalevy/dev/scylla/lister.cc:30
    scylladb#11 0x6cd96e0 in apply /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36
    scylladb#12 0x6cd9854 in apply<lister::visit(seastar::directory_entry)::<lambda(seastar::directory_entry)>&, seastar::directory_entry&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:44
    scylladb#13 0x6cd9953 in apply<lister::visit(seastar::directory_entry)::<lambda(seastar::directory_entry)>&, seastar::directory_entry&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1650
    scylladb#14 0x6cd6ba9 in operator() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1168
    scylladb#15 0x6cdbac0 in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:101
    scylladb#16 0x6ce6178 in seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>::operator()(seastar::directory_entry&&) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:184
    scylladb#17 0x6ce3033 in seastar::apply_helper<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, std::tuple<seastar::directory_entry>&&, std::integer_sequence<unsigned long, 0ul> >::apply(seastar::noncopyable_function<seastar::future<> (seastar::directory_e
ntry&&)>&&, std::tuple<seastar::directory_entry>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36
    scylladb#18 0x6ce3130 in auto seastar::apply<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::directory_entry>(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&, std::tuple<seastar::directory_entry>&&) /local/home/bha
levy/dev/scylla/seastar/include/seastar/core/apply.hh:44
    scylladb#19 0x6ce322f in seastar::future<> seastar::futurize<seastar::future<> >::apply<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::directory_entry>(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&, std::tuple<s
eastar::directory_entry>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1650
    scylladb#20 0x6cdf88d in seastar::future<seastar::directory_entry>::then_impl<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&)::{lambda()scylladb#1}::operator()()
const::{lambda(seastar::future_state<seastar::directory_entry>&&)scylladb#1}::operator()(seastar::future_state<seastar::directory_entry>) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1196
    scylladb#21 0x6cea2e0 in seastar::continuation<seastar::future<seastar::directory_entry>::then_impl<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&)::{lamb
da()scylladb#1}::operator()() const::{lambda(seastar::future_state<seastar::directory_entry>&&)scylladb#1}, seastar::directory_entry>::run_and_dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:511
    scylladb#22 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091
    scylladb#23 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506
    scylladb#24 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666
    scylladb#25 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199
    scylladb#26 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115
    scylladb#27 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486
    scylladb#28 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2)

SUMMARY: AddressSanitizer: heap-use-after-free /usr/include/c++/9/bits/hashtable_policy.h:283 in _M_next
Shadow bytes around the buggy address:
  0x0c068025c100: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c068025c110: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa
  0x0c068025c120: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd
  0x0c068025c130: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c068025c140: fd fd fd fa fa fa 00 00 00 fa fa fa fd fd fd fa
=>0x0c068025c150: fa fa fd fd fd fa fa fa[fd]fd fd fd fa fa fd fd
  0x0c068025c160: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c068025c170: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa
  0x0c068025c180: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd
  0x0c068025c190: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c068025c1a0: fd fd fd fd fa fa fd fd fd fd fa fa 00 00 00 fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==406079==ABORTING

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
bhalevy added a commit to bhalevy/scylla that referenced this issue May 13, 2020
@bhalevy
distributed_loader::do_populate_column_family: do_with verifier
cb88f7b 
Rather than using lw_shared_ptr to keep it around, use do_with.

Fixes a use-after-free seen with interrupt_build_process_with_resharding_half_to_max_test
in debug mode:

`verifier` was not captured in the scan_dir.then() continuation.
We could add a capture in that lambda, but it's cleaner and safer to let do_with hold on to it instead.

==406079==ERROR: AddressSanitizer: heap-use-after-free on address 0x603001320ac0 at pc 0x00000777a062 bp 0x7ffdeb77b010 sp 0x7ffdeb77b000
READ of size 8 at 0x603001320ac0 thread T0
    #0 0x777a061 in _M_next /usr/include/c++/9/bits/hashtable_policy.h:283
    scylladb#1 0x779ead7 in _M_incr /usr/include/c++/9/bits/hashtable_policy.h:299
    scylladb#2 0x777bbac in operator++ /usr/include/c++/9/bits/hashtable_policy.h:362
    scylladb#3 0x77f8384 in run_and_dispose /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future-util.hh:568
    scylladb#4 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091
    scylladb#5 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506
    scylladb#6 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666
    scylladb#7 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199
    scylladb#8 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115
    scylladb#9 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486
    scylladb#10 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2)
    scylladb#11 0x42b69d in _start (/local/home/bhalevy/.dtest/dtest-q85idok7/test/node1/bin/scylla+0x42b69d)

0x603001320ac0 is located 0 bytes inside of 32-byte region [0x603001320ac0,0x603001320ae0)
freed by thread T0 here:
    #0 0x7fb86520996f in operator delete(void*) (/lib64/libasan.so.5+0x11096f)
    scylladb#1 0x77d1889 in deallocate /usr/include/c++/9/ext/new_allocator.h:128
    scylladb#2 0x77bcbb2 in deallocate /usr/include/c++/9/bits/alloc_traits.h:470
    scylladb#3 0x77bccc5 in _M_deallocate_node_ptr /usr/include/c++/9/bits/hashtable_policy.h:2113
    scylladb#4 0x779dc59 in _M_deallocate_node /usr/include/c++/9/bits/hashtable_policy.h:2103
    scylladb#5 0x77bc8b6 in _M_deallocate_nodes /usr/include/c++/9/bits/hashtable_policy.h:2124
    scylladb#6 0x779d18f in clear /usr/include/c++/9/bits/hashtable.h:2028
    scylladb#7 0x7779b8e in ~_Hashtable /usr/include/c++/9/bits/hashtable.h:1352
    scylladb#8 0x774cfed in ~unordered_map /usr/include/c++/9/bits/unordered_map.h:102
    scylladb#9 0x774d288 in ~shared_ptr_no_esft /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:157
    scylladb#10 0x774d2f1 in dispose /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:213
    scylladb#11 0x76f9642 in ~lw_shared_ptr /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:300
    scylladb#12 0x76f9642 in ~<lambda> /local/home/bhalevy/dev/scylla/distributed_loader.cc:743
    scylladb#13 0x771e0a7 in ~<lambda> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1167
    scylladb#14 0x777c64c in destroy /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:133
    scylladb#15 0x586059 in seastar::noncopyable_function<seastar::future<> ()>::~noncopyable_function() /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:165
    scylladb#16 0x5867da in seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}::~future_state() /local/home/bha
levy/dev/scylla/seastar/include/seastar/core/future.hh:1192
    scylladb#17 0xab704d in seastar::continuation<seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}>::~continu
ation() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:507
    scylladb#18 0xab7216 in seastar::continuation<seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}>::run_and_
dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:512
    scylladb#19 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091
    scylladb#20 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506
    scylladb#21 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666
    scylladb#22 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199
    scylladb#23 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115
    scylladb#24 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486
    scylladb#25 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2)

previously allocated by thread T0 here:
    #0 0x7fb8652089d7 in operator new(unsigned long) (/lib64/libasan.so.5+0x10f9d7)
    scylladb#1 0x77d1796 in allocate /usr/include/c++/9/ext/new_allocator.h:114
    scylladb#2 0x77bcab3 in allocate /usr/include/c++/9/bits/alloc_traits.h:444
    scylladb#3 0x779d936 in _M_allocate_node<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable_policy.h:2081
    scylladb#4 0x777a2ed in _M_emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable.h:1638
    scylladb#5 0x774d707 in emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable.h:749
    scylladb#6 0x771ce3e in emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/unordered_map.h:388
    scylladb#7 0x76f8280 in operator() /local/home/bhalevy/dev/scylla/distributed_loader.cc:732
    scylladb#8 0x774dc5d in _M_invoke /usr/include/c++/9/bits/std_function.h:286
    scylladb#9 0x6cde67b in std::function<seastar::future<> (std::filesystem::__cxx11::path, seastar::directory_entry)>::operator()(std::filesystem::__cxx11::path, seastar::directory_entry) const /usr/include/c++/9/bits/std_function.h:690
    scylladb#10 0x6cd345d in operator() /local/home/bhalevy/dev/scylla/lister.cc:30
    scylladb#11 0x6cd96e0 in apply /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36
    scylladb#12 0x6cd9854 in apply<lister::visit(seastar::directory_entry)::<lambda(seastar::directory_entry)>&, seastar::directory_entry&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:44
    scylladb#13 0x6cd9953 in apply<lister::visit(seastar::directory_entry)::<lambda(seastar::directory_entry)>&, seastar::directory_entry&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1650
    scylladb#14 0x6cd6ba9 in operator() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1168
    scylladb#15 0x6cdbac0 in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:101
    scylladb#16 0x6ce6178 in seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>::operator()(seastar::directory_entry&&) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:184
    scylladb#17 0x6ce3033 in seastar::apply_helper<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, std::tuple<seastar::directory_entry>&&, std::integer_sequence<unsigned long, 0ul> >::apply(seastar::noncopyable_function<seastar::future<> (seastar::directory_e
ntry&&)>&&, std::tuple<seastar::directory_entry>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36
    scylladb#18 0x6ce3130 in auto seastar::apply<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::directory_entry>(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&, std::tuple<seastar::directory_entry>&&) /local/home/bha
levy/dev/scylla/seastar/include/seastar/core/apply.hh:44
    scylladb#19 0x6ce322f in seastar::future<> seastar::futurize<seastar::future<> >::apply<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::directory_entry>(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&, std::tuple<s
eastar::directory_entry>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1650
    scylladb#20 0x6cdf88d in seastar::future<seastar::directory_entry>::then_impl<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&)::{lambda()scylladb#1}::operator()()
const::{lambda(seastar::future_state<seastar::directory_entry>&&)scylladb#1}::operator()(seastar::future_state<seastar::directory_entry>) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1196
    scylladb#21 0x6cea2e0 in seastar::continuation<seastar::future<seastar::directory_entry>::then_impl<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&)::{lamb
da()scylladb#1}::operator()() const::{lambda(seastar::future_state<seastar::directory_entry>&&)scylladb#1}, seastar::directory_entry>::run_and_dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:511
    scylladb#22 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091
    scylladb#23 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506
    scylladb#24 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666
    scylladb#25 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199
    scylladb#26 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115
    scylladb#27 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486
    scylladb#28 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2)

SUMMARY: AddressSanitizer: heap-use-after-free /usr/include/c++/9/bits/hashtable_policy.h:283 in _M_next
Shadow bytes around the buggy address:
  0x0c068025c100: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c068025c110: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa
  0x0c068025c120: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd
  0x0c068025c130: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c068025c140: fd fd fd fa fa fa 00 00 00 fa fa fa fd fd fd fa
=>0x0c068025c150: fa fa fd fd fd fa fa fa[fd]fd fd fd fa fa fd fd
  0x0c068025c160: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c068025c170: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa
  0x0c068025c180: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd
  0x0c068025c190: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
  0x0c068025c1a0: fd fd fd fd fa fa fd fd fd fd fa fa 00 00 00 fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==406079==ABORTING

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
bhalevy added a commit to bhalevy/scylla that referenced this issue Oct 4, 2020
@bhalevy
storage_proxy: query_ranges_to_vnodes_generator ranges_to_vnodes: use…
d2bf48e 
… token_metadata_ptr

Fixes use-after-free seen with putget_with_reloaded_certificates_test:
```
==215==ERROR: AddressSanitizer: heap-use-after-free on address 0x603000a8b180 at pc 0x000012eb5a83 bp 0x7ffd2c16d4c0 sp 0x7ffd2c16d4b0
READ of size 8 at 0x603000a8b180 thread T0
    #0 0x12eb5a82 in std::__uniq_ptr_impl<locator::token_metadata_impl, std::default_delete<locator::token_metadata_impl> >::_M_ptr() const /usr/include/c++/10/bits/unique_ptr.h:173
    scylladb#1 0x12ea230d in std::unique_ptr<locator::token_metadata_impl, std::default_delete<locator::token_metadata_impl> >::get() const /usr/include/c++/10/bits/unique_ptr.h:422
    scylladb#2 0x12e8d3e8 in std::unique_ptr<locator::token_metadata_impl, std::default_delete<locator::token_metadata_impl> >::operator->() const /usr/include/c++/10/bits/unique_ptr.h:416
    scylladb#3 0x12e5d0a2 in locator::token_metadata::ring_range(std::optional<interval_bound<dht::ring_position> > const&, bool) const locator/token_metadata.cc:1712
    scylladb#4 0x112d0126 in service::query_ranges_to_vnodes_generator::process_one_range(unsigned long, std::vector<nonwrapping_interval<dht::ring_position>, std::allocator<nonwrapping_interval<dht::ring_position> > >&) service/storage_proxy.cc:4658
    scylladb#5 0x112cf3c5 in service::query_ranges_to_vnodes_generator::operator()(unsigned long) service/storage_proxy.cc:4616
    scylladb#6 0x112b2261 in service::storage_proxy::query_partition_key_range_concurrent(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >, std::vector<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >, std::allocator<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&, seastar::lw_shared_ptr<query::read_command>, db::consistency_level, service::query_ranges_to_vnodes_generator&&, int, tracing::trace_state_ptr, unsigned long, unsigned int, std::unordered_map<nonwrapping_interval<dht::token>, std::vector<utils::UUID, std::allocator<utils::UUID> >, std::hash<nonwrapping_interval<dht::token> >, std::equal_to<nonwrapping_interval<dht::token> >, std::allocator<std::pair<nonwrapping_interval<dht::token> const, std::vector<utils::UUID, std::allocator<utils::UUID> > > > >, service_permit) service/storage_proxy.cc:4023
    scylladb#7 0x112b094e in operator() service/storage_proxy.cc:4160
    scylladb#8 0x1139c8bb in invoke<service::storage_proxy::query_partition_key_range_concurrent(seastar::lowres_clock::time_point, std::vector<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >&&, seastar::lw_shared_ptr<query::read_command>, db::consistency_level, service::query_ranges_to_vnodes_generator&&, int, tracing::trace_state_ptr, uint64_t, uint32_t, service::replicas_per_token_range, service_permit)::<lambda(seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:2088
    scylladb#9 0x1136625b in futurize_invoke<service::storage_proxy::query_partition_key_range_concurrent(seastar::lowres_clock::time_point, std::vector<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >&&, seastar::lw_shared_ptr<query::read_command>, db::consistency_level, service::query_ranges_to_vnodes_generator&&, int, tracing::trace_state_ptr, uint64_t, uint32_t, service::replicas_per_token_range, service_permit)::<lambda(seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:2119
    scylladb#10 0x11366372 in operator()<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1480
    scylladb#11 0x1139cc3b in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:145
    scylladb#12 0x116f4944 in seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>::operator()(seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:201
    scylladb#13 0x116b3397 in seastar::future<service::query_partition_key_range_concurrent_result> std::__invoke_impl<seastar::future<service::query_partition_key_range_concurrent_result>, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >(std::__invoke_other, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&) /usr/include/c++/10/bits/invoke.h:60
    scylladb#14 0x1165c3a6 in std::__invoke_result<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::type std::__invoke<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&) /usr/include/c++/10/bits/invoke.h:96
    scylladb#15 0x115e6542 in decltype(auto) std::__apply_impl<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >, 0ul>(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >&&, std::integer_sequence<unsigned long, 0ul>) /usr/include/c++/10/tuple:1724
    scylladb#16 0x115e6663 in decltype(auto) std::apply<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >&&) /usr/include/c++/10/tuple:1736
    scylladb#17 0x115e63f9 in seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}::operator()(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&) const::{lambda()scylladb#1}::operator()() const /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1530
    scylladb#18 0x1165c4b9 in void seastar::futurize<seastar::future<service::query_partition_key_range_concurrent_result> >::satisfy_with_result_of<seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}::operator()(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&) const::{lambda()scylladb#1}>(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:2073
    scylladb#19 0x115e61f5 in seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}::operator()(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1528
    scylladb#20 0x1176e9cc in seastar::continuation<seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::run_and_dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:746
    scylladb#21 0x16a9a455 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2196
    scylladb#22 0x16a9e691 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2575
    scylladb#23 0x16aa390e in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2730
    scylladb#24 0x168ae4f7 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:207
    scylladb#25 0x168ac541 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115
    scylladb#26 0xd6cd3c4 in main /local/home/bhalevy/dev/scylla/main.cc:504
    scylladb#27 0x7f8d905d8041 in __libc_start_main (/local/home/bhalevy/dev/scylla/build/debug/dynamic_libs/libc.so.6+0x27041)
    scylladb#28 0xd67c9ed in _start (/local/home/bhalevy/.dtest/dtest-o0qoqmkr/test/node3/bin/scylla+0xd67c9ed)

0x603000a8b180 is located 16 bytes inside of 24-byte region [0x603000a8b170,0x603000a8b188)
freed by thread T0 here:
    #0 0x7f8d92a190cf in operator delete(void*, unsigned long) (/local/home/bhalevy/dev/scylla/build/debug/dynamic_libs/libasan.so.6+0xb30cf)
    scylladb#1 0xd7ebe54 in seastar::internal::lw_shared_ptr_accessors_no_esft<locator::token_metadata>::dispose(seastar::lw_shared_ptr_counter_base*) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:213
    scylladb#2 0x112b155d in seastar::lw_shared_ptr<locator::token_metadata const>::~lw_shared_ptr() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:300
    scylladb#3 0x112b155d in ~<lambda> service/storage_proxy.cc:4137
    scylladb#4 0x1132e92d in ~<lambda> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1479
    scylladb#5 0x1139cc91 in destroy /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:148
    scylladb#6 0x11565673 in seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>::~noncopyable_function() /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:181
    scylladb#7 0x1176e783 in seastar::continuation<seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::~continuation() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:729
    scylladb#8 0x1176ea06 in seastar::continuation<seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::run_and_dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:750
    scylladb#9 0x16a9a455 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2196
    scylladb#10 0x16a9e691 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2575
    scylladb#11 0x16aa390e in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2730
    scylladb#12 0x168ae4f7 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:207
    scylladb#13 0x168ac541 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115
    scylladb#14 0xd6cd3c4 in main /local/home/bhalevy/dev/scylla/main.cc:504
    scylladb#15 0x7f8d905d8041 in __libc_start_main (/local/home/bhalevy/dev/scylla/build/debug/dynamic_libs/libc.so.6+0x27041)

previously allocated by thread T0 here:
    #0 0x7f8d92a18067 in operator new(unsigned long) (/local/home/bhalevy/dev/scylla/build/debug/dynamic_libs/libasan.so.6+0xb2067)
    scylladb#1 0x13cf7132 in seastar::lw_shared_ptr<locator::token_metadata> seastar::lw_shared_ptr<locator::token_metadata>::make<locator::token_metadata>(locator::token_metadata&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:266
    scylladb#2 0x13cc3bfa in seastar::lw_shared_ptr<locator::token_metadata> seastar::make_lw_shared<locator::token_metadata>(locator::token_metadata&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:422
    scylladb#3 0x13ca3007 in seastar::lw_shared_ptr<locator::token_metadata> locator::make_token_metadata_ptr<locator::token_metadata>(locator::token_metadata) locator/token_metadata.hh:338
    scylladb#4 0x13c9bdd4 in locator::shared_token_metadata::clone() const locator/token_metadata.hh:358
    scylladb#5 0x13c9c18a in service::storage_service::get_mutable_token_metadata_ptr() service/storage_service.hh:184
    scylladb#6 0x13a5a445 in service::storage_service::handle_state_normal(gms::inet_address) service/storage_service.cc:1129
    scylladb#7 0x13a6371c in service::storage_service::on_change(gms::inet_address, gms::application_state, gms::versioned_value const&) service/storage_service.cc:1421
    scylladb#8 0x12a86269 in operator() gms/gossiper.cc:1639
    scylladb#9 0x12ad3eea in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:145
    scylladb#10 0x12be2aff in seastar::noncopyable_function<void (seastar::shared_ptr<gms::i_endpoint_state_change_subscriber>)>::operator()(seastar::shared_ptr<gms::i_endpoint_state_change_subscriber>) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:201
    scylladb#11 0x12bb8e98 in atomic_vector<seastar::shared_ptr<gms::i_endpoint_state_change_subscriber> >::for_each(seastar::noncopyable_function<void (seastar::shared_ptr<gms::i_endpoint_state_change_subscriber>)>) utils/atomic_vector.hh:62
    scylladb#12 0x12a8662b in gms::gossiper::do_on_change_notifications(gms::inet_address, gms::application_state const&, gms::versioned_value const&) gms/gossiper.cc:1638
    scylladb#13 0x12a9387c in operator() gms/gossiper.cc:1978
    scylladb#14 0x12b49b20 in __invoke_impl<void, gms::gossiper::add_local_application_state(std::__cxx11::list<std::pair<gms::application_state, gms::versioned_value> >)::<lambda(gms::gossiper&)> mutable::<lambda()> > /usr/include/c++/10/bits/invoke.h:60
    scylladb#15 0x12b21fd6 in __invoke<gms::gossiper::add_local_application_state(std::__cxx11::list<std::pair<gms::application_state, gms::versioned_value> >)::<lambda(gms::gossiper&)> mutable::<lambda()> > /usr/include/c++/10/bits/invoke.h:95
    scylladb#16 0x12b02865 in __apply_impl<gms::gossiper::add_local_application_state(std::__cxx11::list<std::pair<gms::application_state, gms::versioned_value> >)::<lambda(gms::gossiper&)> mutable::<lambda()>, std::tuple<> > /usr/include/c++/10/tuple:1723
    scylladb#17 0x12b028d8 in apply<gms::gossiper::add_local_application_state(std::__cxx11::list<std::pair<gms::application_state, gms::versioned_value> >)::<lambda(gms::gossiper&)> mutable::<lambda()>, std::tuple<> > /usr/include/c++/10/tuple:1734
    scylladb#18 0x12b02967 in apply<gms::gossiper::add_local_application_state(std::__cxx11::list<std::pair<gms::application_state, gms::versioned_value> >)::<lambda(gms::gossiper&)> mutable::<lambda()> > /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:2052
    scylladb#19 0x12ad866a in operator() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/thread.hh:258
    scylladb#20 0x12b609c2 in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:116
    scylladb#21 0xdfabb5f in seastar::noncopyable_function<void ()>::operator()() const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:201
    scylladb#22 0x16e21bb4 in seastar::thread_context::main() /local/home/bhalevy/dev/scylla/seastar/src/core/thread.cc:297
    scylladb#23 0x16e2190f in seastar::thread_context::s_main(int, int) /local/home/bhalevy/dev/scylla/seastar/src/core/thread.cc:275
    scylladb#24 0x7f8d9060322f  (/local/home/bhalevy/dev/scylla/build/debug/dynamic_libs/libc.so.6+0x5222f)
```

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
bhalevy added a commit to bhalevy/scylla that referenced this issue Oct 4, 2020
@bhalevy
storage_proxy: query_ranges_to_vnodes_generator ranges_to_vnodes: use…
66522cf 
… token_metadata_ptr

Fixes use-after-free seen with putget_with_reloaded_certificates_test:
```
==215==ERROR: AddressSanitizer: heap-use-after-free on address 0x603000a8b180 at pc 0x000012eb5a83 bp 0x7ffd2c16d4c0 sp 0x7ffd2c16d4b0
READ of size 8 at 0x603000a8b180 thread T0
    #0 0x12eb5a82 in std::__uniq_ptr_impl<locator::token_metadata_impl, std::default_delete<locator::token_metadata_impl> >::_M_ptr() const /usr/include/c++/10/bits/unique_ptr.h:173
    scylladb#1 0x12ea230d in std::unique_ptr<locator::token_metadata_impl, std::default_delete<locator::token_metadata_impl> >::get() const /usr/include/c++/10/bits/unique_ptr.h:422
    scylladb#2 0x12e8d3e8 in std::unique_ptr<locator::token_metadata_impl, std::default_delete<locator::token_metadata_impl> >::operator->() const /usr/include/c++/10/bits/unique_ptr.h:416
    scylladb#3 0x12e5d0a2 in locator::token_metadata::ring_range(std::optional<interval_bound<dht::ring_position> > const&, bool) const locator/token_metadata.cc:1712
    scylladb#4 0x112d0126 in service::query_ranges_to_vnodes_generator::process_one_range(unsigned long, std::vector<nonwrapping_interval<dht::ring_position>, std::allocator<nonwrapping_interval<dht::ring_position> > >&) service/storage_proxy.cc:4658
    scylladb#5 0x112cf3c5 in service::query_ranges_to_vnodes_generator::operator()(unsigned long) service/storage_proxy.cc:4616
    scylladb#6 0x112b2261 in service::storage_proxy::query_partition_key_range_concurrent(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >, std::vector<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >, std::allocator<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&, seastar::lw_shared_ptr<query::read_command>, db::consistency_level, service::query_ranges_to_vnodes_generator&&, int, tracing::trace_state_ptr, unsigned long, unsigned int, std::unordered_map<nonwrapping_interval<dht::token>, std::vector<utils::UUID, std::allocator<utils::UUID> >, std::hash<nonwrapping_interval<dht::token> >, std::equal_to<nonwrapping_interval<dht::token> >, std::allocator<std::pair<nonwrapping_interval<dht::token> const, std::vector<utils::UUID, std::allocator<utils::UUID> > > > >, service_permit) service/storage_proxy.cc:4023
    scylladb#7 0x112b094e in operator() service/storage_proxy.cc:4160
    scylladb#8 0x1139c8bb in invoke<service::storage_proxy::query_partition_key_range_concurrent(seastar::lowres_clock::time_point, std::vector<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >&&, seastar::lw_shared_ptr<query::read_command>, db::consistency_level, service::query_ranges_to_vnodes_generator&&, int, tracing::trace_state_ptr, uint64_t, uint32_t, service::replicas_per_token_range, service_permit)::<lambda(seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:2088
    scylladb#9 0x1136625b in futurize_invoke<service::storage_proxy::query_partition_key_range_concurrent(seastar::lowres_clock::time_point, std::vector<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >&&, seastar::lw_shared_ptr<query::read_command>, db::consistency_level, service::query_ranges_to_vnodes_generator&&, int, tracing::trace_state_ptr, uint64_t, uint32_t, service::replicas_per_token_range, service_permit)::<lambda(seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:2119
    scylladb#10 0x11366372 in operator()<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1480
    scylladb#11 0x1139cc3b in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:145
    scylladb#12 0x116f4944 in seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>::operator()(seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:201
    scylladb#13 0x116b3397 in seastar::future<service::query_partition_key_range_concurrent_result> std::__invoke_impl<seastar::future<service::query_partition_key_range_concurrent_result>, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >(std::__invoke_other, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&) /usr/include/c++/10/bits/invoke.h:60
    scylladb#14 0x1165c3a6 in std::__invoke_result<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::type std::__invoke<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&) /usr/include/c++/10/bits/invoke.h:96
    scylladb#15 0x115e6542 in decltype(auto) std::__apply_impl<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >, 0ul>(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >&&, std::integer_sequence<unsigned long, 0ul>) /usr/include/c++/10/tuple:1724
    scylladb#16 0x115e6663 in decltype(auto) std::apply<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >&&) /usr/include/c++/10/tuple:1736
    scylladb#17 0x115e63f9 in seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}::operator()(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&) const::{lambda()scylladb#1}::operator()() const /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1530
    scylladb#18 0x1165c4b9 in void seastar::futurize<seastar::future<service::query_partition_key_range_concurrent_result> >::satisfy_with_result_of<seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}::operator()(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&) const::{lambda()scylladb#1}>(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:2073
    scylladb#19 0x115e61f5 in seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}::operator()(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1528
    scylladb#20 0x1176e9cc in seastar::continuation<seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::run_and_dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:746
    scylladb#21 0x16a9a455 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2196
    scylladb#22 0x16a9e691 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2575
    scylladb#23 0x16aa390e in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2730
    scylladb#24 0x168ae4f7 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:207
    scylladb#25 0x168ac541 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115
    scylladb#26 0xd6cd3c4 in main /local/home/bhalevy/dev/scylla/main.cc:504
    scylladb#27 0x7f8d905d8041 in __libc_start_main (/local/home/bhalevy/dev/scylla/build/debug/dynamic_libs/libc.so.6+0x27041)
    scylladb#28 0xd67c9ed in _start (/local/home/bhalevy/.dtest/dtest-o0qoqmkr/test/node3/bin/scylla+0xd67c9ed)

0x603000a8b180 is located 16 bytes inside of 24-byte region [0x603000a8b170,0x603000a8b188)
freed by thread T0 here:
    #0 0x7f8d92a190cf in operator delete(void*, unsigned long) (/local/home/bhalevy/dev/scylla/build/debug/dynamic_libs/libasan.so.6+0xb30cf)
    scylladb#1 0xd7ebe54 in seastar::internal::lw_shared_ptr_accessors_no_esft<locator::token_metadata>::dispose(seastar::lw_shared_ptr_counter_base*) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:213
    scylladb#2 0x112b155d in seastar::lw_shared_ptr<locator::token_metadata const>::~lw_shared_ptr() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:300
    scylladb#3 0x112b155d in ~<lambda> service/storage_proxy.cc:4137
    scylladb#4 0x1132e92d in ~<lambda> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1479
    scylladb#5 0x1139cc91 in destroy /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:148
    scylladb#6 0x11565673 in seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>::~noncopyable_function() /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:181
    scylladb#7 0x1176e783 in seastar::continuation<seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::~continuation() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:729
    scylladb#8 0x1176ea06 in seastar::continuation<seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::run_and_dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:750
    scylladb#9 0x16a9a455 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2196
    scylladb#10 0x16a9e691 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2575
    scylladb#11 0x16aa390e in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2730
    scylladb#12 0x168ae4f7 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:207
    scylladb#13 0x168ac541 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115
    scylladb#14 0xd6cd3c4 in main /local/home/bhalevy/dev/scylla/main.cc:504
    scylladb#15 0x7f8d905d8041 in __libc_start_main (/local/home/bhalevy/dev/scylla/build/debug/dynamic_libs/libc.so.6+0x27041)

previously allocated by thread T0 here:
    #0 0x7f8d92a18067 in operator new(unsigned long) (/local/home/bhalevy/dev/scylla/build/debug/dynamic_libs/libasan.so.6+0xb2067)
    scylladb#1 0x13cf7132 in seastar::lw_shared_ptr<locator::token_metadata> seastar::lw_shared_ptr<locator::token_metadata>::make<locator::token_metadata>(locator::token_metadata&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:266
    scylladb#2 0x13cc3bfa in seastar::lw_shared_ptr<locator::token_metadata> seastar::make_lw_shared<locator::token_metadata>(locator::token_metadata&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:422
    scylladb#3 0x13ca3007 in seastar::lw_shared_ptr<locator::token_metadata> locator::make_token_metadata_ptr<locator::token_metadata>(locator::token_metadata) locator/token_metadata.hh:338
    scylladb#4 0x13c9bdd4 in locator::shared_token_metadata::clone() const locator/token_metadata.hh:358
    scylladb#5 0x13c9c18a in service::storage_service::get_mutable_token_metadata_ptr() service/storage_service.hh:184
    scylladb#6 0x13a5a445 in service::storage_service::handle_state_normal(gms::inet_address) service/storage_service.cc:1129
    scylladb#7 0x13a6371c in service::storage_service::on_change(gms::inet_address, gms::application_state, gms::versioned_value const&) service/storage_service.cc:1421
    scylladb#8 0x12a86269 in operator() gms/gossiper.cc:1639
    scylladb#9 0x12ad3eea in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:145
    scylladb#10 0x12be2aff in seastar::noncopyable_function<void (seastar::shared_ptr<gms::i_endpoint_state_change_subscriber>)>::operator()(seastar::shared_ptr<gms::i_endpoint_state_change_subscriber>) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:201
    scylladb#11 0x12bb8e98 in atomic_vector<seastar::shared_ptr<gms::i_endpoint_state_change_subscriber> >::for_each(seastar::noncopyable_function<void (seastar::shared_ptr<gms::i_endpoint_state_change_subscriber>)>) utils/atomic_vector.hh:62
    scylladb#12 0x12a8662b in gms::gossiper::do_on_change_notifications(gms::inet_address, gms::application_state const&, gms::versioned_value const&) gms/gossiper.cc:1638
    scylladb#13 0x12a9387c in operator() gms/gossiper.cc:1978
    scylladb#14 0x12b49b20 in __invoke_impl<void, gms::gossiper::add_local_application_state(std::__cxx11::list<std::pair<gms::application_state, gms::versioned_value> >)::<lambda(gms::gossiper&)> mutable::<lambda()> > /usr/include/c++/10/bits/invoke.h:60
    scylladb#15 0x12b21fd6 in __invoke<gms::gossiper::add_local_application_state(std::__cxx11::list<std::pair<gms::application_state, gms::versioned_value> >)::<lambda(gms::gossiper&)> mutable::<lambda()> > /usr/include/c++/10/bits/invoke.h:95
    scylladb#16 0x12b02865 in __apply_impl<gms::gossiper::add_local_application_state(std::__cxx11::list<std::pair<gms::application_state, gms::versioned_value> >)::<lambda(gms::gossiper&)> mutable::<lambda()>, std::tuple<> > /usr/include/c++/10/tuple:1723
    scylladb#17 0x12b028d8 in apply<gms::gossiper::add_local_application_state(std::__cxx11::list<std::pair<gms::application_state, gms::versioned_value> >)::<lambda(gms::gossiper&)> mutable::<lambda()>, std::tuple<> > /usr/include/c++/10/tuple:1734
    scylladb#18 0x12b02967 in apply<gms::gossiper::add_local_application_state(std::__cxx11::list<std::pair<gms::application_state, gms::versioned_value> >)::<lambda(gms::gossiper&)> mutable::<lambda()> > /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:2052
    scylladb#19 0x12ad866a in operator() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/thread.hh:258
    scylladb#20 0x12b609c2 in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:116
    scylladb#21 0xdfabb5f in seastar::noncopyable_function<void ()>::operator()() const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:201
    scylladb#22 0x16e21bb4 in seastar::thread_context::main() /local/home/bhalevy/dev/scylla/seastar/src/core/thread.cc:297
    scylladb#23 0x16e2190f in seastar::thread_context::s_main(int, int) /local/home/bhalevy/dev/scylla/seastar/src/core/thread.cc:275
    scylladb#24 0x7f8d9060322f  (/local/home/bhalevy/dev/scylla/build/debug/dynamic_libs/libc.so.6+0x5222f)
```

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
bhalevy added a commit to bhalevy/scylla that referenced this issue Oct 22, 2020
@bhalevy
storage_proxy: query_ranges_to_vnodes_generator ranges_to_vnodes: use…
9457a35 
… token_metadata_ptr

Fixes use-after-free seen with putget_with_reloaded_certificates_test:
```
==215==ERROR: AddressSanitizer: heap-use-after-free on address 0x603000a8b180 at pc 0x000012eb5a83 bp 0x7ffd2c16d4c0 sp 0x7ffd2c16d4b0
READ of size 8 at 0x603000a8b180 thread T0
    #0 0x12eb5a82 in std::__uniq_ptr_impl<locator::token_metadata_impl, std::default_delete<locator::token_metadata_impl> >::_M_ptr() const /usr/include/c++/10/bits/unique_ptr.h:173
    scylladb#1 0x12ea230d in std::unique_ptr<locator::token_metadata_impl, std::default_delete<locator::token_metadata_impl> >::get() const /usr/include/c++/10/bits/unique_ptr.h:422
    scylladb#2 0x12e8d3e8 in std::unique_ptr<locator::token_metadata_impl, std::default_delete<locator::token_metadata_impl> >::operator->() const /usr/include/c++/10/bits/unique_ptr.h:416
    scylladb#3 0x12e5d0a2 in locator::token_metadata::ring_range(std::optional<interval_bound<dht::ring_position> > const&, bool) const locator/token_metadata.cc:1712
    scylladb#4 0x112d0126 in service::query_ranges_to_vnodes_generator::process_one_range(unsigned long, std::vector<nonwrapping_interval<dht::ring_position>, std::allocator<nonwrapping_interval<dht::ring_position> > >&) service/storage_proxy.cc:4658
    scylladb#5 0x112cf3c5 in service::query_ranges_to_vnodes_generator::operator()(unsigned long) service/storage_proxy.cc:4616
    scylladb#6 0x112b2261 in service::storage_proxy::query_partition_key_range_concurrent(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >, std::vector<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >, std::allocator<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&, seastar::lw_shared_ptr<query::read_command>, db::consistency_level, service::query_ranges_to_vnodes_generator&&, int, tracing::trace_state_ptr, unsigned long, unsigned int, std::unordered_map<nonwrapping_interval<dht::token>, std::vector<utils::UUID, std::allocator<utils::UUID> >, std::hash<nonwrapping_interval<dht::token> >, std::equal_to<nonwrapping_interval<dht::token> >, std::allocator<std::pair<nonwrapping_interval<dht::token> const, std::vector<utils::UUID, std::allocator<utils::UUID> > > > >, service_permit) service/storage_proxy.cc:4023
    scylladb#7 0x112b094e in operator() service/storage_proxy.cc:4160
    scylladb#8 0x1139c8bb in invoke<service::storage_proxy::query_partition_key_range_concurrent(seastar::lowres_clock::time_point, std::vector<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >&&, seastar::lw_shared_ptr<query::read_command>, db::consistency_level, service::query_ranges_to_vnodes_generator&&, int, tracing::trace_state_ptr, uint64_t, uint32_t, service::replicas_per_token_range, service_permit)::<lambda(seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:2088
    scylladb#9 0x1136625b in futurize_invoke<service::storage_proxy::query_partition_key_range_concurrent(seastar::lowres_clock::time_point, std::vector<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >&&, seastar::lw_shared_ptr<query::read_command>, db::consistency_level, service::query_ranges_to_vnodes_generator&&, int, tracing::trace_state_ptr, uint64_t, uint32_t, service::replicas_per_token_range, service_permit)::<lambda(seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:2119
    scylladb#10 0x11366372 in operator()<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1480
    scylladb#11 0x1139cc3b in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:145
    scylladb#12 0x116f4944 in seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>::operator()(seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:201
    scylladb#13 0x116b3397 in seastar::future<service::query_partition_key_range_concurrent_result> std::__invoke_impl<seastar::future<service::query_partition_key_range_concurrent_result>, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >(std::__invoke_other, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&) /usr/include/c++/10/bits/invoke.h:60
    scylladb#14 0x1165c3a6 in std::__invoke_result<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::type std::__invoke<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&) /usr/include/c++/10/bits/invoke.h:96
    scylladb#15 0x115e6542 in decltype(auto) std::__apply_impl<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >, 0ul>(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >&&, std::integer_sequence<unsigned long, 0ul>) /usr/include/c++/10/tuple:1724
    scylladb#16 0x115e6663 in decltype(auto) std::apply<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >&&) /usr/include/c++/10/tuple:1736
    scylladb#17 0x115e63f9 in seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}::operator()(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&) const::{lambda()scylladb#1}::operator()() const /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1530
    scylladb#18 0x1165c4b9 in void seastar::futurize<seastar::future<service::query_partition_key_range_concurrent_result> >::satisfy_with_result_of<seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}::operator()(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&) const::{lambda()scylladb#1}>(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:2073
    scylladb#19 0x115e61f5 in seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}::operator()(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1528
    scylladb#20 0x1176e9cc in seastar::continuation<seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::run_and_dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:746
    scylladb#21 0x16a9a455 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2196
    scylladb#22 0x16a9e691 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2575
    scylladb#23 0x16aa390e in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2730
    scylladb#24 0x168ae4f7 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:207
    scylladb#25 0x168ac541 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115
    scylladb#26 0xd6cd3c4 in main /local/home/bhalevy/dev/scylla/main.cc:504
    scylladb#27 0x7f8d905d8041 in __libc_start_main (/local/home/bhalevy/dev/scylla/build/debug/dynamic_libs/libc.so.6+0x27041)
    scylladb#28 0xd67c9ed in _start (/local/home/bhalevy/.dtest/dtest-o0qoqmkr/test/node3/bin/scylla+0xd67c9ed)

0x603000a8b180 is located 16 bytes inside of 24-byte region [0x603000a8b170,0x603000a8b188)
freed by thread T0 here:
    #0 0x7f8d92a190cf in operator delete(void*, unsigned long) (/local/home/bhalevy/dev/scylla/build/debug/dynamic_libs/libasan.so.6+0xb30cf)
    scylladb#1 0xd7ebe54 in seastar::internal::lw_shared_ptr_accessors_no_esft<locator::token_metadata>::dispose(seastar::lw_shared_ptr_counter_base*) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:213
    scylladb#2 0x112b155d in seastar::lw_shared_ptr<locator::token_metadata const>::~lw_shared_ptr() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:300
    scylladb#3 0x112b155d in ~<lambda> service/storage_proxy.cc:4137
    scylladb#4 0x1132e92d in ~<lambda> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1479
    scylladb#5 0x1139cc91 in destroy /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:148
    scylladb#6 0x11565673 in seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>::~noncopyable_function() /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:181
    scylladb#7 0x1176e783 in seastar::continuation<seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::~continuation() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:729
    scylladb#8 0x1176ea06 in seastar::continuation<seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::run_and_dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:750
    scylladb#9 0x16a9a455 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2196
    scylladb#10 0x16a9e691 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2575
    scylladb#11 0x16aa390e in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2730
    scylladb#12 0x168ae4f7 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:207
    scylladb#13 0x168ac541 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115
    scylladb#14 0xd6cd3c4 in main /local/home/bhalevy/dev/scylla/main.cc:504
    scylladb#15 0x7f8d905d8041 in __libc_start_main (/local/home/bhalevy/dev/scylla/build/debug/dynamic_libs/libc.so.6+0x27041)

previously allocated by thread T0 here:
    #0 0x7f8d92a18067 in operator new(unsigned long) (/local/home/bhalevy/dev/scylla/build/debug/dynamic_libs/libasan.so.6+0xb2067)
    scylladb#1 0x13cf7132 in seastar::lw_shared_ptr<locator::token_metadata> seastar::lw_shared_ptr<locator::token_metadata>::make<locator::token_metadata>(locator::token_metadata&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:266
    scylladb#2 0x13cc3bfa in seastar::lw_shared_ptr<locator::token_metadata> seastar::make_lw_shared<locator::token_metadata>(locator::token_metadata&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:422
    scylladb#3 0x13ca3007 in seastar::lw_shared_ptr<locator::token_metadata> locator::make_token_metadata_ptr<locator::token_metadata>(locator::token_metadata) locator/token_metadata.hh:338
    scylladb#4 0x13c9bdd4 in locator::shared_token_metadata::clone() const locator/token_metadata.hh:358
    scylladb#5 0x13c9c18a in service::storage_service::get_mutable_token_metadata_ptr() service/storage_service.hh:184
    scylladb#6 0x13a5a445 in service::storage_service::handle_state_normal(gms::inet_address) service/storage_service.cc:1129
    scylladb#7 0x13a6371c in service::storage_service::on_change(gms::inet_address, gms::application_state, gms::versioned_value const&) service/storage_service.cc:1421
    scylladb#8 0x12a86269 in operator() gms/gossiper.cc:1639
    scylladb#9 0x12ad3eea in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:145
    scylladb#10 0x12be2aff in seastar::noncopyable_function<void (seastar::shared_ptr<gms::i_endpoint_state_change_subscriber>)>::operator()(seastar::shared_ptr<gms::i_endpoint_state_change_subscriber>) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:201
    scylladb#11 0x12bb8e98 in atomic_vector<seastar::shared_ptr<gms::i_endpoint_state_change_subscriber> >::for_each(seastar::noncopyable_function<void (seastar::shared_ptr<gms::i_endpoint_state_change_subscriber>)>) utils/atomic_vector.hh:62
    scylladb#12 0x12a8662b in gms::gossiper::do_on_change_notifications(gms::inet_address, gms::application_state const&, gms::versioned_value const&) gms/gossiper.cc:1638
    scylladb#13 0x12a9387c in operator() gms/gossiper.cc:1978
    scylladb#14 0x12b49b20 in __invoke_impl<void, gms::gossiper::add_local_application_state(std::__cxx11::list<std::pair<gms::application_state, gms::versioned_value> >)::<lambda(gms::gossiper&)> mutable::<lambda()> > /usr/include/c++/10/bits/invoke.h:60
    scylladb#15 0x12b21fd6 in __invoke<gms::gossiper::add_local_application_state(std::__cxx11::list<std::pair<gms::application_state, gms::versioned_value> >)::<lambda(gms::gossiper&)> mutable::<lambda()> > /usr/include/c++/10/bits/invoke.h:95
    scylladb#16 0x12b02865 in __apply_impl<gms::gossiper::add_local_application_state(std::__cxx11::list<std::pair<gms::application_state, gms::versioned_value> >)::<lambda(gms::gossiper&)> mutable::<lambda()>, std::tuple<> > /usr/include/c++/10/tuple:1723
    scylladb#17 0x12b028d8 in apply<gms::gossiper::add_local_application_state(std::__cxx11::list<std::pair<gms::application_state, gms::versioned_value> >)::<lambda(gms::gossiper&)> mutable::<lambda()>, std::tuple<> > /usr/include/c++/10/tuple:1734
    scylladb#18 0x12b02967 in apply<gms::gossiper::add_local_application_state(std::__cxx11::list<std::pair<gms::application_state, gms::versioned_value> >)::<lambda(gms::gossiper&)> mutable::<lambda()> > /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:2052
    scylladb#19 0x12ad866a in operator() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/thread.hh:258
    scylladb#20 0x12b609c2 in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:116
    scylladb#21 0xdfabb5f in seastar::noncopyable_function<void ()>::operator()() const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:201
    scylladb#22 0x16e21bb4 in seastar::thread_context::main() /local/home/bhalevy/dev/scylla/seastar/src/core/thread.cc:297
    scylladb#23 0x16e2190f in seastar::thread_context::s_main(int, int) /local/home/bhalevy/dev/scylla/seastar/src/core/thread.cc:275
    scylladb#24 0x7f8d9060322f  (/local/home/bhalevy/dev/scylla/build/debug/dynamic_libs/libc.so.6+0x5222f)
```

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
@bhalevy bhalevy mentioned this issue Nov 8, 2020
Closed
bhalevy added a commit to bhalevy/scylla that referenced this issue Nov 11, 2020
@bhalevy
storage_proxy: query_ranges_to_vnodes_generator ranges_to_vnodes: use…
5726386 
… token_metadata_ptr

Fixes use-after-free seen with putget_with_reloaded_certificates_test:
```
==215==ERROR: AddressSanitizer: heap-use-after-free on address 0x603000a8b180 at pc 0x000012eb5a83 bp 0x7ffd2c16d4c0 sp 0x7ffd2c16d4b0
READ of size 8 at 0x603000a8b180 thread T0
    #0 0x12eb5a82 in std::__uniq_ptr_impl<locator::token_metadata_impl, std::default_delete<locator::token_metadata_impl> >::_M_ptr() const /usr/include/c++/10/bits/unique_ptr.h:173
    scylladb#1 0x12ea230d in std::unique_ptr<locator::token_metadata_impl, std::default_delete<locator::token_metadata_impl> >::get() const /usr/include/c++/10/bits/unique_ptr.h:422
    scylladb#2 0x12e8d3e8 in std::unique_ptr<locator::token_metadata_impl, std::default_delete<locator::token_metadata_impl> >::operator->() const /usr/include/c++/10/bits/unique_ptr.h:416
    scylladb#3 0x12e5d0a2 in locator::token_metadata::ring_range(std::optional<interval_bound<dht::ring_position> > const&, bool) const locator/token_metadata.cc:1712
    scylladb#4 0x112d0126 in service::query_ranges_to_vnodes_generator::process_one_range(unsigned long, std::vector<nonwrapping_interval<dht::ring_position>, std::allocator<nonwrapping_interval<dht::ring_position> > >&) service/storage_proxy.cc:4658
    scylladb#5 0x112cf3c5 in service::query_ranges_to_vnodes_generator::operator()(unsigned long) service/storage_proxy.cc:4616
    scylladb#6 0x112b2261 in service::storage_proxy::query_partition_key_range_concurrent(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >, std::vector<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >, std::allocator<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&, seastar::lw_shared_ptr<query::read_command>, db::consistency_level, service::query_ranges_to_vnodes_generator&&, int, tracing::trace_state_ptr, unsigned long, unsigned int, std::unordered_map<nonwrapping_interval<dht::token>, std::vector<utils::UUID, std::allocator<utils::UUID> >, std::hash<nonwrapping_interval<dht::token> >, std::equal_to<nonwrapping_interval<dht::token> >, std::allocator<std::pair<nonwrapping_interval<dht::token> const, std::vector<utils::UUID, std::allocator<utils::UUID> > > > >, service_permit) service/storage_proxy.cc:4023
    scylladb#7 0x112b094e in operator() service/storage_proxy.cc:4160
    scylladb#8 0x1139c8bb in invoke<service::storage_proxy::query_partition_key_range_concurrent(seastar::lowres_clock::time_point, std::vector<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >&&, seastar::lw_shared_ptr<query::read_command>, db::consistency_level, service::query_ranges_to_vnodes_generator&&, int, tracing::trace_state_ptr, uint64_t, uint32_t, service::replicas_per_token_range, service_permit)::<lambda(seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:2088
    scylladb#9 0x1136625b in futurize_invoke<service::storage_proxy::query_partition_key_range_concurrent(seastar::lowres_clock::time_point, std::vector<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >&&, seastar::lw_shared_ptr<query::read_command>, db::consistency_level, service::query_ranges_to_vnodes_generator&&, int, tracing::trace_state_ptr, uint64_t, uint32_t, service::replicas_per_token_range, service_permit)::<lambda(seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:2119
    scylladb#10 0x11366372 in operator()<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1480
    scylladb#11 0x1139cc3b in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:145
    scylladb#12 0x116f4944 in seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>::operator()(seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:201
    scylladb#13 0x116b3397 in seastar::future<service::query_partition_key_range_concurrent_result> std::__invoke_impl<seastar::future<service::query_partition_key_range_concurrent_result>, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >(std::__invoke_other, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&) /usr/include/c++/10/bits/invoke.h:60
    scylladb#14 0x1165c3a6 in std::__invoke_result<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::type std::__invoke<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&) /usr/include/c++/10/bits/invoke.h:96
    scylladb#15 0x115e6542 in decltype(auto) std::__apply_impl<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >, 0ul>(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >&&, std::integer_sequence<unsigned long, 0ul>) /usr/include/c++/10/tuple:1724
    scylladb#16 0x115e6663 in decltype(auto) std::apply<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >&&) /usr/include/c++/10/tuple:1736
    scylladb#17 0x115e63f9 in seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}::operator()(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&) const::{lambda()scylladb#1}::operator()() const /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1530
    scylladb#18 0x1165c4b9 in void seastar::futurize<seastar::future<service::query_partition_key_range_concurrent_result> >::satisfy_with_result_of<seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}::operator()(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&) const::{lambda()scylladb#1}>(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:2073
    scylladb#19 0x115e61f5 in seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}::operator()(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1528
    scylladb#20 0x1176e9cc in seastar::continuation<seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::run_and_dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:746
    scylladb#21 0x16a9a455 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2196
    scylladb#22 0x16a9e691 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2575
    scylladb#23 0x16aa390e in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2730
    scylladb#24 0x168ae4f7 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:207
    scylladb#25 0x168ac541 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115
    scylladb#26 0xd6cd3c4 in main /local/home/bhalevy/dev/scylla/main.cc:504
    scylladb#27 0x7f8d905d8041 in __libc_start_main (/local/home/bhalevy/dev/scylla/build/debug/dynamic_libs/libc.so.6+0x27041)
    scylladb#28 0xd67c9ed in _start (/local/home/bhalevy/.dtest/dtest-o0qoqmkr/test/node3/bin/scylla+0xd67c9ed)

0x603000a8b180 is located 16 bytes inside of 24-byte region [0x603000a8b170,0x603000a8b188)
freed by thread T0 here:
    #0 0x7f8d92a190cf in operator delete(void*, unsigned long) (/local/home/bhalevy/dev/scylla/build/debug/dynamic_libs/libasan.so.6+0xb30cf)
    scylladb#1 0xd7ebe54 in seastar::internal::lw_shared_ptr_accessors_no_esft<locator::token_metadata>::dispose(seastar::lw_shared_ptr_counter_base*) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:213
    scylladb#2 0x112b155d in seastar::lw_shared_ptr<locator::token_metadata const>::~lw_shared_ptr() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:300
    scylladb#3 0x112b155d in ~<lambda> service/storage_proxy.cc:4137
    scylladb#4 0x1132e92d in ~<lambda> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1479
    scylladb#5 0x1139cc91 in destroy /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:148
    scylladb#6 0x11565673 in seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>::~noncopyable_function() /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:181
    scylladb#7 0x1176e783 in seastar::continuation<seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::~continuation() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:729
    scylladb#8 0x1176ea06 in seastar::continuation<seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::run_and_dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:750
    scylladb#9 0x16a9a455 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2196
    scylladb#10 0x16a9e691 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2575
    scylladb#11 0x16aa390e in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2730
    scylladb#12 0x168ae4f7 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:207
    scylladb#13 0x168ac541 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115
    scylladb#14 0xd6cd3c4 in main /local/home/bhalevy/dev/scylla/main.cc:504
    scylladb#15 0x7f8d905d8041 in __libc_start_main (/local/home/bhalevy/dev/scylla/build/debug/dynamic_libs/libc.so.6+0x27041)

previously allocated by thread T0 here:
    #0 0x7f8d92a18067 in operator new(unsigned long) (/local/home/bhalevy/dev/scylla/build/debug/dynamic_libs/libasan.so.6+0xb2067)
    scylladb#1 0x13cf7132 in seastar::lw_shared_ptr<locator::token_metadata> seastar::lw_shared_ptr<locator::token_metadata>::make<locator::token_metadata>(locator::token_metadata&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:266
    scylladb#2 0x13cc3bfa in seastar::lw_shared_ptr<locator::token_metadata> seastar::make_lw_shared<locator::token_metadata>(locator::token_metadata&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:422
    scylladb#3 0x13ca3007 in seastar::lw_shared_ptr<locator::token_metadata> locator::make_token_metadata_ptr<locator::token_metadata>(locator::token_metadata) locator/token_metadata.hh:338
    scylladb#4 0x13c9bdd4 in locator::shared_token_metadata::clone() const locator/token_metadata.hh:358
    scylladb#5 0x13c9c18a in service::storage_service::get_mutable_token_metadata_ptr() service/storage_service.hh:184
    scylladb#6 0x13a5a445 in service::storage_service::handle_state_normal(gms::inet_address) service/storage_service.cc:1129
    scylladb#7 0x13a6371c in service::storage_service::on_change(gms::inet_address, gms::application_state, gms::versioned_value const&) service/storage_service.cc:1421
    scylladb#8 0x12a86269 in operator() gms/gossiper.cc:1639
    scylladb#9 0x12ad3eea in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:145
    scylladb#10 0x12be2aff in seastar::noncopyable_function<void (seastar::shared_ptr<gms::i_endpoint_state_change_subscriber>)>::operator()(seastar::shared_ptr<gms::i_endpoint_state_change_subscriber>) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:201
    scylladb#11 0x12bb8e98 in atomic_vector<seastar::shared_ptr<gms::i_endpoint_state_change_subscriber> >::for_each(seastar::noncopyable_function<void (seastar::shared_ptr<gms::i_endpoint_state_change_subscriber>)>) utils/atomic_vector.hh:62
    scylladb#12 0x12a8662b in gms::gossiper::do_on_change_notifications(gms::inet_address, gms::application_state const&, gms::versioned_value const&) gms/gossiper.cc:1638
    scylladb#13 0x12a9387c in operator() gms/gossiper.cc:1978
    scylladb#14 0x12b49b20 in __invoke_impl<void, gms::gossiper::add_local_application_state(std::__cxx11::list<std::pair<gms::application_state, gms::versioned_value> >)::<lambda(gms::gossiper&)> mutable::<lambda()> > /usr/include/c++/10/bits/invoke.h:60
    scylladb#15 0x12b21fd6 in __invoke<gms::gossiper::add_local_application_state(std::__cxx11::list<std::pair<gms::application_state, gms::versioned_value> >)::<lambda(gms::gossiper&)> mutable::<lambda()> > /usr/include/c++/10/bits/invoke.h:95
    scylladb#16 0x12b02865 in __apply_impl<gms::gossiper::add_local_application_state(std::__cxx11::list<std::pair<gms::application_state, gms::versioned_value> >)::<lambda(gms::gossiper&)> mutable::<lambda()>, std::tuple<> > /usr/include/c++/10/tuple:1723
    scylladb#17 0x12b028d8 in apply<gms::gossiper::add_local_application_state(std::__cxx11::list<std::pair<gms::application_state, gms::versioned_value> >)::<lambda(gms::gossiper&)> mutable::<lambda()>, std::tuple<> > /usr/include/c++/10/tuple:1734
    scylladb#18 0x12b02967 in apply<gms::gossiper::add_local_application_state(std::__cxx11::list<std::pair<gms::application_state, gms::versioned_value> >)::<lambda(gms::gossiper&)> mutable::<lambda()> > /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:2052
    scylladb#19 0x12ad866a in operator() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/thread.hh:258
    scylladb#20 0x12b609c2 in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:116
    scylladb#21 0xdfabb5f in seastar::noncopyable_function<void ()>::operator()() const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:201
    scylladb#22 0x16e21bb4 in seastar::thread_context::main() /local/home/bhalevy/dev/scylla/seastar/src/core/thread.cc:297
    scylladb#23 0x16e2190f in seastar::thread_context::s_main(int, int) /local/home/bhalevy/dev/scylla/seastar/src/core/thread.cc:275
    scylladb#24 0x7f8d9060322f  (/local/home/bhalevy/dev/scylla/build/debug/dynamic_libs/libc.so.6+0x5222f)
```

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
@bhalevy bhalevy mentioned this issue Mar 11, 2021
Open
asias added a commit to asias/scylla that referenced this issue Jun 7, 2022
@asias
repair: Avoid stall in apply_rows_on_master_in_thread
45bcacf 
Fix the following stall during repair:

```
Reactor stalled for 3 ms on shard 0. Backtrace:
[Backtrace scylladb#20]
{build/release/scylla} 0x4c6deb2: void seastar::backtrace<seastar::backtrace_buffer::append_backtrace_oneline()::{lambda(seastar::frame)scylladb#1}>(seastar::backtrace_buffer::append_backtrace_oneline()::{lambda(seastar::frame)scylladb#1}&&) at ./build/release/seastar/./seastar/include/seastar/util/backtrace.hh:59
 (inlined by) seastar::backtrace_buffer::append_backtrace_oneline() at ./build/release/seastar/./seastar/src/core/reactor.cc:772
 (inlined by) seastar::print_with_backtrace(seastar::backtrace_buffer&, bool) at ./build/release/seastar/./seastar/src/core/reactor.cc:791
{build/release/scylla} 0x4c6cb10: seastar::internal::cpu_stall_detector::generate_trace() at ./build/release/seastar/./seastar/src/core/reactor.cc:1366
{build/release/scylla} 0x4c6ddc0: seastar::internal::cpu_stall_detector::maybe_report() at ./build/release/seastar/./seastar/src/core/reactor.cc:1108
 (inlined by) seastar::internal::cpu_stall_detector::on_signal() at ./build/release/seastar/./seastar/src/core/reactor.cc:1125
 (inlined by) seastar::reactor::block_notifier(int) at ./build/release/seastar/./seastar/src/core/reactor.cc:1349
{build/release/scylla} 0x7f75551bfa1f: ?? ??:0
{build/release/scylla} 0x11293e9: std::default_delete<bytes_ostream::chunk>::operator()(bytes_ostream::chunk*) const at database.cc:?
 (inlined by) std::default_delete<bytes_ostream::chunk>::operator()(bytes_ostream::chunk*) const at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/unique_ptr.h:85
{build/release/scylla} 0x37b18e6: ~unique_ptr at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/unique_ptr.h:361
 (inlined by) ~bytes_ostream at ././bytes_ostream.hh:26
 (inlined by) ~frozen_mutation_fragment at ././frozen_mutation.hh:265
 (inlined by) std::_Optional_payload_base<frozen_mutation_fragment>::_M_destroy() at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/optional:260
 (inlined by) std::_Optional_payload_base<frozen_mutation_fragment>::_M_reset() at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/optional:280
 (inlined by) ~_Optional_payload at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/optional:401
 (inlined by) ~_Optional_base at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/optional:472
 (inlined by) ~repair_row at ././repair/row.hh:24
 (inlined by) void std::destroy_at<repair_row>(repair_row*) at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/stl_construct.h:88
 (inlined by) void std::allocator_traits<std::allocator<std::_List_node<repair_row> > >::destroy<repair_row>(std::allocator<std::_List_node<repair_row> >&, repair_row*) at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/alloc_traits.h:537
 (inlined by) std::__cxx11::_List_base<repair_row, std::allocator<repair_row> >::_M_clear() at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/list.tcc:77
 (inlined by) ~_List_base at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/stl_list.h:499
 (inlined by) repair_meta::apply_rows_on_master_in_thread(std::__cxx11::list<partition_key_and_mutation_fragments, std::allocator<partition_key_and_mutation_fragments> >, gms::inet_address, seastar::bool_class<update_working_row_buf_tag>, seastar::bool_class<update_peer_row_hash_sets_tag>, unsigned int) at ./repair/row_level.cc:1273
{build/release/scylla} 0x37ad9dc: repair_meta::get_row_diff_source_op(seastar::bool_class<update_peer_row_hash_sets_tag>, gms::inet_address, unsigned int, seastar::rpc::sink<repair_hash_with_cmd>&, seastar::rpc::source<repair_row_on_wire_with_cmd>&) at ./repair/row_level.cc:1617
{build/release/scylla} 0x37a2982: repair_meta::get_row_diff_with_rpc_stream(absl::btree_set<repair_hash, std::less<repair_hash>, std::allocator<repair_hash> >, seastar::bool_class<needs_all_rows_tag>, seastar::bool_class<update_peer_row_hash_sets_tag>, gms::inet_address, unsigned int) at ./repair/row_level.cc:1683
```

Found with scylla --blocked-reactor-notify-ms 1

Refs scylladb#10665
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@tgrabiec @avikivity