New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
SIGSEGV when running multilple cassandra-stress with user profile and smp #25
Comments
@penberg with the following head I still get a SIGSEGV
|
@slivne, I am trying to reproduce locally but all I get is this error when I try to run cassandra-stress:
|
Please use the modified cassandra stress available at: we do not support prepared cql over thrift - so I had removed it from the cassandra-stress code |
I am not able to reproduce locally with "--smp 2" which is not that surprising. Can you please try to reproduce the issue with the debug build on Godzilla? Please note that commit b3b0d51 might be related to the issue at hand. |
@penberg There already is my unmerged patchset which adds support for executing "normal" CQL queries over Thrift. That already deals with maintaining proper connection state and translating CQL output to "CQL over Thrift" output. What is lacking to support prepared statements is mostly handling and storing them on server side (the logic is different for CQL over native and CQL over Thrift, the code responsible for the latter is untranslated). |
currently blocked by #51 On Tue, Jul 28, 2015 at 3:27 PM, Paweł Dziepak notifications@github.com
|
with #51 fixed @penberg with debug I am getting the following executing sudo rm -Rf /data2/shlomi/tmp/* ; sudo build/debug/scylla --datadir /data2/shlomi/tmp --commitlog-directory /data2/shlomi/tmp --smp 6 --listen-address 192.168.10.101 --seed-provider-parameters seeds=192.168.10.101 --rpc-address 192.168.10.101 --default-log-level debug
I am not able to run in gdb I added a pair of printouts at the start,end of db::legacy_schema_tables::create_table_from_table_row_and_... and enabled debug logging to get a bit more info so now I get
so the failure point is in the create table function or maybe something after it and some of the cores have not completed the operation another run
|
for reference this is what happens when urchin is run with --smp 6 and a single cassandra-stress client with debug enabled
|
@slivne, does
report anything interesting? @avikivity any ideas why address sanitizer doesn't report a proper backtrace? |
/usr/include/c++/4.9.2/bits/hashtable.h:614 |
Usually it does report a complete backtrace, no idea why it didn't now. |
@slivne That suggests that the query result set is used after it's free'd. @tgrabiec, @avikivity, is that |
It's not safe. In its caller it's a lambda parameter, so it dies as soon as the create_tables_from_tables_partition() returns (which is before the parallel_for_each() completes). Suggest passing it by value to create_tables_from_tables_partition(), and using do_with() to protect it. |
@avikivity, would |
It would, but that's unrelated to the bug. I don't think it's worth it, there isn't a readily-available reducer to use here. You could replace the shared_ptr with do_with (but again that's not related to the bug). |
@avikivity Well, too late, I already sent a patch that uses map_reduce(). That's what we use elsewhere in the code as well. Lets talk more about it on the mailing list. |
2015-07-28 18:21 GMT+02:00 Avi Kivity notifications@github.com:
create_tables_from_tables_partition(proxy.local(), v.second) "v" comes from lambda parameter, also passed by reference: return do_parse_system_tables(proxy, do_parse_system_tables calls that lambda with "v" wrapped in do_with(): return do_with(std::move(v), [func = std::forward(func), cf_name](auto& v) {
It seems to me that we should be fine here. |
The ASan traces point to the |
So what happens is that there's multiple clients bombarding the server with "create table" statements. We start processing them on all cores and end up creating create table mutations with slightly different UUIDs (they're time-based generated). We then attempt to make sense of it all in This causes various kinds of issues and sometimes manifests as use-after-free and a SIGSEGV. |
We do not untend to support alter table But wedi need to support create table It seems we are taking the lock to "late" or missung a check adter taking
|
@slivne Please re-read what I commented above. We take the lock at the right time and the exception we throw is supposed to be harmless. Origin also throws an exception for your scenario, just a different one. However, we seem to drop the lock way too early which causes various issues. You can workaround the error by removing the |
Q: even if we solve it on a single node will it work in a cluster
|
@slivne I don't know. We would first need to determine what happens in Origin. Schema updates are handled via the same code path so I suspect you'd see the same UUID mismatch exception as with the case where single node is bombarded with parallel table creation. |
Commit scylladb/seastar@732dba1 fixed the SIGSEGV for me. @slivne please retest on Godzilla. |
Anyway it looks like stress is generating invalid inserts with 0 rows. |
Rather than using lw_shared_ptr to keep it around, use do_with. Fixes a use-after-free seen with interrupt_build_process_with_resharding_half_to_max_test in debug mode: ==406079==ERROR: AddressSanitizer: heap-use-after-free on address 0x603001320ac0 at pc 0x00000777a062 bp 0x7ffdeb77b010 sp 0x7ffdeb77b000 READ of size 8 at 0x603001320ac0 thread T0 #0 0x777a061 in _M_next /usr/include/c++/9/bits/hashtable_policy.h:283 scylladb#1 0x779ead7 in _M_incr /usr/include/c++/9/bits/hashtable_policy.h:299 scylladb#2 0x777bbac in operator++ /usr/include/c++/9/bits/hashtable_policy.h:362 scylladb#3 0x77f8384 in run_and_dispose /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future-util.hh:568 scylladb#4 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091 scylladb#5 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506 scylladb#6 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666 scylladb#7 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199 scylladb#8 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115 scylladb#9 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486 scylladb#10 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2) scylladb#11 0x42b69d in _start (/local/home/bhalevy/.dtest/dtest-q85idok7/test/node1/bin/scylla+0x42b69d) 0x603001320ac0 is located 0 bytes inside of 32-byte region [0x603001320ac0,0x603001320ae0) freed by thread T0 here: #0 0x7fb86520996f in operator delete(void*) (/lib64/libasan.so.5+0x11096f) scylladb#1 0x77d1889 in deallocate /usr/include/c++/9/ext/new_allocator.h:128 scylladb#2 0x77bcbb2 in deallocate /usr/include/c++/9/bits/alloc_traits.h:470 scylladb#3 0x77bccc5 in _M_deallocate_node_ptr /usr/include/c++/9/bits/hashtable_policy.h:2113 scylladb#4 0x779dc59 in _M_deallocate_node /usr/include/c++/9/bits/hashtable_policy.h:2103 scylladb#5 0x77bc8b6 in _M_deallocate_nodes /usr/include/c++/9/bits/hashtable_policy.h:2124 scylladb#6 0x779d18f in clear /usr/include/c++/9/bits/hashtable.h:2028 scylladb#7 0x7779b8e in ~_Hashtable /usr/include/c++/9/bits/hashtable.h:1352 scylladb#8 0x774cfed in ~unordered_map /usr/include/c++/9/bits/unordered_map.h:102 scylladb#9 0x774d288 in ~shared_ptr_no_esft /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:157 scylladb#10 0x774d2f1 in dispose /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:213 scylladb#11 0x76f9642 in ~lw_shared_ptr /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:300 scylladb#12 0x76f9642 in ~<lambda> /local/home/bhalevy/dev/scylla/distributed_loader.cc:743 scylladb#13 0x771e0a7 in ~<lambda> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1167 scylladb#14 0x777c64c in destroy /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:133 scylladb#15 0x586059 in seastar::noncopyable_function<seastar::future<> ()>::~noncopyable_function() /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:165 scylladb#16 0x5867da in seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}::~future_state() /local/home/bha levy/dev/scylla/seastar/include/seastar/core/future.hh:1192 scylladb#17 0xab704d in seastar::continuation<seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}>::~continu ation() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:507 scylladb#18 0xab7216 in seastar::continuation<seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}>::run_and_ dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:512 scylladb#19 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091 scylladb#20 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506 scylladb#21 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666 scylladb#22 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199 scylladb#23 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115 scylladb#24 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486 scylladb#25 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2) previously allocated by thread T0 here: #0 0x7fb8652089d7 in operator new(unsigned long) (/lib64/libasan.so.5+0x10f9d7) scylladb#1 0x77d1796 in allocate /usr/include/c++/9/ext/new_allocator.h:114 scylladb#2 0x77bcab3 in allocate /usr/include/c++/9/bits/alloc_traits.h:444 scylladb#3 0x779d936 in _M_allocate_node<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable_policy.h:2081 scylladb#4 0x777a2ed in _M_emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable.h:1638 scylladb#5 0x774d707 in emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable.h:749 scylladb#6 0x771ce3e in emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/unordered_map.h:388 scylladb#7 0x76f8280 in operator() /local/home/bhalevy/dev/scylla/distributed_loader.cc:732 scylladb#8 0x774dc5d in _M_invoke /usr/include/c++/9/bits/std_function.h:286 scylladb#9 0x6cde67b in std::function<seastar::future<> (std::filesystem::__cxx11::path, seastar::directory_entry)>::operator()(std::filesystem::__cxx11::path, seastar::directory_entry) const /usr/include/c++/9/bits/std_function.h:690 scylladb#10 0x6cd345d in operator() /local/home/bhalevy/dev/scylla/lister.cc:30 scylladb#11 0x6cd96e0 in apply /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36 scylladb#12 0x6cd9854 in apply<lister::visit(seastar::directory_entry)::<lambda(seastar::directory_entry)>&, seastar::directory_entry&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:44 scylladb#13 0x6cd9953 in apply<lister::visit(seastar::directory_entry)::<lambda(seastar::directory_entry)>&, seastar::directory_entry&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1650 scylladb#14 0x6cd6ba9 in operator() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1168 scylladb#15 0x6cdbac0 in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:101 scylladb#16 0x6ce6178 in seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>::operator()(seastar::directory_entry&&) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:184 scylladb#17 0x6ce3033 in seastar::apply_helper<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, std::tuple<seastar::directory_entry>&&, std::integer_sequence<unsigned long, 0ul> >::apply(seastar::noncopyable_function<seastar::future<> (seastar::directory_e ntry&&)>&&, std::tuple<seastar::directory_entry>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36 scylladb#18 0x6ce3130 in auto seastar::apply<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::directory_entry>(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&, std::tuple<seastar::directory_entry>&&) /local/home/bha levy/dev/scylla/seastar/include/seastar/core/apply.hh:44 scylladb#19 0x6ce322f in seastar::future<> seastar::futurize<seastar::future<> >::apply<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::directory_entry>(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&, std::tuple<s eastar::directory_entry>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1650 scylladb#20 0x6cdf88d in seastar::future<seastar::directory_entry>::then_impl<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<seastar::directory_entry>&&)scylladb#1}::operator()(seastar::future_state<seastar::directory_entry>) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1196 scylladb#21 0x6cea2e0 in seastar::continuation<seastar::future<seastar::directory_entry>::then_impl<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&)::{lamb da()scylladb#1}::operator()() const::{lambda(seastar::future_state<seastar::directory_entry>&&)scylladb#1}, seastar::directory_entry>::run_and_dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:511 scylladb#22 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091 scylladb#23 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506 scylladb#24 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666 scylladb#25 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199 scylladb#26 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115 scylladb#27 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486 scylladb#28 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2) SUMMARY: AddressSanitizer: heap-use-after-free /usr/include/c++/9/bits/hashtable_policy.h:283 in _M_next Shadow bytes around the buggy address: 0x0c068025c100: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c068025c110: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa 0x0c068025c120: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd 0x0c068025c130: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c068025c140: fd fd fd fa fa fa 00 00 00 fa fa fa fd fd fd fa =>0x0c068025c150: fa fa fd fd fd fa fa fa[fd]fd fd fd fa fa fd fd 0x0c068025c160: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c068025c170: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa 0x0c068025c180: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd 0x0c068025c190: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c068025c1a0: fd fd fd fd fa fa fd fd fd fd fa fa 00 00 00 fa Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa Freed heap region: fd Stack left redzone: f1 Stack mid redzone: f2 Stack right redzone: f3 Stack after return: f5 Stack use after scope: f8 Global redzone: f9 Global init order: f6 Poisoned by user: f7 Container overflow: fc Array cookie: ac Intra object redzone: bb ASan internal: fe Left alloca redzone: ca Right alloca redzone: cb Shadow gap: cc ==406079==ABORTING Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Rather than using lw_shared_ptr to keep it around, use do_with. Fixes a use-after-free seen with interrupt_build_process_with_resharding_half_to_max_test in debug mode: ==406079==ERROR: AddressSanitizer: heap-use-after-free on address 0x603001320ac0 at pc 0x00000777a062 bp 0x7ffdeb77b010 sp 0x7ffdeb77b000 READ of size 8 at 0x603001320ac0 thread T0 #0 0x777a061 in _M_next /usr/include/c++/9/bits/hashtable_policy.h:283 scylladb#1 0x779ead7 in _M_incr /usr/include/c++/9/bits/hashtable_policy.h:299 scylladb#2 0x777bbac in operator++ /usr/include/c++/9/bits/hashtable_policy.h:362 scylladb#3 0x77f8384 in run_and_dispose /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future-util.hh:568 scylladb#4 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091 scylladb#5 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506 scylladb#6 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666 scylladb#7 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199 scylladb#8 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115 scylladb#9 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486 scylladb#10 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2) scylladb#11 0x42b69d in _start (/local/home/bhalevy/.dtest/dtest-q85idok7/test/node1/bin/scylla+0x42b69d) 0x603001320ac0 is located 0 bytes inside of 32-byte region [0x603001320ac0,0x603001320ae0) freed by thread T0 here: #0 0x7fb86520996f in operator delete(void*) (/lib64/libasan.so.5+0x11096f) scylladb#1 0x77d1889 in deallocate /usr/include/c++/9/ext/new_allocator.h:128 scylladb#2 0x77bcbb2 in deallocate /usr/include/c++/9/bits/alloc_traits.h:470 scylladb#3 0x77bccc5 in _M_deallocate_node_ptr /usr/include/c++/9/bits/hashtable_policy.h:2113 scylladb#4 0x779dc59 in _M_deallocate_node /usr/include/c++/9/bits/hashtable_policy.h:2103 scylladb#5 0x77bc8b6 in _M_deallocate_nodes /usr/include/c++/9/bits/hashtable_policy.h:2124 scylladb#6 0x779d18f in clear /usr/include/c++/9/bits/hashtable.h:2028 scylladb#7 0x7779b8e in ~_Hashtable /usr/include/c++/9/bits/hashtable.h:1352 scylladb#8 0x774cfed in ~unordered_map /usr/include/c++/9/bits/unordered_map.h:102 scylladb#9 0x774d288 in ~shared_ptr_no_esft /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:157 scylladb#10 0x774d2f1 in dispose /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:213 scylladb#11 0x76f9642 in ~lw_shared_ptr /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:300 scylladb#12 0x76f9642 in ~<lambda> /local/home/bhalevy/dev/scylla/distributed_loader.cc:743 scylladb#13 0x771e0a7 in ~<lambda> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1167 scylladb#14 0x777c64c in destroy /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:133 scylladb#15 0x586059 in seastar::noncopyable_function<seastar::future<> ()>::~noncopyable_function() /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:165 scylladb#16 0x5867da in seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}::~future_state() /local/home/bha levy/dev/scylla/seastar/include/seastar/core/future.hh:1192 scylladb#17 0xab704d in seastar::continuation<seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}>::~continu ation() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:507 scylladb#18 0xab7216 in seastar::continuation<seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}>::run_and_ dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:512 scylladb#19 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091 scylladb#20 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506 scylladb#21 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666 scylladb#22 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199 scylladb#23 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115 scylladb#24 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486 scylladb#25 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2) previously allocated by thread T0 here: #0 0x7fb8652089d7 in operator new(unsigned long) (/lib64/libasan.so.5+0x10f9d7) scylladb#1 0x77d1796 in allocate /usr/include/c++/9/ext/new_allocator.h:114 scylladb#2 0x77bcab3 in allocate /usr/include/c++/9/bits/alloc_traits.h:444 scylladb#3 0x779d936 in _M_allocate_node<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable_policy.h:2081 scylladb#4 0x777a2ed in _M_emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable.h:1638 scylladb#5 0x774d707 in emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable.h:749 scylladb#6 0x771ce3e in emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/unordered_map.h:388 scylladb#7 0x76f8280 in operator() /local/home/bhalevy/dev/scylla/distributed_loader.cc:732 scylladb#8 0x774dc5d in _M_invoke /usr/include/c++/9/bits/std_function.h:286 scylladb#9 0x6cde67b in std::function<seastar::future<> (std::filesystem::__cxx11::path, seastar::directory_entry)>::operator()(std::filesystem::__cxx11::path, seastar::directory_entry) const /usr/include/c++/9/bits/std_function.h:690 scylladb#10 0x6cd345d in operator() /local/home/bhalevy/dev/scylla/lister.cc:30 scylladb#11 0x6cd96e0 in apply /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36 scylladb#12 0x6cd9854 in apply<lister::visit(seastar::directory_entry)::<lambda(seastar::directory_entry)>&, seastar::directory_entry&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:44 scylladb#13 0x6cd9953 in apply<lister::visit(seastar::directory_entry)::<lambda(seastar::directory_entry)>&, seastar::directory_entry&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1650 scylladb#14 0x6cd6ba9 in operator() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1168 scylladb#15 0x6cdbac0 in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:101 scylladb#16 0x6ce6178 in seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>::operator()(seastar::directory_entry&&) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:184 scylladb#17 0x6ce3033 in seastar::apply_helper<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, std::tuple<seastar::directory_entry>&&, std::integer_sequence<unsigned long, 0ul> >::apply(seastar::noncopyable_function<seastar::future<> (seastar::directory_e ntry&&)>&&, std::tuple<seastar::directory_entry>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36 scylladb#18 0x6ce3130 in auto seastar::apply<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::directory_entry>(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&, std::tuple<seastar::directory_entry>&&) /local/home/bha levy/dev/scylla/seastar/include/seastar/core/apply.hh:44 scylladb#19 0x6ce322f in seastar::future<> seastar::futurize<seastar::future<> >::apply<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::directory_entry>(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&, std::tuple<s eastar::directory_entry>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1650 scylladb#20 0x6cdf88d in seastar::future<seastar::directory_entry>::then_impl<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<seastar::directory_entry>&&)scylladb#1}::operator()(seastar::future_state<seastar::directory_entry>) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1196 scylladb#21 0x6cea2e0 in seastar::continuation<seastar::future<seastar::directory_entry>::then_impl<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&)::{lamb da()scylladb#1}::operator()() const::{lambda(seastar::future_state<seastar::directory_entry>&&)scylladb#1}, seastar::directory_entry>::run_and_dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:511 scylladb#22 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091 scylladb#23 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506 scylladb#24 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666 scylladb#25 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199 scylladb#26 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115 scylladb#27 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486 scylladb#28 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2) SUMMARY: AddressSanitizer: heap-use-after-free /usr/include/c++/9/bits/hashtable_policy.h:283 in _M_next Shadow bytes around the buggy address: 0x0c068025c100: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c068025c110: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa 0x0c068025c120: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd 0x0c068025c130: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c068025c140: fd fd fd fa fa fa 00 00 00 fa fa fa fd fd fd fa =>0x0c068025c150: fa fa fd fd fd fa fa fa[fd]fd fd fd fa fa fd fd 0x0c068025c160: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c068025c170: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa 0x0c068025c180: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd 0x0c068025c190: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c068025c1a0: fd fd fd fd fa fa fd fd fd fd fa fa 00 00 00 fa Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa Freed heap region: fd Stack left redzone: f1 Stack mid redzone: f2 Stack right redzone: f3 Stack after return: f5 Stack use after scope: f8 Global redzone: f9 Global init order: f6 Poisoned by user: f7 Container overflow: fc Array cookie: ac Intra object redzone: bb ASan internal: fe Left alloca redzone: ca Right alloca redzone: cb Shadow gap: cc ==406079==ABORTING Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Rather than using lw_shared_ptr to keep it around, use do_with. Fixes a use-after-free seen with interrupt_build_process_with_resharding_half_to_max_test in debug mode: ==406079==ERROR: AddressSanitizer: heap-use-after-free on address 0x603001320ac0 at pc 0x00000777a062 bp 0x7ffdeb77b010 sp 0x7ffdeb77b000 READ of size 8 at 0x603001320ac0 thread T0 #0 0x777a061 in _M_next /usr/include/c++/9/bits/hashtable_policy.h:283 scylladb#1 0x779ead7 in _M_incr /usr/include/c++/9/bits/hashtable_policy.h:299 scylladb#2 0x777bbac in operator++ /usr/include/c++/9/bits/hashtable_policy.h:362 scylladb#3 0x77f8384 in run_and_dispose /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future-util.hh:568 scylladb#4 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091 scylladb#5 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506 scylladb#6 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666 scylladb#7 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199 scylladb#8 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115 scylladb#9 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486 scylladb#10 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2) scylladb#11 0x42b69d in _start (/local/home/bhalevy/.dtest/dtest-q85idok7/test/node1/bin/scylla+0x42b69d) 0x603001320ac0 is located 0 bytes inside of 32-byte region [0x603001320ac0,0x603001320ae0) freed by thread T0 here: #0 0x7fb86520996f in operator delete(void*) (/lib64/libasan.so.5+0x11096f) scylladb#1 0x77d1889 in deallocate /usr/include/c++/9/ext/new_allocator.h:128 scylladb#2 0x77bcbb2 in deallocate /usr/include/c++/9/bits/alloc_traits.h:470 scylladb#3 0x77bccc5 in _M_deallocate_node_ptr /usr/include/c++/9/bits/hashtable_policy.h:2113 scylladb#4 0x779dc59 in _M_deallocate_node /usr/include/c++/9/bits/hashtable_policy.h:2103 scylladb#5 0x77bc8b6 in _M_deallocate_nodes /usr/include/c++/9/bits/hashtable_policy.h:2124 scylladb#6 0x779d18f in clear /usr/include/c++/9/bits/hashtable.h:2028 scylladb#7 0x7779b8e in ~_Hashtable /usr/include/c++/9/bits/hashtable.h:1352 scylladb#8 0x774cfed in ~unordered_map /usr/include/c++/9/bits/unordered_map.h:102 scylladb#9 0x774d288 in ~shared_ptr_no_esft /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:157 scylladb#10 0x774d2f1 in dispose /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:213 scylladb#11 0x76f9642 in ~lw_shared_ptr /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:300 scylladb#12 0x76f9642 in ~<lambda> /local/home/bhalevy/dev/scylla/distributed_loader.cc:743 scylladb#13 0x771e0a7 in ~<lambda> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1167 scylladb#14 0x777c64c in destroy /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:133 scylladb#15 0x586059 in seastar::noncopyable_function<seastar::future<> ()>::~noncopyable_function() /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:165 scylladb#16 0x5867da in seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}::~future_state() /local/home/bha levy/dev/scylla/seastar/include/seastar/core/future.hh:1192 scylladb#17 0xab704d in seastar::continuation<seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}>::~continu ation() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:507 scylladb#18 0xab7216 in seastar::continuation<seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}>::run_and_ dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:512 scylladb#19 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091 scylladb#20 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506 scylladb#21 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666 scylladb#22 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199 scylladb#23 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115 scylladb#24 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486 scylladb#25 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2) previously allocated by thread T0 here: #0 0x7fb8652089d7 in operator new(unsigned long) (/lib64/libasan.so.5+0x10f9d7) scylladb#1 0x77d1796 in allocate /usr/include/c++/9/ext/new_allocator.h:114 scylladb#2 0x77bcab3 in allocate /usr/include/c++/9/bits/alloc_traits.h:444 scylladb#3 0x779d936 in _M_allocate_node<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable_policy.h:2081 scylladb#4 0x777a2ed in _M_emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable.h:1638 scylladb#5 0x774d707 in emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable.h:749 scylladb#6 0x771ce3e in emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/unordered_map.h:388 scylladb#7 0x76f8280 in operator() /local/home/bhalevy/dev/scylla/distributed_loader.cc:732 scylladb#8 0x774dc5d in _M_invoke /usr/include/c++/9/bits/std_function.h:286 scylladb#9 0x6cde67b in std::function<seastar::future<> (std::filesystem::__cxx11::path, seastar::directory_entry)>::operator()(std::filesystem::__cxx11::path, seastar::directory_entry) const /usr/include/c++/9/bits/std_function.h:690 scylladb#10 0x6cd345d in operator() /local/home/bhalevy/dev/scylla/lister.cc:30 scylladb#11 0x6cd96e0 in apply /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36 scylladb#12 0x6cd9854 in apply<lister::visit(seastar::directory_entry)::<lambda(seastar::directory_entry)>&, seastar::directory_entry&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:44 scylladb#13 0x6cd9953 in apply<lister::visit(seastar::directory_entry)::<lambda(seastar::directory_entry)>&, seastar::directory_entry&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1650 scylladb#14 0x6cd6ba9 in operator() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1168 scylladb#15 0x6cdbac0 in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:101 scylladb#16 0x6ce6178 in seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>::operator()(seastar::directory_entry&&) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:184 scylladb#17 0x6ce3033 in seastar::apply_helper<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, std::tuple<seastar::directory_entry>&&, std::integer_sequence<unsigned long, 0ul> >::apply(seastar::noncopyable_function<seastar::future<> (seastar::directory_e ntry&&)>&&, std::tuple<seastar::directory_entry>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36 scylladb#18 0x6ce3130 in auto seastar::apply<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::directory_entry>(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&, std::tuple<seastar::directory_entry>&&) /local/home/bha levy/dev/scylla/seastar/include/seastar/core/apply.hh:44 scylladb#19 0x6ce322f in seastar::future<> seastar::futurize<seastar::future<> >::apply<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::directory_entry>(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&, std::tuple<s eastar::directory_entry>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1650 scylladb#20 0x6cdf88d in seastar::future<seastar::directory_entry>::then_impl<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<seastar::directory_entry>&&)scylladb#1}::operator()(seastar::future_state<seastar::directory_entry>) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1196 scylladb#21 0x6cea2e0 in seastar::continuation<seastar::future<seastar::directory_entry>::then_impl<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&)::{lamb da()scylladb#1}::operator()() const::{lambda(seastar::future_state<seastar::directory_entry>&&)scylladb#1}, seastar::directory_entry>::run_and_dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:511 scylladb#22 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091 scylladb#23 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506 scylladb#24 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666 scylladb#25 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199 scylladb#26 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115 scylladb#27 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486 scylladb#28 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2) SUMMARY: AddressSanitizer: heap-use-after-free /usr/include/c++/9/bits/hashtable_policy.h:283 in _M_next Shadow bytes around the buggy address: 0x0c068025c100: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c068025c110: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa 0x0c068025c120: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd 0x0c068025c130: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c068025c140: fd fd fd fa fa fa 00 00 00 fa fa fa fd fd fd fa =>0x0c068025c150: fa fa fd fd fd fa fa fa[fd]fd fd fd fa fa fd fd 0x0c068025c160: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c068025c170: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa 0x0c068025c180: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd 0x0c068025c190: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c068025c1a0: fd fd fd fd fa fa fd fd fd fd fa fa 00 00 00 fa Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa Freed heap region: fd Stack left redzone: f1 Stack mid redzone: f2 Stack right redzone: f3 Stack after return: f5 Stack use after scope: f8 Global redzone: f9 Global init order: f6 Poisoned by user: f7 Container overflow: fc Array cookie: ac Intra object redzone: bb ASan internal: fe Left alloca redzone: ca Right alloca redzone: cb Shadow gap: cc ==406079==ABORTING Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Rather than using lw_shared_ptr to keep it around, use do_with. Fixes a use-after-free seen with interrupt_build_process_with_resharding_half_to_max_test in debug mode: ==406079==ERROR: AddressSanitizer: heap-use-after-free on address 0x603001320ac0 at pc 0x00000777a062 bp 0x7ffdeb77b010 sp 0x7ffdeb77b000 READ of size 8 at 0x603001320ac0 thread T0 #0 0x777a061 in _M_next /usr/include/c++/9/bits/hashtable_policy.h:283 scylladb#1 0x779ead7 in _M_incr /usr/include/c++/9/bits/hashtable_policy.h:299 scylladb#2 0x777bbac in operator++ /usr/include/c++/9/bits/hashtable_policy.h:362 scylladb#3 0x77f8384 in run_and_dispose /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future-util.hh:568 scylladb#4 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091 scylladb#5 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506 scylladb#6 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666 scylladb#7 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199 scylladb#8 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115 scylladb#9 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486 scylladb#10 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2) scylladb#11 0x42b69d in _start (/local/home/bhalevy/.dtest/dtest-q85idok7/test/node1/bin/scylla+0x42b69d) 0x603001320ac0 is located 0 bytes inside of 32-byte region [0x603001320ac0,0x603001320ae0) freed by thread T0 here: #0 0x7fb86520996f in operator delete(void*) (/lib64/libasan.so.5+0x11096f) scylladb#1 0x77d1889 in deallocate /usr/include/c++/9/ext/new_allocator.h:128 scylladb#2 0x77bcbb2 in deallocate /usr/include/c++/9/bits/alloc_traits.h:470 scylladb#3 0x77bccc5 in _M_deallocate_node_ptr /usr/include/c++/9/bits/hashtable_policy.h:2113 scylladb#4 0x779dc59 in _M_deallocate_node /usr/include/c++/9/bits/hashtable_policy.h:2103 scylladb#5 0x77bc8b6 in _M_deallocate_nodes /usr/include/c++/9/bits/hashtable_policy.h:2124 scylladb#6 0x779d18f in clear /usr/include/c++/9/bits/hashtable.h:2028 scylladb#7 0x7779b8e in ~_Hashtable /usr/include/c++/9/bits/hashtable.h:1352 scylladb#8 0x774cfed in ~unordered_map /usr/include/c++/9/bits/unordered_map.h:102 scylladb#9 0x774d288 in ~shared_ptr_no_esft /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:157 scylladb#10 0x774d2f1 in dispose /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:213 scylladb#11 0x76f9642 in ~lw_shared_ptr /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:300 scylladb#12 0x76f9642 in ~<lambda> /local/home/bhalevy/dev/scylla/distributed_loader.cc:743 scylladb#13 0x771e0a7 in ~<lambda> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1167 scylladb#14 0x777c64c in destroy /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:133 scylladb#15 0x586059 in seastar::noncopyable_function<seastar::future<> ()>::~noncopyable_function() /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:165 scylladb#16 0x5867da in seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}::~future_state() /local/home/bha levy/dev/scylla/seastar/include/seastar/core/future.hh:1192 scylladb#17 0xab704d in seastar::continuation<seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}>::~continu ation() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:507 scylladb#18 0xab7216 in seastar::continuation<seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}>::run_and_ dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:512 scylladb#19 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091 scylladb#20 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506 scylladb#21 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666 scylladb#22 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199 scylladb#23 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115 scylladb#24 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486 scylladb#25 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2) previously allocated by thread T0 here: #0 0x7fb8652089d7 in operator new(unsigned long) (/lib64/libasan.so.5+0x10f9d7) scylladb#1 0x77d1796 in allocate /usr/include/c++/9/ext/new_allocator.h:114 scylladb#2 0x77bcab3 in allocate /usr/include/c++/9/bits/alloc_traits.h:444 scylladb#3 0x779d936 in _M_allocate_node<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable_policy.h:2081 scylladb#4 0x777a2ed in _M_emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable.h:1638 scylladb#5 0x774d707 in emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable.h:749 scylladb#6 0x771ce3e in emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/unordered_map.h:388 scylladb#7 0x76f8280 in operator() /local/home/bhalevy/dev/scylla/distributed_loader.cc:732 scylladb#8 0x774dc5d in _M_invoke /usr/include/c++/9/bits/std_function.h:286 scylladb#9 0x6cde67b in std::function<seastar::future<> (std::filesystem::__cxx11::path, seastar::directory_entry)>::operator()(std::filesystem::__cxx11::path, seastar::directory_entry) const /usr/include/c++/9/bits/std_function.h:690 scylladb#10 0x6cd345d in operator() /local/home/bhalevy/dev/scylla/lister.cc:30 scylladb#11 0x6cd96e0 in apply /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36 scylladb#12 0x6cd9854 in apply<lister::visit(seastar::directory_entry)::<lambda(seastar::directory_entry)>&, seastar::directory_entry&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:44 scylladb#13 0x6cd9953 in apply<lister::visit(seastar::directory_entry)::<lambda(seastar::directory_entry)>&, seastar::directory_entry&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1650 scylladb#14 0x6cd6ba9 in operator() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1168 scylladb#15 0x6cdbac0 in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:101 scylladb#16 0x6ce6178 in seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>::operator()(seastar::directory_entry&&) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:184 scylladb#17 0x6ce3033 in seastar::apply_helper<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, std::tuple<seastar::directory_entry>&&, std::integer_sequence<unsigned long, 0ul> >::apply(seastar::noncopyable_function<seastar::future<> (seastar::directory_e ntry&&)>&&, std::tuple<seastar::directory_entry>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36 scylladb#18 0x6ce3130 in auto seastar::apply<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::directory_entry>(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&, std::tuple<seastar::directory_entry>&&) /local/home/bha levy/dev/scylla/seastar/include/seastar/core/apply.hh:44 scylladb#19 0x6ce322f in seastar::future<> seastar::futurize<seastar::future<> >::apply<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::directory_entry>(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&, std::tuple<s eastar::directory_entry>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1650 scylladb#20 0x6cdf88d in seastar::future<seastar::directory_entry>::then_impl<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<seastar::directory_entry>&&)scylladb#1}::operator()(seastar::future_state<seastar::directory_entry>) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1196 scylladb#21 0x6cea2e0 in seastar::continuation<seastar::future<seastar::directory_entry>::then_impl<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&)::{lamb da()scylladb#1}::operator()() const::{lambda(seastar::future_state<seastar::directory_entry>&&)scylladb#1}, seastar::directory_entry>::run_and_dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:511 scylladb#22 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091 scylladb#23 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506 scylladb#24 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666 scylladb#25 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199 scylladb#26 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115 scylladb#27 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486 scylladb#28 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2) SUMMARY: AddressSanitizer: heap-use-after-free /usr/include/c++/9/bits/hashtable_policy.h:283 in _M_next Shadow bytes around the buggy address: 0x0c068025c100: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c068025c110: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa 0x0c068025c120: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd 0x0c068025c130: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c068025c140: fd fd fd fa fa fa 00 00 00 fa fa fa fd fd fd fa =>0x0c068025c150: fa fa fd fd fd fa fa fa[fd]fd fd fd fa fa fd fd 0x0c068025c160: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c068025c170: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa 0x0c068025c180: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd 0x0c068025c190: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c068025c1a0: fd fd fd fd fa fa fd fd fd fd fa fa 00 00 00 fa Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa Freed heap region: fd Stack left redzone: f1 Stack mid redzone: f2 Stack right redzone: f3 Stack after return: f5 Stack use after scope: f8 Global redzone: f9 Global init order: f6 Poisoned by user: f7 Container overflow: fc Array cookie: ac Intra object redzone: bb ASan internal: fe Left alloca redzone: ca Right alloca redzone: cb Shadow gap: cc ==406079==ABORTING Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Rather than using lw_shared_ptr to keep it around, use do_with. Fixes a use-after-free seen with interrupt_build_process_with_resharding_half_to_max_test in debug mode: ==406079==ERROR: AddressSanitizer: heap-use-after-free on address 0x603001320ac0 at pc 0x00000777a062 bp 0x7ffdeb77b010 sp 0x7ffdeb77b000 READ of size 8 at 0x603001320ac0 thread T0 #0 0x777a061 in _M_next /usr/include/c++/9/bits/hashtable_policy.h:283 scylladb#1 0x779ead7 in _M_incr /usr/include/c++/9/bits/hashtable_policy.h:299 scylladb#2 0x777bbac in operator++ /usr/include/c++/9/bits/hashtable_policy.h:362 scylladb#3 0x77f8384 in run_and_dispose /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future-util.hh:568 scylladb#4 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091 scylladb#5 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506 scylladb#6 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666 scylladb#7 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199 scylladb#8 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115 scylladb#9 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486 scylladb#10 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2) scylladb#11 0x42b69d in _start (/local/home/bhalevy/.dtest/dtest-q85idok7/test/node1/bin/scylla+0x42b69d) 0x603001320ac0 is located 0 bytes inside of 32-byte region [0x603001320ac0,0x603001320ae0) freed by thread T0 here: #0 0x7fb86520996f in operator delete(void*) (/lib64/libasan.so.5+0x11096f) scylladb#1 0x77d1889 in deallocate /usr/include/c++/9/ext/new_allocator.h:128 scylladb#2 0x77bcbb2 in deallocate /usr/include/c++/9/bits/alloc_traits.h:470 scylladb#3 0x77bccc5 in _M_deallocate_node_ptr /usr/include/c++/9/bits/hashtable_policy.h:2113 scylladb#4 0x779dc59 in _M_deallocate_node /usr/include/c++/9/bits/hashtable_policy.h:2103 scylladb#5 0x77bc8b6 in _M_deallocate_nodes /usr/include/c++/9/bits/hashtable_policy.h:2124 scylladb#6 0x779d18f in clear /usr/include/c++/9/bits/hashtable.h:2028 scylladb#7 0x7779b8e in ~_Hashtable /usr/include/c++/9/bits/hashtable.h:1352 scylladb#8 0x774cfed in ~unordered_map /usr/include/c++/9/bits/unordered_map.h:102 scylladb#9 0x774d288 in ~shared_ptr_no_esft /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:157 scylladb#10 0x774d2f1 in dispose /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:213 scylladb#11 0x76f9642 in ~lw_shared_ptr /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:300 scylladb#12 0x76f9642 in ~<lambda> /local/home/bhalevy/dev/scylla/distributed_loader.cc:743 scylladb#13 0x771e0a7 in ~<lambda> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1167 scylladb#14 0x777c64c in destroy /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:133 scylladb#15 0x586059 in seastar::noncopyable_function<seastar::future<> ()>::~noncopyable_function() /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:165 scylladb#16 0x5867da in seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}::~future_state() /local/home/bha levy/dev/scylla/seastar/include/seastar/core/future.hh:1192 scylladb#17 0xab704d in seastar::continuation<seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}>::~continu ation() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:507 scylladb#18 0xab7216 in seastar::continuation<seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}>::run_and_ dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:512 scylladb#19 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091 scylladb#20 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506 scylladb#21 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666 scylladb#22 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199 scylladb#23 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115 scylladb#24 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486 scylladb#25 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2) previously allocated by thread T0 here: #0 0x7fb8652089d7 in operator new(unsigned long) (/lib64/libasan.so.5+0x10f9d7) scylladb#1 0x77d1796 in allocate /usr/include/c++/9/ext/new_allocator.h:114 scylladb#2 0x77bcab3 in allocate /usr/include/c++/9/bits/alloc_traits.h:444 scylladb#3 0x779d936 in _M_allocate_node<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable_policy.h:2081 scylladb#4 0x777a2ed in _M_emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable.h:1638 scylladb#5 0x774d707 in emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable.h:749 scylladb#6 0x771ce3e in emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/unordered_map.h:388 scylladb#7 0x76f8280 in operator() /local/home/bhalevy/dev/scylla/distributed_loader.cc:732 scylladb#8 0x774dc5d in _M_invoke /usr/include/c++/9/bits/std_function.h:286 scylladb#9 0x6cde67b in std::function<seastar::future<> (std::filesystem::__cxx11::path, seastar::directory_entry)>::operator()(std::filesystem::__cxx11::path, seastar::directory_entry) const /usr/include/c++/9/bits/std_function.h:690 scylladb#10 0x6cd345d in operator() /local/home/bhalevy/dev/scylla/lister.cc:30 scylladb#11 0x6cd96e0 in apply /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36 scylladb#12 0x6cd9854 in apply<lister::visit(seastar::directory_entry)::<lambda(seastar::directory_entry)>&, seastar::directory_entry&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:44 scylladb#13 0x6cd9953 in apply<lister::visit(seastar::directory_entry)::<lambda(seastar::directory_entry)>&, seastar::directory_entry&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1650 scylladb#14 0x6cd6ba9 in operator() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1168 scylladb#15 0x6cdbac0 in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:101 scylladb#16 0x6ce6178 in seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>::operator()(seastar::directory_entry&&) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:184 scylladb#17 0x6ce3033 in seastar::apply_helper<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, std::tuple<seastar::directory_entry>&&, std::integer_sequence<unsigned long, 0ul> >::apply(seastar::noncopyable_function<seastar::future<> (seastar::directory_e ntry&&)>&&, std::tuple<seastar::directory_entry>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36 scylladb#18 0x6ce3130 in auto seastar::apply<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::directory_entry>(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&, std::tuple<seastar::directory_entry>&&) /local/home/bha levy/dev/scylla/seastar/include/seastar/core/apply.hh:44 scylladb#19 0x6ce322f in seastar::future<> seastar::futurize<seastar::future<> >::apply<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::directory_entry>(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&, std::tuple<s eastar::directory_entry>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1650 scylladb#20 0x6cdf88d in seastar::future<seastar::directory_entry>::then_impl<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<seastar::directory_entry>&&)scylladb#1}::operator()(seastar::future_state<seastar::directory_entry>) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1196 scylladb#21 0x6cea2e0 in seastar::continuation<seastar::future<seastar::directory_entry>::then_impl<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&)::{lamb da()scylladb#1}::operator()() const::{lambda(seastar::future_state<seastar::directory_entry>&&)scylladb#1}, seastar::directory_entry>::run_and_dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:511 scylladb#22 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091 scylladb#23 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506 scylladb#24 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666 scylladb#25 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199 scylladb#26 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115 scylladb#27 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486 scylladb#28 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2) SUMMARY: AddressSanitizer: heap-use-after-free /usr/include/c++/9/bits/hashtable_policy.h:283 in _M_next Shadow bytes around the buggy address: 0x0c068025c100: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c068025c110: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa 0x0c068025c120: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd 0x0c068025c130: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c068025c140: fd fd fd fa fa fa 00 00 00 fa fa fa fd fd fd fa =>0x0c068025c150: fa fa fd fd fd fa fa fa[fd]fd fd fd fa fa fd fd 0x0c068025c160: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c068025c170: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa 0x0c068025c180: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd 0x0c068025c190: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c068025c1a0: fd fd fd fd fa fa fd fd fd fd fa fa 00 00 00 fa Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa Freed heap region: fd Stack left redzone: f1 Stack mid redzone: f2 Stack right redzone: f3 Stack after return: f5 Stack use after scope: f8 Global redzone: f9 Global init order: f6 Poisoned by user: f7 Container overflow: fc Array cookie: ac Intra object redzone: bb ASan internal: fe Left alloca redzone: ca Right alloca redzone: cb Shadow gap: cc ==406079==ABORTING Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Rather than using lw_shared_ptr to keep it around, use do_with. Fixes a use-after-free seen with interrupt_build_process_with_resharding_half_to_max_test in debug mode: ==406079==ERROR: AddressSanitizer: heap-use-after-free on address 0x603001320ac0 at pc 0x00000777a062 bp 0x7ffdeb77b010 sp 0x7ffdeb77b000 READ of size 8 at 0x603001320ac0 thread T0 #0 0x777a061 in _M_next /usr/include/c++/9/bits/hashtable_policy.h:283 scylladb#1 0x779ead7 in _M_incr /usr/include/c++/9/bits/hashtable_policy.h:299 scylladb#2 0x777bbac in operator++ /usr/include/c++/9/bits/hashtable_policy.h:362 scylladb#3 0x77f8384 in run_and_dispose /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future-util.hh:568 scylladb#4 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091 scylladb#5 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506 scylladb#6 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666 scylladb#7 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199 scylladb#8 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115 scylladb#9 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486 scylladb#10 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2) scylladb#11 0x42b69d in _start (/local/home/bhalevy/.dtest/dtest-q85idok7/test/node1/bin/scylla+0x42b69d) 0x603001320ac0 is located 0 bytes inside of 32-byte region [0x603001320ac0,0x603001320ae0) freed by thread T0 here: #0 0x7fb86520996f in operator delete(void*) (/lib64/libasan.so.5+0x11096f) scylladb#1 0x77d1889 in deallocate /usr/include/c++/9/ext/new_allocator.h:128 scylladb#2 0x77bcbb2 in deallocate /usr/include/c++/9/bits/alloc_traits.h:470 scylladb#3 0x77bccc5 in _M_deallocate_node_ptr /usr/include/c++/9/bits/hashtable_policy.h:2113 scylladb#4 0x779dc59 in _M_deallocate_node /usr/include/c++/9/bits/hashtable_policy.h:2103 scylladb#5 0x77bc8b6 in _M_deallocate_nodes /usr/include/c++/9/bits/hashtable_policy.h:2124 scylladb#6 0x779d18f in clear /usr/include/c++/9/bits/hashtable.h:2028 scylladb#7 0x7779b8e in ~_Hashtable /usr/include/c++/9/bits/hashtable.h:1352 scylladb#8 0x774cfed in ~unordered_map /usr/include/c++/9/bits/unordered_map.h:102 scylladb#9 0x774d288 in ~shared_ptr_no_esft /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:157 scylladb#10 0x774d2f1 in dispose /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:213 scylladb#11 0x76f9642 in ~lw_shared_ptr /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:300 scylladb#12 0x76f9642 in ~<lambda> /local/home/bhalevy/dev/scylla/distributed_loader.cc:743 scylladb#13 0x771e0a7 in ~<lambda> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1167 scylladb#14 0x777c64c in destroy /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:133 scylladb#15 0x586059 in seastar::noncopyable_function<seastar::future<> ()>::~noncopyable_function() /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:165 scylladb#16 0x5867da in seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}::~future_state() /local/home/bha levy/dev/scylla/seastar/include/seastar/core/future.hh:1192 scylladb#17 0xab704d in seastar::continuation<seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}>::~continu ation() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:507 scylladb#18 0xab7216 in seastar::continuation<seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}>::run_and_ dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:512 scylladb#19 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091 scylladb#20 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506 scylladb#21 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666 scylladb#22 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199 scylladb#23 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115 scylladb#24 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486 scylladb#25 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2) previously allocated by thread T0 here: #0 0x7fb8652089d7 in operator new(unsigned long) (/lib64/libasan.so.5+0x10f9d7) scylladb#1 0x77d1796 in allocate /usr/include/c++/9/ext/new_allocator.h:114 scylladb#2 0x77bcab3 in allocate /usr/include/c++/9/bits/alloc_traits.h:444 scylladb#3 0x779d936 in _M_allocate_node<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable_policy.h:2081 scylladb#4 0x777a2ed in _M_emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable.h:1638 scylladb#5 0x774d707 in emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable.h:749 scylladb#6 0x771ce3e in emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/unordered_map.h:388 scylladb#7 0x76f8280 in operator() /local/home/bhalevy/dev/scylla/distributed_loader.cc:732 scylladb#8 0x774dc5d in _M_invoke /usr/include/c++/9/bits/std_function.h:286 scylladb#9 0x6cde67b in std::function<seastar::future<> (std::filesystem::__cxx11::path, seastar::directory_entry)>::operator()(std::filesystem::__cxx11::path, seastar::directory_entry) const /usr/include/c++/9/bits/std_function.h:690 scylladb#10 0x6cd345d in operator() /local/home/bhalevy/dev/scylla/lister.cc:30 scylladb#11 0x6cd96e0 in apply /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36 scylladb#12 0x6cd9854 in apply<lister::visit(seastar::directory_entry)::<lambda(seastar::directory_entry)>&, seastar::directory_entry&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:44 scylladb#13 0x6cd9953 in apply<lister::visit(seastar::directory_entry)::<lambda(seastar::directory_entry)>&, seastar::directory_entry&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1650 scylladb#14 0x6cd6ba9 in operator() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1168 scylladb#15 0x6cdbac0 in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:101 scylladb#16 0x6ce6178 in seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>::operator()(seastar::directory_entry&&) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:184 scylladb#17 0x6ce3033 in seastar::apply_helper<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, std::tuple<seastar::directory_entry>&&, std::integer_sequence<unsigned long, 0ul> >::apply(seastar::noncopyable_function<seastar::future<> (seastar::directory_e ntry&&)>&&, std::tuple<seastar::directory_entry>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36 scylladb#18 0x6ce3130 in auto seastar::apply<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::directory_entry>(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&, std::tuple<seastar::directory_entry>&&) /local/home/bha levy/dev/scylla/seastar/include/seastar/core/apply.hh:44 scylladb#19 0x6ce322f in seastar::future<> seastar::futurize<seastar::future<> >::apply<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::directory_entry>(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&, std::tuple<s eastar::directory_entry>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1650 scylladb#20 0x6cdf88d in seastar::future<seastar::directory_entry>::then_impl<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<seastar::directory_entry>&&)scylladb#1}::operator()(seastar::future_state<seastar::directory_entry>) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1196 scylladb#21 0x6cea2e0 in seastar::continuation<seastar::future<seastar::directory_entry>::then_impl<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&)::{lamb da()scylladb#1}::operator()() const::{lambda(seastar::future_state<seastar::directory_entry>&&)scylladb#1}, seastar::directory_entry>::run_and_dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:511 scylladb#22 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091 scylladb#23 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506 scylladb#24 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666 scylladb#25 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199 scylladb#26 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115 scylladb#27 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486 scylladb#28 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2) SUMMARY: AddressSanitizer: heap-use-after-free /usr/include/c++/9/bits/hashtable_policy.h:283 in _M_next Shadow bytes around the buggy address: 0x0c068025c100: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c068025c110: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa 0x0c068025c120: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd 0x0c068025c130: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c068025c140: fd fd fd fa fa fa 00 00 00 fa fa fa fd fd fd fa =>0x0c068025c150: fa fa fd fd fd fa fa fa[fd]fd fd fd fa fa fd fd 0x0c068025c160: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c068025c170: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa 0x0c068025c180: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd 0x0c068025c190: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c068025c1a0: fd fd fd fd fa fa fd fd fd fd fa fa 00 00 00 fa Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa Freed heap region: fd Stack left redzone: f1 Stack mid redzone: f2 Stack right redzone: f3 Stack after return: f5 Stack use after scope: f8 Global redzone: f9 Global init order: f6 Poisoned by user: f7 Container overflow: fc Array cookie: ac Intra object redzone: bb ASan internal: fe Left alloca redzone: ca Right alloca redzone: cb Shadow gap: cc ==406079==ABORTING Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Rather than using lw_shared_ptr to keep it around, use do_with. Fixes a use-after-free seen with interrupt_build_process_with_resharding_half_to_max_test in debug mode: ==406079==ERROR: AddressSanitizer: heap-use-after-free on address 0x603001320ac0 at pc 0x00000777a062 bp 0x7ffdeb77b010 sp 0x7ffdeb77b000 READ of size 8 at 0x603001320ac0 thread T0 #0 0x777a061 in _M_next /usr/include/c++/9/bits/hashtable_policy.h:283 scylladb#1 0x779ead7 in _M_incr /usr/include/c++/9/bits/hashtable_policy.h:299 scylladb#2 0x777bbac in operator++ /usr/include/c++/9/bits/hashtable_policy.h:362 scylladb#3 0x77f8384 in run_and_dispose /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future-util.hh:568 scylladb#4 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091 scylladb#5 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506 scylladb#6 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666 scylladb#7 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199 scylladb#8 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115 scylladb#9 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486 scylladb#10 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2) scylladb#11 0x42b69d in _start (/local/home/bhalevy/.dtest/dtest-q85idok7/test/node1/bin/scylla+0x42b69d) 0x603001320ac0 is located 0 bytes inside of 32-byte region [0x603001320ac0,0x603001320ae0) freed by thread T0 here: #0 0x7fb86520996f in operator delete(void*) (/lib64/libasan.so.5+0x11096f) scylladb#1 0x77d1889 in deallocate /usr/include/c++/9/ext/new_allocator.h:128 scylladb#2 0x77bcbb2 in deallocate /usr/include/c++/9/bits/alloc_traits.h:470 scylladb#3 0x77bccc5 in _M_deallocate_node_ptr /usr/include/c++/9/bits/hashtable_policy.h:2113 scylladb#4 0x779dc59 in _M_deallocate_node /usr/include/c++/9/bits/hashtable_policy.h:2103 scylladb#5 0x77bc8b6 in _M_deallocate_nodes /usr/include/c++/9/bits/hashtable_policy.h:2124 scylladb#6 0x779d18f in clear /usr/include/c++/9/bits/hashtable.h:2028 scylladb#7 0x7779b8e in ~_Hashtable /usr/include/c++/9/bits/hashtable.h:1352 scylladb#8 0x774cfed in ~unordered_map /usr/include/c++/9/bits/unordered_map.h:102 scylladb#9 0x774d288 in ~shared_ptr_no_esft /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:157 scylladb#10 0x774d2f1 in dispose /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:213 scylladb#11 0x76f9642 in ~lw_shared_ptr /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:300 scylladb#12 0x76f9642 in ~<lambda> /local/home/bhalevy/dev/scylla/distributed_loader.cc:743 scylladb#13 0x771e0a7 in ~<lambda> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1167 scylladb#14 0x777c64c in destroy /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:133 scylladb#15 0x586059 in seastar::noncopyable_function<seastar::future<> ()>::~noncopyable_function() /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:165 scylladb#16 0x5867da in seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}::~future_state() /local/home/bha levy/dev/scylla/seastar/include/seastar/core/future.hh:1192 scylladb#17 0xab704d in seastar::continuation<seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}>::~continu ation() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:507 scylladb#18 0xab7216 in seastar::continuation<seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}>::run_and_ dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:512 scylladb#19 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091 scylladb#20 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506 scylladb#21 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666 scylladb#22 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199 scylladb#23 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115 scylladb#24 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486 scylladb#25 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2) previously allocated by thread T0 here: #0 0x7fb8652089d7 in operator new(unsigned long) (/lib64/libasan.so.5+0x10f9d7) scylladb#1 0x77d1796 in allocate /usr/include/c++/9/ext/new_allocator.h:114 scylladb#2 0x77bcab3 in allocate /usr/include/c++/9/bits/alloc_traits.h:444 scylladb#3 0x779d936 in _M_allocate_node<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable_policy.h:2081 scylladb#4 0x777a2ed in _M_emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable.h:1638 scylladb#5 0x774d707 in emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable.h:749 scylladb#6 0x771ce3e in emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/unordered_map.h:388 scylladb#7 0x76f8280 in operator() /local/home/bhalevy/dev/scylla/distributed_loader.cc:732 scylladb#8 0x774dc5d in _M_invoke /usr/include/c++/9/bits/std_function.h:286 scylladb#9 0x6cde67b in std::function<seastar::future<> (std::filesystem::__cxx11::path, seastar::directory_entry)>::operator()(std::filesystem::__cxx11::path, seastar::directory_entry) const /usr/include/c++/9/bits/std_function.h:690 scylladb#10 0x6cd345d in operator() /local/home/bhalevy/dev/scylla/lister.cc:30 scylladb#11 0x6cd96e0 in apply /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36 scylladb#12 0x6cd9854 in apply<lister::visit(seastar::directory_entry)::<lambda(seastar::directory_entry)>&, seastar::directory_entry&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:44 scylladb#13 0x6cd9953 in apply<lister::visit(seastar::directory_entry)::<lambda(seastar::directory_entry)>&, seastar::directory_entry&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1650 scylladb#14 0x6cd6ba9 in operator() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1168 scylladb#15 0x6cdbac0 in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:101 scylladb#16 0x6ce6178 in seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>::operator()(seastar::directory_entry&&) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:184 scylladb#17 0x6ce3033 in seastar::apply_helper<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, std::tuple<seastar::directory_entry>&&, std::integer_sequence<unsigned long, 0ul> >::apply(seastar::noncopyable_function<seastar::future<> (seastar::directory_e ntry&&)>&&, std::tuple<seastar::directory_entry>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36 scylladb#18 0x6ce3130 in auto seastar::apply<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::directory_entry>(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&, std::tuple<seastar::directory_entry>&&) /local/home/bha levy/dev/scylla/seastar/include/seastar/core/apply.hh:44 scylladb#19 0x6ce322f in seastar::future<> seastar::futurize<seastar::future<> >::apply<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::directory_entry>(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&, std::tuple<s eastar::directory_entry>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1650 scylladb#20 0x6cdf88d in seastar::future<seastar::directory_entry>::then_impl<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<seastar::directory_entry>&&)scylladb#1}::operator()(seastar::future_state<seastar::directory_entry>) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1196 scylladb#21 0x6cea2e0 in seastar::continuation<seastar::future<seastar::directory_entry>::then_impl<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&)::{lamb da()scylladb#1}::operator()() const::{lambda(seastar::future_state<seastar::directory_entry>&&)scylladb#1}, seastar::directory_entry>::run_and_dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:511 scylladb#22 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091 scylladb#23 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506 scylladb#24 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666 scylladb#25 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199 scylladb#26 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115 scylladb#27 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486 scylladb#28 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2) SUMMARY: AddressSanitizer: heap-use-after-free /usr/include/c++/9/bits/hashtable_policy.h:283 in _M_next Shadow bytes around the buggy address: 0x0c068025c100: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c068025c110: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa 0x0c068025c120: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd 0x0c068025c130: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c068025c140: fd fd fd fa fa fa 00 00 00 fa fa fa fd fd fd fa =>0x0c068025c150: fa fa fd fd fd fa fa fa[fd]fd fd fd fa fa fd fd 0x0c068025c160: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c068025c170: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa 0x0c068025c180: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd 0x0c068025c190: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c068025c1a0: fd fd fd fd fa fa fd fd fd fd fa fa 00 00 00 fa Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa Freed heap region: fd Stack left redzone: f1 Stack mid redzone: f2 Stack right redzone: f3 Stack after return: f5 Stack use after scope: f8 Global redzone: f9 Global init order: f6 Poisoned by user: f7 Container overflow: fc Array cookie: ac Intra object redzone: bb ASan internal: fe Left alloca redzone: ca Right alloca redzone: cb Shadow gap: cc ==406079==ABORTING Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Rather than using lw_shared_ptr to keep it around, use do_with. Fixes a use-after-free seen with interrupt_build_process_with_resharding_half_to_max_test in debug mode: `verifier` was not captured in the scan_dir.then() continuation. We could add a capture in that lambda, but it's cleaner and safer to let do_with hold on to it instead. ==406079==ERROR: AddressSanitizer: heap-use-after-free on address 0x603001320ac0 at pc 0x00000777a062 bp 0x7ffdeb77b010 sp 0x7ffdeb77b000 READ of size 8 at 0x603001320ac0 thread T0 #0 0x777a061 in _M_next /usr/include/c++/9/bits/hashtable_policy.h:283 scylladb#1 0x779ead7 in _M_incr /usr/include/c++/9/bits/hashtable_policy.h:299 scylladb#2 0x777bbac in operator++ /usr/include/c++/9/bits/hashtable_policy.h:362 scylladb#3 0x77f8384 in run_and_dispose /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future-util.hh:568 scylladb#4 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091 scylladb#5 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506 scylladb#6 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666 scylladb#7 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199 scylladb#8 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115 scylladb#9 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486 scylladb#10 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2) scylladb#11 0x42b69d in _start (/local/home/bhalevy/.dtest/dtest-q85idok7/test/node1/bin/scylla+0x42b69d) 0x603001320ac0 is located 0 bytes inside of 32-byte region [0x603001320ac0,0x603001320ae0) freed by thread T0 here: #0 0x7fb86520996f in operator delete(void*) (/lib64/libasan.so.5+0x11096f) scylladb#1 0x77d1889 in deallocate /usr/include/c++/9/ext/new_allocator.h:128 scylladb#2 0x77bcbb2 in deallocate /usr/include/c++/9/bits/alloc_traits.h:470 scylladb#3 0x77bccc5 in _M_deallocate_node_ptr /usr/include/c++/9/bits/hashtable_policy.h:2113 scylladb#4 0x779dc59 in _M_deallocate_node /usr/include/c++/9/bits/hashtable_policy.h:2103 scylladb#5 0x77bc8b6 in _M_deallocate_nodes /usr/include/c++/9/bits/hashtable_policy.h:2124 scylladb#6 0x779d18f in clear /usr/include/c++/9/bits/hashtable.h:2028 scylladb#7 0x7779b8e in ~_Hashtable /usr/include/c++/9/bits/hashtable.h:1352 scylladb#8 0x774cfed in ~unordered_map /usr/include/c++/9/bits/unordered_map.h:102 scylladb#9 0x774d288 in ~shared_ptr_no_esft /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:157 scylladb#10 0x774d2f1 in dispose /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:213 scylladb#11 0x76f9642 in ~lw_shared_ptr /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:300 scylladb#12 0x76f9642 in ~<lambda> /local/home/bhalevy/dev/scylla/distributed_loader.cc:743 scylladb#13 0x771e0a7 in ~<lambda> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1167 scylladb#14 0x777c64c in destroy /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:133 scylladb#15 0x586059 in seastar::noncopyable_function<seastar::future<> ()>::~noncopyable_function() /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:165 scylladb#16 0x5867da in seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}::~future_state() /local/home/bha levy/dev/scylla/seastar/include/seastar/core/future.hh:1192 scylladb#17 0xab704d in seastar::continuation<seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}>::~continu ation() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:507 scylladb#18 0xab7216 in seastar::continuation<seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}>::run_and_ dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:512 scylladb#19 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091 scylladb#20 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506 scylladb#21 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666 scylladb#22 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199 scylladb#23 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115 scylladb#24 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486 scylladb#25 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2) previously allocated by thread T0 here: #0 0x7fb8652089d7 in operator new(unsigned long) (/lib64/libasan.so.5+0x10f9d7) scylladb#1 0x77d1796 in allocate /usr/include/c++/9/ext/new_allocator.h:114 scylladb#2 0x77bcab3 in allocate /usr/include/c++/9/bits/alloc_traits.h:444 scylladb#3 0x779d936 in _M_allocate_node<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable_policy.h:2081 scylladb#4 0x777a2ed in _M_emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable.h:1638 scylladb#5 0x774d707 in emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable.h:749 scylladb#6 0x771ce3e in emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/unordered_map.h:388 scylladb#7 0x76f8280 in operator() /local/home/bhalevy/dev/scylla/distributed_loader.cc:732 scylladb#8 0x774dc5d in _M_invoke /usr/include/c++/9/bits/std_function.h:286 scylladb#9 0x6cde67b in std::function<seastar::future<> (std::filesystem::__cxx11::path, seastar::directory_entry)>::operator()(std::filesystem::__cxx11::path, seastar::directory_entry) const /usr/include/c++/9/bits/std_function.h:690 scylladb#10 0x6cd345d in operator() /local/home/bhalevy/dev/scylla/lister.cc:30 scylladb#11 0x6cd96e0 in apply /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36 scylladb#12 0x6cd9854 in apply<lister::visit(seastar::directory_entry)::<lambda(seastar::directory_entry)>&, seastar::directory_entry&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:44 scylladb#13 0x6cd9953 in apply<lister::visit(seastar::directory_entry)::<lambda(seastar::directory_entry)>&, seastar::directory_entry&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1650 scylladb#14 0x6cd6ba9 in operator() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1168 scylladb#15 0x6cdbac0 in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:101 scylladb#16 0x6ce6178 in seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>::operator()(seastar::directory_entry&&) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:184 scylladb#17 0x6ce3033 in seastar::apply_helper<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, std::tuple<seastar::directory_entry>&&, std::integer_sequence<unsigned long, 0ul> >::apply(seastar::noncopyable_function<seastar::future<> (seastar::directory_e ntry&&)>&&, std::tuple<seastar::directory_entry>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36 scylladb#18 0x6ce3130 in auto seastar::apply<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::directory_entry>(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&, std::tuple<seastar::directory_entry>&&) /local/home/bha levy/dev/scylla/seastar/include/seastar/core/apply.hh:44 scylladb#19 0x6ce322f in seastar::future<> seastar::futurize<seastar::future<> >::apply<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::directory_entry>(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&, std::tuple<s eastar::directory_entry>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1650 scylladb#20 0x6cdf88d in seastar::future<seastar::directory_entry>::then_impl<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<seastar::directory_entry>&&)scylladb#1}::operator()(seastar::future_state<seastar::directory_entry>) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1196 scylladb#21 0x6cea2e0 in seastar::continuation<seastar::future<seastar::directory_entry>::then_impl<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&)::{lamb da()scylladb#1}::operator()() const::{lambda(seastar::future_state<seastar::directory_entry>&&)scylladb#1}, seastar::directory_entry>::run_and_dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:511 scylladb#22 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091 scylladb#23 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506 scylladb#24 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666 scylladb#25 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199 scylladb#26 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115 scylladb#27 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486 scylladb#28 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2) SUMMARY: AddressSanitizer: heap-use-after-free /usr/include/c++/9/bits/hashtable_policy.h:283 in _M_next Shadow bytes around the buggy address: 0x0c068025c100: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c068025c110: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa 0x0c068025c120: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd 0x0c068025c130: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c068025c140: fd fd fd fa fa fa 00 00 00 fa fa fa fd fd fd fa =>0x0c068025c150: fa fa fd fd fd fa fa fa[fd]fd fd fd fa fa fd fd 0x0c068025c160: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c068025c170: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa 0x0c068025c180: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd 0x0c068025c190: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c068025c1a0: fd fd fd fd fa fa fd fd fd fd fa fa 00 00 00 fa Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa Freed heap region: fd Stack left redzone: f1 Stack mid redzone: f2 Stack right redzone: f3 Stack after return: f5 Stack use after scope: f8 Global redzone: f9 Global init order: f6 Poisoned by user: f7 Container overflow: fc Array cookie: ac Intra object redzone: bb ASan internal: fe Left alloca redzone: ca Right alloca redzone: cb Shadow gap: cc ==406079==ABORTING Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Rather than using lw_shared_ptr to keep it around, use do_with. Fixes a use-after-free seen with interrupt_build_process_with_resharding_half_to_max_test in debug mode: `verifier` was not captured in the scan_dir.then() continuation. We could add a capture in that lambda, but it's cleaner and safer to let do_with hold on to it instead. ==406079==ERROR: AddressSanitizer: heap-use-after-free on address 0x603001320ac0 at pc 0x00000777a062 bp 0x7ffdeb77b010 sp 0x7ffdeb77b000 READ of size 8 at 0x603001320ac0 thread T0 #0 0x777a061 in _M_next /usr/include/c++/9/bits/hashtable_policy.h:283 scylladb#1 0x779ead7 in _M_incr /usr/include/c++/9/bits/hashtable_policy.h:299 scylladb#2 0x777bbac in operator++ /usr/include/c++/9/bits/hashtable_policy.h:362 scylladb#3 0x77f8384 in run_and_dispose /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future-util.hh:568 scylladb#4 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091 scylladb#5 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506 scylladb#6 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666 scylladb#7 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199 scylladb#8 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115 scylladb#9 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486 scylladb#10 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2) scylladb#11 0x42b69d in _start (/local/home/bhalevy/.dtest/dtest-q85idok7/test/node1/bin/scylla+0x42b69d) 0x603001320ac0 is located 0 bytes inside of 32-byte region [0x603001320ac0,0x603001320ae0) freed by thread T0 here: #0 0x7fb86520996f in operator delete(void*) (/lib64/libasan.so.5+0x11096f) scylladb#1 0x77d1889 in deallocate /usr/include/c++/9/ext/new_allocator.h:128 scylladb#2 0x77bcbb2 in deallocate /usr/include/c++/9/bits/alloc_traits.h:470 scylladb#3 0x77bccc5 in _M_deallocate_node_ptr /usr/include/c++/9/bits/hashtable_policy.h:2113 scylladb#4 0x779dc59 in _M_deallocate_node /usr/include/c++/9/bits/hashtable_policy.h:2103 scylladb#5 0x77bc8b6 in _M_deallocate_nodes /usr/include/c++/9/bits/hashtable_policy.h:2124 scylladb#6 0x779d18f in clear /usr/include/c++/9/bits/hashtable.h:2028 scylladb#7 0x7779b8e in ~_Hashtable /usr/include/c++/9/bits/hashtable.h:1352 scylladb#8 0x774cfed in ~unordered_map /usr/include/c++/9/bits/unordered_map.h:102 scylladb#9 0x774d288 in ~shared_ptr_no_esft /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:157 scylladb#10 0x774d2f1 in dispose /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:213 scylladb#11 0x76f9642 in ~lw_shared_ptr /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:300 scylladb#12 0x76f9642 in ~<lambda> /local/home/bhalevy/dev/scylla/distributed_loader.cc:743 scylladb#13 0x771e0a7 in ~<lambda> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1167 scylladb#14 0x777c64c in destroy /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:133 scylladb#15 0x586059 in seastar::noncopyable_function<seastar::future<> ()>::~noncopyable_function() /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:165 scylladb#16 0x5867da in seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}::~future_state() /local/home/bha levy/dev/scylla/seastar/include/seastar/core/future.hh:1192 scylladb#17 0xab704d in seastar::continuation<seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}>::~continu ation() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:507 scylladb#18 0xab7216 in seastar::continuation<seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}>::run_and_ dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:512 scylladb#19 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091 scylladb#20 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506 scylladb#21 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666 scylladb#22 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199 scylladb#23 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115 scylladb#24 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486 scylladb#25 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2) previously allocated by thread T0 here: #0 0x7fb8652089d7 in operator new(unsigned long) (/lib64/libasan.so.5+0x10f9d7) scylladb#1 0x77d1796 in allocate /usr/include/c++/9/ext/new_allocator.h:114 scylladb#2 0x77bcab3 in allocate /usr/include/c++/9/bits/alloc_traits.h:444 scylladb#3 0x779d936 in _M_allocate_node<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable_policy.h:2081 scylladb#4 0x777a2ed in _M_emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable.h:1638 scylladb#5 0x774d707 in emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable.h:749 scylladb#6 0x771ce3e in emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/unordered_map.h:388 scylladb#7 0x76f8280 in operator() /local/home/bhalevy/dev/scylla/distributed_loader.cc:732 scylladb#8 0x774dc5d in _M_invoke /usr/include/c++/9/bits/std_function.h:286 scylladb#9 0x6cde67b in std::function<seastar::future<> (std::filesystem::__cxx11::path, seastar::directory_entry)>::operator()(std::filesystem::__cxx11::path, seastar::directory_entry) const /usr/include/c++/9/bits/std_function.h:690 scylladb#10 0x6cd345d in operator() /local/home/bhalevy/dev/scylla/lister.cc:30 scylladb#11 0x6cd96e0 in apply /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36 scylladb#12 0x6cd9854 in apply<lister::visit(seastar::directory_entry)::<lambda(seastar::directory_entry)>&, seastar::directory_entry&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:44 scylladb#13 0x6cd9953 in apply<lister::visit(seastar::directory_entry)::<lambda(seastar::directory_entry)>&, seastar::directory_entry&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1650 scylladb#14 0x6cd6ba9 in operator() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1168 scylladb#15 0x6cdbac0 in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:101 scylladb#16 0x6ce6178 in seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>::operator()(seastar::directory_entry&&) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:184 scylladb#17 0x6ce3033 in seastar::apply_helper<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, std::tuple<seastar::directory_entry>&&, std::integer_sequence<unsigned long, 0ul> >::apply(seastar::noncopyable_function<seastar::future<> (seastar::directory_e ntry&&)>&&, std::tuple<seastar::directory_entry>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36 scylladb#18 0x6ce3130 in auto seastar::apply<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::directory_entry>(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&, std::tuple<seastar::directory_entry>&&) /local/home/bha levy/dev/scylla/seastar/include/seastar/core/apply.hh:44 scylladb#19 0x6ce322f in seastar::future<> seastar::futurize<seastar::future<> >::apply<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::directory_entry>(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&, std::tuple<s eastar::directory_entry>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1650 scylladb#20 0x6cdf88d in seastar::future<seastar::directory_entry>::then_impl<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<seastar::directory_entry>&&)scylladb#1}::operator()(seastar::future_state<seastar::directory_entry>) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1196 scylladb#21 0x6cea2e0 in seastar::continuation<seastar::future<seastar::directory_entry>::then_impl<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&)::{lamb da()scylladb#1}::operator()() const::{lambda(seastar::future_state<seastar::directory_entry>&&)scylladb#1}, seastar::directory_entry>::run_and_dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:511 scylladb#22 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091 scylladb#23 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506 scylladb#24 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666 scylladb#25 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199 scylladb#26 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115 scylladb#27 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486 scylladb#28 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2) SUMMARY: AddressSanitizer: heap-use-after-free /usr/include/c++/9/bits/hashtable_policy.h:283 in _M_next Shadow bytes around the buggy address: 0x0c068025c100: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c068025c110: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa 0x0c068025c120: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd 0x0c068025c130: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c068025c140: fd fd fd fa fa fa 00 00 00 fa fa fa fd fd fd fa =>0x0c068025c150: fa fa fd fd fd fa fa fa[fd]fd fd fd fa fa fd fd 0x0c068025c160: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c068025c170: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa 0x0c068025c180: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd 0x0c068025c190: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c068025c1a0: fd fd fd fd fa fa fd fd fd fd fa fa 00 00 00 fa Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa Freed heap region: fd Stack left redzone: f1 Stack mid redzone: f2 Stack right redzone: f3 Stack after return: f5 Stack use after scope: f8 Global redzone: f9 Global init order: f6 Poisoned by user: f7 Container overflow: fc Array cookie: ac Intra object redzone: bb ASan internal: fe Left alloca redzone: ca Right alloca redzone: cb Shadow gap: cc ==406079==ABORTING Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Rather than using lw_shared_ptr to keep it around, use do_with. Fixes a use-after-free seen with interrupt_build_process_with_resharding_half_to_max_test in debug mode: `verifier` was not captured in the scan_dir.then() continuation. We could add a capture in that lambda, but it's cleaner and safer to let do_with hold on to it instead. ==406079==ERROR: AddressSanitizer: heap-use-after-free on address 0x603001320ac0 at pc 0x00000777a062 bp 0x7ffdeb77b010 sp 0x7ffdeb77b000 READ of size 8 at 0x603001320ac0 thread T0 #0 0x777a061 in _M_next /usr/include/c++/9/bits/hashtable_policy.h:283 scylladb#1 0x779ead7 in _M_incr /usr/include/c++/9/bits/hashtable_policy.h:299 scylladb#2 0x777bbac in operator++ /usr/include/c++/9/bits/hashtable_policy.h:362 scylladb#3 0x77f8384 in run_and_dispose /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future-util.hh:568 scylladb#4 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091 scylladb#5 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506 scylladb#6 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666 scylladb#7 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199 scylladb#8 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115 scylladb#9 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486 scylladb#10 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2) scylladb#11 0x42b69d in _start (/local/home/bhalevy/.dtest/dtest-q85idok7/test/node1/bin/scylla+0x42b69d) 0x603001320ac0 is located 0 bytes inside of 32-byte region [0x603001320ac0,0x603001320ae0) freed by thread T0 here: #0 0x7fb86520996f in operator delete(void*) (/lib64/libasan.so.5+0x11096f) scylladb#1 0x77d1889 in deallocate /usr/include/c++/9/ext/new_allocator.h:128 scylladb#2 0x77bcbb2 in deallocate /usr/include/c++/9/bits/alloc_traits.h:470 scylladb#3 0x77bccc5 in _M_deallocate_node_ptr /usr/include/c++/9/bits/hashtable_policy.h:2113 scylladb#4 0x779dc59 in _M_deallocate_node /usr/include/c++/9/bits/hashtable_policy.h:2103 scylladb#5 0x77bc8b6 in _M_deallocate_nodes /usr/include/c++/9/bits/hashtable_policy.h:2124 scylladb#6 0x779d18f in clear /usr/include/c++/9/bits/hashtable.h:2028 scylladb#7 0x7779b8e in ~_Hashtable /usr/include/c++/9/bits/hashtable.h:1352 scylladb#8 0x774cfed in ~unordered_map /usr/include/c++/9/bits/unordered_map.h:102 scylladb#9 0x774d288 in ~shared_ptr_no_esft /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:157 scylladb#10 0x774d2f1 in dispose /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:213 scylladb#11 0x76f9642 in ~lw_shared_ptr /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:300 scylladb#12 0x76f9642 in ~<lambda> /local/home/bhalevy/dev/scylla/distributed_loader.cc:743 scylladb#13 0x771e0a7 in ~<lambda> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1167 scylladb#14 0x777c64c in destroy /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:133 scylladb#15 0x586059 in seastar::noncopyable_function<seastar::future<> ()>::~noncopyable_function() /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:165 scylladb#16 0x5867da in seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}::~future_state() /local/home/bha levy/dev/scylla/seastar/include/seastar/core/future.hh:1192 scylladb#17 0xab704d in seastar::continuation<seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}>::~continu ation() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:507 scylladb#18 0xab7216 in seastar::continuation<seastar::future<>::then_impl<seastar::noncopyable_function<seastar::future<> ()>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> ()>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<>&&)scylladb#1}>::run_and_ dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:512 scylladb#19 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091 scylladb#20 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506 scylladb#21 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666 scylladb#22 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199 scylladb#23 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115 scylladb#24 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486 scylladb#25 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2) previously allocated by thread T0 here: #0 0x7fb8652089d7 in operator new(unsigned long) (/lib64/libasan.so.5+0x10f9d7) scylladb#1 0x77d1796 in allocate /usr/include/c++/9/ext/new_allocator.h:114 scylladb#2 0x77bcab3 in allocate /usr/include/c++/9/bits/alloc_traits.h:444 scylladb#3 0x779d936 in _M_allocate_node<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable_policy.h:2081 scylladb#4 0x777a2ed in _M_emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable.h:1638 scylladb#5 0x774d707 in emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/hashtable.h:749 scylladb#6 0x771ce3e in emplace<long int&, distributed_loader::cleanup_column_family_sstables(seastar::distributed<database>&, seastar::sstring, seastar::sstring, seastar::sstring)::sstable_descriptor> /usr/include/c++/9/bits/unordered_map.h:388 scylladb#7 0x76f8280 in operator() /local/home/bhalevy/dev/scylla/distributed_loader.cc:732 scylladb#8 0x774dc5d in _M_invoke /usr/include/c++/9/bits/std_function.h:286 scylladb#9 0x6cde67b in std::function<seastar::future<> (std::filesystem::__cxx11::path, seastar::directory_entry)>::operator()(std::filesystem::__cxx11::path, seastar::directory_entry) const /usr/include/c++/9/bits/std_function.h:690 scylladb#10 0x6cd345d in operator() /local/home/bhalevy/dev/scylla/lister.cc:30 scylladb#11 0x6cd96e0 in apply /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36 scylladb#12 0x6cd9854 in apply<lister::visit(seastar::directory_entry)::<lambda(seastar::directory_entry)>&, seastar::directory_entry&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:44 scylladb#13 0x6cd9953 in apply<lister::visit(seastar::directory_entry)::<lambda(seastar::directory_entry)>&, seastar::directory_entry&&> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1650 scylladb#14 0x6cd6ba9 in operator() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1168 scylladb#15 0x6cdbac0 in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:101 scylladb#16 0x6ce6178 in seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>::operator()(seastar::directory_entry&&) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:184 scylladb#17 0x6ce3033 in seastar::apply_helper<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, std::tuple<seastar::directory_entry>&&, std::integer_sequence<unsigned long, 0ul> >::apply(seastar::noncopyable_function<seastar::future<> (seastar::directory_e ntry&&)>&&, std::tuple<seastar::directory_entry>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/apply.hh:36 scylladb#18 0x6ce3130 in auto seastar::apply<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::directory_entry>(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&, std::tuple<seastar::directory_entry>&&) /local/home/bha levy/dev/scylla/seastar/include/seastar/core/apply.hh:44 scylladb#19 0x6ce322f in seastar::future<> seastar::futurize<seastar::future<> >::apply<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::directory_entry>(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&, std::tuple<s eastar::directory_entry>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1650 scylladb#20 0x6cdf88d in seastar::future<seastar::directory_entry>::then_impl<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&)::{lambda()scylladb#1}::operator()() const::{lambda(seastar::future_state<seastar::directory_entry>&&)scylladb#1}::operator()(seastar::future_state<seastar::directory_entry>) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1196 scylladb#21 0x6cea2e0 in seastar::continuation<seastar::future<seastar::directory_entry>::then_impl<seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>, seastar::future<> >(seastar::noncopyable_function<seastar::future<> (seastar::directory_entry&&)>&&)::{lamb da()scylladb#1}::operator()() const::{lambda(seastar::future_state<seastar::directory_entry>&&)scylladb#1}, seastar::directory_entry>::run_and_dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:511 scylladb#22 0x9808883 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2091 scylladb#23 0x980c831 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2506 scylladb#24 0x9811c44 in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2666 scylladb#25 0x967db93 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:199 scylladb#26 0x967bbb9 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115 scylladb#27 0x475fcb in main /local/home/bhalevy/dev/scylla/main.cc:486 scylladb#28 0x7fb862d981a2 in __libc_start_main (/lib64/libc.so.6+0x271a2) SUMMARY: AddressSanitizer: heap-use-after-free /usr/include/c++/9/bits/hashtable_policy.h:283 in _M_next Shadow bytes around the buggy address: 0x0c068025c100: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c068025c110: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa 0x0c068025c120: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd 0x0c068025c130: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c068025c140: fd fd fd fa fa fa 00 00 00 fa fa fa fd fd fd fa =>0x0c068025c150: fa fa fd fd fd fa fa fa[fd]fd fd fd fa fa fd fd 0x0c068025c160: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c068025c170: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa 0x0c068025c180: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd 0x0c068025c190: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c068025c1a0: fd fd fd fd fa fa fd fd fd fd fa fa 00 00 00 fa Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa Freed heap region: fd Stack left redzone: f1 Stack mid redzone: f2 Stack right redzone: f3 Stack after return: f5 Stack use after scope: f8 Global redzone: f9 Global init order: f6 Poisoned by user: f7 Container overflow: fc Array cookie: ac Intra object redzone: bb ASan internal: fe Left alloca redzone: ca Right alloca redzone: cb Shadow gap: cc ==406079==ABORTING Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
… token_metadata_ptr Fixes use-after-free seen with putget_with_reloaded_certificates_test: ``` ==215==ERROR: AddressSanitizer: heap-use-after-free on address 0x603000a8b180 at pc 0x000012eb5a83 bp 0x7ffd2c16d4c0 sp 0x7ffd2c16d4b0 READ of size 8 at 0x603000a8b180 thread T0 #0 0x12eb5a82 in std::__uniq_ptr_impl<locator::token_metadata_impl, std::default_delete<locator::token_metadata_impl> >::_M_ptr() const /usr/include/c++/10/bits/unique_ptr.h:173 scylladb#1 0x12ea230d in std::unique_ptr<locator::token_metadata_impl, std::default_delete<locator::token_metadata_impl> >::get() const /usr/include/c++/10/bits/unique_ptr.h:422 scylladb#2 0x12e8d3e8 in std::unique_ptr<locator::token_metadata_impl, std::default_delete<locator::token_metadata_impl> >::operator->() const /usr/include/c++/10/bits/unique_ptr.h:416 scylladb#3 0x12e5d0a2 in locator::token_metadata::ring_range(std::optional<interval_bound<dht::ring_position> > const&, bool) const locator/token_metadata.cc:1712 scylladb#4 0x112d0126 in service::query_ranges_to_vnodes_generator::process_one_range(unsigned long, std::vector<nonwrapping_interval<dht::ring_position>, std::allocator<nonwrapping_interval<dht::ring_position> > >&) service/storage_proxy.cc:4658 scylladb#5 0x112cf3c5 in service::query_ranges_to_vnodes_generator::operator()(unsigned long) service/storage_proxy.cc:4616 scylladb#6 0x112b2261 in service::storage_proxy::query_partition_key_range_concurrent(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >, std::vector<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >, std::allocator<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&, seastar::lw_shared_ptr<query::read_command>, db::consistency_level, service::query_ranges_to_vnodes_generator&&, int, tracing::trace_state_ptr, unsigned long, unsigned int, std::unordered_map<nonwrapping_interval<dht::token>, std::vector<utils::UUID, std::allocator<utils::UUID> >, std::hash<nonwrapping_interval<dht::token> >, std::equal_to<nonwrapping_interval<dht::token> >, std::allocator<std::pair<nonwrapping_interval<dht::token> const, std::vector<utils::UUID, std::allocator<utils::UUID> > > > >, service_permit) service/storage_proxy.cc:4023 scylladb#7 0x112b094e in operator() service/storage_proxy.cc:4160 scylladb#8 0x1139c8bb in invoke<service::storage_proxy::query_partition_key_range_concurrent(seastar::lowres_clock::time_point, std::vector<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >&&, seastar::lw_shared_ptr<query::read_command>, db::consistency_level, service::query_ranges_to_vnodes_generator&&, int, tracing::trace_state_ptr, uint64_t, uint32_t, service::replicas_per_token_range, service_permit)::<lambda(seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:2088 scylladb#9 0x1136625b in futurize_invoke<service::storage_proxy::query_partition_key_range_concurrent(seastar::lowres_clock::time_point, std::vector<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >&&, seastar::lw_shared_ptr<query::read_command>, db::consistency_level, service::query_ranges_to_vnodes_generator&&, int, tracing::trace_state_ptr, uint64_t, uint32_t, service::replicas_per_token_range, service_permit)::<lambda(seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:2119 scylladb#10 0x11366372 in operator()<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1480 scylladb#11 0x1139cc3b in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:145 scylladb#12 0x116f4944 in seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>::operator()(seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:201 scylladb#13 0x116b3397 in seastar::future<service::query_partition_key_range_concurrent_result> std::__invoke_impl<seastar::future<service::query_partition_key_range_concurrent_result>, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >(std::__invoke_other, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&) /usr/include/c++/10/bits/invoke.h:60 scylladb#14 0x1165c3a6 in std::__invoke_result<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::type std::__invoke<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&) /usr/include/c++/10/bits/invoke.h:96 scylladb#15 0x115e6542 in decltype(auto) std::__apply_impl<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >, 0ul>(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >&&, std::integer_sequence<unsigned long, 0ul>) /usr/include/c++/10/tuple:1724 scylladb#16 0x115e6663 in decltype(auto) std::apply<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >&&) /usr/include/c++/10/tuple:1736 scylladb#17 0x115e63f9 in seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}::operator()(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&) const::{lambda()scylladb#1}::operator()() const /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1530 scylladb#18 0x1165c4b9 in void seastar::futurize<seastar::future<service::query_partition_key_range_concurrent_result> >::satisfy_with_result_of<seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}::operator()(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&) const::{lambda()scylladb#1}>(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:2073 scylladb#19 0x115e61f5 in seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}::operator()(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1528 scylladb#20 0x1176e9cc in seastar::continuation<seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::run_and_dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:746 scylladb#21 0x16a9a455 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2196 scylladb#22 0x16a9e691 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2575 scylladb#23 0x16aa390e in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2730 scylladb#24 0x168ae4f7 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:207 scylladb#25 0x168ac541 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115 scylladb#26 0xd6cd3c4 in main /local/home/bhalevy/dev/scylla/main.cc:504 scylladb#27 0x7f8d905d8041 in __libc_start_main (/local/home/bhalevy/dev/scylla/build/debug/dynamic_libs/libc.so.6+0x27041) scylladb#28 0xd67c9ed in _start (/local/home/bhalevy/.dtest/dtest-o0qoqmkr/test/node3/bin/scylla+0xd67c9ed) 0x603000a8b180 is located 16 bytes inside of 24-byte region [0x603000a8b170,0x603000a8b188) freed by thread T0 here: #0 0x7f8d92a190cf in operator delete(void*, unsigned long) (/local/home/bhalevy/dev/scylla/build/debug/dynamic_libs/libasan.so.6+0xb30cf) scylladb#1 0xd7ebe54 in seastar::internal::lw_shared_ptr_accessors_no_esft<locator::token_metadata>::dispose(seastar::lw_shared_ptr_counter_base*) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:213 scylladb#2 0x112b155d in seastar::lw_shared_ptr<locator::token_metadata const>::~lw_shared_ptr() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:300 scylladb#3 0x112b155d in ~<lambda> service/storage_proxy.cc:4137 scylladb#4 0x1132e92d in ~<lambda> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1479 scylladb#5 0x1139cc91 in destroy /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:148 scylladb#6 0x11565673 in seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>::~noncopyable_function() /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:181 scylladb#7 0x1176e783 in seastar::continuation<seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::~continuation() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:729 scylladb#8 0x1176ea06 in seastar::continuation<seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::run_and_dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:750 scylladb#9 0x16a9a455 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2196 scylladb#10 0x16a9e691 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2575 scylladb#11 0x16aa390e in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2730 scylladb#12 0x168ae4f7 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:207 scylladb#13 0x168ac541 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115 scylladb#14 0xd6cd3c4 in main /local/home/bhalevy/dev/scylla/main.cc:504 scylladb#15 0x7f8d905d8041 in __libc_start_main (/local/home/bhalevy/dev/scylla/build/debug/dynamic_libs/libc.so.6+0x27041) previously allocated by thread T0 here: #0 0x7f8d92a18067 in operator new(unsigned long) (/local/home/bhalevy/dev/scylla/build/debug/dynamic_libs/libasan.so.6+0xb2067) scylladb#1 0x13cf7132 in seastar::lw_shared_ptr<locator::token_metadata> seastar::lw_shared_ptr<locator::token_metadata>::make<locator::token_metadata>(locator::token_metadata&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:266 scylladb#2 0x13cc3bfa in seastar::lw_shared_ptr<locator::token_metadata> seastar::make_lw_shared<locator::token_metadata>(locator::token_metadata&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:422 scylladb#3 0x13ca3007 in seastar::lw_shared_ptr<locator::token_metadata> locator::make_token_metadata_ptr<locator::token_metadata>(locator::token_metadata) locator/token_metadata.hh:338 scylladb#4 0x13c9bdd4 in locator::shared_token_metadata::clone() const locator/token_metadata.hh:358 scylladb#5 0x13c9c18a in service::storage_service::get_mutable_token_metadata_ptr() service/storage_service.hh:184 scylladb#6 0x13a5a445 in service::storage_service::handle_state_normal(gms::inet_address) service/storage_service.cc:1129 scylladb#7 0x13a6371c in service::storage_service::on_change(gms::inet_address, gms::application_state, gms::versioned_value const&) service/storage_service.cc:1421 scylladb#8 0x12a86269 in operator() gms/gossiper.cc:1639 scylladb#9 0x12ad3eea in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:145 scylladb#10 0x12be2aff in seastar::noncopyable_function<void (seastar::shared_ptr<gms::i_endpoint_state_change_subscriber>)>::operator()(seastar::shared_ptr<gms::i_endpoint_state_change_subscriber>) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:201 scylladb#11 0x12bb8e98 in atomic_vector<seastar::shared_ptr<gms::i_endpoint_state_change_subscriber> >::for_each(seastar::noncopyable_function<void (seastar::shared_ptr<gms::i_endpoint_state_change_subscriber>)>) utils/atomic_vector.hh:62 scylladb#12 0x12a8662b in gms::gossiper::do_on_change_notifications(gms::inet_address, gms::application_state const&, gms::versioned_value const&) gms/gossiper.cc:1638 scylladb#13 0x12a9387c in operator() gms/gossiper.cc:1978 scylladb#14 0x12b49b20 in __invoke_impl<void, gms::gossiper::add_local_application_state(std::__cxx11::list<std::pair<gms::application_state, gms::versioned_value> >)::<lambda(gms::gossiper&)> mutable::<lambda()> > /usr/include/c++/10/bits/invoke.h:60 scylladb#15 0x12b21fd6 in __invoke<gms::gossiper::add_local_application_state(std::__cxx11::list<std::pair<gms::application_state, gms::versioned_value> >)::<lambda(gms::gossiper&)> mutable::<lambda()> > /usr/include/c++/10/bits/invoke.h:95 scylladb#16 0x12b02865 in __apply_impl<gms::gossiper::add_local_application_state(std::__cxx11::list<std::pair<gms::application_state, gms::versioned_value> >)::<lambda(gms::gossiper&)> mutable::<lambda()>, std::tuple<> > /usr/include/c++/10/tuple:1723 scylladb#17 0x12b028d8 in apply<gms::gossiper::add_local_application_state(std::__cxx11::list<std::pair<gms::application_state, gms::versioned_value> >)::<lambda(gms::gossiper&)> mutable::<lambda()>, std::tuple<> > /usr/include/c++/10/tuple:1734 scylladb#18 0x12b02967 in apply<gms::gossiper::add_local_application_state(std::__cxx11::list<std::pair<gms::application_state, gms::versioned_value> >)::<lambda(gms::gossiper&)> mutable::<lambda()> > /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:2052 scylladb#19 0x12ad866a in operator() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/thread.hh:258 scylladb#20 0x12b609c2 in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:116 scylladb#21 0xdfabb5f in seastar::noncopyable_function<void ()>::operator()() const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:201 scylladb#22 0x16e21bb4 in seastar::thread_context::main() /local/home/bhalevy/dev/scylla/seastar/src/core/thread.cc:297 scylladb#23 0x16e2190f in seastar::thread_context::s_main(int, int) /local/home/bhalevy/dev/scylla/seastar/src/core/thread.cc:275 scylladb#24 0x7f8d9060322f (/local/home/bhalevy/dev/scylla/build/debug/dynamic_libs/libc.so.6+0x5222f) ``` Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
… token_metadata_ptr Fixes use-after-free seen with putget_with_reloaded_certificates_test: ``` ==215==ERROR: AddressSanitizer: heap-use-after-free on address 0x603000a8b180 at pc 0x000012eb5a83 bp 0x7ffd2c16d4c0 sp 0x7ffd2c16d4b0 READ of size 8 at 0x603000a8b180 thread T0 #0 0x12eb5a82 in std::__uniq_ptr_impl<locator::token_metadata_impl, std::default_delete<locator::token_metadata_impl> >::_M_ptr() const /usr/include/c++/10/bits/unique_ptr.h:173 scylladb#1 0x12ea230d in std::unique_ptr<locator::token_metadata_impl, std::default_delete<locator::token_metadata_impl> >::get() const /usr/include/c++/10/bits/unique_ptr.h:422 scylladb#2 0x12e8d3e8 in std::unique_ptr<locator::token_metadata_impl, std::default_delete<locator::token_metadata_impl> >::operator->() const /usr/include/c++/10/bits/unique_ptr.h:416 scylladb#3 0x12e5d0a2 in locator::token_metadata::ring_range(std::optional<interval_bound<dht::ring_position> > const&, bool) const locator/token_metadata.cc:1712 scylladb#4 0x112d0126 in service::query_ranges_to_vnodes_generator::process_one_range(unsigned long, std::vector<nonwrapping_interval<dht::ring_position>, std::allocator<nonwrapping_interval<dht::ring_position> > >&) service/storage_proxy.cc:4658 scylladb#5 0x112cf3c5 in service::query_ranges_to_vnodes_generator::operator()(unsigned long) service/storage_proxy.cc:4616 scylladb#6 0x112b2261 in service::storage_proxy::query_partition_key_range_concurrent(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >, std::vector<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >, std::allocator<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&, seastar::lw_shared_ptr<query::read_command>, db::consistency_level, service::query_ranges_to_vnodes_generator&&, int, tracing::trace_state_ptr, unsigned long, unsigned int, std::unordered_map<nonwrapping_interval<dht::token>, std::vector<utils::UUID, std::allocator<utils::UUID> >, std::hash<nonwrapping_interval<dht::token> >, std::equal_to<nonwrapping_interval<dht::token> >, std::allocator<std::pair<nonwrapping_interval<dht::token> const, std::vector<utils::UUID, std::allocator<utils::UUID> > > > >, service_permit) service/storage_proxy.cc:4023 scylladb#7 0x112b094e in operator() service/storage_proxy.cc:4160 scylladb#8 0x1139c8bb in invoke<service::storage_proxy::query_partition_key_range_concurrent(seastar::lowres_clock::time_point, std::vector<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >&&, seastar::lw_shared_ptr<query::read_command>, db::consistency_level, service::query_ranges_to_vnodes_generator&&, int, tracing::trace_state_ptr, uint64_t, uint32_t, service::replicas_per_token_range, service_permit)::<lambda(seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:2088 scylladb#9 0x1136625b in futurize_invoke<service::storage_proxy::query_partition_key_range_concurrent(seastar::lowres_clock::time_point, std::vector<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >&&, seastar::lw_shared_ptr<query::read_command>, db::consistency_level, service::query_ranges_to_vnodes_generator&&, int, tracing::trace_state_ptr, uint64_t, uint32_t, service::replicas_per_token_range, service_permit)::<lambda(seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:2119 scylladb#10 0x11366372 in operator()<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1480 scylladb#11 0x1139cc3b in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:145 scylladb#12 0x116f4944 in seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>::operator()(seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:201 scylladb#13 0x116b3397 in seastar::future<service::query_partition_key_range_concurrent_result> std::__invoke_impl<seastar::future<service::query_partition_key_range_concurrent_result>, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >(std::__invoke_other, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&) /usr/include/c++/10/bits/invoke.h:60 scylladb#14 0x1165c3a6 in std::__invoke_result<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::type std::__invoke<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&) /usr/include/c++/10/bits/invoke.h:96 scylladb#15 0x115e6542 in decltype(auto) std::__apply_impl<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >, 0ul>(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >&&, std::integer_sequence<unsigned long, 0ul>) /usr/include/c++/10/tuple:1724 scylladb#16 0x115e6663 in decltype(auto) std::apply<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >&&) /usr/include/c++/10/tuple:1736 scylladb#17 0x115e63f9 in seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}::operator()(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&) const::{lambda()scylladb#1}::operator()() const /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1530 scylladb#18 0x1165c4b9 in void seastar::futurize<seastar::future<service::query_partition_key_range_concurrent_result> >::satisfy_with_result_of<seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}::operator()(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&) const::{lambda()scylladb#1}>(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:2073 scylladb#19 0x115e61f5 in seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}::operator()(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1528 scylladb#20 0x1176e9cc in seastar::continuation<seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::run_and_dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:746 scylladb#21 0x16a9a455 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2196 scylladb#22 0x16a9e691 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2575 scylladb#23 0x16aa390e in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2730 scylladb#24 0x168ae4f7 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:207 scylladb#25 0x168ac541 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115 scylladb#26 0xd6cd3c4 in main /local/home/bhalevy/dev/scylla/main.cc:504 scylladb#27 0x7f8d905d8041 in __libc_start_main (/local/home/bhalevy/dev/scylla/build/debug/dynamic_libs/libc.so.6+0x27041) scylladb#28 0xd67c9ed in _start (/local/home/bhalevy/.dtest/dtest-o0qoqmkr/test/node3/bin/scylla+0xd67c9ed) 0x603000a8b180 is located 16 bytes inside of 24-byte region [0x603000a8b170,0x603000a8b188) freed by thread T0 here: #0 0x7f8d92a190cf in operator delete(void*, unsigned long) (/local/home/bhalevy/dev/scylla/build/debug/dynamic_libs/libasan.so.6+0xb30cf) scylladb#1 0xd7ebe54 in seastar::internal::lw_shared_ptr_accessors_no_esft<locator::token_metadata>::dispose(seastar::lw_shared_ptr_counter_base*) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:213 scylladb#2 0x112b155d in seastar::lw_shared_ptr<locator::token_metadata const>::~lw_shared_ptr() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:300 scylladb#3 0x112b155d in ~<lambda> service/storage_proxy.cc:4137 scylladb#4 0x1132e92d in ~<lambda> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1479 scylladb#5 0x1139cc91 in destroy /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:148 scylladb#6 0x11565673 in seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>::~noncopyable_function() /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:181 scylladb#7 0x1176e783 in seastar::continuation<seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::~continuation() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:729 scylladb#8 0x1176ea06 in seastar::continuation<seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::run_and_dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:750 scylladb#9 0x16a9a455 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2196 scylladb#10 0x16a9e691 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2575 scylladb#11 0x16aa390e in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2730 scylladb#12 0x168ae4f7 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:207 scylladb#13 0x168ac541 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115 scylladb#14 0xd6cd3c4 in main /local/home/bhalevy/dev/scylla/main.cc:504 scylladb#15 0x7f8d905d8041 in __libc_start_main (/local/home/bhalevy/dev/scylla/build/debug/dynamic_libs/libc.so.6+0x27041) previously allocated by thread T0 here: #0 0x7f8d92a18067 in operator new(unsigned long) (/local/home/bhalevy/dev/scylla/build/debug/dynamic_libs/libasan.so.6+0xb2067) scylladb#1 0x13cf7132 in seastar::lw_shared_ptr<locator::token_metadata> seastar::lw_shared_ptr<locator::token_metadata>::make<locator::token_metadata>(locator::token_metadata&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:266 scylladb#2 0x13cc3bfa in seastar::lw_shared_ptr<locator::token_metadata> seastar::make_lw_shared<locator::token_metadata>(locator::token_metadata&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:422 scylladb#3 0x13ca3007 in seastar::lw_shared_ptr<locator::token_metadata> locator::make_token_metadata_ptr<locator::token_metadata>(locator::token_metadata) locator/token_metadata.hh:338 scylladb#4 0x13c9bdd4 in locator::shared_token_metadata::clone() const locator/token_metadata.hh:358 scylladb#5 0x13c9c18a in service::storage_service::get_mutable_token_metadata_ptr() service/storage_service.hh:184 scylladb#6 0x13a5a445 in service::storage_service::handle_state_normal(gms::inet_address) service/storage_service.cc:1129 scylladb#7 0x13a6371c in service::storage_service::on_change(gms::inet_address, gms::application_state, gms::versioned_value const&) service/storage_service.cc:1421 scylladb#8 0x12a86269 in operator() gms/gossiper.cc:1639 scylladb#9 0x12ad3eea in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:145 scylladb#10 0x12be2aff in seastar::noncopyable_function<void (seastar::shared_ptr<gms::i_endpoint_state_change_subscriber>)>::operator()(seastar::shared_ptr<gms::i_endpoint_state_change_subscriber>) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:201 scylladb#11 0x12bb8e98 in atomic_vector<seastar::shared_ptr<gms::i_endpoint_state_change_subscriber> >::for_each(seastar::noncopyable_function<void (seastar::shared_ptr<gms::i_endpoint_state_change_subscriber>)>) utils/atomic_vector.hh:62 scylladb#12 0x12a8662b in gms::gossiper::do_on_change_notifications(gms::inet_address, gms::application_state const&, gms::versioned_value const&) gms/gossiper.cc:1638 scylladb#13 0x12a9387c in operator() gms/gossiper.cc:1978 scylladb#14 0x12b49b20 in __invoke_impl<void, gms::gossiper::add_local_application_state(std::__cxx11::list<std::pair<gms::application_state, gms::versioned_value> >)::<lambda(gms::gossiper&)> mutable::<lambda()> > /usr/include/c++/10/bits/invoke.h:60 scylladb#15 0x12b21fd6 in __invoke<gms::gossiper::add_local_application_state(std::__cxx11::list<std::pair<gms::application_state, gms::versioned_value> >)::<lambda(gms::gossiper&)> mutable::<lambda()> > /usr/include/c++/10/bits/invoke.h:95 scylladb#16 0x12b02865 in __apply_impl<gms::gossiper::add_local_application_state(std::__cxx11::list<std::pair<gms::application_state, gms::versioned_value> >)::<lambda(gms::gossiper&)> mutable::<lambda()>, std::tuple<> > /usr/include/c++/10/tuple:1723 scylladb#17 0x12b028d8 in apply<gms::gossiper::add_local_application_state(std::__cxx11::list<std::pair<gms::application_state, gms::versioned_value> >)::<lambda(gms::gossiper&)> mutable::<lambda()>, std::tuple<> > /usr/include/c++/10/tuple:1734 scylladb#18 0x12b02967 in apply<gms::gossiper::add_local_application_state(std::__cxx11::list<std::pair<gms::application_state, gms::versioned_value> >)::<lambda(gms::gossiper&)> mutable::<lambda()> > /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:2052 scylladb#19 0x12ad866a in operator() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/thread.hh:258 scylladb#20 0x12b609c2 in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:116 scylladb#21 0xdfabb5f in seastar::noncopyable_function<void ()>::operator()() const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:201 scylladb#22 0x16e21bb4 in seastar::thread_context::main() /local/home/bhalevy/dev/scylla/seastar/src/core/thread.cc:297 scylladb#23 0x16e2190f in seastar::thread_context::s_main(int, int) /local/home/bhalevy/dev/scylla/seastar/src/core/thread.cc:275 scylladb#24 0x7f8d9060322f (/local/home/bhalevy/dev/scylla/build/debug/dynamic_libs/libc.so.6+0x5222f) ``` Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
… token_metadata_ptr Fixes use-after-free seen with putget_with_reloaded_certificates_test: ``` ==215==ERROR: AddressSanitizer: heap-use-after-free on address 0x603000a8b180 at pc 0x000012eb5a83 bp 0x7ffd2c16d4c0 sp 0x7ffd2c16d4b0 READ of size 8 at 0x603000a8b180 thread T0 #0 0x12eb5a82 in std::__uniq_ptr_impl<locator::token_metadata_impl, std::default_delete<locator::token_metadata_impl> >::_M_ptr() const /usr/include/c++/10/bits/unique_ptr.h:173 scylladb#1 0x12ea230d in std::unique_ptr<locator::token_metadata_impl, std::default_delete<locator::token_metadata_impl> >::get() const /usr/include/c++/10/bits/unique_ptr.h:422 scylladb#2 0x12e8d3e8 in std::unique_ptr<locator::token_metadata_impl, std::default_delete<locator::token_metadata_impl> >::operator->() const /usr/include/c++/10/bits/unique_ptr.h:416 scylladb#3 0x12e5d0a2 in locator::token_metadata::ring_range(std::optional<interval_bound<dht::ring_position> > const&, bool) const locator/token_metadata.cc:1712 scylladb#4 0x112d0126 in service::query_ranges_to_vnodes_generator::process_one_range(unsigned long, std::vector<nonwrapping_interval<dht::ring_position>, std::allocator<nonwrapping_interval<dht::ring_position> > >&) service/storage_proxy.cc:4658 scylladb#5 0x112cf3c5 in service::query_ranges_to_vnodes_generator::operator()(unsigned long) service/storage_proxy.cc:4616 scylladb#6 0x112b2261 in service::storage_proxy::query_partition_key_range_concurrent(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >, std::vector<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >, std::allocator<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&, seastar::lw_shared_ptr<query::read_command>, db::consistency_level, service::query_ranges_to_vnodes_generator&&, int, tracing::trace_state_ptr, unsigned long, unsigned int, std::unordered_map<nonwrapping_interval<dht::token>, std::vector<utils::UUID, std::allocator<utils::UUID> >, std::hash<nonwrapping_interval<dht::token> >, std::equal_to<nonwrapping_interval<dht::token> >, std::allocator<std::pair<nonwrapping_interval<dht::token> const, std::vector<utils::UUID, std::allocator<utils::UUID> > > > >, service_permit) service/storage_proxy.cc:4023 scylladb#7 0x112b094e in operator() service/storage_proxy.cc:4160 scylladb#8 0x1139c8bb in invoke<service::storage_proxy::query_partition_key_range_concurrent(seastar::lowres_clock::time_point, std::vector<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >&&, seastar::lw_shared_ptr<query::read_command>, db::consistency_level, service::query_ranges_to_vnodes_generator&&, int, tracing::trace_state_ptr, uint64_t, uint32_t, service::replicas_per_token_range, service_permit)::<lambda(seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:2088 scylladb#9 0x1136625b in futurize_invoke<service::storage_proxy::query_partition_key_range_concurrent(seastar::lowres_clock::time_point, std::vector<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >&&, seastar::lw_shared_ptr<query::read_command>, db::consistency_level, service::query_ranges_to_vnodes_generator&&, int, tracing::trace_state_ptr, uint64_t, uint32_t, service::replicas_per_token_range, service_permit)::<lambda(seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:2119 scylladb#10 0x11366372 in operator()<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1480 scylladb#11 0x1139cc3b in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:145 scylladb#12 0x116f4944 in seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>::operator()(seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:201 scylladb#13 0x116b3397 in seastar::future<service::query_partition_key_range_concurrent_result> std::__invoke_impl<seastar::future<service::query_partition_key_range_concurrent_result>, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >(std::__invoke_other, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&) /usr/include/c++/10/bits/invoke.h:60 scylladb#14 0x1165c3a6 in std::__invoke_result<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::type std::__invoke<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&) /usr/include/c++/10/bits/invoke.h:96 scylladb#15 0x115e6542 in decltype(auto) std::__apply_impl<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >, 0ul>(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >&&, std::integer_sequence<unsigned long, 0ul>) /usr/include/c++/10/tuple:1724 scylladb#16 0x115e6663 in decltype(auto) std::apply<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >&&) /usr/include/c++/10/tuple:1736 scylladb#17 0x115e63f9 in seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}::operator()(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&) const::{lambda()scylladb#1}::operator()() const /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1530 scylladb#18 0x1165c4b9 in void seastar::futurize<seastar::future<service::query_partition_key_range_concurrent_result> >::satisfy_with_result_of<seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}::operator()(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&) const::{lambda()scylladb#1}>(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:2073 scylladb#19 0x115e61f5 in seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}::operator()(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1528 scylladb#20 0x1176e9cc in seastar::continuation<seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::run_and_dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:746 scylladb#21 0x16a9a455 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2196 scylladb#22 0x16a9e691 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2575 scylladb#23 0x16aa390e in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2730 scylladb#24 0x168ae4f7 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:207 scylladb#25 0x168ac541 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115 scylladb#26 0xd6cd3c4 in main /local/home/bhalevy/dev/scylla/main.cc:504 scylladb#27 0x7f8d905d8041 in __libc_start_main (/local/home/bhalevy/dev/scylla/build/debug/dynamic_libs/libc.so.6+0x27041) scylladb#28 0xd67c9ed in _start (/local/home/bhalevy/.dtest/dtest-o0qoqmkr/test/node3/bin/scylla+0xd67c9ed) 0x603000a8b180 is located 16 bytes inside of 24-byte region [0x603000a8b170,0x603000a8b188) freed by thread T0 here: #0 0x7f8d92a190cf in operator delete(void*, unsigned long) (/local/home/bhalevy/dev/scylla/build/debug/dynamic_libs/libasan.so.6+0xb30cf) scylladb#1 0xd7ebe54 in seastar::internal::lw_shared_ptr_accessors_no_esft<locator::token_metadata>::dispose(seastar::lw_shared_ptr_counter_base*) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:213 scylladb#2 0x112b155d in seastar::lw_shared_ptr<locator::token_metadata const>::~lw_shared_ptr() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:300 scylladb#3 0x112b155d in ~<lambda> service/storage_proxy.cc:4137 scylladb#4 0x1132e92d in ~<lambda> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1479 scylladb#5 0x1139cc91 in destroy /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:148 scylladb#6 0x11565673 in seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>::~noncopyable_function() /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:181 scylladb#7 0x1176e783 in seastar::continuation<seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::~continuation() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:729 scylladb#8 0x1176ea06 in seastar::continuation<seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::run_and_dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:750 scylladb#9 0x16a9a455 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2196 scylladb#10 0x16a9e691 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2575 scylladb#11 0x16aa390e in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2730 scylladb#12 0x168ae4f7 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:207 scylladb#13 0x168ac541 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115 scylladb#14 0xd6cd3c4 in main /local/home/bhalevy/dev/scylla/main.cc:504 scylladb#15 0x7f8d905d8041 in __libc_start_main (/local/home/bhalevy/dev/scylla/build/debug/dynamic_libs/libc.so.6+0x27041) previously allocated by thread T0 here: #0 0x7f8d92a18067 in operator new(unsigned long) (/local/home/bhalevy/dev/scylla/build/debug/dynamic_libs/libasan.so.6+0xb2067) scylladb#1 0x13cf7132 in seastar::lw_shared_ptr<locator::token_metadata> seastar::lw_shared_ptr<locator::token_metadata>::make<locator::token_metadata>(locator::token_metadata&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:266 scylladb#2 0x13cc3bfa in seastar::lw_shared_ptr<locator::token_metadata> seastar::make_lw_shared<locator::token_metadata>(locator::token_metadata&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:422 scylladb#3 0x13ca3007 in seastar::lw_shared_ptr<locator::token_metadata> locator::make_token_metadata_ptr<locator::token_metadata>(locator::token_metadata) locator/token_metadata.hh:338 scylladb#4 0x13c9bdd4 in locator::shared_token_metadata::clone() const locator/token_metadata.hh:358 scylladb#5 0x13c9c18a in service::storage_service::get_mutable_token_metadata_ptr() service/storage_service.hh:184 scylladb#6 0x13a5a445 in service::storage_service::handle_state_normal(gms::inet_address) service/storage_service.cc:1129 scylladb#7 0x13a6371c in service::storage_service::on_change(gms::inet_address, gms::application_state, gms::versioned_value const&) service/storage_service.cc:1421 scylladb#8 0x12a86269 in operator() gms/gossiper.cc:1639 scylladb#9 0x12ad3eea in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:145 scylladb#10 0x12be2aff in seastar::noncopyable_function<void (seastar::shared_ptr<gms::i_endpoint_state_change_subscriber>)>::operator()(seastar::shared_ptr<gms::i_endpoint_state_change_subscriber>) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:201 scylladb#11 0x12bb8e98 in atomic_vector<seastar::shared_ptr<gms::i_endpoint_state_change_subscriber> >::for_each(seastar::noncopyable_function<void (seastar::shared_ptr<gms::i_endpoint_state_change_subscriber>)>) utils/atomic_vector.hh:62 scylladb#12 0x12a8662b in gms::gossiper::do_on_change_notifications(gms::inet_address, gms::application_state const&, gms::versioned_value const&) gms/gossiper.cc:1638 scylladb#13 0x12a9387c in operator() gms/gossiper.cc:1978 scylladb#14 0x12b49b20 in __invoke_impl<void, gms::gossiper::add_local_application_state(std::__cxx11::list<std::pair<gms::application_state, gms::versioned_value> >)::<lambda(gms::gossiper&)> mutable::<lambda()> > /usr/include/c++/10/bits/invoke.h:60 scylladb#15 0x12b21fd6 in __invoke<gms::gossiper::add_local_application_state(std::__cxx11::list<std::pair<gms::application_state, gms::versioned_value> >)::<lambda(gms::gossiper&)> mutable::<lambda()> > /usr/include/c++/10/bits/invoke.h:95 scylladb#16 0x12b02865 in __apply_impl<gms::gossiper::add_local_application_state(std::__cxx11::list<std::pair<gms::application_state, gms::versioned_value> >)::<lambda(gms::gossiper&)> mutable::<lambda()>, std::tuple<> > /usr/include/c++/10/tuple:1723 scylladb#17 0x12b028d8 in apply<gms::gossiper::add_local_application_state(std::__cxx11::list<std::pair<gms::application_state, gms::versioned_value> >)::<lambda(gms::gossiper&)> mutable::<lambda()>, std::tuple<> > /usr/include/c++/10/tuple:1734 scylladb#18 0x12b02967 in apply<gms::gossiper::add_local_application_state(std::__cxx11::list<std::pair<gms::application_state, gms::versioned_value> >)::<lambda(gms::gossiper&)> mutable::<lambda()> > /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:2052 scylladb#19 0x12ad866a in operator() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/thread.hh:258 scylladb#20 0x12b609c2 in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:116 scylladb#21 0xdfabb5f in seastar::noncopyable_function<void ()>::operator()() const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:201 scylladb#22 0x16e21bb4 in seastar::thread_context::main() /local/home/bhalevy/dev/scylla/seastar/src/core/thread.cc:297 scylladb#23 0x16e2190f in seastar::thread_context::s_main(int, int) /local/home/bhalevy/dev/scylla/seastar/src/core/thread.cc:275 scylladb#24 0x7f8d9060322f (/local/home/bhalevy/dev/scylla/build/debug/dynamic_libs/libc.so.6+0x5222f) ``` Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
… token_metadata_ptr Fixes use-after-free seen with putget_with_reloaded_certificates_test: ``` ==215==ERROR: AddressSanitizer: heap-use-after-free on address 0x603000a8b180 at pc 0x000012eb5a83 bp 0x7ffd2c16d4c0 sp 0x7ffd2c16d4b0 READ of size 8 at 0x603000a8b180 thread T0 #0 0x12eb5a82 in std::__uniq_ptr_impl<locator::token_metadata_impl, std::default_delete<locator::token_metadata_impl> >::_M_ptr() const /usr/include/c++/10/bits/unique_ptr.h:173 scylladb#1 0x12ea230d in std::unique_ptr<locator::token_metadata_impl, std::default_delete<locator::token_metadata_impl> >::get() const /usr/include/c++/10/bits/unique_ptr.h:422 scylladb#2 0x12e8d3e8 in std::unique_ptr<locator::token_metadata_impl, std::default_delete<locator::token_metadata_impl> >::operator->() const /usr/include/c++/10/bits/unique_ptr.h:416 scylladb#3 0x12e5d0a2 in locator::token_metadata::ring_range(std::optional<interval_bound<dht::ring_position> > const&, bool) const locator/token_metadata.cc:1712 scylladb#4 0x112d0126 in service::query_ranges_to_vnodes_generator::process_one_range(unsigned long, std::vector<nonwrapping_interval<dht::ring_position>, std::allocator<nonwrapping_interval<dht::ring_position> > >&) service/storage_proxy.cc:4658 scylladb#5 0x112cf3c5 in service::query_ranges_to_vnodes_generator::operator()(unsigned long) service/storage_proxy.cc:4616 scylladb#6 0x112b2261 in service::storage_proxy::query_partition_key_range_concurrent(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >, std::vector<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >, std::allocator<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&, seastar::lw_shared_ptr<query::read_command>, db::consistency_level, service::query_ranges_to_vnodes_generator&&, int, tracing::trace_state_ptr, unsigned long, unsigned int, std::unordered_map<nonwrapping_interval<dht::token>, std::vector<utils::UUID, std::allocator<utils::UUID> >, std::hash<nonwrapping_interval<dht::token> >, std::equal_to<nonwrapping_interval<dht::token> >, std::allocator<std::pair<nonwrapping_interval<dht::token> const, std::vector<utils::UUID, std::allocator<utils::UUID> > > > >, service_permit) service/storage_proxy.cc:4023 scylladb#7 0x112b094e in operator() service/storage_proxy.cc:4160 scylladb#8 0x1139c8bb in invoke<service::storage_proxy::query_partition_key_range_concurrent(seastar::lowres_clock::time_point, std::vector<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >&&, seastar::lw_shared_ptr<query::read_command>, db::consistency_level, service::query_ranges_to_vnodes_generator&&, int, tracing::trace_state_ptr, uint64_t, uint32_t, service::replicas_per_token_range, service_permit)::<lambda(seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:2088 scylladb#9 0x1136625b in futurize_invoke<service::storage_proxy::query_partition_key_range_concurrent(seastar::lowres_clock::time_point, std::vector<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >&&, seastar::lw_shared_ptr<query::read_command>, db::consistency_level, service::query_ranges_to_vnodes_generator&&, int, tracing::trace_state_ptr, uint64_t, uint32_t, service::replicas_per_token_range, service_permit)::<lambda(seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:2119 scylladb#10 0x11366372 in operator()<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1480 scylladb#11 0x1139cc3b in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:145 scylladb#12 0x116f4944 in seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>::operator()(seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:201 scylladb#13 0x116b3397 in seastar::future<service::query_partition_key_range_concurrent_result> std::__invoke_impl<seastar::future<service::query_partition_key_range_concurrent_result>, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >(std::__invoke_other, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&) /usr/include/c++/10/bits/invoke.h:60 scylladb#14 0x1165c3a6 in std::__invoke_result<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::type std::__invoke<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&) /usr/include/c++/10/bits/invoke.h:96 scylladb#15 0x115e6542 in decltype(auto) std::__apply_impl<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >, 0ul>(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >&&, std::integer_sequence<unsigned long, 0ul>) /usr/include/c++/10/tuple:1724 scylladb#16 0x115e6663 in decltype(auto) std::apply<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >&&) /usr/include/c++/10/tuple:1736 scylladb#17 0x115e63f9 in seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}::operator()(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&) const::{lambda()scylladb#1}::operator()() const /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1530 scylladb#18 0x1165c4b9 in void seastar::futurize<seastar::future<service::query_partition_key_range_concurrent_result> >::satisfy_with_result_of<seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}::operator()(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&) const::{lambda()scylladb#1}>(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:2073 scylladb#19 0x115e61f5 in seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}::operator()(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1528 scylladb#20 0x1176e9cc in seastar::continuation<seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::run_and_dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:746 scylladb#21 0x16a9a455 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2196 scylladb#22 0x16a9e691 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2575 scylladb#23 0x16aa390e in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2730 scylladb#24 0x168ae4f7 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:207 scylladb#25 0x168ac541 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115 scylladb#26 0xd6cd3c4 in main /local/home/bhalevy/dev/scylla/main.cc:504 scylladb#27 0x7f8d905d8041 in __libc_start_main (/local/home/bhalevy/dev/scylla/build/debug/dynamic_libs/libc.so.6+0x27041) scylladb#28 0xd67c9ed in _start (/local/home/bhalevy/.dtest/dtest-o0qoqmkr/test/node3/bin/scylla+0xd67c9ed) 0x603000a8b180 is located 16 bytes inside of 24-byte region [0x603000a8b170,0x603000a8b188) freed by thread T0 here: #0 0x7f8d92a190cf in operator delete(void*, unsigned long) (/local/home/bhalevy/dev/scylla/build/debug/dynamic_libs/libasan.so.6+0xb30cf) scylladb#1 0xd7ebe54 in seastar::internal::lw_shared_ptr_accessors_no_esft<locator::token_metadata>::dispose(seastar::lw_shared_ptr_counter_base*) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:213 scylladb#2 0x112b155d in seastar::lw_shared_ptr<locator::token_metadata const>::~lw_shared_ptr() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:300 scylladb#3 0x112b155d in ~<lambda> service/storage_proxy.cc:4137 scylladb#4 0x1132e92d in ~<lambda> /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:1479 scylladb#5 0x1139cc91 in destroy /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:148 scylladb#6 0x11565673 in seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>::~noncopyable_function() /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:181 scylladb#7 0x1176e783 in seastar::continuation<seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::~continuation() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:729 scylladb#8 0x1176ea06 in seastar::continuation<seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::then_impl_nrvo<seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>, seastar::future<service::query_partition_key_range_concurrent_result> >(seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<service::query_partition_key_range_concurrent_result>&&, seastar::noncopyable_function<seastar::future<service::query_partition_key_range_concurrent_result> (seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> >&&)>&, seastar::future_state<std::tuple<seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > > >&&)scylladb#1}, seastar::foreign_ptr<seastar::lw_shared_ptr<query::result> > >::run_and_dispose() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:750 scylladb#9 0x16a9a455 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2196 scylladb#10 0x16a9e691 in seastar::reactor::run_some_tasks() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2575 scylladb#11 0x16aa390e in seastar::reactor::run() /local/home/bhalevy/dev/scylla/seastar/src/core/reactor.cc:2730 scylladb#12 0x168ae4f7 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:207 scylladb#13 0x168ac541 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /local/home/bhalevy/dev/scylla/seastar/src/core/app-template.cc:115 scylladb#14 0xd6cd3c4 in main /local/home/bhalevy/dev/scylla/main.cc:504 scylladb#15 0x7f8d905d8041 in __libc_start_main (/local/home/bhalevy/dev/scylla/build/debug/dynamic_libs/libc.so.6+0x27041) previously allocated by thread T0 here: #0 0x7f8d92a18067 in operator new(unsigned long) (/local/home/bhalevy/dev/scylla/build/debug/dynamic_libs/libasan.so.6+0xb2067) scylladb#1 0x13cf7132 in seastar::lw_shared_ptr<locator::token_metadata> seastar::lw_shared_ptr<locator::token_metadata>::make<locator::token_metadata>(locator::token_metadata&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:266 scylladb#2 0x13cc3bfa in seastar::lw_shared_ptr<locator::token_metadata> seastar::make_lw_shared<locator::token_metadata>(locator::token_metadata&&) /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/shared_ptr.hh:422 scylladb#3 0x13ca3007 in seastar::lw_shared_ptr<locator::token_metadata> locator::make_token_metadata_ptr<locator::token_metadata>(locator::token_metadata) locator/token_metadata.hh:338 scylladb#4 0x13c9bdd4 in locator::shared_token_metadata::clone() const locator/token_metadata.hh:358 scylladb#5 0x13c9c18a in service::storage_service::get_mutable_token_metadata_ptr() service/storage_service.hh:184 scylladb#6 0x13a5a445 in service::storage_service::handle_state_normal(gms::inet_address) service/storage_service.cc:1129 scylladb#7 0x13a6371c in service::storage_service::on_change(gms::inet_address, gms::application_state, gms::versioned_value const&) service/storage_service.cc:1421 scylladb#8 0x12a86269 in operator() gms/gossiper.cc:1639 scylladb#9 0x12ad3eea in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:145 scylladb#10 0x12be2aff in seastar::noncopyable_function<void (seastar::shared_ptr<gms::i_endpoint_state_change_subscriber>)>::operator()(seastar::shared_ptr<gms::i_endpoint_state_change_subscriber>) const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:201 scylladb#11 0x12bb8e98 in atomic_vector<seastar::shared_ptr<gms::i_endpoint_state_change_subscriber> >::for_each(seastar::noncopyable_function<void (seastar::shared_ptr<gms::i_endpoint_state_change_subscriber>)>) utils/atomic_vector.hh:62 scylladb#12 0x12a8662b in gms::gossiper::do_on_change_notifications(gms::inet_address, gms::application_state const&, gms::versioned_value const&) gms/gossiper.cc:1638 scylladb#13 0x12a9387c in operator() gms/gossiper.cc:1978 scylladb#14 0x12b49b20 in __invoke_impl<void, gms::gossiper::add_local_application_state(std::__cxx11::list<std::pair<gms::application_state, gms::versioned_value> >)::<lambda(gms::gossiper&)> mutable::<lambda()> > /usr/include/c++/10/bits/invoke.h:60 scylladb#15 0x12b21fd6 in __invoke<gms::gossiper::add_local_application_state(std::__cxx11::list<std::pair<gms::application_state, gms::versioned_value> >)::<lambda(gms::gossiper&)> mutable::<lambda()> > /usr/include/c++/10/bits/invoke.h:95 scylladb#16 0x12b02865 in __apply_impl<gms::gossiper::add_local_application_state(std::__cxx11::list<std::pair<gms::application_state, gms::versioned_value> >)::<lambda(gms::gossiper&)> mutable::<lambda()>, std::tuple<> > /usr/include/c++/10/tuple:1723 scylladb#17 0x12b028d8 in apply<gms::gossiper::add_local_application_state(std::__cxx11::list<std::pair<gms::application_state, gms::versioned_value> >)::<lambda(gms::gossiper&)> mutable::<lambda()>, std::tuple<> > /usr/include/c++/10/tuple:1734 scylladb#18 0x12b02967 in apply<gms::gossiper::add_local_application_state(std::__cxx11::list<std::pair<gms::application_state, gms::versioned_value> >)::<lambda(gms::gossiper&)> mutable::<lambda()> > /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/future.hh:2052 scylladb#19 0x12ad866a in operator() /local/home/bhalevy/dev/scylla/seastar/include/seastar/core/thread.hh:258 scylladb#20 0x12b609c2 in call /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:116 scylladb#21 0xdfabb5f in seastar::noncopyable_function<void ()>::operator()() const /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:201 scylladb#22 0x16e21bb4 in seastar::thread_context::main() /local/home/bhalevy/dev/scylla/seastar/src/core/thread.cc:297 scylladb#23 0x16e2190f in seastar::thread_context::s_main(int, int) /local/home/bhalevy/dev/scylla/seastar/src/core/thread.cc:275 scylladb#24 0x7f8d9060322f (/local/home/bhalevy/dev/scylla/build/debug/dynamic_libs/libc.so.6+0x5222f) ``` Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
I get the following when running multiple cassandra-stress with user profile (which means that we are doing create_table viq cql and not via thrift
on monster
sudo rm -Rf /data2/shlomi/tmp/* ; sudo gdb --args build/release/seastar --datadir /data2/shlomi/tmp --commitlog-directory /data2/shlomi/tmp --smp 6 --network-stack native --dpdk-pmd --dhcp 0 --host-ipv4-addr 192.168.10.101 --netmask-ipv4-addr 255.255.255.0 --seed-provider-parameters seeds=127.0.0.1
on godzilla
sleep 1
taskset -c 1,17 ~/cassandra/tools/bin/cassandra-stress user duration=2m profile=./urchin-qa-internal/simple_test_no_compression.yaml 'ops(insert=1)' -mode cql3 native -rate threads=16 -node 192.168.10.101 &
taskset -c 2,18 ~/cassandra/tools/bin/cassandra-stress user duration=2m profile=./urchin-qa-internal/simple_test_no_compression.yaml 'ops(insert=1)' -mode cql3 native -rate threads=16 -node 192.168.10.101 &
taskset -c 3,19 ~/cassandra/tools/bin/cassandra-stress user duration=2m profile=./urchin-qa-internal/simple_test_no_compression.yaml 'ops(insert=1)' -mode cql3 native -rate threads=16 -node 192.168.10.101 &
taskset -c 4,20 ~/cassandra/tools/bin/cassandra-stress user duration=2m profile=./urchin-qa-internal/simple_test_no_compression.yaml 'ops(insert=1)' -mode cql3 native -rate threads=16 -node 192.168.10.101 &
taskset -c 5,21 ~/cassandra/tools/bin/cassandra-stress user duration=2m profile=./urchin-qa-internal/simple_test_no_compression.yaml 'ops(insert=1)' -mode cql3 native -rate threads=16 -node 192.168.10.101 &
taskset -c 6,22 ~/cassandra/tools/bin/cassandra-stress user duration=2m profile=./urchin-qa-internal/simple_test_no_compression.yaml 'ops(insert=1)' -mode cql3 native -rate threads=16 -node 192.168.10.101 &
taskset -c 7,23 ~/cassandra/tools/bin/cassandra-stress user duration=2m profile=./urchin-qa-internal/simple_test_no_compression.yaml 'ops(insert=1)' -mode cql3 native -rate threads=16 -node 192.168.10.101 &
taskset -c 8,24 ~/cassandra/tools/bin/cassandra-stress user duration=2m profile=./urchin-qa-internal/simple_test_no_compression.yaml 'ops(insert=1)' -mode cql3 native -rate threads=16 -node 192.168.10.101 &
taskset -c 9,25 ~/cassandra/tools/bin/cassandra-stress user duration=2m profile=./urchin-qa-internal/simple_test_no_compression.yaml 'ops(insert=1)' -mode cql3 native -rate threads=16 -node 192.168.10.101 &
taskset -c 10,26 ~/cassandra/tools/bin/cassandra-stress user duration=2m profile=./urchin-qa-internal/simple_test_no_compression.yaml 'ops(insert=1)' -mode cql3 native -rate threads=16 -node 192.168.10.101 &
taskset -c 11,27 ~/cassandra/tools/bin/cassandra-stress user duration=2m profile=./urchin-qa-internal/simple_test_no_compression.yaml 'ops(insert=1)' -mode cql3 native -rate threads=16 -node 192.168.10.101 &
taskset -c 12,28 ~/cassandra/tools/bin/cassandra-stress user duration=2m profile=./urchin-qa-internal/simple_test_no_compression.yaml 'ops(insert=1)' -mode cql3 native -rate threads=16 -node 192.168.10.101 &
taskset -c 13,29 ~/cassandra/tools/bin/cassandra-stress user duration=2m profile=./urchin-qa-internal/simple_test_no_compression.yaml 'ops(insert=1)' -mode cql3 native -rate threads=16 -node 192.168.10.101 &
before the SIGSEGV there are some
WARNING: exceptional future ignored of type 'query::no_such_column': keyspace_name
WARNING: exceptional future ignored of type 'query::no_such_column': keyspace_name
WARNING: exceptional future ignored of type 'query::no_such_column': keyspace_name
which I'll try to catch and open a new bug for them
The text was updated successfully, but these errors were encountered: