Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

use after free in time_window_backlog_tracker::backlog #4465

Closed
bhalevy opened this issue Apr 29, 2019 · 6 comments

Comments

Projects
None yet
3 participants
@bhalevy
Copy link
Contributor

commented Apr 29, 2019

scylla version @ee9f3388f6b5e29e682bf78c085b8495ca839602

Seen in byo_build_tests_dtest/273/artifact/logs-debug.2/1556466252802_compaction_test.TestCompaction_with_TimeWindowCompactionStrategy.sstable_deletion_test/node1.log:

==8010==ERROR: AddressSanitizer: heap-use-after-free on address 0x60700037df00 at pc 0x0000032bcaaf bp 0x7ffc9a1600c0 sp 0x7ffc9a1600b0
READ of size 8 at 0x60700037df00 thread T0
    #0 0x32bcaae  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0x32bcaae)
    #1 0x32a8d02  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0x32a8d02)
    #2 0x3286f23  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0x3286f23)
    #3 0x32642f0  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0x32642f0)
    #4 0x32650c7  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0x32650c7)
    #5 0x3274b95  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0x3274b95)
    #6 0x3395cdb  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0x3395cdb)
    #7 0x3398aa2  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0x3398aa2)
    #8 0x33f763e  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0x33f763e)
    #9 0x3366503  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0x3366503)
    #10 0x33b6fe3  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0x33b6fe3)
    #11 0xdddf6a  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0xdddf6a)
    #12 0xbc3332  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0xbc3332)
    #13 0xd7cee1  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0xd7cee1)
    #14 0xe1c4a1  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0xe1c4a1)
    #15 0x1b3484d  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0x1b3484d)
    #16 0xb7c90eb  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0xb7c90eb)
    #17 0xb7593b0  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0xb7593b0)
    #18 0xb6c9945  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0xb6c9945)
    #19 0xb7d9c66  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0xb7d9c66)
    #20 0x34166dc  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0x34166dc)
    #21 0xb6cb4d5  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0xb6cb4d5)
    #22 0xb9471af  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0xb9471af)
    #23 0xb762fe9  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0xb762fe9)
    #24 0xb75c37b  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0xb75c37b)
    #25 0xb80c5bd  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0xb80c5bd)
    #26 0x1db5112  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0x1db5112)
    #27 0xb760903  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0xb760903)
    #28 0xb4d8bf9  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0xb4d8bf9)
    #29 0xb4d328b  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0xb4d328b)
    #30 0x4eae3f  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0x4eae3f)
    #31 0x7f005bbe1412 in __libc_start_main (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla-dtest/../scylla/dynamic_libs/libc.so.6+0x24412)
    #32 0x424acd  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0x424acd)

std::_Hashtable<seastar::lw_shared_ptr<sstables::sstable>, std::pair<seastar::lw_shared_ptr<sstables::sstable> const, backlog_write_progress_manager*>, std::allocator<std::pair<seastar::lw_shared_ptr<sstables::sstable> const, backlog_write_progress_manager*> >, std::__detail::_Select1st, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::_M_begin() const at /usr/include/c++/8/bits/hashtable.h:389
std::_Hashtable<seastar::lw_shared_ptr<sstables::sstable>, std::pair<seastar::lw_shared_ptr<sstables::sstable> const, backlog_write_progress_manager*>, std::allocator<std::pair<seastar::lw_shared_ptr<sstables::sstable> const, backlog_write_progress_manager*> >, std::__detail::_Select1st, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::begin() const at /usr/include/c++/8/bits/hashtable.h:508
std::unordered_map<seastar::lw_shared_ptr<sstables::sstable>, backlog_write_progress_manager*, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<std::pair<seastar::lw_shared_ptr<sstables::sstable> const, backlog_write_progress_manager*> > >::begin() const at /usr/include/c++/8/bits/unordered_map.h:334
size_tiered_backlog_tracker::partial_backlog(std::unordered_map<seastar::lw_shared_ptr<sstables::sstable>, backlog_write_progress_manager*, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<std::pair<seastar::lw_shared_ptr<sstables::sstable> const, backlog_write_progress_manager*> > > const&) const at /local/home/bhalevy/dev/scylla/./sstables/size_tiered_backlog_tracker.hh:89
size_tiered_backlog_tracker::backlog(std::unordered_map<seastar::lw_shared_ptr<sstables::sstable>, backlog_write_progress_manager*, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<std::pair<seastar::lw_shared_ptr<sstables::sstable> const, backlog_write_progress_manager*> > > const&, std::unordered_map<seastar::lw_shared_ptr<sstables::sstable>, backlog_read_progress_manager*, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<std::pair<seastar::lw_shared_ptr<sstables::sstable> const, backlog_read_progress_manager*> > > const&) const at /local/home/bhalevy/dev/scylla/./sstables/size_tiered_backlog_tracker.hh:114
sstables::time_window_backlog_tracker::backlog(std::unordered_map<seastar::lw_shared_ptr<sstables::sstable>, backlog_write_progress_manager*, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<std::pair<seastar::lw_shared_ptr<sstables::sstable> const, backlog_write_progress_manager*> > > const&, std::unordered_map<seastar::lw_shared_ptr<sstables::sstable>, backlog_read_progress_manager*, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<std::pair<seastar::lw_shared_ptr<sstables::sstable> const, backlog_read_progress_manager*> > > const&) const at /local/home/bhalevy/dev/scylla/sstables/compaction_strategy.cc:493
compaction_backlog_tracker::backlog() const at /local/home/bhalevy/dev/scylla/sstables/compaction_manager.cc:740 (discriminator 2)
compaction_backlog_manager::backlog() const at /local/home/bhalevy/dev/scylla/sstables/compaction_manager.cc:831 (discriminator 2)
compaction_manager::backlog() at /local/home/bhalevy/dev/scylla/sstables/compaction_manager.hh:225
compaction_manager::compaction_manager(seastar::scheduling_group, seastar::io_priority_class const&, unsigned long)::{lambda()#1}::operator()() const at /local/home/bhalevy/dev/scylla/sstables/compaction_manager.cc:343
std::_Function_handler<float (), compaction_manager::compaction_manager(seastar::scheduling_group, seastar::io_priority_class const&, unsigned long)::{lambda()#1}>::_M_invoke(std::_Any_data const&) at /usr/include/c++/8/bits/std_function.h:282
std::function<float ()>::operator()() const at /usr/include/c++/8/bits/std_function.h:687
backlog_controller::adjust() at /local/home/bhalevy/dev/scylla/database.cc:240
backlog_controller::backlog_controller(seastar::scheduling_group, seastar::io_priority_class const&, std::chrono::duration<long, std::ratio<1l, 1000l> >, std::vector<backlog_controller::control_point, std::allocator<backlog_controller::control_point> >, std::function<float ()>)::{lambda()#1}::operator()() const at /local/home/bhalevy/dev/scylla/backlog_controller.hh:79
seastar::noncopyable_function<void ()>::direct_vtable_for<backlog_controller::backlog_controller(seastar::scheduling_group, seastar::io_priority_class const&, std::chrono::duration<long, std::ratio<1l, 1000l> >, std::vector<backlog_controller::control_point, std::allocator<backlog_controller::control_point> >, std::function<float ()>)::{lambda()#1}>::call(seastar::noncopyable_function<void ()> const*) at /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:71
seastar::noncopyable_function<void ()>::operator()() const at /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:145
void seastar::reactor::complete_timers<seastar::timer_set<seastar::timer<std::chrono::_V2::steady_clock>, &seastar::timer<std::chrono::_V2::steady_clock>::_link>, boost::intrusive::list<seastar::timer<std::chrono::_V2::steady_clock>, boost::intrusive::member_hook<seastar::timer<std::chrono::_V2::steady_clock>, boost::intrusive::list_member_hook<void, void, void>, &seastar::timer<std::chrono::_V2::steady_clock>::_link>, void, void, void>, seastar::reactor::service_highres_timer()::{lambda()#1}>(seastar::timer_set<seastar::timer<std::chrono::_V2::steady_clock>, &seastar::timer<std::chrono::_V2::steady_clock>::_link>&, boost::intrusive::list<seastar::timer<std::chrono::_V2::steady_clock>, boost::intrusive::member_hook<seastar::timer<std::chrono::_V2::steady_clock>, boost::intrusive::list_member_hook<void, void, void>, &seastar::timer<std::chrono::_V2::steady_clock>::_link>, void, void, void>&, seastar::reactor::service_highres_timer()::{lambda()#1}&&) at /local/home/bhalevy/dev/scylla/seastar/build/debug/../../src/core/reactor.cc:1620
seastar::reactor::service_highres_timer()::{lambda()#1}::operator()() const at /local/home/bhalevy/dev/scylla/seastar/build/debug/../../src/core/reactor.cc:4136
seastar::reactor_backend_epoll::stop_tick() at /local/home/bhalevy/dev/scylla/seastar/build/debug/../../src/core/reactor.cc:948
_ZN7seastar20noncopyable_functionIFNS_6futureIJmEEEvEEC1IZNS1_IJEE4thenIZNS_11pollable_fd6sendtoENS_14socket_addressEPKvmEUlvE_S2_EET0_OT_EUlvE_EESE_ at /local/home/bhalevy/dev/scylla/seastar/build/debug/../../include/seastar/util/noncopyable_function.hh:117
std::function<void ()>::operator()() const at /usr/include/c++/8/bits/std_function.h:687
seastar::reactor::signals::poll_signal() at /usr/include/c++/8/bits/atomic_base.h:396
 (inlined by) seastar::reactor::signals::poll_signal() at /local/home/bhalevy/dev/scylla/seastar/build/debug/../../src/core/reactor.cc:1006
seastar::reactor::io_pollfn::try_enter_interrupt_mode() at /local/home/bhalevy/dev/scylla/seastar/build/debug/../../src/core/reactor.cc:3770
seastar::reactor::poll_once() at /local/home/bhalevy/dev/scylla/seastar/build/debug/../../src/core/reactor.cc:4335
seastar::reactor::run()::{lambda()#5}::operator()() at /local/home/bhalevy/dev/scylla/seastar/build/debug/../../src/core/reactor.cc:4203
seastar::noncopyable_function<void ()>::direct_vtable_for<seastar::reactor::run()::{lambda()#5}>::move(seastar::noncopyable_function<void ()>*, seastar::noncopyable_function<void ()>::direct_vtable_for<seastar::reactor::run()::{lambda()#5}>) at /local/home/bhalevy/dev/scylla/seastar/build/debug/../../include/seastar/util/noncopyable_function.hh:74
std::function<bool ()>::operator()() const at /usr/include/c++/8/bits/std_function.h:687
seastar::reactor::run() at /local/home/bhalevy/dev/scylla/seastar/build/debug/../../src/core/reactor.cc:4245
seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) at /local/home/bhalevy/dev/scylla/seastar/build/debug/../../include/seastar/core/future.hh:804
 (inlined by) seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) at /local/home/bhalevy/dev/scylla/seastar/build/debug/../../src/core/app-template.cc:179
seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&)::{lambda()#1}::operator()() at /local/home/bhalevy/dev/scylla/seastar/build/debug/../../src/core/app-template.cc:115
main at /local/home/bhalevy/dev/scylla/main.cc:377 (discriminator 1)
?? ??:0
_start at ??:?


0x60700037df00 is located 32 bytes inside of 72-byte region [0x60700037dee0,0x60700037df28)
freed by thread T0 here:
    #0 0x7f005decb708 in operator delete(void*) (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla-dtest/../scylla/dynamic_libs/libasan.so.5+0xf2708)
    #1 0x331db27  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0x331db27)
    #2 0x3301847  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0x3301847)
    #3 0x3300d63  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0x3300d63)
    #4 0x3305887  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0x3305887)
    #5 0x32dfa71  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0x32dfa71)
    #6 0x32b8228  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0x32b8228)
    #7 0x3292e1e  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0x3292e1e)
    #8 0x32748fa  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0x32748fa)
    #9 0x3395cdb  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0x3395cdb)
    #10 0x3398aa2  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0x3398aa2)
    #11 0x33f763e  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0x33f763e)
    #12 0x3366503  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0x3366503)
    #13 0x33b6fe3  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0x33b6fe3)
    #14 0xdddf6a  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0xdddf6a)
    #15 0xbc3332  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0xbc3332)
    #16 0xd7cee1  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0xd7cee1)
    #17 0xe1c4a1  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0xe1c4a1)
    #18 0x1b3484d  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0x1b3484d)
    #19 0xb7c90eb  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0xb7c90eb)
    #20 0xb7593b0  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0xb7593b0)
    #21 0xb6c9945  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0xb6c9945)
    #22 0xb7d9c66  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0xb7d9c66)
    #23 0x34166dc  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0x34166dc)
    #24 0xb6cb4d5  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0xb6cb4d5)
    #25 0xb9471af  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0xb9471af)
    #26 0xb762fe9  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0xb762fe9)
    #27 0xb75c37b  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0xb75c37b)
    #28 0xb80c5bd  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0xb80c5bd)
    #29 0x1db5112  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0x1db5112)

__gnu_cxx::new_allocator<std::__detail::_Hash_node<std::pair<long const, std::unordered_map<seastar::lw_shared_ptr<sstables::sstable>, backlog_write_progress_manager*, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<std::pair<seastar::lw_shared_ptr<sstables::sstable> const, backlog_write_progress_manager*> > > >, false> >::deallocate(std::__detail::_Hash_node<std::pair<long const, std::unordered_map<seastar::lw_shared_ptr<sstables::sstable>, backlog_write_progress_manager*, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<std::pair<seastar::lw_shared_ptr<sstables::sstable> const, backlog_write_progress_manager*> > > >, false>*, unsigned long) at /usr/include/c++/8/ext/new_allocator.h:125
std::allocator_traits<std::allocator<std::__detail::_Hash_node<std::pair<long const, std::unordered_map<seastar::lw_shared_ptr<sstables::sstable>, backlog_write_progress_manager*, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<std::pair<seastar::lw_shared_ptr<sstables::sstable> const, backlog_write_progress_manager*> > > >, false> > >::deallocate(std::allocator<std::__detail::_Hash_node<std::pair<long const, std::unordered_map<seastar::lw_shared_ptr<sstables::sstable>, backlog_write_progress_manager*, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<std::pair<seastar::lw_shared_ptr<sstables::sstable> const, backlog_write_progress_manager*> > > >, false> >&, std::__detail::_Hash_node<std::pair<long const, std::unordered_map<seastar::lw_shared_ptr<sstables::sstable>, backlog_write_progress_manager*, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<std::pair<seastar::lw_shared_ptr<sstables::sstable> const, backlog_write_progress_manager*> > > >, false>*, unsigned long) at /usr/include/c++/8/bits/alloc_traits.h:462
std::__detail::_Hashtable_alloc<std::allocator<std::__detail::_Hash_node<std::pair<long const, std::unordered_map<seastar::lw_shared_ptr<sstables::sstable>, backlog_write_progress_manager*, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<std::pair<seastar::lw_shared_ptr<sstables::sstable> const, backlog_write_progress_manager*> > > >, false> > >::_M_deallocate_node(std::__detail::_Hash_node<std::pair<long const, std::unordered_map<seastar::lw_shared_ptr<sstables::sstable>, backlog_write_progress_manager*, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<std::pair<seastar::lw_shared_ptr<sstables::sstable> const, backlog_write_progress_manager*> > > >, false>*) at /usr/include/c++/8/bits/hashtable_policy.h:2102
std::_Hashtable<long, std::pair<long const, std::unordered_map<seastar::lw_shared_ptr<sstables::sstable>, backlog_write_progress_manager*, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<std::pair<seastar::lw_shared_ptr<sstables::sstable> const, backlog_write_progress_manager*> > > >, std::allocator<std::pair<long const, std::unordered_map<seastar::lw_shared_ptr<sstables::sstable>, backlog_write_progress_manager*, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<std::pair<seastar::lw_shared_ptr<sstables::sstable> const, backlog_write_progress_manager*> > > > >, std::__detail::_Select1st, std::equal_to<long>, std::hash<long>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::_M_erase(unsigned long, std::__detail::_Hash_node_base*, std::__detail::_Hash_node<std::pair<long const, std::unordered_map<seastar::lw_shared_ptr<sstables::sstable>, backlog_write_progress_manager*, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<std::pair<seastar::lw_shared_ptr<sstables::sstable> const, backlog_write_progress_manager*> > > >, false>*) at /usr/include/c++/8/bits/hashtable.h:1908
std::_Hashtable<long, std::pair<long const, std::unordered_map<seastar::lw_shared_ptr<sstables::sstable>, backlog_write_progress_manager*, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<std::pair<seastar::lw_shared_ptr<sstables::sstable> const, backlog_write_progress_manager*> > > >, std::allocator<std::pair<long const, std::unordered_map<seastar::lw_shared_ptr<sstables::sstable>, backlog_write_progress_manager*, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<std::pair<seastar::lw_shared_ptr<sstables::sstable> const, backlog_write_progress_manager*> > > > >, std::__detail::_Select1st, std::equal_to<long>, std::hash<long>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::erase(std::__detail::_Node_const_iterator<std::pair<long const, std::unordered_map<seastar::lw_shared_ptr<sstables::sstable>, backlog_write_progress_manager*, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<std::pair<seastar::lw_shared_ptr<sstables::sstable> const, backlog_write_progress_manager*> > > >, false, false>) at /usr/include/c++/8/bits/hashtable.h:1883
std::_Hashtable<long, std::pair<long const, std::unordered_map<seastar::lw_shared_ptr<sstables::sstable>, backlog_write_progress_manager*, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<std::pair<seastar::lw_shared_ptr<sstables::sstable> const, backlog_write_progress_manager*> > > >, std::allocator<std::pair<long const, std::unordered_map<seastar::lw_shared_ptr<sstables::sstable>, backlog_write_progress_manager*, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<std::pair<seastar::lw_shared_ptr<sstables::sstable> const, backlog_write_progress_manager*> > > > >, std::__detail::_Select1st, std::equal_to<long>, std::hash<long>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::erase(std::__detail::_Node_iterator<std::pair<long const, std::unordered_map<seastar::lw_shared_ptr<sstables::sstable>, backlog_write_progress_manager*, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<std::pair<seastar::lw_shared_ptr<sstables::sstable> const, backlog_write_progress_manager*> > > >, false, false>) at /usr/include/c++/8/bits/hashtable.h:767
std::unordered_map<long, std::unordered_map<seastar::lw_shared_ptr<sstables::sstable>, backlog_write_progress_manager*, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<std::pair<seastar::lw_shared_ptr<sstables::sstable> const, backlog_write_progress_manager*> > >, std::hash<long>, std::equal_to<long>, std::allocator<std::pair<long const, std::unordered_map<seastar::lw_shared_ptr<sstables::sstable>, backlog_write_progress_manager*, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<std::pair<seastar::lw_shared_ptr<sstables::sstable> const, backlog_write_progress_manager*> > > > > >::erase(std::__detail::_Node_iterator<std::pair<long const, std::unordered_map<seastar::lw_shared_ptr<sstables::sstable>, backlog_write_progress_manager*, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<std::pair<seastar::lw_shared_ptr<sstables::sstable> const, backlog_write_progress_manager*> > > >, false, false>) at /usr/include/c++/8/bits/unordered_map.h:798
sstables::time_window_backlog_tracker::backlog(std::unordered_map<seastar::lw_shared_ptr<sstables::sstable>, backlog_write_progress_manager*, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<std::pair<seastar::lw_shared_ptr<sstables::sstable> const, backlog_write_progress_manager*> > > const&, std::unordered_map<seastar::lw_shared_ptr<sstables::sstable>, backlog_read_progress_manager*, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<std::pair<seastar::lw_shared_ptr<sstables::sstable> const, backlog_read_progress_manager*> > > const&) const at /local/home/bhalevy/dev/scylla/sstables/compaction_strategy.cc:486
compaction_backlog_tracker::backlog() const at /local/home/bhalevy/dev/scylla/sstables/compaction_manager.cc:740 (discriminator 2)
compaction_backlog_manager::backlog() const at /local/home/bhalevy/dev/scylla/sstables/compaction_manager.cc:831 (discriminator 2)
compaction_manager::backlog() at /local/home/bhalevy/dev/scylla/sstables/compaction_manager.hh:225
compaction_manager::compaction_manager(seastar::scheduling_group, seastar::io_priority_class const&, unsigned long)::{lambda()#1}::operator()() const at /local/home/bhalevy/dev/scylla/sstables/compaction_manager.cc:343
std::_Function_handler<float (), compaction_manager::compaction_manager(seastar::scheduling_group, seastar::io_priority_class const&, unsigned long)::{lambda()#1}>::_M_invoke(std::_Any_data const&) at /usr/include/c++/8/bits/std_function.h:282
std::function<float ()>::operator()() const at /usr/include/c++/8/bits/std_function.h:687
backlog_controller::adjust() at /local/home/bhalevy/dev/scylla/database.cc:240
backlog_controller::backlog_controller(seastar::scheduling_group, seastar::io_priority_class const&, std::chrono::duration<long, std::ratio<1l, 1000l> >, std::vector<backlog_controller::control_point, std::allocator<backlog_controller::control_point> >, std::function<float ()>)::{lambda()#1}::operator()() const at /local/home/bhalevy/dev/scylla/backlog_controller.hh:79
seastar::noncopyable_function<void ()>::direct_vtable_for<backlog_controller::backlog_controller(seastar::scheduling_group, seastar::io_priority_class const&, std::chrono::duration<long, std::ratio<1l, 1000l> >, std::vector<backlog_controller::control_point, std::allocator<backlog_controller::control_point> >, std::function<float ()>)::{lambda()#1}>::call(seastar::noncopyable_function<void ()> const*) at /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:71
seastar::noncopyable_function<void ()>::operator()() const at /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:145
void seastar::reactor::complete_timers<seastar::timer_set<seastar::timer<std::chrono::_V2::steady_clock>, &seastar::timer<std::chrono::_V2::steady_clock>::_link>, boost::intrusive::list<seastar::timer<std::chrono::_V2::steady_clock>, boost::intrusive::member_hook<seastar::timer<std::chrono::_V2::steady_clock>, boost::intrusive::list_member_hook<void, void, void>, &seastar::timer<std::chrono::_V2::steady_clock>::_link>, void, void, void>, seastar::reactor::service_highres_timer()::{lambda()#1}>(seastar::timer_set<seastar::timer<std::chrono::_V2::steady_clock>, &seastar::timer<std::chrono::_V2::steady_clock>::_link>&, boost::intrusive::list<seastar::timer<std::chrono::_V2::steady_clock>, boost::intrusive::member_hook<seastar::timer<std::chrono::_V2::steady_clock>, boost::intrusive::list_member_hook<void, void, void>, &seastar::timer<std::chrono::_V2::steady_clock>::_link>, void, void, void>&, seastar::reactor::service_highres_timer()::{lambda()#1}&&) at /local/home/bhalevy/dev/scylla/seastar/build/debug/../../src/core/reactor.cc:1620
seastar::reactor::service_highres_timer()::{lambda()#1}::operator()() const at /local/home/bhalevy/dev/scylla/seastar/build/debug/../../src/core/reactor.cc:4136
seastar::reactor_backend_epoll::stop_tick() at /local/home/bhalevy/dev/scylla/seastar/build/debug/../../src/core/reactor.cc:948
_ZN7seastar20noncopyable_functionIFNS_6futureIJmEEEvEEC1IZNS1_IJEE4thenIZNS_11pollable_fd6sendtoENS_14socket_addressEPKvmEUlvE_S2_EET0_OT_EUlvE_EESE_ at /local/home/bhalevy/dev/scylla/seastar/build/debug/../../include/seastar/util/noncopyable_function.hh:117
std::function<void ()>::operator()() const at /usr/include/c++/8/bits/std_function.h:687
seastar::reactor::signals::poll_signal() at /usr/include/c++/8/bits/atomic_base.h:396
 (inlined by) seastar::reactor::signals::poll_signal() at /local/home/bhalevy/dev/scylla/seastar/build/debug/../../src/core/reactor.cc:1006
seastar::reactor::io_pollfn::try_enter_interrupt_mode() at /local/home/bhalevy/dev/scylla/seastar/build/debug/../../src/core/reactor.cc:3770
seastar::reactor::poll_once() at /local/home/bhalevy/dev/scylla/seastar/build/debug/../../src/core/reactor.cc:4335
seastar::reactor::run()::{lambda()#5}::operator()() at /local/home/bhalevy/dev/scylla/seastar/build/debug/../../src/core/reactor.cc:4203
seastar::noncopyable_function<void ()>::direct_vtable_for<seastar::reactor::run()::{lambda()#5}>::move(seastar::noncopyable_function<void ()>*, seastar::noncopyable_function<void ()>::direct_vtable_for<seastar::reactor::run()::{lambda()#5}>) at /local/home/bhalevy/dev/scylla/seastar/build/debug/../../include/seastar/util/noncopyable_function.hh:74
std::function<bool ()>::operator()() const at /usr/include/c++/8/bits/std_function.h:687

previously allocated by thread T0 here:
    #0 0x7f005deca830 in operator new(unsigned long) (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla-dtest/../scylla/dynamic_libs/libasan.so.5+0xf1830)
    #1 0x331d83c  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0x331d83c)
    #2 0x3301733  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0x3301733)
    #3 0x32dd1be  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0x32dd1be)
    #4 0x32b6dce  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0x32b6dce)
    #5 0x32925cf  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0x32925cf)
    #6 0x3273e60  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0x3273e60)
    #7 0x3395cdb  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0x3395cdb)
    #8 0x3398aa2  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0x3398aa2)
    #9 0x33f763e  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0x33f763e)
    #10 0x3366503  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0x3366503)
    #11 0x33b6fe3  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0x33b6fe3)
    #12 0xdddf6a  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0xdddf6a)
    #13 0xbc3332  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0xbc3332)
    #14 0xd7cee1  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0xd7cee1)
    #15 0xe1c4a1  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0xe1c4a1)
    #16 0x1b3484d  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0x1b3484d)
    #17 0xb7c90eb  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0xb7c90eb)
    #18 0xb7593b0  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0xb7593b0)
    #19 0xb6c9945  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0xb6c9945)
    #20 0xb7d9c66  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0xb7d9c66)
    #21 0x34166dc  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0x34166dc)
    #22 0xb6cb4d5  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0xb6cb4d5)
    #23 0xb9471af  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0xb9471af)
    #24 0xb762fe9  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0xb762fe9)
    #25 0xb75c37b  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0xb75c37b)
    #26 0xb80c5bd  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0xb80c5bd)
    #27 0x1db5112  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0x1db5112)
    #28 0xb760903  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0xb760903)
    #29 0xb4d8bf9  (/jenkins/slave/workspace/scylla-master/manual-and-scheduled-tests/byo_build_tests_dtest/scylla/dynamic_libs/ld-linux-x86-64.so.2+0xb4d8bf9)


__gnu_cxx::new_allocator<std::__detail::_Hash_node<std::pair<long const, std::unordered_map<seastar::lw_shared_ptr<sstables::sstable>, backlog_write_progress_manager*, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<std::pair<seastar::lw_shared_ptr<sstables::sstable> const, backlog_write_progress_manager*> > > >, false> >::allocate(unsigned long, void const*) at /usr/include/c++/8/ext/new_allocator.h:111
std::allocator_traits<std::allocator<std::__detail::_Hash_node<std::pair<long const, std::unordered_map<seastar::lw_shared_ptr<sstables::sstable>, backlog_write_progress_manager*, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<std::pair<seastar::lw_shared_ptr<sstables::sstable> const, backlog_write_progress_manager*> > > >, false> > >::allocate(std::allocator<std::__detail::_Hash_node<std::pair<long const, std::unordered_map<seastar::lw_shared_ptr<sstables::sstable>, backlog_write_progress_manager*, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<std::pair<seastar::lw_shared_ptr<sstables::sstable> const, backlog_write_progress_manager*> > > >, false> >&, unsigned long) at /usr/include/c++/8/bits/alloc_traits.h:436
std::__detail::_Hash_node<std::pair<long const, std::unordered_map<seastar::lw_shared_ptr<sstables::sstable>, backlog_write_progress_manager*, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<std::pair<seastar::lw_shared_ptr<sstables::sstable> const, backlog_write_progress_manager*> > > >, false>* std::__detail::_Hashtable_alloc<std::allocator<std::__detail::_Hash_node<std::pair<long const, std::unordered_map<seastar::lw_shared_ptr<sstables::sstable>, backlog_write_progress_manager*, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<std::pair<seastar::lw_shared_ptr<sstables::sstable> const, backlog_write_progress_manager*> > > >, false> > >::_M_allocate_node<std::piecewise_construct_t const&, std::tuple<long const&>, std::tuple<> >(std::piecewise_construct_t const&, std::tuple<long const&>&&, std::tuple<>&&) at /usr/include/c++/8/bits/hashtable_policy.h:2077
std::__detail::_Map_base<long, std::pair<long const, std::unordered_map<seastar::lw_shared_ptr<sstables::sstable>, backlog_write_progress_manager*, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<std::pair<seastar::lw_shared_ptr<sstables::sstable> const, backlog_write_progress_manager*> > > >, std::allocator<std::pair<long const, std::unordered_map<seastar::lw_shared_ptr<sstables::sstable>, backlog_write_progress_manager*, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<std::pair<seastar::lw_shared_ptr<sstables::sstable> const, backlog_write_progress_manager*> > > > >, std::__detail::_Select1st, std::equal_to<long>, std::hash<long>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true>, true>::operator[](long const&) at /usr/include/c++/8/bits/hashtable_policy.h:711 (discriminator 2)
std::unordered_map<long, std::unordered_map<seastar::lw_shared_ptr<sstables::sstable>, backlog_write_progress_manager*, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<std::pair<seastar::lw_shared_ptr<sstables::sstable> const, backlog_write_progress_manager*> > >, std::hash<long>, std::equal_to<long>, std::allocator<std::pair<long const, std::unordered_map<seastar::lw_shared_ptr<sstables::sstable>, backlog_write_progress_manager*, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<std::pair<seastar::lw_shared_ptr<sstables::sstable> const, backlog_write_progress_manager*> > > > > >::operator[](long const&) at /usr/include/c++/8/bits/unordered_map.h:974
sstables::time_window_backlog_tracker::backlog(std::unordered_map<seastar::lw_shared_ptr<sstables::sstable>, backlog_write_progress_manager*, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<std::pair<seastar::lw_shared_ptr<sstables::sstable> const, backlog_write_progress_manager*> > > const&, std::unordered_map<seastar::lw_shared_ptr<sstables::sstable>, backlog_read_progress_manager*, std::hash<seastar::lw_shared_ptr<sstables::sstable> >, std::equal_to<seastar::lw_shared_ptr<sstables::sstable> >, std::allocator<std::pair<seastar::lw_shared_ptr<sstables::sstable> const, backlog_read_progress_manager*> > > const&) const at /local/home/bhalevy/dev/scylla/sstables/compaction_strategy.cc:464 (discriminator 2)
compaction_backlog_tracker::backlog() const at /local/home/bhalevy/dev/scylla/sstables/compaction_manager.cc:740 (discriminator 2)
compaction_backlog_manager::backlog() const at /local/home/bhalevy/dev/scylla/sstables/compaction_manager.cc:831 (discriminator 2)
compaction_manager::backlog() at /local/home/bhalevy/dev/scylla/sstables/compaction_manager.hh:225
compaction_manager::compaction_manager(seastar::scheduling_group, seastar::io_priority_class const&, unsigned long)::{lambda()#1}::operator()() const at /local/home/bhalevy/dev/scylla/sstables/compaction_manager.cc:343
std::_Function_handler<float (), compaction_manager::compaction_manager(seastar::scheduling_group, seastar::io_priority_class const&, unsigned long)::{lambda()#1}>::_M_invoke(std::_Any_data const&) at /usr/include/c++/8/bits/std_function.h:282
std::function<float ()>::operator()() const at /usr/include/c++/8/bits/std_function.h:687
backlog_controller::adjust() at /local/home/bhalevy/dev/scylla/database.cc:240
backlog_controller::backlog_controller(seastar::scheduling_group, seastar::io_priority_class const&, std::chrono::duration<long, std::ratio<1l, 1000l> >, std::vector<backlog_controller::control_point, std::allocator<backlog_controller::control_point> >, std::function<float ()>)::{lambda()#1}::operator()() const at /local/home/bhalevy/dev/scylla/backlog_controller.hh:79
seastar::noncopyable_function<void ()>::direct_vtable_for<backlog_controller::backlog_controller(seastar::scheduling_group, seastar::io_priority_class const&, std::chrono::duration<long, std::ratio<1l, 1000l> >, std::vector<backlog_controller::control_point, std::allocator<backlog_controller::control_point> >, std::function<float ()>)::{lambda()#1}>::call(seastar::noncopyable_function<void ()> const*) at /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:71
seastar::noncopyable_function<void ()>::operator()() const at /local/home/bhalevy/dev/scylla/seastar/include/seastar/util/noncopyable_function.hh:145
void seastar::reactor::complete_timers<seastar::timer_set<seastar::timer<std::chrono::_V2::steady_clock>, &seastar::timer<std::chrono::_V2::steady_clock>::_link>, boost::intrusive::list<seastar::timer<std::chrono::_V2::steady_clock>, boost::intrusive::member_hook<seastar::timer<std::chrono::_V2::steady_clock>, boost::intrusive::list_member_hook<void, void, void>, &seastar::timer<std::chrono::_V2::steady_clock>::_link>, void, void, void>, seastar::reactor::service_highres_timer()::{lambda()#1}>(seastar::timer_set<seastar::timer<std::chrono::_V2::steady_clock>, &seastar::timer<std::chrono::_V2::steady_clock>::_link>&, boost::intrusive::list<seastar::timer<std::chrono::_V2::steady_clock>, boost::intrusive::member_hook<seastar::timer<std::chrono::_V2::steady_clock>, boost::intrusive::list_member_hook<void, void, void>, &seastar::timer<std::chrono::_V2::steady_clock>::_link>, void, void, void>&, seastar::reactor::service_highres_timer()::{lambda()#1}&&) at /local/home/bhalevy/dev/scylla/seastar/build/debug/../../src/core/reactor.cc:1620
seastar::reactor::service_highres_timer()::{lambda()#1}::operator()() const at /local/home/bhalevy/dev/scylla/seastar/build/debug/../../src/core/reactor.cc:4136
seastar::reactor_backend_epoll::stop_tick() at /local/home/bhalevy/dev/scylla/seastar/build/debug/../../src/core/reactor.cc:948
_ZN7seastar20noncopyable_functionIFNS_6futureIJmEEEvEEC1IZNS1_IJEE4thenIZNS_11pollable_fd6sendtoENS_14socket_addressEPKvmEUlvE_S2_EET0_OT_EUlvE_EESE_ at /local/home/bhalevy/dev/scylla/seastar/build/debug/../../include/seastar/util/noncopyable_function.hh:117
std::function<void ()>::operator()() const at /usr/include/c++/8/bits/std_function.h:687
seastar::reactor::signals::poll_signal() at /usr/include/c++/8/bits/atomic_base.h:396
 (inlined by) seastar::reactor::signals::poll_signal() at /local/home/bhalevy/dev/scylla/seastar/build/debug/../../src/core/reactor.cc:1006
seastar::reactor::io_pollfn::try_enter_interrupt_mode() at /local/home/bhalevy/dev/scylla/seastar/build/debug/../../src/core/reactor.cc:3770
seastar::reactor::poll_once() at /local/home/bhalevy/dev/scylla/seastar/build/debug/../../src/core/reactor.cc:4335
seastar::reactor::run()::{lambda()#5}::operator()() at /local/home/bhalevy/dev/scylla/seastar/build/debug/../../src/core/reactor.cc:4203
seastar::noncopyable_function<void ()>::direct_vtable_for<seastar::reactor::run()::{lambda()#5}>::move(seastar::noncopyable_function<void ()>*, seastar::noncopyable_function<void ()>::direct_vtable_for<seastar::reactor::run()::{lambda()#5}>) at /local/home/bhalevy/dev/scylla/seastar/build/debug/../../include/seastar/util/noncopyable_function.hh:74
std::function<bool ()>::operator()() const at /usr/include/c++/8/bits/std_function.h:687
seastar::reactor::run() at /local/home/bhalevy/dev/scylla/seastar/build/debug/../../src/core/reactor.cc:4245
seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) at /local/home/bhalevy/dev/scylla/seastar/build/debug/../../include/seastar/core/future.hh:804
 (inlined by) seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) at /local/home/bhalevy/dev/scylla/seastar/build/debug/../../src/core/app-template.cc:179
457     virtual double backlog(const compaction_backlog_tracker::ongoing_writes& ow, const compaction_backlog_tracker::ongoing_compactions& oc) const override {
458         std::unordered_map<api::timestamp_type, compaction_backlog_tracker::ongoing_writes> writes_per_window;
459         std::unordered_map<api::timestamp_type, compaction_backlog_tracker::ongoing_compactions> compactions_per_window;
460         double b = 0;
461 
462         for (auto& wp : ow) {
463             auto bound = lower_bound_of(wp.second->maximum_timestamp());
464             writes_per_window[bound].insert(wp);

*** allocated here

465         }
466 
467         for (auto& cp : oc) {
468             auto bound = lower_bound_of(cp.first->get_stats_metadata().max_timestamp);
469             compactions_per_window[bound].insert(cp);
470         }
471 
472         auto no_ow = compaction_backlog_tracker::ongoing_writes();
473         auto no_oc = compaction_backlog_tracker::ongoing_compactions();
474         // Match the in-progress backlogs to existing windows. Compactions should always match an
475         // existing windows. Writes in progress can fall into an non-existent window.
476         for (auto& windows : _windows) {
477             auto bound = windows.first;
478             auto* ow_this_window = &no_ow;
479             auto itw = writes_per_window.find(bound);
480             if (itw != writes_per_window.end()) {
481                 ow_this_window = &itw->second;
482                 // We will erase here so we can keep track of which
483                 // writes belong to existing windows. Writes that don't belong to any window
484                 // are writes in progress to new windows and will be accounted in the final
485                 // loop before we return
486                 writes_per_window.erase(itw);

*** freed here

487             }
488             auto* oc_this_window = &no_oc;
489             auto itc = compactions_per_window.find(bound);
490             if (itc != compactions_per_window.end()) {
491                 oc_this_window = &itc->second;
492             }
493             b += windows.second.backlog(*ow_this_window, *oc_this_window);

*** *ow_this_window used after free

494         }

@bhalevy bhalevy added the dtest label Apr 29, 2019

@bhalevy

This comment has been minimized.

Copy link
Contributor Author

commented Apr 29, 2019

@glommer looks like you're the author of this code, can you please confirm?

@bhalevy

This comment has been minimized.

Copy link
Contributor Author

commented Apr 29, 2019

@tzach

This comment has been minimized.

Copy link
Contributor

commented May 2, 2019

@bhalevy since we do not all have access to the testing system, can you please copy a simple reproducer here?

@bhalevy

This comment has been minimized.

Copy link
Contributor Author

commented May 2, 2019

@bhalevy since we do not all have access to the testing system, can you please copy a simple reproducer here?

@tzach here it goes... Consider the python code below as pseudo code. It requires a single node cluster.

    session.execute("CREATE KEYSPACE ks WITH replication={'class':'SimpleStrategy', 'replication_factor':1}")

    session.execute("CREATE TABLE cf (key int PRIMARY KEY, val int) with gc_grace_seconds = 0 and compaction= {'class':'TimeWindowCompactionStrategy'}")

    for x in range(0, 100):
        session.execute('insert into cf (key, val) values (' + str(x) + ',1)')
    node1.flush()
    for x in range(0, 100):
        session.execute('delete from cf where key = ' + str(x))
@tzach

This comment has been minimized.

Copy link
Contributor

commented May 2, 2019

Thanks, @bhalevy finally I can understand what you are talking about :)

@slivne slivne added this to the 3.2 milestone May 5, 2019

@slivne

This comment has been minimized.

Copy link
Contributor

commented May 5, 2019

[PATCH 1/1] time_window_backlog_tracker: fix use after free

avikivity added a commit that referenced this issue May 6, 2019

time_window_backlog_tracker: fix use after free
Fixes #4465

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Message-Id: <20190430094209.13958-1-bhalevy@scylladb.com>
(cherry picked from commit 3a2fa82)

avikivity added a commit that referenced this issue May 6, 2019

time_window_backlog_tracker: fix use after free
Fixes #4465

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Message-Id: <20190430094209.13958-1-bhalevy@scylladb.com>
(cherry picked from commit 3a2fa82)

avikivity added a commit that referenced this issue May 6, 2019

time_window_backlog_tracker: fix use after free
Fixes #4465

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Message-Id: <20190430094209.13958-1-bhalevy@scylladb.com>
(cherry picked from commit 3a2fa82)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.