Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Segfault during resharding_by_same_smp_test #4425

Closed
bhalevy opened this issue Apr 11, 2019 · 4 comments

Comments

Projects
None yet
2 participants
@bhalevy
Copy link
Contributor

commented Apr 11, 2019

Scylla version e3a5ac2

Seen in dtest-release/88/artifact/logs-release.2/1554951166057_resharding_test.ReshardingTest_nodes1_with_SizeTieredCompactionStrategy.resharding_by_same_smp_test/node1.log:

INFO  2019-04-11 02:52:03,618 [shard 0] repair - Starting shutdown of repair
INFO  2019-04-11 02:52:03,618 [shard 0] repair - Completed shutdown of repair
scylla: /jenkins/workspace/scylla-master/dtest-release@2/scylla/seastar/include/seastar/core/sharded.hh:560: Service& seastar::sharded<T>::local() [with Service = database]: Assertion `local_is_initialized()' failed.
scylla: /jenkins/workspace/scylla-master/dtest-release@2/scylla/seastar/include/seastar/core/sharded.hh:560: Service& seastar::sharded<T>::local() [with Service = database]: Assertion `local_is_initialized()' failed.
Aborting on shard 2.
Backtrace:
  0x0000000004110772
  0x000000000400f455
  0x000000000400f755
  0x000000000400f803
  0x00007f08532c502f
  /jenkins/workspace/scylla-master/dtest-release@2/scylla-dtest/../scylla/dynamic_libs/libc.so.6+0x000000000003853e
  /jenkins/workspace/scylla-master/dtest-release@2/scylla-dtest/../scylla/dynamic_libs/libc.so.6+0x0000000000022894
  /jenkins/workspace/scylla-master/dtest-release@2/scylla-dtest/../scylla/dynamic_libs/libc.so.6+0x0000000000022768
  /jenkins/workspace/scylla-master/dtest-release@2/scylla-dtest/../scylla/dynamic_libs/libc.so.6+0x00000000000309f5
  0x000000000314b5a8
  0x0000000000b186e8
  0x0000000000b19789
  0x0000000004008fdf
  0x0000000004011b44
  0x00000000041137c5
  0x0000000004007e54
  0x00000000040e0e7d
  0x000000000410791b
  0x0000000003fdc9fd
  /jenkins/workspace/scylla-master/dtest-release@2/scylla-dtest/../scylla/dynamic_libs/libpthread.so.0+0x000000000000858d
  /jenkins/workspace/scylla-master/dtest-release@2/scylla-dtest/../scylla/dynamic_libs/libc.so.6+0x00000000000fd6a2
Segmentation fault on shard 4.
Backtrace:
  0x0000000004110772
  0x000000000400f455
  0x000000000400f755
  0x000000000400f7a3
  0x00007f08532c502f
  0x0000000000c3d5f0
  0x0000000000bdffeb
  0x000000000314b361
  0x0000000000b186e8
  0x0000000000b19789
  0x0000000004008fdf
  0x0000000004011b44
  0x00000000041137c5
  0x0000000004007e54
  0x00000000040e0e7d
  0x000000000410791b
  0x0000000003fdc9fd
  /jenkins/workspace/scylla-master/dtest-release@2/scylla-dtest/../scylla/dynamic_libs/libpthread.so.0+0x000000000000858d
  /jenkins/workspace/scylla-master/dtest-release@2/scylla-dtest/../scylla/dynamic_libs/libc.so.6+0x00000000000fd6a2

Shard 2:

void seastar::backtrace<seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}>(seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}&&) at /jenkins/workspace/scylla-master/dtest-release@2/scylla/seastar/build/release/../../include/seastar/util/backtrace.hh:55
seastar::print_with_backtrace(seastar::backtrace_buffer&) at /jenkins/workspace/scylla-master/dtest-release@2/scylla/seastar/build/release/../../src/core/reactor.cc:1083
 (inlined by) print_with_backtrace at /jenkins/workspace/scylla-master/dtest-release@2/scylla/seastar/build/release/../../src/core/reactor.cc:1104
seastar::print_with_backtrace(char const*) at /jenkins/workspace/scylla-master/dtest-release@2/scylla/seastar/build/release/../../src/core/reactor.cc:1111
seastar::install_oneshot_signal_handler<6, &seastar::sigabrt_action>()::{lambda(int, siginfo_t*, void*)#1}::_FUN(int, siginfo_t*, void*) at /jenkins/workspace/scylla-master/dtest-release@2/scylla/seastar/build/release/../../src/core/reactor.cc:4981
 (inlined by) operator() at /jenkins/workspace/scylla-master/dtest-release@2/scylla/seastar/build/release/../../src/core/reactor.cc:4963
 (inlined by) _FUN at /jenkins/workspace/scylla-master/dtest-release@2/scylla/seastar/build/release/../../src/core/reactor.cc:4959
?? ??:0
?? ??:0
?? ??:0
?? ??:0
?? ??:0
?? ??:0
?? ??:0
?? ??:0
?? ??:0
std::_Function_handler<seastar::future<> (database&), seastar::sharded<database>::invoke_on_all<service::cache_hitrate_calculator::recalculate_hitrates()::{lambda(std::unordered_map<utils::UUID, service::cache_hitrate_calculator::recalculate_hitrates()::stat, std::hash<utils::UUID>, std::equal_to<utils::UUID>, std::allocator<std::pair<utils::UUID const, service::cache_hitrate_calculator::recalculate_hitrates()::stat> > >)#4}::operator()(std::unordered_map<utils::UUID, service::cache_hitrate_calculator::recalculate_hitrates()::stat, std::hash<utils::UUID>, std::equal_to<utils::UUID>, std::allocator<std::pair<utils::UUID const, service::cache_hitrate_calculator::recalculate_hitrates()::stat> > >)::{lambda(database&)#1}>(service::cache_hitrate_calculator::recalculate_hitrates()::{lambda(std::unordered_map<utils::UUID, service::cache_hitrate_calculator::recalculate_hitrates()::stat, std::hash<utils::UUID>, std::equal_to<utils::UUID>, std::allocator<std::pair<utils::UUID const, service::cache_hitrate_calculator::recalculate_hitrates()::stat> > >)#4}::operator()(std::unordered_map<utils::UUID, service::cache_hitrate_calculator::recalculate_hitrates()::stat, std::hash<utils::UUID>, std::equal_to<utils::UUID>, std::allocator<std::pair<utils::UUID const, service::cache_hitrate_calculator::recalculate_hitrates()::stat> > >)::{lambda(database&)#1}&&)::{lambda(database&)#1}>::_M_invoke(std::_Any_data const&, database&) at /jenkins/workspace/scylla-master/dtest-release@2/scylla/seastar/include/seastar/core/sharded.hh:560
 (inlined by) operator() at /jenkins/workspace/scylla-master/dtest-release@2/scylla/service/misc_services.cc:127
 (inlined by) ?? at /usr/include/boost/range/detail/default_constructible_unary_fn.hpp:45
 (inlined by) ?? at /usr/include/boost/iterator/filter_iterator.hpp:100
 (inlined by) ?? at /usr/include/boost/iterator/filter_iterator.hpp:59
 (inlined by) ?? at /usr/include/boost/iterator/filter_iterator.hpp:114
 (inlined by) ?? at /usr/include/boost/range/adaptor/filtered.hpp:48
 (inlined by) ?? at /usr/include/boost/range/adaptor/filtered.hpp:65
 (inlined by) operator() at /jenkins/workspace/scylla-master/dtest-release@2/scylla/service/misc_services.cc:150
 (inlined by) ?? at /jenkins/workspace/scylla-master/dtest-release@2/scylla/seastar/include/seastar/core/future.hh:1441
 (inlined by) ?? at /jenkins/workspace/scylla-master/dtest-release@2/scylla/seastar/include/seastar/core/future.hh:1467
 (inlined by) operator() at /jenkins/workspace/scylla-master/dtest-release@2/scylla/seastar/include/seastar/core/sharded.hh:536
 (inlined by) _M_invoke at /usr/include/c++/8/bits/std_function.h:283
seastar::smp_message_queue::async_work_item<seastar::sharded<database>::invoke_on_all(std::function<seastar::future<> (database&)>)::{lambda(unsigned int)#1}::operator()(unsigned int) const::{lambda()#1}>::process()::{lambda()#1}::operator()() const at /usr/include/c++/8/bits/std_function.h:687
 (inlined by) seastar::sharded<database>::invoke_on_all(std::function<seastar::future<> (database&)>)::{lambda(unsigned int)#1}::operator()(unsigned int) const::{lambda()#1}::operator()() const at /jenkins/workspace/scylla-master/dtest-release@2/scylla/seastar/include/seastar/core/sharded.hh:502
 (inlined by) ?? at /jenkins/workspace/scylla-master/dtest-release@2/scylla/seastar/include/seastar/core/future.hh:1484
 (inlined by) seastar::smp_message_queue::async_work_item<seastar::sharded<database>::invoke_on_all(std::function<seastar::future<> (database&)>)::{lambda(unsigned int)#1}::operator()(unsigned int) const::{lambda()#1}>::process()::{lambda()#1}::operator()() const at /jenkins/workspace/scylla-master/dtest-release@2/scylla/seastar/include/seastar/core/reactor.hh:188
seastar::smp_message_queue::async_work_item<seastar::sharded<database>::invoke_on_all(std::function<seastar::future<> (database&)>)::{lambda(unsigned int)#1}::operator()(unsigned int) const::{lambda()#1}>::process() at /jenkins/workspace/scylla-master/dtest-release@2/scylla/seastar/include/seastar/core/future.hh:1418
 (inlined by) ?? at /jenkins/workspace/scylla-master/dtest-release@2/scylla/seastar/include/seastar/core/future.hh:1467
 (inlined by) ?? at /jenkins/workspace/scylla-master/dtest-release@2/scylla/seastar/include/seastar/core/future-util.hh:80
 (inlined by) seastar::smp_message_queue::async_work_item<seastar::sharded<database>::invoke_on_all(std::function<seastar::future<> (database&)>)::{lambda(unsigned int)#1}::operator()(unsigned int) const::{lambda()#1}>::process() at /jenkins/workspace/scylla-master/dtest-release@2/scylla/seastar/include/seastar/core/reactor.hh:187
operator() at /jenkins/workspace/scylla-master/dtest-release@2/scylla/seastar/build/release/../../src/core/reactor.cc:4614
 (inlined by) ?? at /jenkins/workspace/scylla-master/dtest-release@2/scylla/seastar/build/release/../../src/core/reactor.cc:4587
seastar::smp::poll_queues() at /jenkins/workspace/scylla-master/dtest-release@2/scylla/seastar/build/release/../../src/core/reactor.cc:4613
 (inlined by) seastar::smp::poll_queues() at /jenkins/workspace/scylla-master/dtest-release@2/scylla/seastar/build/release/../../src/core/reactor.cc:5416
seastar::reactor::smp_pollfn::poll() at /jenkins/workspace/scylla-master/dtest-release@2/scylla/seastar/build/release/../../src/core/reactor.cc:3882
std::_Function_handler<bool (), seastar::reactor::run()::{lambda()#6}>::_M_invoke(std::_Any_data const&) at /jenkins/workspace/scylla-master/dtest-release@2/scylla/seastar/build/release/../../src/core/reactor.cc:4301
 (inlined by) operator() at /jenkins/workspace/scylla-master/dtest-release@2/scylla/seastar/build/release/../../src/core/reactor.cc:4195
 (inlined by) _M_invoke at /usr/include/c++/8/bits/std_function.h:282
seastar::reactor::run() at /usr/include/c++/8/bits/std_function.h:687
 (inlined by) seastar::reactor::run() at /jenkins/workspace/scylla-master/dtest-release@2/scylla/seastar/build/release/../../src/core/reactor.cc:4220
seastar::smp::configure(boost::program_options::variables_map)::{lambda()#3}::operator()() const at /jenkins/workspace/scylla-master/dtest-release@2/scylla/seastar/build/release/../../src/core/reactor.cc:5369
seastar::posix_thread::start_routine(void*) at /usr/include/c++/8/bits/std_function.h:687
 (inlined by) seastar::posix_thread::start_routine(void*) at /jenkins/workspace/scylla-master/dtest-release@2/scylla/seastar/build/release/../../src/core/posix.cc:52

Shard 4:

void seastar::backtrace<seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}>(seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}&&) at /jenkins/workspace/scylla-master/dtest-release@2/scylla/seastar/build/release/../../include/seastar/util/backtrace.hh:55
seastar::print_with_backtrace(seastar::backtrace_buffer&) at /jenkins/workspace/scylla-master/dtest-release@2/scylla/seastar/build/release/../../src/core/reactor.cc:1083
 (inlined by) print_with_backtrace at /jenkins/workspace/scylla-master/dtest-release@2/scylla/seastar/build/release/../../src/core/reactor.cc:1104
seastar::print_with_backtrace(char const*) at /jenkins/workspace/scylla-master/dtest-release@2/scylla/seastar/build/release/../../src/core/reactor.cc:1111
seastar::install_oneshot_signal_handler<11, &seastar::sigsegv_action>()::{lambda(int, siginfo_t*, void*)#1}::_FUN(int, siginfo_t*, void*) at /jenkins/workspace/scylla-master/dtest-release@2/scylla/seastar/build/release/../../src/core/reactor.cc:4977
 (inlined by) operator() at /jenkins/workspace/scylla-master/dtest-release@2/scylla/seastar/build/release/../../src/core/reactor.cc:4963
 (inlined by) _FUN at /jenkins/workspace/scylla-master/dtest-release@2/scylla/seastar/build/release/../../src/core/reactor.cc:4959
?? ??:0
std::_Hashtable<seastar::basic_sstring<char, unsigned int, 15u, true>, std::pair<seastar::basic_sstring<char, unsigned int, 15u, true> const, keyspace>, std::allocator<std::pair<seastar::basic_sstring<char, unsigned int, 15u, true> const, keyspace> >, std::__detail::_Select1st, std::equal_to<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::hash<seastar::basic_sstring<char, unsigned int, 15u, true> >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::_M_find_before_node(unsigned long, seastar::basic_sstring<char, unsigned int, 15u, true> const&, unsigned long) const at /usr/include/c++/8/bits/hashtable.h:1560
database::find_keyspace(seastar::basic_sstring<char, unsigned int, 15u, true> const&) at /usr/include/c++/8/bits/hashtable.h:654
 (inlined by) ?? at /usr/include/c++/8/bits/hashtable_policy.h:757
 (inlined by) ?? at /usr/include/c++/8/bits/unordered_map.h:991
 (inlined by) database::find_keyspace(seastar::basic_sstring<char, unsigned int, 15u, true> const&) at /jenkins/workspace/scylla-master/dtest-release@2/scylla/database.cc:768
operator() at /jenkins/workspace/scylla-master/dtest-release@2/scylla/service/misc_services.cc:127
 (inlined by) ?? at /usr/include/boost/range/detail/default_constructible_unary_fn.hpp:45
 (inlined by) ?? at /usr/include/boost/iterator/filter_iterator.hpp:100
 (inlined by) ?? at /usr/include/boost/iterator/filter_iterator.hpp:59
 (inlined by) ?? at /usr/include/boost/iterator/filter_iterator.hpp:114
 (inlined by) ?? at /usr/include/boost/range/adaptor/filtered.hpp:48
 (inlined by) ?? at /usr/include/boost/range/adaptor/filtered.hpp:65
 (inlined by) operator() at /jenkins/workspace/scylla-master/dtest-release@2/scylla/service/misc_services.cc:150
 (inlined by) ?? at /jenkins/workspace/scylla-master/dtest-release@2/scylla/seastar/include/seastar/core/future.hh:1441
 (inlined by) ?? at /jenkins/workspace/scylla-master/dtest-release@2/scylla/seastar/include/seastar/core/future.hh:1467
 (inlined by) operator() at /jenkins/workspace/scylla-master/dtest-release@2/scylla/seastar/include/seastar/core/sharded.hh:536
 (inlined by) _M_invoke at /usr/include/c++/8/bits/std_function.h:283
seastar::smp_message_queue::async_work_item<seastar::sharded<database>::invoke_on_all(std::function<seastar::future<> (database&)>)::{lambda(unsigned int)#1}::operator()(unsigned int) const::{lambda()#1}>::process()::{lambda()#1}::operator()() const at /usr/include/c++/8/bits/std_function.h:687
 (inlined by) seastar::sharded<database>::invoke_on_all(std::function<seastar::future<> (database&)>)::{lambda(unsigned int)#1}::operator()(unsigned int) const::{lambda()#1}::operator()() const at /jenkins/workspace/scylla-master/dtest-release@2/scylla/seastar/include/seastar/core/sharded.hh:502
 (inlined by) ?? at /jenkins/workspace/scylla-master/dtest-release@2/scylla/seastar/include/seastar/core/future.hh:1484
 (inlined by) seastar::smp_message_queue::async_work_item<seastar::sharded<database>::invoke_on_all(std::function<seastar::future<> (database&)>)::{lambda(unsigned int)#1}::operator()(unsigned int) const::{lambda()#1}>::process()::{lambda()#1}::operator()() const at /jenkins/workspace/scylla-master/dtest-release@2/scylla/seastar/include/seastar/core/reactor.hh:188
seastar::smp_message_queue::async_work_item<seastar::sharded<database>::invoke_on_all(std::function<seastar::future<> (database&)>)::{lambda(unsigned int)#1}::operator()(unsigned int) const::{lambda()#1}>::process() at /jenkins/workspace/scylla-master/dtest-release@2/scylla/seastar/include/seastar/core/future.hh:1418
 (inlined by) ?? at /jenkins/workspace/scylla-master/dtest-release@2/scylla/seastar/include/seastar/core/future.hh:1467
 (inlined by) ?? at /jenkins/workspace/scylla-master/dtest-release@2/scylla/seastar/include/seastar/core/future-util.hh:80
 (inlined by) seastar::smp_message_queue::async_work_item<seastar::sharded<database>::invoke_on_all(std::function<seastar::future<> (database&)>)::{lambda(unsigned int)#1}::operator()(unsigned int) const::{lambda()#1}>::process() at /jenkins/workspace/scylla-master/dtest-release@2/scylla/seastar/include/seastar/core/reactor.hh:187
operator() at /jenkins/workspace/scylla-master/dtest-release@2/scylla/seastar/build/release/../../src/core/reactor.cc:4614
 (inlined by) ?? at /jenkins/workspace/scylla-master/dtest-release@2/scylla/seastar/build/release/../../src/core/reactor.cc:4587
seastar::smp::poll_queues() at /jenkins/workspace/scylla-master/dtest-release@2/scylla/seastar/build/release/../../src/core/reactor.cc:4613
 (inlined by) seastar::smp::poll_queues() at /jenkins/workspace/scylla-master/dtest-release@2/scylla/seastar/build/release/../../src/core/reactor.cc:5416
seastar::reactor::smp_pollfn::poll() at /jenkins/workspace/scylla-master/dtest-release@2/scylla/seastar/build/release/../../src/core/reactor.cc:3882
std::_Function_handler<bool (), seastar::reactor::run()::{lambda()#6}>::_M_invoke(std::_Any_data const&) at /jenkins/workspace/scylla-master/dtest-release@2/scylla/seastar/build/release/../../src/core/reactor.cc:4301
 (inlined by) operator() at /jenkins/workspace/scylla-master/dtest-release@2/scylla/seastar/build/release/../../src/core/reactor.cc:4195
 (inlined by) _M_invoke at /usr/include/c++/8/bits/std_function.h:282
seastar::reactor::run() at /usr/include/c++/8/bits/std_function.h:687
 (inlined by) seastar::reactor::run() at /jenkins/workspace/scylla-master/dtest-release@2/scylla/seastar/build/release/../../src/core/reactor.cc:4220
seastar::smp::configure(boost::program_options::variables_map)::{lambda()#3}::operator()() const at /jenkins/workspace/scylla-master/dtest-release@2/scylla/seastar/build/release/../../src/core/reactor.cc:5369
seastar::posix_thread::start_routine(void*) at /usr/include/c++/8/bits/std_function.h:687
 (inlined by) seastar::posix_thread::start_routine(void*) at /jenkins/workspace/scylla-master/dtest-release@2/scylla/seastar/build/release/../../src/core/posix.cc:52

Core dump here: node1-reactor-2.28413.1554951123.core.gz

@bhalevy bhalevy added the dtest label Apr 11, 2019

@bhalevy

This comment has been minimized.

Copy link
Contributor Author

commented Apr 11, 2019

@gleb-cloudius Can you please look at this?

@gleb-cloudius

This comment has been minimized.

Copy link
Contributor

commented Apr 11, 2019

@gleb-cloudius

This comment has been minimized.

Copy link
Contributor

commented Apr 21, 2019

@bhalevy

This comment has been minimized.

Copy link
Contributor Author

commented Apr 21, 2019

Where is the binary?

-- Gleb.

@gleb-cloudius binary is here, core dump is here

avikivity added a commit that referenced this issue Apr 22, 2019

cache_hitrate_calculator: fix use after free in non_system_filter lambda
non_system_filter lambda is defined static which means it is initialized
only once, so the 'this' that is will capture will belong to a shard
where the function runs first. During service destruction the function
may run on different shard and access already other's shard service that
may be already freed.

Fixed #4425

Message-Id: <20190421152139.GN21208@scylladb.com>
(cherry picked from commit 306f5b9)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.