Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Segfaults during compaction - master #4085

Closed
veramine opened this issue Jan 10, 2019 · 10 comments

Comments

@veramine
Copy link

commented Jan 10, 2019

This is Scylla's bug tracker, to be used for reporting bugs only.
If you have a question about Scylla, and not a bug, please ask it in
our mailing-list at scylladb-dev@googlegroups.com or in our slack channel.

  • I have read the disclaimer above, and I am reporting a suspected malfunction in Scylla.

Installation details
Scylla version (or git commit hash): 666.development-0.20190109.f5301990f
Cluster size: 1
OS (RHEL/CentOS/Ubuntu/AWS AMI): AWS AMI (CentOS 7)

Hardware details (for performance issues)
Platform (physical/VM/cloud instance type/docker): AWS i3.xlarge

We have been testing the latest bits on a single node test cluster. During our normal stress test, we noticed several segmentation faults. I've attached the full journalctl log. I've pasted in the decode stack traces below. I also saw some messages about corrupted sstables so its possible there was really only one legitimate issue and the other segfaults are the result of corrupted data or state.

void seastar::backtrace<seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}>(seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}&&) at /scylla/seastar/include/seastar/util/backtrace.hh:56
seastar::print_with_backtrace(seastar::backtrace_buffer&) at /scylla/seastar/src/core/reactor.cc:410
seastar::print_with_backtrace(char const*) at /scylla/seastar/src/core/reactor.cc:438
void seastar::install_oneshot_signal_handler<11, &seastar::sigsegv_action>()::{lambda(int, siginfo_t*, void*)#1}::_FUN(int, siginfo_t*, void*) at /scylla/seastar/src/core/reactor.cc:4025
?? ??:0
sstables::bag_sstable_set::~bag_sstable_set() at /usr/include/c++/8/ostream:167
sstables::regular_compaction::~regular_compaction() at /usr/include/c++/8/ostream:167
sstables::regular_compaction::~regular_compaction() at /usr/include/c++/8/ostream:167
seastar::internal::do_with_state<seastar::futurize<std::result_of<std::decay<sstables::compaction::run(std::unique_ptr<sstables::compaction, std::default_delete<sstables::compaction> >)::{lambda()#1}>::type ()>::type>::type seastar::async<sstables::compaction::run(std::unique_ptr<sstables::compaction, std::default_delete<sstables::compaction> >)::{lambda()#1}>(seastar::thread_attributes, std::decay&&, (std::decay<sstables::compaction::run(std::unique_ptr<sstables::compaction, std::default_delete<sstables::compaction> >)::{lambda()#1}>::type&&)...)::work, seastar::future<sstables::compaction_info> >::run_and_dispose() at /usr/include/c++/8/ostream:167
seastar::reactor::run_tasks(seastar::reactor::task_queue&) at /scylla/seastar/src/core/reactor.cc:2700
seastar::reactor::run_some_tasks() [clone .part.2943] at /scylla/seastar/src/core/reactor.cc:3123
seastar::reactor::run() at /scylla/seastar/src/core/reactor.cc:3107
seastar::smp::configure(boost::program_options::variables_map)::{lambda()#3}::operator()() const at /scylla/seastar/src/core/reactor.cc:4339
seastar::posix_thread::start_routine(void*) at /usr/include/c++/8/bits/std_function.h:687
void seastar::backtrace<seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}>(seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}&&) at /scylla/seastar/include/seastar/util/backtrace.hh:56
seastar::print_with_backtrace(seastar::backtrace_buffer&) at /scylla/seastar/src/core/reactor.cc:410
seastar::print_with_backtrace(char const*) at /scylla/seastar/src/core/reactor.cc:438
void seastar::install_oneshot_signal_handler<11, &seastar::sigsegv_action>()::{lambda(int, siginfo_t*, void*)#1}::_FUN(int, siginfo_t*, void*) at /scylla/seastar/src/core/reactor.cc:4025
?? ??:0
sstables::bag_sstable_set::erase(seastar::lw_shared_ptr<sstables::sstable>) at /usr/include/c++/8/ostream:167
sstables::sstable_set::erase(seastar::lw_shared_ptr<sstables::sstable>) at /usr/include/c++/8/ostream:167
sstables::regular_compaction::maybe_replace_exhausted_sstables() at /usr/include/c++/8/ostream:167
sstables::regular_compaction::finish_sstable_writer() at /usr/include/c++/8/ostream:167
seastar::futurize<std::result_of<std::decay<sstables::compaction::run(std::unique_ptr<sstables::compaction, std::default_delete<sstables::compaction> >)::{lambda()#1}>::type ()>::type>::type seastar::async<sstables::compaction::run(std::unique_ptr<sstables::compaction, std::default_delete<sstables::compaction> >)::{lambda()#1}>(seastar::thread_attributes, std::decay&&, (std::decay<sstables::compaction::run(std::unique_ptr<sstables::compaction, std::default_delete<sstables::compaction> >)::{lambda()#1}>::type&&)...)::{lambda(seastar::futurize<std::result_of<std::decay<{lambda()#1}>::type ()>::type> seastar::async<{lambda()#1}>(seastar::futurize<std::result_of<std::decay<{lambda()#1}>::type ()>::type>::type, seastar::thread_attributes, std::decay<{lambda()#1}>::type&&)::work&)#1}::operator()(seastar::futurize<std::result_of<std::decay<{lambda()#1}>::type ()>::type> seastar::async<{lambda()#1}>(seastar::futurize<std::result_of<std::decay<{lambda()#1}>::type ()>::type>::type, seastar::thread_attributes, std::decay<{lambda()#1}>::type&&)::work)::{lambda()#1}::operator()() const [clone .isra.2050] at /usr/include/c++/8/ostream:167
seastar::thread_context::main() at /scylla/seastar/src/core/thread.cc:224
void seastar::backtrace<seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}>(seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}&&) at /scylla/seastar/include/seastar/util/backtrace.hh:56
seastar::print_with_backtrace(seastar::backtrace_buffer&) at /scylla/seastar/src/core/reactor.cc:410
seastar::print_with_backtrace(char const*) at /scylla/seastar/src/core/reactor.cc:438
void seastar::install_oneshot_signal_handler<11, &seastar::sigsegv_action>()::{lambda(int, siginfo_t*, void*)#1}::_FUN(int, siginfo_t*, void*) at /scylla/seastar/src/core/reactor.cc:4025
?? ??:0
_ZN7seastar12continuationIZNS_6futureIJEE12then_wrappedIZN8sstables7sstable15remove_temp_dirEvEUlS2_E_S2_EET0_OT_EUlS9_E_JEE15run_and_disposeEv at /usr/include/c++/8/bits/stl_vector.h:1643
seastar::reactor::run_tasks(seastar::reactor::task_queue&) at /scylla/seastar/src/core/reactor.cc:2700
seastar::reactor::run_some_tasks() [clone .part.2943] at /scylla/seastar/src/core/reactor.cc:3123
seastar::reactor::run() at /scylla/seastar/src/core/reactor.cc:3107
seastar::smp::configure(boost::program_options::variables_map)::{lambda()#3}::operator()() const at /scylla/seastar/src/core/reactor.cc:4339
seastar::posix_thread::start_routine(void*) at /usr/include/c++/8/bits/std_function.h:687
void seastar::backtrace<seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}>(seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}&&) at /scylla/seastar/include/seastar/util/backtrace.hh:56
seastar::print_with_backtrace(seastar::backtrace_buffer&) at /scylla/seastar/src/core/reactor.cc:410
seastar::print_with_backtrace(char const*) at /scylla/seastar/src/core/reactor.cc:438
void seastar::install_oneshot_signal_handler<11, &seastar::sigsegv_action>()::{lambda(int, siginfo_t*, void*)#1}::_FUN(int, siginfo_t*, void*) at /scylla/seastar/src/core/reactor.cc:4025
?? ??:0
_ZN7seastar12continuationIZNS_6futureIJEE12then_wrappedIZN8sstables7sstable15remove_temp_dirEvEUlS2_E_S2_EET0_OT_EUlS9_E_JEE15run_and_disposeEv at /usr/include/c++/8/bits/stl_vector.h:1643
seastar::reactor::run_tasks(seastar::reactor::task_queue&) at /scylla/seastar/src/core/reactor.cc:2700
seastar::reactor::run_some_tasks() [clone .part.2943] at /scylla/seastar/src/core/reactor.cc:3123
seastar::reactor::run() at /scylla/seastar/src/core/reactor.cc:3107
seastar::smp::configure(boost::program_options::variables_map)::{lambda()#3}::operator()() const at /scylla/seastar/src/core/reactor.cc:4339
seastar::posix_thread::start_routine(void*) at /usr/include/c++/8/bits/std_function.h:687
void seastar::backtrace<seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}>(seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}&&) at /scylla/seastar/include/seastar/util/backtrace.hh:56
seastar::print_with_backtrace(seastar::backtrace_buffer&) at /scylla/seastar/src/core/reactor.cc:410
seastar::print_with_backtrace(char const*) at /scylla/seastar/src/core/reactor.cc:438
void seastar::install_oneshot_signal_handler<11, &seastar::sigsegv_action>()::{lambda(int, siginfo_t*, void*)#1}::_FUN(int, siginfo_t*, void*) at /scylla/seastar/src/core/reactor.cc:4025
?? ??:0
_ZN7seastar12continuationIZNS_6futureIJEE12then_wrappedIZN8sstables7sstable15remove_temp_dirEvEUlS2_E_S2_EET0_OT_EUlS9_E_JEE15run_and_disposeEv at /usr/include/c++/8/bits/stl_vector.h:1643
seastar::reactor::run_tasks(seastar::reactor::task_queue&) at /scylla/seastar/src/core/reactor.cc:2700
seastar::reactor::run_some_tasks() [clone .part.2943] at /scylla/seastar/src/core/reactor.cc:3123
seastar::reactor::run() at /scylla/seastar/src/core/reactor.cc:3107
seastar::smp::configure(boost::program_options::variables_map)::{lambda()#3}::operator()() const at /scylla/seastar/src/core/reactor.cc:4339
seastar::posix_thread::start_routine(void*) at /usr/include/c++/8/bits/std_function.h:687

segfault.txt

@veramine

This comment has been minimized.

Copy link
Author

commented Jan 10, 2019

We are still using the legacy scylla sstable format, not the new mc format. It seems like Scylla got stuck in a loop trying to boot with errors such as the following:

Jan 10 23:23:15 ip-172-31-39-55.us-west-2.compute.internal scylla.bin[1531]:  [shard 0] commitlog - Cannot parse the version of the file: Recycled-CommitLog-1-1269206218.log
Jan 10 23:23:15 ip-172-31-39-55.us-west-2.compute.internal scylla.bin[1531]:  [shard 0] commitlog - Cannot parse the version of the file: Recycled-CommitLog-1-1269206219.log
Jan 10 23:23:15 ip-172-31-39-55.us-west-2.compute.internal scylla.bin[1531]:  [shard 0] commitlog - Cannot parse the version of the file: Recycled-CommitLog-1-1269206220.log
Jan 10 23:23:15 ip-172-31-39-55.us-west-2.compute.internal scylla.bin[1531]:  [shard 0] commitlog - Cannot parse the version of the file: Recycled-CommitLog-1-1269206221.log

and

Jan 10 23:23:48 ip-172-31-39-55.us-west-2.compute.internal scylla.bin[1531]:  [shard 3] commitlog_replayer - Corrupted file: /var/lib/scylla/commitlog/CommitLog-1-54043196868600320.log. 33484792 bytes skipped.
Jan 10 23:23:59 ip-172-31-39-55.us-west-2.compute.internal scylla.bin[1531]:  [shard 2] commitlog_replayer - Corrupted file: /var/lib/scylla/commitlog/CommitLog-1-36028798359328477.log. 33488888 bytes skipped.
Jan 10 23:24:05 ip-172-31-39-55.us-west-2.compute.internal scylla.bin[1531]:  [shard 0] commitlog_replayer - Corrupted file: /var/lib/scylla/commitlog/CommitLog-1-1269601079.log. 27791352 bytes skipped.

I removed all files in /var/lib/scylla/commitlog and now it seems to be starting ok.

@raphaelsc

This comment has been minimized.

Copy link
Contributor

commented Jan 11, 2019

@veramine Commitlog got corrupted, not sstable. @elcallio, I think that belongs to you. One of the segfault is related to my recent changes in compaction. I'll investigate. Thanks

@slivne

This comment has been minimized.

Copy link
Contributor

commented Jan 20, 2019

@veramine on master mc format is enabled by default

your logs have compactions logs with file name such as mc-2638-big-Data.db

@raphaelsc - there are two issues

  • the segfault they are hitting is in compaction
  • we do have 1-2 items you are trying to clear related compaction/sstable_set - lets see if the backtraces look the same

@elcallio there is a huge amount of recycled commitlogs such as Recycled-CommitLog-1-1269206218.log that are kept boot after boot

  • are we supposed to clear commitlogs after replay on boot or not ?

  • are we going to use them - they are left unused ?

  • do we cap the amount of Recycled commitlogs we keep on disk to make sure it does not grow ?

  • another item I noticed is that all shards report the same logs

Jan 10 01:19:58 ip-172-31-39-55.us-west-2.compute.internal scylla.bin[18551]:  [shard 0] commitlog - Cannot parse the version of the file: Recycled-CommitLog-1-1269206218.log
Jan 10 01:19:58 ip-172-31-39-55.us-west-2.compute.internal scylla.bin[18551]:  [shard 1] commitlog - Cannot parse the version of the file: Recycled-CommitLog-1-1269206218.log
Jan 10 01:19:58 ip-172-31-39-55.us-west-2.compute.internal scylla.bin[18551]:  [shard 3] commitlog - Cannot parse the version of the file: Recycled-CommitLog-1-1269206218.log
Jan 10 01:19:58 ip-172-31-39-55.us-west-2.compute.internal scylla.bin[18551]:  [shard 2] commitlog - Cannot parse the version of the file: Recycled-CommitLog-1-1269206218.log

are recycled commitlogs split to shards ? if not its strange (as commitlogs are).

@veramine it will really help if you can upload a core(s) and debug symbols so we can take a look.

@slivne slivne added this to the 3.1 milestone Jan 20, 2019

@raphaelsc

This comment has been minimized.

Copy link
Contributor

commented Jan 21, 2019

@bhalevy

This comment has been minimized.

Copy link
Contributor

commented Jan 22, 2019

@raphaelsc

This comment has been minimized.

Copy link
Contributor

commented Jan 22, 2019

$ addr2line -Cfpi -e ./scylla
  0x00000000006c9872
  0x00000000005dbebc
  0x00000000005dc165
  0x00000000005dc1b3
  /lib64/libpthread.so.0+0x00000000000121af
  0x0000000001990ad9
  0x0000000001974dfd
  0x00000000019607a4
  0x0000000001963aa0
  0x000000000194415c
  0x000000000054cca5
void seastar::backtrace<seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}>(seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}&&) at /jenkins/workspace/scylla-master/dtest/label/scylla-dtest/mode/release/smp/2/scylla/seastar/include/seastar/util/backtrace.hh:56
seastar::backtrace_buffer::append_backtrace() at /jenkins/workspace/scylla-master/dtest/label/scylla-dtest/mode/release/smp/2/scylla/seastar/src/core/reactor.cc:410
 (inlined by) print_with_backtrace at /jenkins/workspace/scylla-master/dtest/label/scylla-dtest/mode/release/smp/2/scylla/seastar/src/core/reactor.cc:431
seastar::print_with_backtrace(char const*) at /jenkins/workspace/scylla-master/dtest/label/scylla-dtest/mode/release/smp/2/scylla/seastar/src/core/reactor.cc:438
sigsegv_action at /jenkins/workspace/scylla-master/dtest/label/scylla-dtest/mode/release/smp/2/scylla/seastar/src/core/reactor.cc:4025
 (inlined by) operator() at /jenkins/workspace/scylla-master/dtest/label/scylla-dtest/mode/release/smp/2/scylla/seastar/src/core/reactor.cc:4011
 (inlined by) _FUN at /jenkins/workspace/scylla-master/dtest/label/scylla-dtest/mode/release/smp/2/scylla/seastar/src/core/reactor.cc:4007
?? ??:0
seastar::lw_shared_ptr<sstables::sstable>::~lw_shared_ptr() at /jenkins/workspace/scylla-master/dtest/label/scylla-dtest/mode/release/smp/2/scylla/seastar/include/seastar/core/shared_ptr.hh:299
 (inlined by) void __gnu_cxx::new_allocator<seastar::lw_shared_ptr<sstables::sstable> >::destroy<seastar::lw_shared_ptr<sstables::sstable> >(seastar::lw_shared_ptr<sstables::sstable>*) at /usr/include/c++/7/ext/new_allocator.h:140
 (inlined by) void std::allocator_traits<std::allocator<seastar::lw_shared_ptr<sstables::sstable> > >::destroy<seastar::lw_shared_ptr<sstables::sstable> >(std::allocator<seastar::lw_shared_ptr<sstables::sstable> >&, seastar::lw_shared_ptr<sstables::sstable>*) at /usr/include/c++/7/bits/alloc_traits.h:487
 (inlined by) std::vector<seastar::lw_shared_ptr<sstables::sstable>, std::allocator<seastar::lw_shared_ptr<sstables::sstable> > >::_M_erase(__gnu_cxx::__normal_iterator<seastar::lw_shared_ptr<sstables::sstable>*, std::vector<seastar::lw_shared_ptr<sstables::sstable>, std::allocator<seastar::lw_shared_ptr<sstables::sstable> > > >) at /usr/include/c++/7/bits/vector.tcc:159
 (inlined by) std::vector<seastar::lw_shared_ptr<sstables::sstable>, std::allocator<seastar::lw_shared_ptr<sstables::sstable> > >::erase(__gnu_cxx::__normal_iterator<seastar::lw_shared_ptr<sstables::sstable> const*, std::vector<seastar::lw_shared_ptr<sstables::sstable>, std::allocator<seastar::lw_shared_ptr<sstables::sstable> > > >) at /usr/include/c++/7/bits/stl_vector.h:1180
 (inlined by) sstables::bag_sstable_set::erase(seastar::lw_shared_ptr<sstables::sstable>) at /jenkins/workspace/scylla-master/dtest/label/scylla-dtest/mode/release/smp/2/scylla/sstables/compaction_strategy.cc:201
sstables::sstable_set::erase(seastar::lw_shared_ptr<sstables::sstable>) at /jenkins/workspace/scylla-master/dtest/label/scylla-dtest/mode/release/smp/2/scylla/sstables/compaction_strategy.cc:156
sstables::regular_compaction::maybe_replace_exhausted_sstables()::{lambda(seastar::lw_shared_ptr<sstables::sstable>&)#3}::operator()(seastar::lw_shared_ptr<sstables::sstable>&) const at /jenkins/workspace/scylla-master/dtest/label/scylla-dtest/mode/release/smp/2/scylla/sstables/compaction.cc:658
 (inlined by) sstables::regular_compaction::maybe_replace_exhausted_sstables()::{lambda(seastar::lw_shared_ptr<sstables::sstable>&)#3} std::for_each<__gnu_cxx::__normal_iterator<seastar::lw_shared_ptr<sstables::sstable>*, std::vector<seastar::lw_shared_ptr<sstables::sstable>, std::allocator<seastar::lw_shared_ptr<sstables::sstable> > > >, sstables::regular_compaction::maybe_replace_exhausted_sstables()::{lambda(seastar::lw_shared_ptr<sstables::sstable>&)#3}>(__gnu_cxx::__normal_iterator<seastar::lw_shared_ptr<sstables::sstable>*, std::vector<seastar::lw_shared_ptr<sstables::sstable>, std::allocator<seastar::lw_shared_ptr<sstables::sstable> > > >, sstables::regular_compaction::maybe_replace_exhausted_sstables()::{lambda(seastar::lw_shared_ptr<sstables::sstable>&)#3}, sstables::regular_compaction::maybe_replace_exhausted_sstables()::{lambda(seastar::lw_shared_ptr<sstables::sstable>&)#3}) at /usr/include/c++/7/bits/stl_algo.h:3884
 (inlined by) sstables::regular_compaction::maybe_replace_exhausted_sstables() at /jenkins/workspace/scylla-master/dtest/label/scylla-dtest/mode/release/smp/2/scylla/sstables/compaction.cc:656
sstables::regular_compaction::stop_sstable_writer() at /jenkins/workspace/scylla-master/dtest/label/scylla-dtest/mode/release/smp/2/scylla/sstables/compaction.cc:601
 (inlined by) sstables::regular_compaction::finish_sstable_writer() at /jenkins/workspace/scylla-master/dtest/label/scylla-dtest/mode/release/smp/2/scylla/sstables/compaction.cc:607
sstables::compacting_sstable_writer::consume_end_of_stream() at /jenkins/workspace/scylla-master/dtest/label/scylla-dtest/mode/release/smp/2/scylla/sstables/compaction.cc:507
 (inlined by) auto compact_mutation_state<(emit_only_live_rows)0, (compact_for_sstables)1>::consume_end_of_stream<sstables::compacting_sstable_writer>(sstables::compacting_sstable_writer&) at /jenkins/workspace/scylla-master/dtest/label/scylla-dtest/mode/release/smp/2/scylla/./mutation_compactor.hh:292
 (inlined by) compact_mutation<(emit_only_live_rows)0, (compact_for_sstables)1, sstables::compacting_sstable_writer>::consume_end_of_stream() at /jenkins/workspace/scylla-master/dtest/label/scylla-dtest/mode/release/smp/2/scylla/./mutation_compactor.hh:397
 (inlined by) stable_flattened_mutations_consumer<compact_for_compaction<sstables::compacting_sstable_writer> >::consume_end_of_stream() at /jenkins/workspace/scylla-master/dtest/label/scylla-dtest/mode/release/smp/2/scylla/./mutation_reader.hh:366
 (inlined by) auto flat_mutation_reader::impl::consume_in_thread<stable_flattened_mutations_consumer<compact_for_compaction<sstables::compacting_sstable_writer> >, std::function<bool (dht::decorated_key const&)> >(stable_flattened_mutations_consumer<compact_for_compaction<sstables::compacting_sstable_writer> >, std::function<bool (dht::decorated_key const&)>, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at /jenkins/workspace/scylla-master/dtest/label/scylla-dtest/mode/release/smp/2/scylla/./flat_mutation_reader.hh:288
 (inlined by) auto flat_mutation_reader::consume_in_thread<stable_flattened_mutations_consumer<compact_for_compaction<sstables::compacting_sstable_writer> >, std::function<bool (dht::decorated_key const&)> >(stable_flattened_mutations_consumer<compact_for_compaction<sstables::compacting_sstable_writer> >, std::function<bool (dht::decorated_key const&)>, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at /jenkins/workspace/scylla-master/dtest/label/scylla-dtest/mode/release/smp/2/scylla/./flat_mutation_reader.hh:368
 (inlined by) operator() at /jenkins/workspace/scylla-master/dtest/label/scylla-dtest/mode/release/smp/2/scylla/sstables/compaction.cc:863
 (inlined by) apply at /jenkins/workspace/scylla-master/dtest/label/scylla-dtest/mode/release/smp/2/scylla/seastar/include/seastar/core/apply.hh:35
 (inlined by) apply<sstables::compaction::run(std::unique_ptr<sstables::compaction>)::<lambda()> > at /jenkins/workspace/scylla-master/dtest/label/scylla-dtest/mode/release/smp/2/scylla/seastar/include/seastar/core/apply.hh:43
 (inlined by) apply<sstables::compaction::run(std::unique_ptr<sstables::compaction>)::<lambda()> > at /jenkins/workspace/scylla-master/dtest/label/scylla-dtest/mode/release/smp/2/scylla/seastar/include/seastar/core/future.hh:1309
 (inlined by) operator() at /jenkins/workspace/scylla-master/dtest/label/scylla-dtest/mode/release/smp/2/scylla/seastar/include/seastar/core/thread.hh:315
 (inlined by) _M_invoke at /usr/include/c++/7/bits/std_function.h:316
std::function<void ()>::operator()() const at /usr/include/c++/7/bits/std_function.h:706
 (inlined by) seastar::thread_context::main() at /jenkins/workspace/scylla-master/dtest/label/scylla-dtest/mode/release/smp/2/scylla/seastar/src/core/thread.cc:313

@bhalevy

This comment has been minimized.

Copy link
Contributor

commented Jan 27, 2019

See in partitioner_tests.TestPartitioner.random_partitioner_test:

$ addr2line -Cfpi -e archive/urchin-dtest/logs-release.2/scylla 
  0x00000000041c0782
  0x00000000040d9a8c
  0x00000000040d9d35
  0x00000000040d9d83
  /lib64/libpthread.so.0+0x00000000000121af
  0x00000000013c4ad1
  0x00000000013a831d
  0x0000000001393b07
  0x0000000001396f48
  0x000000000137a9ec
  0x00000000005af065
...
sstables::bag_sstable_set::erase(seastar::lw_shared_ptr<sstables::sstable>) at /jenkins/workspace/scylla-master/dtest/label/scylla-dtest/mode/release/smp/2/scylla/seastar/include/seastar/core/shared_ptr.hh:299
 (inlined by) ?? at /usr/include/c++/7/ext/new_allocator.h:140
 (inlined by) ?? at /usr/include/c++/7/bits/alloc_traits.h:487
 (inlined by) ?? at /usr/include/c++/7/bits/vector.tcc:159
 (inlined by) ?? at /usr/include/c++/7/bits/stl_vector.h:1180
 (inlined by) sstables::bag_sstable_set::erase(seastar::lw_shared_ptr<sstables::sstable>) at /jenkins/workspace/scylla-master/dtest/label/scylla-dtest/mode/release/smp/2/scylla/sstables/compaction_strategy.cc:201
sstables::sstable_set::erase(seastar::lw_shared_ptr<sstables::sstable>) at /jenkins/workspace/scylla-master/dtest/label/scylla-dtest/mode/release/smp/2/scylla/sstables/compaction_strategy.cc:156
sstables::regular_compaction::maybe_replace_exhausted_sstables()::{lambda(seastar::lw_shared_ptr<sstables::sstable>&)#3}::operator()(seastar::lw_shared_ptr<sstables::sstable>&) const at /jenkins/workspace/scylla-master/dtest/label/scylla-dtest/mode/release/smp/2/scylla/sstables/compaction.cc:665
 (inlined by) ?? at /usr/include/c++/7/bits/stl_algo.h:3884
 (inlined by) sstables::regular_compaction::maybe_replace_exhausted_sstables() at /jenkins/workspace/scylla-master/dtest/label/scylla-dtest/mode/release/smp/2/scylla/sstables/compaction.cc:663
sstables::regular_compaction::finish_sstable_writer() at /jenkins/workspace/scylla-master/dtest/label/scylla-dtest/mode/release/smp/2/scylla/sstables/compaction.cc:608
 (inlined by) sstables::regular_compaction::finish_sstable_writer() at /jenkins/workspace/scylla-master/dtest/label/scylla-dtest/mode/release/smp/2/scylla/sstables/compaction.cc:614
std::_Function_handler<void (), seastar::async<sstables::compaction::run(std::unique_ptr<sstables::compaction, std::default_delete<sstables::compaction> >)::{lambda()#1}>(seastar::thread_attributes, std::decay&&, (std::decay<sstables::compaction::run(std::unique_ptr<sstables::compaction, std::default_delete<sstables::compaction> >)::{lambda()#1}>::type&&)...)::{lambda(seastar::async<{lambda()#1}>(seastar::futurize<std::result_of<std::decay<auto:1>::type ()>::type>::type, seastar::thread_attributes, std::decay<auto:1>::type&&)::work&)#1}::operator()(seastar::async<{lambda()#1}>(seastar::futurize<std::result_of<std::decay<{lambda()#1}>::type ()>::type>::type, seastar::thread_attributes, std::decay<{lambda()#1}>::type&&)::work)::{lambda()#1}>::_M_invoke(std::_Any_data const&) at /jenkins/workspace/scylla-master/dtest/label/scylla-dtest/mode/release/smp/2/scylla/sstables/compaction.cc:515
@bhalevy

This comment has been minimized.

Copy link
Contributor

commented Jan 27, 2019

Possibly related backtrace seen in compaction_additional_test.CompactionAdditionalTest.compaction_removes_ttld_data_by_time_windows_test
and compaction_additional_test.CompactionAdditionalTest.compaction_delete_with_smp_change_test

$ addr2line -Cfpi -e archive/urchin-dtest/logs-release.2/scylla 
  0x00000000013af9a7
  0x0000000001383c53
  0x0000000001384008
  0x0000000001368b3b
  0x00000000040ba39e
  0x00000000040ba561
  0x000000000418c247
  0x00000000041b4cea
  0x0000000004215bed
sstables::bag_sstable_set::~bag_sstable_set() at /jenkins/workspace/scylla-master/dtest/label/scylla-dtest/mode/release/smp/2/scylla/seastar/include/seastar/core/shared_ptr.hh:299
 (inlined by) ?? at /usr/include/c++/7/bits/stl_construct.h:98
 (inlined by) ?? at /usr/include/c++/7/bits/stl_construct.h:108
 (inlined by) ?? at /usr/include/c++/7/bits/stl_construct.h:137
 (inlined by) ?? at /usr/include/c++/7/bits/stl_construct.h:206
 (inlined by) ?? at /usr/include/c++/7/bits/stl_vector.h:434
 (inlined by) ?? at /jenkins/workspace/scylla-master/dtest/label/scylla-dtest/mode/release/smp/2/scylla/sstables/compaction_strategy.cc:187
 (inlined by) sstables::bag_sstable_set::~bag_sstable_set() at /jenkins/workspace/scylla-master/dtest/label/scylla-dtest/mode/release/smp/2/scylla/sstables/compaction_strategy.cc:187
sstables::regular_compaction::~regular_compaction() at /jenkins/workspace/scylla-master/dtest/label/scylla-dtest/mode/release/smp/2/scylla/seastar/include/seastar/core/shared_ptr.hh:157
 (inlined by) ?? at /jenkins/workspace/scylla-master/dtest/label/scylla-dtest/mode/release/smp/2/scylla/seastar/include/seastar/core/shared_ptr.hh:213
 (inlined by) ?? at /jenkins/workspace/scylla-master/dtest/label/scylla-dtest/mode/release/smp/2/scylla/seastar/include/seastar/core/shared_ptr.hh:300
 (inlined by) ?? at /jenkins/workspace/scylla-master/dtest/label/scylla-dtest/mode/release/smp/2/scylla/sstables/compaction.cc:374
 (inlined by) sstables::regular_compaction::~regular_compaction() at /jenkins/workspace/scylla-master/dtest/label/scylla-dtest/mode/release/smp/2/scylla/sstables/compaction.cc:518
sstables::regular_compaction::~regular_compaction() at /jenkins/workspace/scylla-master/dtest/label/scylla-dtest/mode/release/smp/2/scylla/sstables/compaction.cc:518
seastar::internal::do_with_state<seastar::async<sstables::compaction::run(std::unique_ptr<sstables::compaction, std::default_delete<sstables::compaction> >)::{lambda()#1}>(seastar::thread_attributes, std::decay&&, (std::decay<sstables::compaction::run(std::unique_ptr<sstables::compaction, std::default_delete<sstables::compaction> >)::{lambda()#1}>::type&&)...)::work, seastar::future<sstables::compaction_info> >::run_and_dispose() at /usr/include/c++/7/bits/unique_ptr.h:78
 (inlined by) ?? at /usr/include/c++/7/bits/unique_ptr.h:268
 (inlined by) ~<lambda> at /jenkins/workspace/scylla-master/dtest/label/scylla-dtest/mode/release/smp/2/scylla/sstables/compaction.cc:856

pdziepak added a commit that referenced this issue Jan 30, 2019

sstables/compaction: Fix segfault when replacing expired sstable in i…
…ncremental compaction

Fully expired sstable is not added to compacting set, meaning it's not actually
compacted, but it's kept in a list of sstables which incremental compaction
uses to check if any sstable can be replaced.
Incremental compaction was unconditionally removing expired sstable from compacting
set, which led to segfault because end iterator was given.

The fix is about changing sstable_set::erase() behavior to follow standard one
for erase functions which will works if the target element is not present.

Fixes #4085.

Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
Message-Id: <20190129152620.28183-1-raphaelsc@scylladb.com>

@slivne slivne reopened this Jan 30, 2019

@slivne

This comment has been minimized.

Copy link
Contributor

commented Jan 30, 2019

the fix was for the comapction part.

calle - still needs to look at the commitlog issues - keeping this open for this

avikivity added a commit that referenced this issue Feb 12, 2019

commitlog: Accept recycled (not yet re-used) segments in replay
Refs #4085

Changes commitlog descriptor to both accept "Recycled-Commitlog..."
file names, and preserve said name in the descriptor.

This ensures we pick up the not-yet-used recycled segments left
from a crash for replay. The replay in turn will simply ignore
the recycled files, and post actual replay they will be deleted
as needed.

Message-Id: <20190129123311.16050-1-calle@scylladb.com>
@slivne

This comment has been minimized.

Copy link
Contributor

commented Feb 12, 2019

both issues have been fixed closing the issue

@slivne slivne closed this Feb 12, 2019

avikivity pushed a commit that referenced this issue Feb 12, 2019

Calle Wilund
commitlog: Accept recycled (not yet re-used) segments in replay
Refs #4085

Changes commitlog descriptor to both accept "Recycled-Commitlog..."
file names, and preserve said name in the descriptor.

This ensures we pick up the not-yet-used recycled segments left
from a crash for replay. The replay in turn will simply ignore
the recycled files, and post actual replay they will be deleted
as needed.

avikivity added a commit that referenced this issue Jul 22, 2019

sstables/compaction: Fix segfault when replacing expired sstable in i…
…ncremental compaction

Fully expired sstable is not added to compacting set, meaning it's not actually
compacted, but it's kept in a list of sstables which incremental compaction
uses to check if any sstable can be replaced.
Incremental compaction was unconditionally removing expired sstable from compacting
set, which led to segfault because end iterator was given.

The fix is about changing sstable_set::erase() behavior to follow standard one
for erase functions which will works if the target element is not present.

Fixes #4085.

Signed-off-by: Raphael S. Carvalho <raphaelsc@scylladb.com>
Message-Id: <20190130163100.5824-1-raphaelsc@scylladb.com>
(cherry picked from commit 930f8ca)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.