Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SIGSEGV when running dtest: limits_test.py:TestLimits.max_cells_test #3532

Closed
tgrabiec opened this Issue Jun 20, 2018 · 1 comment

Comments

Projects
None yet
2 participants
@tgrabiec
Copy link
Contributor

tgrabiec commented Jun 20, 2018

Installation details
Scylla version (or git commit hash): master (7827427)
Cluster size: 1
args: --smp 2 -m1G

boost::intrusive::pointer_plus_bits<boost::intrusive::compact_rbtree_node<void*>*, 1ul>::get_pointer(boost::intrusive::compact_rbtree_node<void*>*) at /usr/include/boost/intrusive/pointer_plus_bits.hpp:71
 (inlined by) boost::intrusive::compact_rbtree_node_traits_impl<void*>::get_parent(boost::intrusive::compact_rbtree_node<void*>* const&) at /usr/include/boost/intrusive/detail/rbtree_node.hpp:137
 (inlined by) boost::intrusive::bstree_algorithms_base<boost::intrusive::rbtree_node_traits<void*, true> >::next_node(boost::intrusive::compact_rbtree_node<void*>* const&) at /usr/include/boost/intrusive/detail/bstree_algorithms_base.hpp:53
 (inlined by) boost::intrusive::tree_iterator<boost::intrusive::mhtraits<rows_entry, intrusive_set_external_comparator_member_hook, &rows_entry::_link>, true>::operator++() at /usr/include/boost/intrusive/detail/tree_iterator.hpp:86
 (inlined by) void std::__advance<boost::intrusive::tree_iterator<boost::intrusive::mhtraits<rows_entry, intrusive_set_external_comparator_member_hook, &rows_entry::_link>, true>, long>(boost::intrusive::tree_iterator<boost::intrusive::mhtraits<rows_entry, intrusive_set_external_comparator_member_hook, &rows_entry::_link>, true>&, long, std::bidirectional_iterator_tag) at /usr/include/c++/7/bits/stl_iterator_base_funcs.h:166
 (inlined by) void std::advance<boost::intrusive::tree_iterator<boost::intrusive::mhtraits<rows_entry, intrusive_set_external_comparator_member_hook, &rows_entry::_link>, true>, long>(boost::intrusive::tree_iterator<boost::intrusive::mhtraits<rows_entry, intrusive_set_external_comparator_member_hook, &rows_entry::_link>, true>&, long) at /usr/include/c++/7/bits/stl_iterator_base_funcs.h:201
 (inlined by) boost::intrusive::tree_iterator<boost::intrusive::mhtraits<rows_entry, intrusive_set_external_comparator_member_hook, &rows_entry::_link>, true> std::next<boost::intrusive::tree_iterator<boost::intrusive::mhtraits<rows_entry, intrusive_set_external_comparator_member_hook, &rows_entry::_link>, true> >(boost::intrusive::tree_iterator<boost::intrusive::mhtraits<rows_entry, intrusive_set_external_comparator_member_hook, &rows_entry::_link>, true>, std::iterator_traits<boost::intrusive::tree_iterator<boost::intrusive::mhtraits<rows_entry, intrusive_set_external_comparator_member_hook, &rows_entry::_link>, true> >::difference_type) at /usr/include/c++/7/bits/stl_iterator_base_funcs.h:214
 (inlined by) partition_snapshot_flat_reader<partition_snapshot_reader_dummy_accounter>::lsa_partition_reader::pop_clustering_row() at /home/tgrabiec/src/scylla2/partition_snapshot_reader.hh:142
 (inlined by) partition_snapshot_flat_reader<partition_snapshot_reader_dummy_accounter>::lsa_partition_reader::next_row(nonwrapping_range<clustering_key_prefix> const&, std::experimental::fundamentals_v1::optional<position_in_partition> const&, range_tombstone_stream&)::{lambda()#1}::operator()() const at /home/tgrabiec/src/scylla2/partition_snapshot_reader.hh:208
_ZZZN30partition_snapshot_flat_readerI41partition_snapshot_reader_dummy_accounterE20lsa_partition_reader16in_alloc_sectionIZNS2_8next_rowERK17nonwrapping_rangeI21clustering_key_prefixERKNSt12experimental15fundamentals_v18optionalI21position_in_partitionEER22range_tombstone_streamEUlvE_EEDcOT_ENKUlvE_clEvENKUlvE_clEv at /home/tgrabiec/src/scylla2/partition_snapshot_reader.hh:105
 (inlined by) _Z29with_linearized_managed_bytesIZZN30partition_snapshot_flat_readerI41partition_snapshot_reader_dummy_accounterE20lsa_partition_reader16in_alloc_sectionIZNS3_8next_rowERK17nonwrapping_rangeI21clustering_key_prefixERKNSt12experimental15fundamentals_v18optionalI21position_in_partitionEER22range_tombstone_streamEUlvE_EEDcOT_ENKUlvE_clEvEUlvE_ENSt9result_ofIFSK_vEE4typeESL_ at /home/tgrabiec/src/scylla2/./utils/managed_bytes.hh:423
 (inlined by) _ZZN30partition_snapshot_flat_readerI41partition_snapshot_reader_dummy_accounterE20lsa_partition_reader16in_alloc_sectionIZNS2_8next_rowERK17nonwrapping_rangeI21clustering_key_prefixERKNSt12experimental15fundamentals_v18optionalI21position_in_partitionEER22range_tombstone_streamEUlvE_EEDcOT_ENKUlvE_clEv at /home/tgrabiec/src/scylla2/partition_snapshot_reader.hh:106
 (inlined by) _ZN8logalloc18allocating_section24with_reclaiming_disabledIZN30partition_snapshot_flat_readerI41partition_snapshot_reader_dummy_accounterE20lsa_partition_reader16in_alloc_sectionIZNS5_8next_rowERK17nonwrapping_rangeI21clustering_key_prefixERKNSt12experimental15fundamentals_v18optionalI21position_in_partitionEER22range_tombstone_streamEUlvE_EEDcOT_EUlvE_EEDcRNS_6regionESN_ at /home/tgrabiec/src/scylla2/./utils/logalloc.hh:706
 (inlined by) _ZN30partition_snapshot_flat_readerI41partition_snapshot_reader_dummy_accounterE20lsa_partition_reader16in_alloc_sectionIZNS2_8next_rowERK17nonwrapping_rangeI21clustering_key_prefixERKNSt12experimental15fundamentals_v18optionalI21position_in_partitionEER22range_tombstone_streamEUlvE_EEDcOT_ at /home/tgrabiec/src/scylla2/partition_snapshot_reader.hh:106
 (inlined by) partition_snapshot_flat_reader<partition_snapshot_reader_dummy_accounter>::lsa_partition_reader::next_row(nonwrapping_range<clustering_key_prefix> const&, std::experimental::fundamentals_v1::optional<position_in_partition> const&, range_tombstone_stream&) at /home/tgrabiec/src/scylla2/partition_snapshot_reader.hh:226
 (inlined by) partition_snapshot_flat_reader<partition_snapshot_reader_dummy_accounter>::read_next() at /home/tgrabiec/src/scylla2/partition_snapshot_reader.hh:258
 (inlined by) partition_snapshot_flat_reader<partition_snapshot_reader_dummy_accounter>::do_fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at /home/tgrabiec/src/scylla2/partition_snapshot_reader.hh:289
partition_snapshot_flat_reader<partition_snapshot_reader_dummy_accounter>::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#1}::operator()() const at /home/tgrabiec/src/scylla2/partition_snapshot_reader.hh:324
 (inlined by) _ZN8logalloc18allocating_section12with_reserveIZN30partition_snapshot_flat_readerI41partition_snapshot_reader_dummy_accounterE11fill_bufferENSt6chrono10time_pointIN7seastar12lowres_clockENS5_8durationIlSt5ratioILl1ELl1000EEEEEEEUlvE_EEDcOT_ at /home/tgrabiec/src/scylla2/./utils/logalloc.hh:679
 (inlined by) _ZN30partition_snapshot_flat_readerI41partition_snapshot_reader_dummy_accounterE20lsa_partition_reader12with_reserveIZNS1_11fill_bufferENSt6chrono10time_pointIN7seastar12lowres_clockENS4_8durationIlSt5ratioILl1ELl1000EEEEEEEUlvE_EEDcOT_ at /home/tgrabiec/src/scylla2/partition_snapshot_reader.hh:177
 (inlined by) partition_snapshot_flat_reader<partition_snapshot_reader_dummy_accounter>::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at /home/tgrabiec/src/scylla2/partition_snapshot_reader.hh:323
flat_mutation_reader::impl::operator()() at /home/tgrabiec/src/scylla2/flat_mutation_reader.hh:144
 (inlined by) flat_mutation_reader::operator()() at /home/tgrabiec/src/scylla2/flat_mutation_reader.hh:315
 (inlined by) operator() at /home/tgrabiec/src/scylla2/mutation_reader.cc:311
 (inlined by) apply<mutation_reader_merger::prepare_next()::<lambda(mutation_reader_merger::reader_and_last_fragment_kind)>, mutation_reader_merger::reader_and_last_fragment_kind&> at /home/tgrabiec/src/scylla2/./seastar/core/future.hh:1387
 (inlined by) futurize_apply<mutation_reader_merger::prepare_next()::<lambda(mutation_reader_merger::reader_and_last_fragment_kind)>, mutation_reader_merger::reader_and_last_fragment_kind&> at /home/tgrabiec/src/scylla2/./seastar/core/future.hh:1445
 (inlined by) parallel_for_each<__gnu_cxx::__normal_iterator<mutation_reader_merger::reader_and_last_fragment_kind*, std::vector<mutation_reader_merger::reader_and_last_fragment_kind> >, mutation_reader_merger::prepare_next()::<lambda(mutation_reader_merger::reader_and_last_fragment_kind)> > at /home/tgrabiec/src/scylla2/./seastar/core/future-util.hh:130
parallel_for_each<std::vector<mutation_reader_merger::reader_and_last_fragment_kind>&, mutation_reader_merger::prepare_next()::<lambda(mutation_reader_merger::reader_and_last_fragment_kind)> > at /home/tgrabiec/src/scylla2/./seastar/core/future-util.hh:176
 (inlined by) mutation_reader_merger::prepare_next() at /home/tgrabiec/src/scylla2/mutation_reader.cc:310
mutation_reader_merger::operator()() at /home/tgrabiec/src/scylla2/mutation_reader.cc:398
mutation_fragment_merger<mutation_reader_merger>::fetch() at /home/tgrabiec/src/scylla2/mutation_reader.cc:84
 (inlined by) mutation_fragment_merger<mutation_reader_merger>::operator()() at /home/tgrabiec/src/scylla2/mutation_reader.cc:115
 (inlined by) operator() at /home/tgrabiec/src/scylla2/mutation_reader.cc:489
repeat<combined_mutation_reader::fill_buffer(seastar::lowres_clock::time_point)::<lambda()> > at /home/tgrabiec/src/scylla2/./seastar/core/future-util.hh:281
 (inlined by) combined_mutation_reader::fill_buffer(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >) at /home/tgrabiec/src/scylla2/mutation_reader.cc:500
seastar::future<> flat_mutation_reader::impl::consume_pausable<std::reference_wrapper<flat_mutation_reader::impl::consumer_adapter<stable_flattened_mutations_consumer<compact_for_query<(emit_only_live_rows)1, querier::clustering_position_tracker<query_result_builder> > > > > >(std::reference_wrapper<flat_mutation_reader::impl::consumer_adapter<stable_flattened_mutations_consumer<compact_for_query<(emit_only_live_rows)1, querier::clustering_position_tracker<query_result_builder> > > > >, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#2}::operator()() at /home/tgrabiec/src/scylla2/flat_mutation_reader.hh:160
 (inlined by) seastar::internal::do_until_state<seastar::future<> flat_mutation_reader::impl::consume_pausable<std::reference_wrapper<flat_mutation_reader::impl::consumer_adapter<stable_flattened_mutations_consumer<compact_for_query<(emit_only_live_rows)1, querier::clustering_position_tracker<query_result_builder> > > > > >(std::reference_wrapper<flat_mutation_reader::impl::consumer_adapter<stable_flattened_mutations_consumer<compact_for_query<(emit_only_live_rows)1, querier::clustering_position_tracker<query_result_builder> > > > >, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#1}, seastar::future<> flat_mutation_reader::impl::consume_pausable<std::reference_wrapper<flat_mutation_reader::impl::consumer_adapter<stable_flattened_mutations_consumer<compact_for_query<(emit_only_live_rows)1, querier::clustering_position_tracker<query_result_builder> > > > > >(std::reference_wrapper<flat_mutation_reader::impl::consumer_adapter<stable_flattened_mutations_consumer<compact_for_query<(emit_only_live_rows)1, querier::clustering_position_tracker<query_result_builder> > > > >, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000l> > >)::{lambda()#2}>::run_and_dispose() at /home/tgrabiec/src/scylla2/./seastar/core/future-util.hh:463
seastar::reactor::run_tasks(seastar::reactor::task_queue&) at /home/tgrabiec/src/scylla2/seastar/core/reactor.cc:2609
seastar::reactor::run_some_tasks() at /home/tgrabiec/src/scylla2/seastar/core/reactor.cc:3021
seastar::reactor::run_some_tasks() at /usr/include/c++/7/chrono:377
 (inlined by) seastar::reactor::run() at /home/tgrabiec/src/scylla2/seastar/core/reactor.cc:3168
seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) at /home/tgrabiec/src/scylla2/seastar/core/app-template.cc:185
main at /home/tgrabiec/src/scylla2/main.cc:779

@tgrabiec tgrabiec added the bug label Jun 20, 2018

@tgrabiec tgrabiec self-assigned this Jun 20, 2018

@slivne slivne added this to the 2.3 milestone Jun 21, 2018

@tgrabiec

This comment has been minimized.

Copy link
Contributor Author

tgrabiec commented Jun 25, 2018

Introduced in 5b59df3

avikivity added a commit that referenced this issue Jul 1, 2018

row_cache: Fix memtable reads concurrent with cache update missing wr…
…ites

Introduced in 5b59df3.

It is incorrect to erase entries from the memtable being moved to
cache if partition update can be preempted because a later memtable
read may create a snapshot in the memtable before memtable writes for
that partition are made visible through cache. As a result the read
may miss some of the writes which were in the memtable. The code was
checking for presence of snapshots when entering the partition, but
this condition may change if update is preempted. The fix is to not
allow erasing if update is preemptible.

This also caused SIGSEGVs because we were assuming that no such
snapshots will be created and hence were not invalidating iterators on
removal of the entries, which results in undefined behavior when such
snapshots are actually created.

Fixes SIGSEGV in dtest: limits_test.py:TestLimits.max_cells_test

Fixes #3532

Message-Id: <1530129009-13716-1-git-send-email-tgrabiec@scylladb.com>
(cherry picked from commit b464b66)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.