Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Assertion `_closing_state == state::closed' failed during add_dc_during_mv_insert_test #4948

Closed
bhalevy opened this issue Sep 3, 2019 · 4 comments
Assignees

Comments

@bhalevy
Copy link
Contributor

@bhalevy bhalevy commented Sep 3, 2019

scylla version 6c4ad93
scylladb/scylla-dtest@2846bff

dtest-release/231/artifact/logs-release.2/1567422929197_materialized_views_test.TestMaterializedViews.add_dc_during_mv_insert_test/node5.log:

ERROR 2019-09-02 11:14:11,630 [shard 1] storage_proxy - exception during mutation write to 127.0.49.5: std::bad_alloc (std::bad_alloc)
ERROR 2019-09-02 11:14:11,630 [shard 1] system_keyspace - update compaction history failed: exceptions::mutation_write_failure_exception (Operation failed for system.compaction_history - received 0 responses and 1 failures from 1 CL=ONE.): ignored
scylla: /jenkins/workspace/scylla-master/dtest-release/scylla/seastar/src/core/reactor.cc:2631: virtual seastar::append_challenged_posix_file_impl::~append_challenged_posix_file_impl(): Assertion `_closing_state == state::closed' failed.
Aborting on shard 0.
Backtrace:
  0x00000000028dcd02
  0x00000000027fa92b
  0x00000000027fac25
  0x00000000027facc0
  0x00007f7587326e7f
  /jenkins/workspace/scylla-master/dtest-release/scylla-dtest/../scylla/dynamic_libs/libc.so.6+0x0000000000037e74
  /jenkins/workspace/scylla-master/dtest-release/scylla-dtest/../scylla/dynamic_libs/libc.so.6+0x0000000000022894
  /jenkins/workspace/scylla-master/dtest-release/scylla-dtest/../scylla/dynamic_libs/libc.so.6+0x0000000000022768
  /jenkins/workspace/scylla-master/dtest-release/scylla-dtest/../scylla/dynamic_libs/libc.so.6+0x0000000000030565
  0x000000000282f12f
  0x00000000009656f8
  0x00000000027aeed1
  0x0000000000980a94
  0x0000000000d53b89
  0x0000000000db4735
  0x0000000000ca3a4c
  0x000000000070ea23
non-virtual thunk to seastar::append_challenged_posix_file_impl::~append_challenged_posix_file_impl() at /usr/include/boost/container/vector.hpp:1605
seastar::shared_ptr_count_for<checked_file_impl>::~shared_ptr_count_for() at /jenkins/workspace/scylla-master/dtest-release/scylla/seastar/include/seastar/core/shared_ptr.hh:535
 (inlined by) ?? at /jenkins/workspace/scylla-master/dtest-release/scylla/seastar/include/seastar/core/file.hh:155
 (inlined by) ?? at /jenkins/workspace/scylla-master/dtest-release/scylla/checked-file-impl.hh:27
 (inlined by) ?? at /jenkins/workspace/scylla-master/dtest-release/scylla/seastar/include/seastar/core/shared_ptr.hh:463
 (inlined by) seastar::shared_ptr_count_for<checked_file_impl>::~shared_ptr_count_for() at /jenkins/workspace/scylla-master/dtest-release/scylla/seastar/include/seastar/core/shared_ptr.hh:463
seastar::file_data_sink_impl::~file_data_sink_impl() at /jenkins/workspace/scylla-master/dtest-release/scylla/seastar/include/seastar/core/shared_ptr.hh:535
 (inlined by) ?? at /jenkins/workspace/scylla-master/dtest-release/scylla/seastar/include/seastar/core/file.hh:155
 (inlined by) ?? at /jenkins/workspace/scylla-master/dtest-release/scylla/seastar/src/core/fstream.cc:312
 (inlined by) seastar::file_data_sink_impl::~file_data_sink_impl() at /jenkins/workspace/scylla-master/dtest-release/scylla/seastar/src/core/fstream.cc:312
seastar::output_stream<char>::~output_stream() at /usr/include/c++/9/bits/unique_ptr.h:81
 (inlined by) ?? at /usr/include/c++/9/bits/unique_ptr.h:284
 (inlined by) ?? at /jenkins/workspace/scylla-master/dtest-release/scylla/seastar/include/seastar/core/iostream.hh:93
 (inlined by) seastar::output_stream<char>::~output_stream() at /jenkins/workspace/scylla-master/dtest-release/scylla/seastar/include/seastar/core/iostream.hh:304
void sstables::sstable::write_simple<(sstables::component_type)1, sstables::compression>(sstables::compression const&, seastar::io_priority_class const&) at /jenkins/workspace/scylla-master/dtest-release/scylla/sstables/writer.hh:51
 (inlined by) void sstables::sstable::write_simple<(sstables::component_type)1, sstables::compression>(sstables::compression const&, seastar::io_priority_class const&) at /jenkins/workspace/scylla-master/dtest-release/scylla/sstables/sstables.cc:1058
sstables::mc::writer::consume_end_of_stream() at /jenkins/workspace/scylla-master/dtest-release/scylla/sstables/mc/writer.cc:1414
seastar::noncopyable_function<void ()>::direct_vtable_for<seastar::async<sstables::sstable::write_components(flat_mutation_reader, unsigned long, seastar::lw_shared_ptr<schema const>, sstables::sstable_writer_config const&, encoding_stats, seastar::io_priority_class const&)::{lambda()#1}>(seastar::thread_attributes, std::decay&&, (std::decay<sstables::sstable::write_components(flat_mutation_reader, unsigned long, seastar::lw_shared_ptr<schema const>, sstables::sstable_writer_config const&, encoding_stats, seastar::io_priority_class const&)::{lambda()#1}>::type&&)...)::{lambda(seastar::async<{lambda()#1}>(seastar::futurize<std::result_of<std::decay<auto:1>::type ()>::type>::type, seastar::thread_attributes, std::decay<auto:1>::type&&)::work&)#1}::operator()(seastar::async<{lambda()#1}>(seastar::futurize<std::result_of<std::decay<{lambda()#1}>::type ()>::type>::type, seastar::thread_attributes, std::decay<{lambda()#1}>::type&&)::work)::{lambda()#1}>::call(seastar::noncopyable_function<void ()> const*) at /jenkins/workspace/scylla-master/dtest-release/scylla/sstables/sstables.cc:2360
 (inlined by) ?? at /jenkins/workspace/scylla-master/dtest-release/scylla/./flat_mutation_reader.hh:288
 (inlined by) ?? at /jenkins/workspace/scylla-master/dtest-release/scylla/./flat_mutation_reader.hh:368
 (inlined by) ?? at /jenkins/workspace/scylla-master/dtest-release/scylla/./flat_mutation_reader.hh:376
 (inlined by) operator() at /jenkins/workspace/scylla-master/dtest-release/scylla/sstables/sstables.cc:2415
 (inlined by) ?? at /jenkins/workspace/scylla-master/dtest-release/scylla/seastar/include/seastar/core/apply.hh:35
 (inlined by) ?? at /jenkins/workspace/scylla-master/dtest-release/scylla/seastar/include/seastar/core/apply.hh:43
 (inlined by) ?? at /jenkins/workspace/scylla-master/dtest-release/scylla/seastar/include/seastar/core/future.hh:1450
 (inlined by) ?? at /jenkins/workspace/scylla-master/dtest-release/scylla/seastar/include/seastar/core/future.hh:1484
 (inlined by) operator() at /jenkins/workspace/scylla-master/dtest-release/scylla/seastar/include/seastar/core/thread.hh:265
 (inlined by) call at /jenkins/workspace/scylla-master/dtest-release/scylla/seastar/include/seastar/util/noncopyable_function.hh:71
seastar::thread_context::main() at /jenkins/workspace/scylla-master/dtest-release/scylla/seastar/include/seastar/util/noncopyable_function.hh:145
 (inlined by) seastar::thread_context::main() at /jenkins/workspace/scylla-master/dtest-release/scylla/seastar/src/core/thread.cc:277
@bhalevy

This comment has been minimized.

Copy link
Contributor Author

@bhalevy bhalevy commented Sep 3, 2019

@espindola please look into this issue

@espindola

This comment has been minimized.

Copy link
Contributor

@espindola espindola commented Sep 3, 2019

Are you sure this is the right backtrace? The raw one shows libc.so.6 on the stack, but the decoded one does not.
Are the binary and the core file available somewhere?

@espindola

This comment has been minimized.

Copy link
Contributor

@espindola espindola commented Sep 3, 2019

Decoded backtrace:
backtrace.txt

avikivity added a commit that referenced this issue Sep 5, 2019
The previous code was not exception safe and would eventually cause a
file to be destroyed without being closed, causing an assert failure.

Unfortunately it doesn't seem to be possible to test this without
error injection, since using an invalid directory fails before this
code is executed.

Fixes #4948

Signed-off-by: Rafael Ávila de Espíndola <espindola@scylladb.com>
Message-Id: <20190904002314.79591-1-espindola@scylladb.com>
(cherry picked from commit 000514e)
elcallio pushed a commit to elcallio/scylla that referenced this issue Oct 2, 2019
The previous code was not exception safe and would eventually cause a
file to be destroyed without being closed, causing an assert failure.

Unfortunately it doesn't seem to be possible to test this without
error injection, since using an invalid directory fails before this
code is executed.

Fixes scylladb#4948

Signed-off-by: Rafael Ávila de Espíndola <espindola@scylladb.com>
Message-Id: <20190904002314.79591-1-espindola@scylladb.com>
avikivity added a commit that referenced this issue Nov 17, 2019
The previous code was not exception safe and would eventually cause a
file to be destroyed without being closed, causing an assert failure.

Unfortunately it doesn't seem to be possible to test this without
error injection, since using an invalid directory fails before this
code is executed.

Fixes #4948

Signed-off-by: Rafael Ávila de Espíndola <espindola@scylladb.com>
Message-Id: <20190904002314.79591-1-espindola@scylladb.com>
(cherry picked from commit 000514e)
@avikivity

This comment has been minimized.

Copy link
Contributor

@avikivity avikivity commented Nov 17, 2019

Backported to 3.0 (3.1 was already fixed)

avikivity added a commit that referenced this issue Nov 17, 2019
The previous code was not exception safe and would eventually cause a
file to be destroyed without being closed, causing an assert failure.

Unfortunately it doesn't seem to be possible to test this without
error injection, since using an invalid directory fails before this
code is executed.

Fixes #4948

Signed-off-by: Rafael Ávila de Espíndola <espindola@scylladb.com>
Message-Id: <20190904002314.79591-1-espindola@scylladb.com>
(cherry picked from commit 000514e)
avikivity added a commit that referenced this issue Nov 19, 2019
The previous code was not exception safe and would eventually cause a
file to be destroyed without being closed, causing an assert failure.

Unfortunately it doesn't seem to be possible to test this without
error injection, since using an invalid directory fails before this
code is executed.

Fixes #4948

Signed-off-by: Rafael Ávila de Espíndola <espindola@scylladb.com>
Message-Id: <20190904002314.79591-1-espindola@scylladb.com>
(cherry picked from commit 000514e)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.