Skip to content

Commit

Permalink
repair: Avoid stall in apply_rows_on_master_in_thread
Browse files Browse the repository at this point in the history
Fix the following stall during repair:

```
Reactor stalled for 3 ms on shard 0. Backtrace:
[Backtrace scylladb#20]
{build/release/scylla} 0x4c6deb2: void seastar::backtrace<seastar::backtrace_buffer::append_backtrace_oneline()::{lambda(seastar::frame)scylladb#1}>(seastar::backtrace_buffer::append_backtrace_oneline()::{lambda(seastar::frame)scylladb#1}&&) at ./build/release/seastar/./seastar/include/seastar/util/backtrace.hh:59
 (inlined by) seastar::backtrace_buffer::append_backtrace_oneline() at ./build/release/seastar/./seastar/src/core/reactor.cc:772
 (inlined by) seastar::print_with_backtrace(seastar::backtrace_buffer&, bool) at ./build/release/seastar/./seastar/src/core/reactor.cc:791
{build/release/scylla} 0x4c6cb10: seastar::internal::cpu_stall_detector::generate_trace() at ./build/release/seastar/./seastar/src/core/reactor.cc:1366
{build/release/scylla} 0x4c6ddc0: seastar::internal::cpu_stall_detector::maybe_report() at ./build/release/seastar/./seastar/src/core/reactor.cc:1108
 (inlined by) seastar::internal::cpu_stall_detector::on_signal() at ./build/release/seastar/./seastar/src/core/reactor.cc:1125
 (inlined by) seastar::reactor::block_notifier(int) at ./build/release/seastar/./seastar/src/core/reactor.cc:1349
{build/release/scylla} 0x7f75551bfa1f: ?? ??:0
{build/release/scylla} 0x11293e9: std::default_delete<bytes_ostream::chunk>::operator()(bytes_ostream::chunk*) const at database.cc:?
 (inlined by) std::default_delete<bytes_ostream::chunk>::operator()(bytes_ostream::chunk*) const at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/unique_ptr.h:85
{build/release/scylla} 0x37b18e6: ~unique_ptr at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/unique_ptr.h:361
 (inlined by) ~bytes_ostream at ././bytes_ostream.hh:26
 (inlined by) ~frozen_mutation_fragment at ././frozen_mutation.hh:265
 (inlined by) std::_Optional_payload_base<frozen_mutation_fragment>::_M_destroy() at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/optional:260
 (inlined by) std::_Optional_payload_base<frozen_mutation_fragment>::_M_reset() at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/optional:280
 (inlined by) ~_Optional_payload at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/optional:401
 (inlined by) ~_Optional_base at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/optional:472
 (inlined by) ~repair_row at ././repair/row.hh:24
 (inlined by) void std::destroy_at<repair_row>(repair_row*) at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/stl_construct.h:88
 (inlined by) void std::allocator_traits<std::allocator<std::_List_node<repair_row> > >::destroy<repair_row>(std::allocator<std::_List_node<repair_row> >&, repair_row*) at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/alloc_traits.h:537
 (inlined by) std::__cxx11::_List_base<repair_row, std::allocator<repair_row> >::_M_clear() at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/list.tcc:77
 (inlined by) ~_List_base at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/stl_list.h:499
 (inlined by) repair_meta::apply_rows_on_master_in_thread(std::__cxx11::list<partition_key_and_mutation_fragments, std::allocator<partition_key_and_mutation_fragments> >, gms::inet_address, seastar::bool_class<update_working_row_buf_tag>, seastar::bool_class<update_peer_row_hash_sets_tag>, unsigned int) at ./repair/row_level.cc:1273
{build/release/scylla} 0x37ad9dc: repair_meta::get_row_diff_source_op(seastar::bool_class<update_peer_row_hash_sets_tag>, gms::inet_address, unsigned int, seastar::rpc::sink<repair_hash_with_cmd>&, seastar::rpc::source<repair_row_on_wire_with_cmd>&) at ./repair/row_level.cc:1617
{build/release/scylla} 0x37a2982: repair_meta::get_row_diff_with_rpc_stream(absl::btree_set<repair_hash, std::less<repair_hash>, std::allocator<repair_hash> >, seastar::bool_class<needs_all_rows_tag>, seastar::bool_class<update_peer_row_hash_sets_tag>, gms::inet_address, unsigned int) at ./repair/row_level.cc:1683
```

Found with scylla --blocked-reactor-notify-ms 1

Refs scylladb#10665
  • Loading branch information
asias committed Jun 7, 2022
1 parent adda43e commit 45bcacf
Showing 1 changed file with 2 additions and 0 deletions.
2 changes: 2 additions & 0 deletions repair/row_level.cc
Original file line number Diff line number Diff line change
Expand Up @@ -1270,6 +1270,8 @@ class repair_meta {
// Repair rows in row_diff will be flushed to disk by flush_rows_in_working_row_buf,
// so we skip calling do_apply_rows here.
_dirty_on_master = is_dirty_on_master::yes;
// Clear gently to avoid stalls
utils::clear_gently(row_diff).get();
}
public:
// Must run inside a seastar thread
Expand Down

0 comments on commit 45bcacf

Please sign in to comment.