Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
repair: Avoid stall in apply_rows_on_master_in_thread
Fix the following stall during repair: ``` Reactor stalled for 3 ms on shard 0. Backtrace: [Backtrace scylladb#20] {build/release/scylla} 0x4c6deb2: void seastar::backtrace<seastar::backtrace_buffer::append_backtrace_oneline()::{lambda(seastar::frame)scylladb#1}>(seastar::backtrace_buffer::append_backtrace_oneline()::{lambda(seastar::frame)scylladb#1}&&) at ./build/release/seastar/./seastar/include/seastar/util/backtrace.hh:59 (inlined by) seastar::backtrace_buffer::append_backtrace_oneline() at ./build/release/seastar/./seastar/src/core/reactor.cc:772 (inlined by) seastar::print_with_backtrace(seastar::backtrace_buffer&, bool) at ./build/release/seastar/./seastar/src/core/reactor.cc:791 {build/release/scylla} 0x4c6cb10: seastar::internal::cpu_stall_detector::generate_trace() at ./build/release/seastar/./seastar/src/core/reactor.cc:1366 {build/release/scylla} 0x4c6ddc0: seastar::internal::cpu_stall_detector::maybe_report() at ./build/release/seastar/./seastar/src/core/reactor.cc:1108 (inlined by) seastar::internal::cpu_stall_detector::on_signal() at ./build/release/seastar/./seastar/src/core/reactor.cc:1125 (inlined by) seastar::reactor::block_notifier(int) at ./build/release/seastar/./seastar/src/core/reactor.cc:1349 {build/release/scylla} 0x7f75551bfa1f: ?? ??:0 {build/release/scylla} 0x11293e9: std::default_delete<bytes_ostream::chunk>::operator()(bytes_ostream::chunk*) const at database.cc:? (inlined by) std::default_delete<bytes_ostream::chunk>::operator()(bytes_ostream::chunk*) const at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/unique_ptr.h:85 {build/release/scylla} 0x37b18e6: ~unique_ptr at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/unique_ptr.h:361 (inlined by) ~bytes_ostream at ././bytes_ostream.hh:26 (inlined by) ~frozen_mutation_fragment at ././frozen_mutation.hh:265 (inlined by) std::_Optional_payload_base<frozen_mutation_fragment>::_M_destroy() at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/optional:260 (inlined by) std::_Optional_payload_base<frozen_mutation_fragment>::_M_reset() at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/optional:280 (inlined by) ~_Optional_payload at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/optional:401 (inlined by) ~_Optional_base at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/optional:472 (inlined by) ~repair_row at ././repair/row.hh:24 (inlined by) void std::destroy_at<repair_row>(repair_row*) at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/stl_construct.h:88 (inlined by) void std::allocator_traits<std::allocator<std::_List_node<repair_row> > >::destroy<repair_row>(std::allocator<std::_List_node<repair_row> >&, repair_row*) at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/alloc_traits.h:537 (inlined by) std::__cxx11::_List_base<repair_row, std::allocator<repair_row> >::_M_clear() at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/list.tcc:77 (inlined by) ~_List_base at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/stl_list.h:499 (inlined by) repair_meta::apply_rows_on_master_in_thread(std::__cxx11::list<partition_key_and_mutation_fragments, std::allocator<partition_key_and_mutation_fragments> >, gms::inet_address, seastar::bool_class<update_working_row_buf_tag>, seastar::bool_class<update_peer_row_hash_sets_tag>, unsigned int) at ./repair/row_level.cc:1273 {build/release/scylla} 0x37ad9dc: repair_meta::get_row_diff_source_op(seastar::bool_class<update_peer_row_hash_sets_tag>, gms::inet_address, unsigned int, seastar::rpc::sink<repair_hash_with_cmd>&, seastar::rpc::source<repair_row_on_wire_with_cmd>&) at ./repair/row_level.cc:1617 {build/release/scylla} 0x37a2982: repair_meta::get_row_diff_with_rpc_stream(absl::btree_set<repair_hash, std::less<repair_hash>, std::allocator<repair_hash> >, seastar::bool_class<needs_all_rows_tag>, seastar::bool_class<update_peer_row_hash_sets_tag>, gms::inet_address, unsigned int) at ./repair/row_level.cc:1683 ``` Found with scylla --blocked-reactor-notify-ms 1 Refs scylladb#10665
- Loading branch information