Skip to content

Commit

Permalink
repair: Avoid stall in working_row_hashes
Browse files Browse the repository at this point in the history
Fix the following stall during repair:

```
Reactor stalled for 1 ms on shard 0. Backtrace:
[Backtrace scylladb#11]
{build/release/scylla} 0x4c6deb2: void seastar::backtrace<seastar::backtrace_buffer::append_backtrace_oneline()::{lambda(seastar::frame)scylladb#1}>(seastar::backtrace_buffer::append_backtrace_oneline()::{lambda(seastar::frame)scylladb#1}&&) at ./
 (inlined by) seastar::backtrace_buffer::append_backtrace_oneline() at ./build/release/seastar/./seastar/src/core/reactor.cc:772
 (inlined by) seastar::print_with_backtrace(seastar::backtrace_buffer&, bool) at ./build/release/seastar/./seastar/src/core/reactor.cc:791
{build/release/scylla} 0x4c6cb10: seastar::internal::cpu_stall_detector::generate_trace() at ./build/release/seastar/./seastar/src/core/reactor.cc:1366
{build/release/scylla} 0x4c6ddc0: seastar::internal::cpu_stall_detector::maybe_report() at ./build/release/seastar/./seastar/src/core/reactor.cc:1108
 (inlined by) seastar::internal::cpu_stall_detector::on_signal() at ./build/release/seastar/./seastar/src/core/reactor.cc:1125
 (inlined by) seastar::reactor::block_notifier(int) at ./build/release/seastar/./seastar/src/core/reactor.cc:1349
{build/release/scylla} 0x7f75551bfa1f: ?? ??:0
{build/release/scylla} 0x37abf12: repair_hash::operator<(repair_hash const&) const at ././repair/hash.hh:30
 (inlined by) std::less<repair_hash>::operator()(repair_hash const&, repair_hash const&) const at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/stl_function.h:400
 (inlined by) bool absl::container_internal::key_compare_adapter<std::less<repair_hash>, repair_hash>::checked_compare::operator()<repair_hash, repair_hash, 0>(repair_hash const&, repair_hash const&) const at ./abseil/absl/containe
 (inlined by) absl::container_internal::SearchResult<int, false> absl::container_internal::btree_node<absl::container_internal::set_params<repair_hash, std::less<repair_hash>, std::allocator<repair_hash>, 256, false> >::binary_sear
 (inlined by) _ZNK4absl18container_internal10btree_nodeINS0_10set_paramsI11repair_hashSt4lessIS3_ESaIS3_ELi256ELb0EEEE13binary_searchIS3_NS0_19key_compare_adapterIS5_S3_E15checked_compareEEENS0_12SearchResultIiXsr23btree_is_key_com (inlined by) absl::container_internal::SearchResult<int, false> absl::container_internal::btree_node<absl::container_internal::set_params<repair_hash, std::less<repair_hash>, std::allocator<repair_hash>, 256, false> >::lower_bound
 (inlined by) absl::container_internal::SearchResult<absl::container_internal::btree_iterator<absl::container_internal::btree_node<absl::container_internal::set_params<repair_hash, std::less<repair_hash>, std::allocator<repair_hash (inlined by) std::pair<absl::container_internal::btree_iterator<absl::container_internal::btree_node<absl::container_internal::set_params<repair_hash, std::less<repair_hash>, std::allocator<repair_hash>, 256, false> >, repair_hash (inlined by) std::pair<absl::container_internal::btree_iterator<absl::container_internal::btree_node<absl::container_internal::set_params<repair_hash, std::less<repair_hash>, std::allocator<repair_hash>, 256, false> >, repair_hash
 (inlined by) operator() at ./repair/row_level.cc:896
 (inlined by) seastar::future<void> seastar::futurize<void>::invoke<repair_meta::working_row_hashes()::{lambda(absl::btree_set<repair_hash, std::less<repair_hash>, std::allocator<repair_hash> >&)scylladb#1}::operator()(absl::btree_set<repa
 (inlined by) auto seastar::futurize_invoke<repair_meta::working_row_hashes()::{lambda(absl::btree_set<repair_hash, std::less<repair_hash>, std::allocator<repair_hash> >&)scylladb#1}::operator()(absl::btree_set<repair_hash, std::less<repai{build/release/scylla} 0x37ac70f: seastar::internal::do_for_each_state<std::_List_iterator<repair_row>, repair_meta::working_row_hashes()::{lambda(absl::btree_set<repair_hash, std::less<repair_hash>, std::allocator<repair_hash> >&)
{build/release/scylla} 0x4c7ee64: seastar::reactor::run_tasks(seastar::reactor::task_queue&) at ./build/release/seastar/./seastar/src/core/reactor.cc:2356
 (inlined by) seastar::reactor::run_some_tasks() at ./build/release/seastar/./seastar/src/core/reactor.cc:2769
{build/release/scylla} 0x4c80247: seastar::reactor::do_run() at ./build/release/seastar/./seastar/src/core/reactor.cc:2938
{build/release/scylla} 0x4c7f49c: seastar::reactor::run() at ./build/release/seastar/./seastar/src/core/reactor.cc:2821
{build/release/scylla} 0x4c264d8: seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) at ./build/release/seastar/./seastar/src/core/app-template.cc:265
{build/release/scylla} 0x4c259b1: seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) at ./build/release/seastar/./seastar/src/core/app-template.cc:156
{build/release/scylla} 0xf5c16f: scylla_main(int, char**) at ./main.cc:535
{build/release/scylla} 0xf5999a: std::function<int (int, char**)>::operator()(int, char**) const at /usr/lib/gcc/x86_64-redhat-linux/11/../../../../include/c++/11/bits/std_function.h:590
 (inlined by) main at ./main.cc:1575
{build/release/scylla} 0x27b74: ?? ??:0
{build/release/scylla} 0xf5892d: _start at ??:?
```

Found with scylla --blocked-reactor-notify-ms 1

Refs scylladb#10665
  • Loading branch information
asias committed Jun 7, 2022
1 parent 45bcacf commit f2c05e2
Showing 1 changed file with 2 additions and 2 deletions.
4 changes: 2 additions & 2 deletions repair/row_level.cc
Original file line number Diff line number Diff line change
Expand Up @@ -892,9 +892,9 @@ class repair_meta {
future<repair_hash_set>
working_row_hashes() {
return do_with(repair_hash_set(), [this] (repair_hash_set& hashes) {
return do_for_each(_working_row_buf, [&hashes] (repair_row& r) {
return do_for_each(_working_row_buf, [&hashes] (repair_row& r) mutable {
hashes.emplace(r.hash());
}).then([&hashes] {
}).then([&hashes] () mutable {
return std::move(hashes);
});
});
Expand Down

0 comments on commit f2c05e2

Please sign in to comment.