latency degradation during decommission nodes on write workload - ~77% degradation #14537

fgelcer · 2023-07-05T19:08:08Z

Installation details
Scylla version (or git commit hash): 5.4.0~dev.20230629.f6f974cdeb11 with build-id 7afc85749bdc68e7ee32eead35d51badd480c79f
Cluster size: 3
OS (RHEL/CentOS/Ubuntu/AWS AMI): ami-0b7891d4fe168d4b1 (eu-west-1)
test_id: 4493c5fb-0803-47a0-a6b1-9e86837a2019

based on previous runs of this operation, we have a degradation of ~~77% in the latency (this run had 4.74 ms (this is the AVG of 3.32, 4.67 and 6.23, while previous one was 2.68 ms for 5.3.0~~dev, and 3.92 ms for 5.4.0~dev)

we have 308 stalls during the 3 decommission nodes, where they started and ended as:

2023-07-02T23:23:14+00:00 perf-latency-nemesis-ubuntu-db-node-4493c5fb-4     !INFO | scylla[5333]: Reactor stalled for 6 ms on shard 6. Backtrace: 0x56aa573 0x56a9920 0x56aad4f 0x3cb1f 0x568656f 0x56864bc 0x56835e2 0x5681222 0x568ebad 0x56bb654 0x56bc8d7 0x56df0a1 0x568be6a 0x8b12c 0x10cbbf
................
2023-07-03T01:56:44+00:00 perf-latency-nemesis-ubuntu-db-node-4493c5fb-5     !INFO | scylla[5377]: Reactor stalled for 6 ms on shard 3. Backtrace: 0x56aa573 0x56a9920 0x56aad4f 0x3cb1f 0x56777b7 0x567a90a 0x4103be8 0x1a41d66 0x56bb654 0x56bc8d7 0x56df0a1 0x568be6a 0x8b12c 0x10cbbf

decoded:

Processed 308 stalls lasting a total of 1848 milliseconds.
min=6 avg=6.0 median=6 p95=None p99=None p999=None max=6

This graph is printed in bottom-up order, where callees are printed first.
Use --direction=top-down to print callers first.

[level#index/out_of pct%] below denotes:
  level  - nesting level in the graph
  index  - index of node among to its siblings
  out_of - number of siblings
  pct    - percentage of total stall time of this call relative to its siblings

Command line options: --address-threshold=0x100000000 --executable=scylla/libexec/scylla --direction=bottom-up --branch-threshold=0.05 --file=write/decommissions_reactor_stalls.log

++[0#1/1 100%] addr=0x56aa573 total=1848 count=308 avg=6:
|              seastar::backtrace<seastar::backtrace_buffer::append_backtrace_oneline()::{lambda(seastar::frame)#1}> at ./build/release/seastar/./seastar/include/seastar/util/backtrace.hh:64
|              (inlined by) ?? at ./build/release/seastar/./seastar/src/core/reactor.cc:825
|              (inlined by) seastar::print_with_backtrace at ./build/release/seastar/./seastar/src/core/reactor.cc:844
++           - addr=0x56a9920:
|              seastar::internal::cpu_stall_detector::generate_trace at ./build/release/seastar/./seastar/src/core/reactor.cc:1414
++           - addr=0x56aad4f:
|              seastar::reactor::block_notifier at ./build/release/seastar/./seastar/src/core/reactor.cc:1151
|              (inlined by) ?? at ./build/release/seastar/./seastar/src/core/reactor.cc:1171
|              (inlined by) seastar::reactor::block_notifier at ./build/release/seastar/./seastar/src/core/reactor.cc:1451
++           - addr=0x3cb1f: ?? ??:0
  ++[1#1/46 41%] addr=0x1459903 total=756 count=126 avg=6:
  |              bytes_ostream::free_chain at ././bytes_ostream.hh:163
  ++           - addr=0x4103be0:
  |              bytes_ostream::~bytes_ostream at ././bytes_ostream.hh:199
  |              (inlined by) frozen_mutation_fragment::~frozen_mutation_fragment at ././mutation/frozen_mutation.hh:273
  |              (inlined by) std::destroy_at<frozen_mutation_fragment> at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/stl_construct.h:88
  |              (inlined by) ?? at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/alloc_traits.h:537
  |              (inlined by) ?? at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/list.tcc:77
  |              (inlined by) std::__cxx11::_List_base<frozen_mutation_fragment, std::allocator<frozen_mutation_fragment> >::~_List_base at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/stl_list.h:575
  |              (inlined by) partition_key_and_mutation_fragments::~partition_key_and_mutation_fragments at ././repair/repair.hh:203
  |              (inlined by) std::destroy_at<partition_key_and_mutation_fragments> at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/stl_construct.h:88
  |              (inlined by) ?? at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/alloc_traits.h:537
  |              (inlined by) ?? at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/list.tcc:77
  |              (inlined by) std::__cxx11::_List_base<partition_key_and_mutation_fragments, std::allocator<partition_key_and_mutation_fragments> >::~_List_base at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/stl_list.h:575
  |              (inlined by) to_repair_rows_list at ./repair/row_level.cc:597
  | ++[2#1/1 100%] addr=0x1a41d66 total=1056 count=176 avg=6:
  | |              seastar::coroutine::internal::maybe_yield_awaiter::run_and_dispose at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/coroutine:135
  | |              (inlined by) seastar::coroutine::internal::maybe_yield_awaiter::run_and_dispose at ././seastar/include/seastar/coroutine/maybe_yield.hh:53
  |   ++[3#1/1 100%] addr=0x56bb654 total=1824 count=304 avg=6:
  |   |              seastar::reactor::run_some_tasks at ./build/release/seastar/./seastar/src/core/reactor.cc:2590
  |   |              (inlined by) seastar::reactor::run_some_tasks at ./build/release/seastar/./seastar/src/core/reactor.cc:3053
  |     ++[4#1/1 100%] addr=0x56bc8d7 total=1848 count=308 avg=6:
  |     |              seastar::reactor::do_run at ./build/release/seastar/./seastar/src/core/reactor.cc:3222
  |       ++[5#1/2 86%] addr=0x56df0a1 total=1584 count=264 avg=6:
  |       |             seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_96::operator() at ./build/release/seastar/./seastar/src/core/reactor.cc:4426
  |       |             (inlined by) std::__invoke_impl<void, seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_96&> at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:61
  |       |             (inlined by) std::__invoke_r<void, seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_96&> at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/invoke.h:111
  |       |             (inlined by) std::_Function_handler<void (), seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_96>::_M_invoke at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/std_function.h:290
  |       ++          - addr=0x568be6a:
  |       |             seastar::posix_thread::start_routine at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/std_function.h:591
  |       |             (inlined by) seastar::posix_thread::start_routine at ./build/release/seastar/./seastar/src/core/posix.cc:90
  |       ++          - addr=0x8b12c: ?? ??:0
  |       ++          - addr=0x10cbbf: ?? ??:0
  |       ++[5#2/2 14%] addr=0x56bbc19 total=264 count=44 avg=6:
  |       |             seastar::reactor::run at ./build/release/seastar/./seastar/src/core/reactor.cc:3105
  |       ++          - addr=0x565fb35:
  |       |             seastar::app_template::run_deprecated at ./build/release/seastar/./seastar/src/core/app-template.cc:276
  |       ++          - addr=0x565eca8:
  |       |             seastar::app_template::run at ./build/release/seastar/./seastar/src/core/app-template.cc:167
  |       ++          - addr=0x122de13:
  |       |             scylla_main at ./main.cc:627
  |       ++          - addr=0x122f9a0:
  |       |             std::function<int (int, char**)>::operator() at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/std_function.h:591
  |       ++          - addr=0x122c237:
  |       |             main at ./main.cc:1995
  |       ++          - addr=0x2750f: ?? ??:0
  |       ++          - addr=0x275c8: ?? ??:0
  |       ++          - addr=0x1229de4:
  |       |             _start at ??:?
  |+[1#2/46 16%] addr=0x4103bd3 total=288 count=48 avg=6:
  |              to_repair_rows_list at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/list.tcc:74
  |              (inlined by) std::__cxx11::_List_base<frozen_mutation_fragment, std::allocator<frozen_mutation_fragment> >::~_List_base at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/stl_list.h:575
  |              (inlined by) partition_key_and_mutation_fragments::~partition_key_and_mutation_fragments at ././repair/repair.hh:203
  |              (inlined by) std::destroy_at<partition_key_and_mutation_fragments> at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/stl_construct.h:88
  |              (inlined by) ?? at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/alloc_traits.h:537
  |              (inlined by) ?? at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/list.tcc:77
  |              (inlined by) std::__cxx11::_List_base<partition_key_and_mutation_fragments, std::allocator<partition_key_and_mutation_fragments> >::~_List_base at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/stl_list.h:575
  |              (inlined by) to_repair_rows_list at ./repair/row_level.cc:597
  |+           - addr=0x1a41d66:
  |              seastar::coroutine::internal::maybe_yield_awaiter::run_and_dispose at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/coroutine:135
  |              (inlined by) seastar::coroutine::internal::maybe_yield_awaiter::run_and_dispose at ././seastar/include/seastar/coroutine/maybe_yield.hh:53
  -> continued at addr=0x1a41d66 above
  |+[1#3/46 13%] addr=0x4103bb7 total=234 count=39 avg=6:
  |              to_repair_rows_list at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/list.tcc:74
  |              (inlined by) std::__cxx11::_List_base<partition_key_and_mutation_fragments, std::allocator<partition_key_and_mutation_fragments> >::~_List_base at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/stl_list.h:575
  |              (inlined by) to_repair_rows_list at ./repair/row_level.cc:597
  |+           - addr=0x1a41d66:
  |              seastar::coroutine::internal::maybe_yield_awaiter::run_and_dispose at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/coroutine:135
  |              (inlined by) seastar::coroutine::internal::maybe_yield_awaiter::run_and_dispose at ././seastar/include/seastar/coroutine/maybe_yield.hh:53
  -> continued at addr=0x1a41d66 above
  |+[1#4/46 7%] addr=0x56759f2 total=138 count=23 avg=6:
  |             seastar::memory::cpu_pages::free_span at ./build/release/seastar/./seastar/src/core/memory.cc:686
  |             (inlined by) seastar::memory::cpu_pages::free_span at ./build/release/seastar/./seastar/src/core/memory.cc:698
  |+          - addr=0x5677769:
  |             seastar::memory::small_pool::trim_free_list at ./build/release/seastar/./seastar/src/core/memory.cc:1346
  | ++[2#1/2 96%] addr=0x567a4ca total=144 count=24 avg=6:
  | |             free at ./build/release/seastar/./seastar/src/core/memory.cc:1910
  | | ++[3#1/1 100%] addr=0x1459908 total=270 count=45 avg=6:
  | | |              bytes_ostream::free_chain at ././bytes_ostream.hh:165
  | |   ++[4#1/1 100%] addr=0x4103be0 total=300 count=50 avg=6:
  | |   |              bytes_ostream::~bytes_ostream at ././bytes_ostream.hh:199
  | |   |              (inlined by) frozen_mutation_fragment::~frozen_mutation_fragment at ././mutation/frozen_mutation.hh:273
  | |   |              (inlined by) std::destroy_at<frozen_mutation_fragment> at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/stl_construct.h:88
  | |   |              (inlined by) ?? at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/alloc_traits.h:537
  | |   |              (inlined by) ?? at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/list.tcc:77
  | |   |              (inlined by) std::__cxx11::_List_base<frozen_mutation_fragment, std::allocator<frozen_mutation_fragment> >::~_List_base at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/stl_list.h:575
  | |   |              (inlined by) partition_key_and_mutation_fragments::~partition_key_and_mutation_fragments at ././repair/repair.hh:203
  | |   |              (inlined by) std::destroy_at<partition_key_and_mutation_fragments> at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/stl_construct.h:88
  | |   |              (inlined by) ?? at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/alloc_traits.h:537
  | |   |              (inlined by) ?? at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/list.tcc:77
  | |   |              (inlined by) std::__cxx11::_List_base<partition_key_and_mutation_fragments, std::allocator<partition_key_and_mutation_fragments> >::~_List_base at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/stl_list.h:575
  | |   |              (inlined by) to_repair_rows_list at ./repair/row_level.cc:597
  | |   -> continued at addr=0x4103be0 above
  | ++[2#2/2 4%] 1 more branches total=6 count=1 avg=6
  ++[1#5/46 23%] 42 more branches total=432 count=72 avg=6

logs can be found:
db_logs
loader_logs
monitor_logs
sct_logs

The text was updated successfully, but these errors were encountered:

fgelcer · 2023-07-05T19:08:15Z

@asias

Although to_repair_rows_list may yield if needed between rows and mutation fragments, the input `repair_rows_on_wire` is freed in one shot and that may cause stalls as seen in qa: ``` | bytes_ostream::free_chain at ././bytes_ostream.hh:163 ++ - addr=0x4103be0: | bytes_ostream::~bytes_ostream at ././bytes_ostream.hh:199 | (inlined by) frozen_mutation_fragment::~frozen_mutation_fragment at ././mutation/frozen_mutation.hh:273 | (inlined by) std::destroy_at<frozen_mutation_fragment> at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/stl_construct.h:88 | (inlined by) ?? at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/alloc_traits.h:537 | (inlined by) ?? at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/list.tcc:77 | (inlined by) std::__cxx11::_List_base<frozen_mutation_fragment, std::allocator<frozen_mutation_fragment> >::~_List_base at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/stl_list.h:575 | (inlined by) partition_key_and_mutation_fragments::~partition_key_and_mutation_fragments at ././repair/repair.hh:203 | (inlined by) std::destroy_at<partition_key_and_mutation_fragments> at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/stl_construct.h:88 | (inlined by) ?? at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/alloc_traits.h:537 | (inlined by) ?? at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/list.tcc:77 | (inlined by) std::__cxx11::_List_base<partition_key_and_mutation_fragments, std::allocator<partition_key_and_mutation_fragments> >::~_List_base at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/stl_list.h:575 | (inlined by) to_repair_rows_list at ./repair/row_level.cc:597 ``` This change consumes the rows and frozen mutation fragments incrementally, freeing each after being processed. Fixes scylladb#14537 Signed-off-by: Benny Halevy <bhalevy@scylladb.com>

bhalevy · 2023-08-21T08:39:08Z

Last change in this area in to_repair_rows_list was @avikivity's coroutinization done in e482cb1
But it looks like the stall reported here on return could have happened just as well before the change
when the input or output lists are are destroyed when do_with completes.

Although to_repair_rows_list may yield if needed between rows and mutation fragments, the input `repair_rows_on_wire` is freed in one shot and that may cause stalls as seen in qa: ``` | bytes_ostream::free_chain at ././bytes_ostream.hh:163 ++ - addr=0x4103be0: | bytes_ostream::~bytes_ostream at ././bytes_ostream.hh:199 | (inlined by) frozen_mutation_fragment::~frozen_mutation_fragment at ././mutation/frozen_mutation.hh:273 | (inlined by) std::destroy_at<frozen_mutation_fragment> at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/stl_construct.h:88 | (inlined by) ?? at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/alloc_traits.h:537 | (inlined by) ?? at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/list.tcc:77 | (inlined by) std::__cxx11::_List_base<frozen_mutation_fragment, std::allocator<frozen_mutation_fragment> >::~_List_base at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/stl_list.h:575 | (inlined by) partition_key_and_mutation_fragments::~partition_key_and_mutation_fragments at ././repair/repair.hh:203 | (inlined by) std::destroy_at<partition_key_and_mutation_fragments> at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/stl_construct.h:88 | (inlined by) ?? at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/alloc_traits.h:537 | (inlined by) ?? at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/list.tcc:77 | (inlined by) std::__cxx11::_List_base<partition_key_and_mutation_fragments, std::allocator<partition_key_and_mutation_fragments> >::~_List_base at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/stl_list.h:575 | (inlined by) to_repair_rows_list at ./repair/row_level.cc:597 ``` This change consumes the rows and frozen mutation fragments incrementally, freeing each after being processed. Fixes scylladb#14537 Signed-off-by: Benny Halevy <bhalevy@scylladb.com>

This sort series deals with two stall sources in row-level repair `to_repair_rows_list`: 1. Freeing the input `repair_rows_on_wire` in one shot on return (as seen in #14537) 2. Freeing the result `row_list` in one shot on error. this hasn't been seen in testing but I have no reason to believe it is not susceptible to stalls exactly like repair_rows_on_wire with the same number of rows and mutations. Fixes #14537 Closes #15102 * github.com:scylladb/scylladb: repair: reindent to_repair_rows_list repair: to_repair_rows_list: clear_gently on error repair: to_repair_rows_list: consume frozen rows gently

Although to_repair_rows_list may yield if needed between rows and mutation fragments, the input `repair_rows_on_wire` is freed in one shot and that may cause stalls as seen in qa: ``` | bytes_ostream::free_chain at ././bytes_ostream.hh:163 ++ - addr=0x4103be0: | bytes_ostream::~bytes_ostream at ././bytes_ostream.hh:199 | (inlined by) frozen_mutation_fragment::~frozen_mutation_fragment at ././mutation/frozen_mutation.hh:273 | (inlined by) std::destroy_at<frozen_mutation_fragment> at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/stl_construct.h:88 | (inlined by) ?? at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/alloc_traits.h:537 | (inlined by) ?? at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/list.tcc:77 | (inlined by) std::__cxx11::_List_base<frozen_mutation_fragment, std::allocator<frozen_mutation_fragment> >::~_List_base at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/stl_list.h:575 | (inlined by) partition_key_and_mutation_fragments::~partition_key_and_mutation_fragments at ././repair/repair.hh:203 | (inlined by) std::destroy_at<partition_key_and_mutation_fragments> at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/stl_construct.h:88 | (inlined by) ?? at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/alloc_traits.h:537 | (inlined by) ?? at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/list.tcc:77 | (inlined by) std::__cxx11::_List_base<partition_key_and_mutation_fragments, std::allocator<partition_key_and_mutation_fragments> >::~_List_base at /usr/bin/../lib/gcc/x86_64-redhat-linux/12/../../../../include/c++/12/bits/stl_list.h:575 | (inlined by) to_repair_rows_list at ./repair/row_level.cc:597 ``` This change consumes the rows and frozen mutation fragments incrementally, freeing each after being processed. Fixes scylladb#14537 Signed-off-by: Benny Halevy <bhalevy@scylladb.com>

avikivity · 2023-10-26T10:55:36Z

Performance only, not backporting.

fgelcer added the symptom/latency label Jul 5, 2023

This was referenced Aug 21, 2023

latency degradation during add nodes on read workload - >40% degradation #14540

Closed

latency degradation during decommission nodes on mixed workload - 17% degradation #14543

Closed

bhalevy mentioned this issue Aug 21, 2023

Prevent reactor stalls in to_repair_rows_list #15102

Merged

bhalevy self-assigned this Aug 21, 2023

mykaul added this to the 5.4 milestone Aug 21, 2023

mykaul added the P1 Urgent label Aug 21, 2023

scylladb-promoter closed this as completed in e551431 Aug 23, 2023

scylladb-promoter closed this as completed in #15102 Aug 23, 2023

scylladb-promoter added the Backport candidate label Aug 23, 2023

avikivity removed the Backport candidate label Oct 26, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

latency degradation during decommission nodes on write workload - ~77% degradation #14537

latency degradation during decommission nodes on write workload - ~77% degradation #14537

fgelcer commented Jul 5, 2023

fgelcer commented Jul 5, 2023

bhalevy commented Aug 21, 2023

avikivity commented Oct 26, 2023

latency degradation during decommission nodes on write workload - ~77% degradation #14537

latency degradation during decommission nodes on write workload - ~77% degradation #14537

Comments

fgelcer commented Jul 5, 2023

fgelcer commented Jul 5, 2023

bhalevy commented Aug 21, 2023

avikivity commented Oct 26, 2023