Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Driver performance degradation with TokenAwarePolicy when (nodes < RF) #452

Closed
michoecho opened this issue May 26, 2022 · 2 comments
Closed

Driver performance degradation with TokenAwarePolicy when (nodes < RF) #452

michoecho opened this issue May 26, 2022 · 2 comments

Comments

@michoecho
Copy link

scylla-rust-driver/scylla/src/transport/load_balancing/token_aware.rs

Lines 26 to 29 in 652f131

cluster
.ring_range(token)
.unique()
.take(replication_factor)

Scrutinize the above. Usually (when nodes >= RF) the above ends after just a few iterations. Since tokens are distributed between nodes randomly, we will quickly find RF unique nodes. But if the number of nodes in the ring is below RF, this code will iterate over all tokens, which takes significant work.

I bumped into this during a benchmark. It caused the driver to spend more than 50% of its CPU time in plan() when driving a 2-node cluster with RF=3.
@piodul
Copy link
Collaborator

piodul commented May 27, 2022

Good catch. Looking at the code of network_topology_strategy_replicas I suspect we might have a similar problem with NetworkTopologyStrategy, too (but not 100% sure, didn't try to reproduce).

@havaker is refactoring load balancing right now (#449), so I think we should either fix it during or after the refactor.

michoecho added a commit to michoecho/scylla-rust-driver that referenced this issue Feb 17, 2023
@michoecho
token_aware: avoid iterating over the entire ring when RF is unsatisf…
91449f9 
…iable

token_aware code uses cluster.ring_range(token).unique() to iterate over
candidate replicas until enough candidates are found to satisfy the RF.
This behaves badly when the number of candidates is smaller than RF --
we always iterate over the entire ring, which is very wasteful (it was
seen to slow down the driver by a factor of >2 in a simple performance
test).

Fix that by ending the iteration early when all unique candidate nodes were
already considered.

Fixes scylladb#452
@michoecho michoecho mentioned this issue Feb 17, 2023
Closed
6 tasks
@havaker
Copy link
Contributor

havaker commented Mar 23, 2023

Fixed in #612.

@havaker havaker closed this as completed Mar 23, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
3 participants
@piodul @havaker @michoecho