/cockroach

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

backport-2.1: workload/roachtest: Fix load-based rebalancing test flakes by splitting kv keyspace evenly #30332

Merged
merged 3 commits into from Sep 18, 2018

Conversation

Reviewers

@nvanbenschoten nvanbenschoten

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
3 participants
@a-robinson @cockroach-teamcity @nvanbenschoten
@a-robinson
Member

a-robinson commented Sep 17, 2018

Backport 3/3 commits from #30255.

/cc @cockroachdb/release

workload/kv: Add option to split key space evenly

The default splitting behavior randomly chooses split points, which can
lead to ranges that encompass drastically different amounts of the key
space (e.g. I saw a 9x size difference in the run I examined that
motivated this). Having drastically different range sizes is a pain for
any tests that expect roughly even qps across the ranges.

If you'd prefer I can just write this into the rebalancing roachtest,
but it seemed like something that may be useful more broadly as well.

roachtest: Split kv ranges evenly in load-based rebalancing tests

Fixes #29969. The rebalance-replicas-by-load roachtest sometimes failed
to properly rebalance one range leaseholder for the kv table to each
node because the ranges could receive drastically different qps (where
two smaller ranges don't even come close to the qps of one larger one).

Also, while I'm here, improve the output of the test a bit and send SQL load to all of the nodes, not just the first three.

a-robinson added some commits Sep 15, 2018

@a-robinson
workload/kv: Split key space evenly instead of randomly 
The old splitting behavior randomly chose split points, which could
lead to ranges that encompass drastically different amounts of the key
space (e.g. I saw a 9x size difference in the run I examined that
motivated this). Having drastically different range sizes is a pain for
any tests that expect roughly even qps across the ranges.

Now we will instead always split the key space evenly based on the
number of split points, such that each should get roughly the same
amount of load.

Fixes #29969. The rebalance-replicas-by-load roachtest sometimes failed
to properly rebalance one range leaseholder for the kv table to each
node because the ranges could receive drastically different qps (where
two smaller ranges don't even come close to the qps of one larger one).

Release note: None
2cd5e22
@a-robinson
roachtest: Improve output of load-based rebalancing tests 
Release note: None
63add77
@a-robinson
roachtest: Send load to all nodes in load-based rebalancing tests 
Previously it was hard-coded to send load to just the first 3 nodes.

Release note: None
Loading status checks…
1442623

@a-robinson a-robinson requested a review from nvanbenschoten Sep 17, 2018

@cockroach-teamcity

This comment has been minimized.

Show comment
Hide comment
@cockroach-teamcity

cockroach-teamcity Sep 17, 2018

Member

This change is Reviewable
Member

cockroach-teamcity commented Sep 17, 2018

This change is Reviewable

@a-robinson a-robinson merged commit fd52067 into cockroachdb:release-2.1 Sep 18, 2018
2 checks passed

2 checks passed

GitHub CI (Cockroach) TeamCity build finished
Details
license/cla Contributor License Agreement is signed.
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment