Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kvserver: avoid moving parts in closedts tests #107613

Merged
merged 4 commits into from
Jul 27, 2023

Conversation

tbg
Copy link
Member

@tbg tbg commented Jul 26, 2023

Many tests (two just this last week12) are flaky because the replicate queue
can make rebalancing decisions that undermine the state the test is trying to
set up. Often, this happens "accidentally" because ReplicationAuto is our default
ReplicationMode.

This PR improves the situation at least for closed timestamp integration tests
by switching them all over to ReplicationManual (and preventing any new ones
from accidentally using ReplicationAuto in the future).

This can be seen as a small step towards #107528, which I am increasingly
convinced is an ocean worth boiling.

Epic: None
Release note: None

Footnotes

  1. https://github.com/cockroachdb/cockroach/issues/107179

  2. https://github.com/cockroachdb/cockroach/issues/101824

@tbg tbg added the db-cy-23 label Jul 26, 2023
@cockroach-teamcity
Copy link
Member

This change is Reviewable

@tbg tbg marked this pull request as ready for review July 26, 2023 13:42
@tbg tbg requested review from a team and erikgrinaker July 26, 2023 13:42
tbg added 4 commits July 26, 2023 17:08
This commit does a few things:

- add "ManuallyReplicated" into aggressiveResolvedTimestampReplicatedClusterArgs
- actually make that struct use `ReplicationMode: ReplicationManual`
- go through all of the callers (visible via the rename) and set the replication
  mode to automatic again (with a TODO to be addressed in a follow-up commit)
  where it initially was, and remove explicitly setting it to manual where
  appropriate.

In short, a no-op change but one that will be built upon next by allowing more
of the callers to use the "manual" mode.
pickRandomTarget would livelock if there was only one replica. Now it panics, which is better.
@tbg tbg force-pushed the closedts-test-use-manual branch from 6080df3 to e3e9b77 Compare July 26, 2023 15:08
Copy link
Contributor

@erikgrinaker erikgrinaker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for going the extra mile!

@tbg
Copy link
Member Author

tbg commented Jul 27, 2023

bors r=erikgrinaker

@craig
Copy link
Contributor

craig bot commented Jul 27, 2023

Build succeeded:

@craig craig bot merged commit 310951b into cockroachdb:master Jul 27, 2023
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants