Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

roachtest: add failover/non-system/blackhole tests #92991

Merged
merged 2 commits into from
Dec 5, 2022

Conversation

erikgrinaker
Copy link
Contributor

@erikgrinaker erikgrinaker commented Dec 3, 2022

This patch adds roachtests to benchmark the maximum unavailability during leaseholder network outages on non-system ranges, both symmetric and asymmetric outages. Initial results, with a query timeout of 30 s:

Test pMax read pMax write
crash 14.5 s 14.5 s
blackhole 16.6 s 18.3 s
blackhole-recv 30.1 s 30.1 s
blackhole-send 30.1 s 30.1 s

Touches #79494.

Epic: None
Release note: None

@erikgrinaker erikgrinaker requested review from tbg and a team December 3, 2022 14:59
@erikgrinaker erikgrinaker self-assigned this Dec 3, 2022
@cockroach-teamcity
Copy link
Member

This change is Reviewable

@erikgrinaker erikgrinaker force-pushed the roachtest-blackhole branch 3 times, most recently from 36bbf4c to 4052f8e Compare December 3, 2022 17:25
Epic: None
Release note: None
Copy link
Member

@tbg tbg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 1 of 1 files at r1, 1 of 1 files at r2, all commit messages.
Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @erikgrinaker)


pkg/cmd/roachtest/tests/failover.go line 81 at r2 (raw file):

	require.Equal(t, 7, c.Spec().NodeCount)

	seed := int64(1666467482296309000)

Add a comment here that you wanted predictable randomness.

Copy link
Contributor Author

@erikgrinaker erikgrinaker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @tbg)


pkg/cmd/roachtest/tests/failover.go line 81 at r2 (raw file):

Previously, tbg (Tobias Grieger) wrote…

Add a comment here that you wanted predictable randomness.

Not sure that we do, on second thought. There's enough randomness in the environment here that it likely doesn't matter. Switched it to NewTestRand().

This patch adds roachtests to benchmark the maximum unavailability
during leaseholder network outages on non-system ranges, both symmetric
and asymmetric outages. Initial results, with a query timeout of 30 s:

| Test             | pMax read | pMax write |
|------------------|-----------|------------|
| `crash`          | 14.5 s    | 14.5 s     |
| `blackhole`      | 16.6 s    | 18.3 s     |
| `blackhole-recv` | 30.1 s    | 30.1 s     |
| `blackhole-send` | 30.1 s    | 30.1 s     |

Epic: None
Release note: None
@erikgrinaker
Copy link
Contributor Author

CI failures appear unrelated.

bors r+

@craig
Copy link
Contributor

craig bot commented Dec 5, 2022

Build failed:

@erikgrinaker
Copy link
Contributor Author

bors retry

@craig
Copy link
Contributor

craig bot commented Dec 5, 2022

Build failed:

@erikgrinaker
Copy link
Contributor Author

bors retry

@craig
Copy link
Contributor

craig bot commented Dec 5, 2022

Build succeeded:

@craig craig bot merged commit 0c4a2e0 into cockroachdb:master Dec 5, 2022
@erikgrinaker erikgrinaker deleted the roachtest-blackhole branch December 28, 2022 11:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants