cloudtest: Fortify the test_oom_clusterd test #18861
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
philip-stoev
commented
Apr 20, 2023
| @@ -132,6 +135,16 @@ impl Default for ClusterReplicaSizeMap { | |||
| workers: scale.into(), | |||
| }, | |||
| ); | |||
|
|
|||
| inner.insert( | |||
There was a problem hiding this comment.
The new SIZE type , along with all the other types in this file, are defaults that are only used for testing. In the cloud, a completely separate set of SIZEs is installed and used.
There was a problem hiding this comment.
I'd feel more confident if we had this behind a if testing
There was a problem hiding this comment.
this entire thing is the default when executing bin/environmentd unless the entire SIZE list is overriden with a command-line option, which is what happens in the cloud
there is no easy way to distinguish "test" from "non-test" invocations of bin/environmentd at this time
jkosh44
approved these changes
Apr 20, 2023
The test was forcing an clusterd to OOM, but this could also cause the entire Buildkite instance that it is running on to become unresponsive. Fix by running the test against a memory-constrained Mz cluster that will OOM without bringing down the entire machine.
philip-stoev
force-pushed
the
cloudtest-memory-limit
branch
from
e1fe556
to
7fca33a
Compare
def-
approved these changes
Apr 20, 2023
| @@ -132,6 +135,16 @@ impl Default for ClusterReplicaSizeMap { | |||
| workers: scale.into(), | |||
| }, | |||
| ); | |||
|
|
|||
| inner.insert( | |||
There was a problem hiding this comment.
I'd feel more confident if we had this behind a if testing
|
Thanks, Philip! |
12 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The test was forcing an clusterd to OOM, but this could also cause the entire Buildkite instance that it is running on to become unresponsive.
Fix by running the test against a memory-constrained Mz cluster that will OOM without bringing down the entire machine.
Motivation
CI was failing with "Agent lost" error, indicating a runaway process.