Add test for oom-killed reason in mz_cluster_replica_statuses
#18806
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
umanwizard
force-pushed
the
test_cluster_oom
branch
from
5d4e36d
to
044af8a
Compare
umanwizard
changed the title
oom-killed reason in mz_cluster_replica_statuses
benesch
reviewed
Apr 18, 2023
| def verify_status(status: str, reason: Optional[str]) -> None: | ||
| while True: | ||
| (status, reason) = mz.environmentd.sql_query( | ||
| """ |
There was a problem hiding this comment.
you may wish to use dedent` for the SQL for aeeaaesthetic reasons
| verify_status("not-ready", "oom-killed") | ||
|
|
||
| mz.environmentd.sql("DROP VIEW v CASCADE") | ||
| # Now that we've dropped the problematic view, the replica should come back |
There was a problem hiding this comment.
you may wish to confirm the successful comeback of the replica in question by issuing a SELECT against that replica.
philip-stoev
approved these changes
Apr 18, 2023
|
This is still too slow to converge; opened a thread on Slack to discuss:
https://materializeinc.slack.com/archives/C01LKF361MZ/p1681825775251649
…On Tue, Apr 18, 2023 at 2:02 AM Philip Stoev ***@***.***> wrote:
***@***.**** approved this pull request.
Thank you! If there is any flakiness in this test, we will address it as
it happens.
—
Reply to this email directly, view it on GitHub
<#18806 (review)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAOOGJ6P23GSEG2PNBI4GTTXBYU53ANCNFSM6AAAAAAXBTYE5E>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
umanwizard
force-pushed
the
test_cluster_oom
branch
from
47ad335
to
19ae3a2
Compare
umanwizard
commented
Apr 19, 2023
12 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This test creates an index on a query that is known to OOM, repeatedly checks
mz_cluster_replica_statusesuntil the OOM is observed, clears the query, and then repeatedly checks until the cluster is observed ready.It's possible to imagine that this test can have false negatives (i.e., the test is green but the underlying behavior is broken) if certain patterns of flapping occur in the data. I think this is unlikely to be a concern in practice. It would be more precise/elegant if we could control how long it takes a pod to restart after OOMing, but there doesn't currently seem to be a way to do so.