rptest: fix cluster healthy check #16337

andrewhsu · 2024-01-27T04:16:26Z

Fixes https://github.com/redpanda-data/core-internal/issues/1003

New methodRedpandaServiceCloud.cluster_healthy() according to instructions on ensuring cluster is healthy.

Added assertions with cluster_healthy() to the start and end of the heavy tests that target cloud clusters.

Also added a simple self test for the new method:

ducktape \
  --debug \
  --globals=/home/ubuntu/redpanda/tests/globals.json \
  --cluster=ducktape.cluster.json.JsonCluster \
  --cluster-file=/home/ubuntu/redpanda/tests/cluster.json \
  tests/rptest/redpanda_cloud_tests/cloud_self_test.py

output:

test_id:    rptest.redpanda_cloud_tests.cloud_self_test.SelfRedpandaCloudTest.test_simple
status:     PASS
run time:   30.319 seconds
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
========================================================================================================================================================================================
SESSION REPORT (ALL TESTS)
ducktape version: 0.8.18
session_id:       2024-02-08--008
run time:         30.335 seconds
tests run:        1
passed:           1
flaky:            0
failed:           0
ignored:          0
opassed:          0
ofailed:          0
========================================================================================================================================================================================

Backports Required

Release Notes

none

andrewhsu · 2024-01-27T04:52:04Z

example test run in description, ready for review

vbotbuildovich · 2024-01-27T07:02:13Z

ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/44395#018d4972-da49-4b32-8bd2-259f1738a09a

ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/44858#018d8660-433f-45d5-af8f-765ec59e3e51

ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/44858#018d8671-0d59-4053-9180-fb87c8571dd8

andrewhsu · 2024-02-01T14:59:10Z

tests/rptest/services/redpanda.py

+        # Under-replicated partitions (0):  []
+
+        lines = ret.decode().splitlines()
+        self.logger.debug(f'rpk cluster health lines: {lines}')


feedback from travis: compare this healthy check with the one that is performed in bare metal situation

travisdowns · 2024-02-01T16:20:36Z

tests/rptest/redpanda_cloud_tests/high_throughput_test.py

@@ -478,6 +478,8 @@ def get_broker_pod(self):

    @cluster(num_nodes=2)
    def test_throughput_simple(self):
+        assert self.redpanda.cluster_healthy(


These need to go in one place which will be called automatically before and after the test. Maybe the "before" check in make_redpanda_service_cloud? and the after check in tearDown?

since the make_redpanda_service_cloud() looks to be called in just the __init__() method, i think a more suitable place would be setup() which is called before every test_* method and failures in setup() will fail the test.

it looks like ducktape won't fail a test if there is a failure in the tearDown() method, however. it will just log a warning: https://github.com/redpanda-data/ducktape/blob/62e0285f6b3a2f22fd4a43b5fdbc13be8d4290d9/ducktape/tests/runner_client.py#L295

so, i'll leave it at the end of each individual test unless there is a more suitable place.

it looks like ducktape won't fail a test if there is a failure in the tearDown() method, however. it will just log a warning:

Ah, that explains something that John had written in our variation of the @cluster decorator: they use it exactly to avoid this problem with tearDown.

So let's do this for now but we can switch to a decorator soon enough.

tests/rptest/redpanda_cloud_tests/simple_redpanda_service_cloud_test.py

travisdowns · 2024-02-01T16:28:38Z

tests/rptest/services/redpanda.py

+                    return True
+                break
+
+        return False


This seems good for now, but ideally we want the unhealthy reasons in the error. A variation of this which threw an exception could include that info.

travisdowns

See inline comments.

andrewhsu · 2024-02-06T14:04:14Z

@travisdowns PR updated; ready for review

andrewhsu · 2024-02-06T14:48:19Z

i add git commit acc2b01 because it seems like it can be used to satisfy https://github.com/redpanda-data/core-internal/issues/1013

vbotbuildovich · 2024-02-06T16:55:22Z

new failures in https://buildkite.com/redpanda/redpanda/builds/44755#018d7f15-e115-4465-870c-3c3ec48a5043:

"rptest.tests.partition_move_interruption_test.PartitionMoveInterruption.test_cancelling_partition_move_x_core.replication_factor=3.unclean_abort=False.recovery=restart_recovery.compacted=False"

travisdowns

LGTM

travisdowns · 2024-02-07T15:03:08Z

/ci-repeat

andrewhsu · 2024-02-08T00:28:03Z

rebased with tip of dev to resolve merge conflicts. no functionality change.

savex

lgtm

andrewhsu · 2024-02-08T15:52:41Z

rebased again with tip of dev to resolve merge conflicts. no functionality change.

savex

lgtm

andrewhsu · 2024-02-08T16:05:28Z

i had to redo the rebase because it looks like the previous rebase to 367c7ea was missing 4 commits (not sure why).

i've also re-run the tests and updated output in the PR description to verify.

andrewhsu marked this pull request as ready for review January 27, 2024 04:51

andrewhsu requested review from savex and travisdowns January 27, 2024 04:51

andrewhsu commented Feb 1, 2024

View reviewed changes

travisdowns reviewed Feb 1, 2024

View reviewed changes

tests/rptest/redpanda_cloud_tests/simple_redpanda_service_cloud_test.py Outdated Show resolved Hide resolved

travisdowns reviewed Feb 1, 2024

View reviewed changes

travisdowns requested changes Feb 1, 2024

View reviewed changes

andrewhsu force-pushed the cloud-healthy branch from 1ba5ae3 to 620d9b9 Compare February 1, 2024 23:22

andrewhsu requested a review from travisdowns February 6, 2024 14:03

travisdowns previously approved these changes Feb 7, 2024

View reviewed changes

andrewhsu dismissed travisdowns’s stale review via 1f61f73 February 8, 2024 00:23

andrewhsu force-pushed the cloud-healthy branch 2 times, most recently from 1f61f73 to 9b151dc Compare February 8, 2024 00:26

andrewhsu requested a review from travisdowns February 8, 2024 00:27

travisdowns previously approved these changes Feb 8, 2024

View reviewed changes

savex previously approved these changes Feb 8, 2024

View reviewed changes

andrewhsu dismissed stale reviews from savex and travisdowns via 367c7ea February 8, 2024 15:51

andrewhsu force-pushed the cloud-healthy branch from 9b151dc to 367c7ea Compare February 8, 2024 15:51

andrewhsu requested review from savex and travisdowns February 8, 2024 15:52

savex previously approved these changes Feb 8, 2024

View reviewed changes

andrewhsu added 8 commits February 8, 2024 15:58

rptest: add RedpandaServiceCloud.cluster_healthy()

b7b42c3

rptest: add cloud self tests

6af0d0c

rptest: add cloud_self_test.py to cloud suite

fdd456f

rptest: use cluster_healthy() in high_throughput_test.py

4f574ce

rptest: assert cluster healthy in RedpandaCloudTest.setup()

a9b8769

rptest: assert cluster healthy at end of HighThroughputTest tests

cf48ccc

rptest: assert cluster healthy at end of OMBValidationTest tests

ce5ce26

rptest: use cluster_healthy() for all_up()

7c2c448

andrewhsu dismissed savex’s stale review via 7c2c448 February 8, 2024 15:59

andrewhsu force-pushed the cloud-healthy branch from 367c7ea to 7c2c448 Compare February 8, 2024 15:59

andrewhsu requested a review from savex February 8, 2024 16:03

savex approved these changes Feb 8, 2024

View reviewed changes

travisdowns approved these changes Feb 8, 2024

View reviewed changes

piyushredpanda merged commit 9545c8e into redpanda-data:dev Feb 8, 2024
18 checks passed

andrewhsu deleted the cloud-healthy branch February 8, 2024 21:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

rptest: fix cluster healthy check #16337

rptest: fix cluster healthy check #16337

andrewhsu commented Jan 27, 2024 •

edited

andrewhsu commented Jan 27, 2024

vbotbuildovich commented Jan 27, 2024 •

edited

andrewhsu Feb 1, 2024

travisdowns Feb 1, 2024

andrewhsu Feb 1, 2024

travisdowns Feb 7, 2024

travisdowns Feb 1, 2024

travisdowns left a comment

andrewhsu commented Feb 6, 2024

andrewhsu commented Feb 6, 2024

vbotbuildovich commented Feb 6, 2024

travisdowns left a comment

travisdowns commented Feb 7, 2024

andrewhsu commented Feb 8, 2024

savex left a comment

andrewhsu commented Feb 8, 2024

savex left a comment

andrewhsu commented Feb 8, 2024

rptest: fix cluster healthy check #16337

rptest: fix cluster healthy check #16337

Conversation

andrewhsu commented Jan 27, 2024 • edited

Backports Required

Release Notes

andrewhsu commented Jan 27, 2024

vbotbuildovich commented Jan 27, 2024 • edited

andrewhsu Feb 1, 2024

Choose a reason for hiding this comment

travisdowns Feb 1, 2024

Choose a reason for hiding this comment

andrewhsu Feb 1, 2024

Choose a reason for hiding this comment

travisdowns Feb 7, 2024

Choose a reason for hiding this comment

travisdowns Feb 1, 2024

Choose a reason for hiding this comment

travisdowns left a comment

Choose a reason for hiding this comment

andrewhsu commented Feb 6, 2024

andrewhsu commented Feb 6, 2024

vbotbuildovich commented Feb 6, 2024

travisdowns left a comment

Choose a reason for hiding this comment

travisdowns commented Feb 7, 2024

andrewhsu commented Feb 8, 2024

savex left a comment

Choose a reason for hiding this comment

andrewhsu commented Feb 8, 2024

savex left a comment

Choose a reason for hiding this comment

andrewhsu commented Feb 8, 2024

andrewhsu commented Jan 27, 2024 •

edited

vbotbuildovich commented Jan 27, 2024 •

edited