Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

rptest: clean cloudv2 cluster topics in tearDown() #16049

Merged
merged 5 commits into from
Jan 22, 2024
Merged

rptest: clean cloudv2 cluster topics in tearDown() #16049

merged 5 commits into from
Jan 22, 2024

Conversation

andrewhsu
Copy link
Member

@andrewhsu andrewhsu commented Jan 10, 2024 •
edited

related issue: https://github.com/redpanda-data/cloudv2/issues/11563

Have OMBValidationTest and HighThroughputTest clean cluster in tearDown() and startup() by deleting topics that may have been leftover from a test case. The clean_cluster() method can be added to later to clean more things we want to ensure is gone before the next test case runs.

Since clean_cluster() simply deletes all topics, it should only be called if there are no other tests running at the same time expecting topics to be there.

example run:

ducktape \ 
 --debug \ 
 --globals=/home/ubuntu/redpanda/tests/globals.json \ 
 --cluster=ducktape.cluster.json.JsonCluster \ 
 --cluster-file=/home/ubuntu/redpanda/tests/cluster.json \  
 --test-runner-timeout=3600000 \
 tests/rptest/redpanda_cloud_tests/omb_validation_test.py::OMBValidationTest.test_common_workload

output:

[DEBUG - 2024-01-19 02:19:16,802 - redpanda - clean_cluster - lineno:1681]: found topics to delete (2): ['test-topic-to-delete1', 'test-topic-to-delete2']
...
[DEBUG - 2024-01-19 02:30:28,011 - redpanda - clean_cluster - lineno:1681]: found topics to delete (1): ['test-topic-co1Cozg-0000']
...
test_id:    rptest.redpanda_cloud_tests.omb_validation_test.OMBValidationTest.test_common_workload
status:     PASS
run time:   11 minutes 41.601 seconds
-----------------------------------------------------------------------
======================================================
SESSION REPORT (ALL TESTS)
ducktape version: 0.8.18
session_id:       2024-01-19--012
run time:         11 minutes 41.621 seconds
tests run:        1
passed:           1
flaky:            0
failed:           0
ignored:          0
opassed:          0
ofailed:          0
=====================================================

Backports Required

  • none - not a bug fix
  • none - this is a backport
  • none - issue does not exist in previous branches
  • none - papercut/not impactful enough to backport
  • v23.3.x
  • v23.2.x
  • v23.1.x

Release Notes

  • none

@vbotbuildovich
Copy link
Collaborator

vbotbuildovich commented Jan 10, 2024 •
edited

new failures in https://buildkite.com/redpanda/redpanda/builds/43623#018cf128-e5d0-43bc-bd6f-e00c697b9d7d:

"rptest.tests.cluster_features_test.FeaturesMultiNodeUpgradeTest.test_rollback"

new failures in https://buildkite.com/redpanda/redpanda/builds/43623#018cf128-e5c5-4405-a1b4-4e3a1ea3253f:

"rptest.tests.cluster_features_test.FeaturesMultiNodeUpgradeTest.test_upgrade"

new failures in https://buildkite.com/redpanda/redpanda/builds/43623#018cf128-e5c9-49e1-b95c-e82ca0cc6223:

"rptest.tests.cluster_features_test.FeaturesSingleNodeUpgradeTest.test_upgrade"
"rptest.tests.cluster_features_test.FeaturesNodeJoinTest.test_old_node_join"

new failures in https://buildkite.com/redpanda/redpanda/builds/43623#018cf13a-2c75-4d71-bab6-5c21940c70b1:

"rptest.tests.cluster_features_test.FeaturesSingleNodeUpgradeTest.test_upgrade"
"rptest.tests.cluster_features_test.FeaturesNodeJoinTest.test_old_node_join"

new failures in https://buildkite.com/redpanda/redpanda/builds/43623#018cf13a-2c6e-42b1-98ab-30609c1f3a51:

"rptest.tests.cluster_features_test.FeaturesMultiNodeUpgradeTest.test_rollback"

new failures in https://buildkite.com/redpanda/redpanda/builds/43623#018cf13a-2c72-42b3-b919-c64d3fb41049:

"rptest.tests.cluster_features_test.FeaturesMultiNodeUpgradeTest.test_upgrade"

@andrewhsu
Copy link
Member Author

9f667ec is just a rebase with latest on dev

@andrewhsu andrewhsu force-pushed the clean-cluster branch 2 times, most recently from 8ddafbe to 56ba3ee Compare January 12, 2024 16:43
@vbotbuildovich
Copy link
Collaborator

ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/43726#018cfee3-7ebb-4c4e-bdee-c0964ee9a9f6

travisdowns
travisdowns reviewed Jan 16, 2024
View reviewed changes
tests/rptest/services/redpanda.py Outdated
rpk = RpkTool(self)
topics = rpk.list_topics()
deletable = [x for x in topics if not x.startswith('_')]
self.logger.debug(f'found {len(deletable)} topics to delete')
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's worth listing them.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would even list some high level stats, but that's an overkill.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a way to do rpk cluster topic delete -r ".*" in our wrapper which does the right thing?

@travisdowns
Copy link
Member

So we delete on teardown, I'm wondering what we should do before each test? Should we look for topics without _ and fail in the same way if we seen any?

savex
savex reviewed Jan 16, 2024
View reviewed changes
Copy link
Contributor

@savex savex left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All good. One request, Let us check if topics exists on startup/teardown and clean if test_context/parallel is < 2

tests/rptest/redpanda_cloud_tests/high_throughput_test.py Outdated Show resolved Hide resolved
tests/rptest/services/redpanda.py Show resolved Hide resolved
tests/rptest/services/redpanda.py Outdated
rpk = RpkTool(self)
topics = rpk.list_topics()
deletable = [x for x in topics if not x.startswith('_')]
self.logger.debug(f'found {len(deletable)} topics to delete')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would even list some high level stats, but that's an overkill.

@andrewhsu andrewhsu self-assigned this Jan 18, 2024
@andrewhsu andrewhsu force-pushed the clean-cluster branch 4 times, most recently from e50633f to 46058b2 Compare January 18, 2024 23:24
@andrewhsu
Copy link
Member Author

putting to draft to make adjustments based on feedback

@andrewhsu andrewhsu marked this pull request as draft January 18, 2024 23:26
@andrewhsu
Copy link
Member Author

ready for review. i have example output in description that interestingly shows leftover topic that was cleaned up from the test_common_workload() test.

@andrewhsu andrewhsu marked this pull request as ready for review January 19, 2024 02:59
@travisdowns travisdowns merged commit 5409819 into redpanda-data:dev Jan 22, 2024
20 checks passed
@andrewhsu andrewhsu deleted the clean-cluster branch January 22, 2024 16:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@travisdowns travisdowns travisdowns approved these changes

@savex savex Awaiting requested review from savex

@StephanDollberg StephanDollberg Awaiting requested review from StephanDollberg

Assignees

@andrewhsu andrewhsu

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
5 participants
@andrewhsu @vbotbuildovich @travisdowns @savex @StephanDollberg