rptest: partially deflake test_stress_consumer_group_commits
#26185
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
WillemKauf
force-pushed
the
stress_group_consumer_offsets_fix
branch
from
e961f3a to
12c29b8
Compare
Collaborator
CI test resultstest results on build#66142
test results on build#66150
test results on build#66155
|
Contributor
Author
|
/ci-repeat 1 |
WillemKauf
force-pushed
the
stress_group_consumer_offsets_fix
branch
from
12c29b8 to
d775923
Compare
Contributor
Author
|
/ci-repeat 1 |
Collaborator
Retry command for Build#66154please wait until all jobs are finished before running the slash command |
This test can race with adjacent segment compaction while collecting a
consumer offsets summary, with a backtrace like
```
File "/root/tests/rptest/tests/consumer_group_test.py", line 1690, in test_stress_consumer_group_commits
summary = olv.consumer_offsets_summary(n)
File "/root/tests/rptest/clients/offline_log_viewer.py", line 59, in consumer_offsets_summary
return self._json_cmd(node, "--type consumer_offsets_summary")
File "/root/tests/rptest/clients/offline_log_viewer.py", line 34, in _json_cmd
json_out = node.account.ssh_output(cmd, combine_stderr=False)
File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/cluster/remoteaccount.py", line 41, in wrapper
return method(self, *args, **kwargs)
File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/cluster/remoteaccount.py", line 421, in ssh_output
raise RemoteCommandError(self, cmd, exit_status, stderr.read())
FileNotFoundError: [Errno 2] No such file or directory: \'/var/lib/redpanda/data/kafka/__consumer_offsets/0_22/2239422-4-v1.log\'\n'
```
Use a `wait_until()` wrapper instead to retry this case.
This test could generate enough output to fill up the agent nodes in CI:
```
[WARNING - 2025-05-18 15:24:36,604 - test - copy_service_logs - lineno:162]: Error copying log backtraces from
{'path': '/var/lib/redpanda/redpanda_backtrace.log', 'collect_default': True} to /build/tests/results/2025-05-18--002/ConsumerGroupOffsetResetTest/test_stress_consumer_group_commits/15/RedpandaService-0-140178931023632/docker-rp-18.service <RedpandaService-0-140178931023632: num_nodes: 3, nodes: ['docker-rp-16', 'docker-rp-17', 'docker-rp-18']>: [Errno 28] No space left on device
```
Use a custom `LoggingConfig` to reduce the amount of output generated.
WillemKauf
force-pushed
the
stress_group_consumer_offsets_fix
branch
from
63ad603 to
a28cdec
Compare
mmaslankaprv
approved these changes
May 20, 2025
bashtanov
reviewed
May 21, 2025
| lambda: get_summary(n), | ||
| timeout_sec=60, | ||
| backoff_sec=1, | ||
| retry_on_exc=True, |
Contributor
There was a problem hiding this comment.
This way we catch the asserts above. Are they unlikely to be transient?
Contributor
Author
There was a problem hiding this comment.
Yikes good point. I forget that asserts in Python just throw. i'll have a follow-up fix to this.
Contributor
Author
There was a problem hiding this comment.
Though I doubt these errors are transient, we should fail the test on the first sighting of a failed assert.
7 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This test can race with adjacent segment compaction while collecting a node's consumer offsets summary, with a backtrace like
Use a
wait_until()wrapper instead to retry this exceptional case.The second commit also adds a
LoggingConfigargument to pass to theRedpandaServicebase class for this test, since it generates large enough logs to fill up the CI agent node's disks.Backports Required
Release Notes