CI Failure (consumer timed out waiting for offsets) in ControllerLogLimitPartitionBalancerTests.test_partition_balancer_with_limits

[dlex]

• on (arm64, container) in job https://buildkite.com/redpanda/redpanda/builds/29140#01881e16-3b51-4dbd-9be8-32fde45c7120
• on (arm64, container) in job https://buildkite.com/redpanda/redpanda/builds/29136#01881e02-5281-428d-95e1-801cd2d4a43a
• on (arm64, container) in job https://buildkite.com/redpanda/redpanda/builds/29137#01881e0d-d248-4b03-be91-761c41a6c950
• on (arm64, container) in job https://buildkite.com/redpanda/redpanda/builds/29132#01881df5-b57c-491a-bb65-4dc4fa6d1e63
• on (arm64, container) in job https://buildkite.com/redpanda/redpanda/builds/29133#01881dfb-5a51-43fd-ab70-cfc5783e91e8
• on (arm64, container) in job https://buildkite.com/redpanda/redpanda/builds/29126#018818ed-e6e9-4a42-acf5-4ec20e0b7dc0
• on (arm64, container) in job https://buildkite.com/redpanda/redpanda/builds/29125#018818e5-bd7b-46f5-a59e-81ee3e4ca713
• on (arm64, container) in job https://buildkite.com/redpanda/redpanda/builds/29126#018818ed-e6e9-4a42-acf5-4ec20e0b7dc0
• on (arm64, container) in job https://buildkite.com/redpanda/redpanda/builds/29122#018818dc-932d-4129-9243-76cde84889c2
• on (arm64, container) in job https://buildkite.com/redpanda/redpanda/builds/29121#018818d3-fb9b-4767-8946-aa6693caa35f
• on (arm64, container) in job https://buildkite.com/redpanda/redpanda/builds/29118#018818cd-af91-4642-b584-79a93f7a37e1
• on (arm64, container) in job https://buildkite.com/redpanda/redpanda/builds/29121#018818d3-fb9b-4767-8946-aa6693caa35f
• on (arm64, container) in job https://buildkite.com/redpanda/redpanda/builds/29103#018813bf-52dd-497b-8ead-0880140ab9fd
• on (arm64, container) in job https://buildkite.com/redpanda/redpanda/builds/29106#018813c8-f129-4862-9eb8-f7f2bf1a56a7
• on (arm64, container) in job https://buildkite.com/redpanda/redpanda/builds/29102#018813b6-094d-43f5-9e23-250b97ca10d2
• on (arm64, container) in job https://buildkite.com/redpanda/redpanda/builds/29098#018813ae-cbcc-45b1-add6-c930fbfceb99
• on (arm64, container) in job https://buildkite.com/redpanda/redpanda/builds/29097#018813a6-2f45-4f23-8d7c-dd878f1b209f
• on (amd64, VM) in job https://buildkite.com/redpanda/vtools/builds/7540#01881e71-9407-4786-bb22-88891b75ce9e
• on (arm64, VM) in job https://buildkite.com/redpanda/vtools/builds/7525#01881191-5dc6-41b7-a832-7b704c2f4eab
• on (amd64, VM) in job https://buildkite.com/redpanda/vtools/builds/7528#01881429-6a58-4a0b-b50e-51408b80ff78
• on (amd64, VM) in job https://buildkite.com/redpanda/vtools/builds/7534#0188194a-1f1c-4617-8281-2bc625f0dd7e
• on (amd64, VM) in job https://buildkite.com/redpanda/vtools/builds/7528#01881429-6a58-4a0b-b50e-51408b80ff78
• on (arm64, VM) in job https://buildkite.com/redpanda/vtools/builds/7525#01881191-5dc6-41b7-a832-7b704c2f4eab

(19/69 runs)

Module: rptest.tests.controller_log_limiting_test
Class:  ControllerLogLimitPartitionBalancerTests
Method: test_partition_balancer_with_limits

test_id:    rptest.tests.controller_log_limiting_test.ControllerLogLimitPartitionBalancerTests.test_partition_balancer_with_limits
status:     FAIL
run time:   2 minutes 52.469 seconds
 
    TimeoutError("Consumer failed to consume up to offsets {TopicPartition(topic='topic-pnwnfwpegc', partition=94): 132, TopicPartition(topic='topic-pnwnfwpegc', partition=52): 287, TopicPartition(topic='topic-pnwnfwpegc', partition=63): 108, ....
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 135, in run
    data = self.run_test()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 227, in run_test
    return self.test_context.function(self.test)
  File "/root/tests/rptest/services/cluster.py", line 49, in wrapped
    r = f(self, *args, **kwargs)
  File "/root/tests/rptest/tests/controller_log_limiting_test.py", line 415, in test_partition_balancer_with_limits
    self.run_validation(consumer_timeout_sec=CONSUMER_TIMEOUT)
  File "/root/tests/rptest/tests/end_to_end.py", line 269, in run_validation
    self.run_consumer_validation(
  File "/root/tests/rptest/tests/end_to_end.py", line 290, in run_consumer_validation
    self.await_consumed_offsets(last_acked_offsets,
  File "/root/tests/rptest/tests/end_to_end.py", line 221, in await_consumed_offsets
    wait_until(
  File "/usr/local/lib/python3.10/dist-packages/ducktape/utils/util.py", line 57, in wait_until
    raise TimeoutError(err_msg() if callable(err_msg) else err_msg) from last_exception

Similar to #10502 but in a different test. The tests look really similar too. Posting as a separate issue for visibility.

[piyushredpanda]

Most likely was just the debug slowness; closing.