CI Failure (Timeout - Failed to start) in MultiTopicAutomaticLeadershipBalancingTest.test_topic_aware_rebalance

[r-vasquez]

https://buildkite.com/redpanda/redpanda/builds/29920#0188548a-60df-4a2e-97f8-5a8375161457/6-5906

Module: rptest.tests.leadership_transfer_test
Class:  MultiTopicAutomaticLeadershipBalancingTest
Method: test_topic_aware_rebalance

====================================================================================================
test_id:    rptest.tests.leadership_transfer_test.MultiTopicAutomaticLeadershipBalancingTest.test_topic_aware_rebalance
status:     FAIL
run time:   2 minutes 38.155 seconds


    TimeoutError('Redpanda service docker-rp-8 failed to start within 20 sec')
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 135, in run
    data = self.run_test()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 227, in run_test
    return self.test_context.function(self.test)
  File "/root/tests/rptest/services/cluster.py", line 49, in wrapped
    r = f(self, *args, **kwargs)
  File "/root/tests/rptest/tests/leadership_transfer_test.py", line 189, in test_topic_aware_rebalance
    self.redpanda.start_node(node)
  File "/root/tests/rptest/services/redpanda.py", line 1796, in start_node
    self.start_service(node, start_rp)
  File "/root/tests/rptest/services/redpanda.py", line 1868, in start_service
    start()
  File "/root/tests/rptest/services/redpanda.py", line 1787, in start_rp
    wait_until(
  File "/usr/local/lib/python3.10/dist-packages/ducktape/utils/util.py", line 57, in wait_until
    raise TimeoutError(err_msg() if callable(err_msg) else err_msg) from last_exception
ducktape.errors.TimeoutError: Redpanda service docker-rp-8 failed to start within 20 sec

[NyaliaLui]

FAIL test: MultiTopicAutomaticLeadershipBalancingTest.test_topic_aware_rebalance (6/30 runs)
  failure at 2023-05-26T07:02:05.442Z: TimeoutError('Redpanda service docker-rp-4 failed to start within 20 sec')
      on (amd64, container) in job https://buildkite.com/redpanda/redpanda/builds/29952#018856a6-00da-403c-8eb9-66fb914026d1

[abhijat]

https://buildkite.com/redpanda/redpanda/builds/30108#018868e9-8cfa-4239-a449-fa4c1bc3d5c4

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 135, in run
    data = self.run_test()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 227, in run_test
    return self.test_context.function(self.test)
  File "/root/tests/rptest/services/cluster.py", line 49, in wrapped
    r = f(self, *args, **kwargs)
  File "/root/tests/rptest/tests/leadership_transfer_test.py", line 189, in test_topic_aware_rebalance
    self.redpanda.start_node(node)
  File "/root/tests/rptest/services/redpanda.py", line 2144, in start_node
    self.start_service(node, start_rp)
  File "/root/tests/rptest/services/redpanda.py", line 2216, in start_service
    start()
  File "/root/tests/rptest/services/redpanda.py", line 2135, in start_rp
    wait_until(
  File "/usr/local/lib/python3.10/dist-packages/ducktape/utils/util.py", line 57, in wait_until
    raise TimeoutError(err_msg() if callable(err_msg) else err_msg) from last_exception
ducktape.errors.TimeoutError: Redpanda service docker-rp-11 failed to start within 20 sec

[michael-redpanda]

https://buildkite.com/redpanda/redpanda/builds/30231#01886fa2-9b7b-4aa7-aa83-baa660555270

[ballard26]

In https://buildkite.com/redpanda/redpanda/builds/30231#01886fa2-9b7b-4
Redpanda's process starts at 3:25:15 , the admin api service starts at 3:25:33, and the test last tries to ping the RP instance at 03:25:33. It looks like Redpanda is starting slower than normal and racing with the test. I'm not seeing anything specific to this test failing so far.

[piyushredpanda]

Yeah possibly tied to clang-16 upgrade, or something like that. A bunch of tests in this state.

[andijcr]

https://buildkite.com/redpanda/redpanda/builds/30528#0188826b-c3b0-4def-9143-bbb6170e95b9

[andijcr]

https://buildkite.com/redpanda/redpanda/builds/30741#0188933f-38cb-4f52-af51-42ea5332e5e2

[rystsov]

https://buildkite.com/redpanda/redpanda/builds/30854#01889a70-5d0c-4881-a945-1a61c2f6db05

[michael-redpanda]

https://buildkite.com/redpanda/redpanda/builds/31188#0188b564-dd68-494c-a276-0d5970f11373

[michael-redpanda]

https://buildkite.com/redpanda/redpanda/builds/31188#0188b564-dd68-494c-a276-0d5970f11373

[abhijat]

https://buildkite.com/redpanda/redpanda/builds/31299#0188bab3-73d6-45c3-a3a8-b2863e018581

[twmb]

https://buildkite.com/redpanda/redpanda/builds/31331#0188bd78-3379-4c4f-a3af-658a7192e4b1

[michael-redpanda]

https://buildkite.com/redpanda/redpanda/builds/31350#0188bf0b-ce2e-46a3-9e58-88bfd832c85d