Ci Failure (Failed to get metadata: Local: Timed out) in ThroughputLimitsSnc.test_configuration

[NyaliaLui]

Version & Environment

Redpanda version: dev:

This happened during cluster startup in CI

https://buildkite.com/redpanda/redpanda/builds/22997#01863c84-7997-4df4-a53b-3be8d31d445d/6-2309

Module: rptest.tests.throughput_limits_snc_test
Class:  ThroughputLimitsSnc
Method: test_configuration

test_id:    rptest.tests.throughput_limits_snc_test.ThroughputLimitsSnc.test_configuration
status:     FAIL
run time:   15.756 seconds


    KafkaException(KafkaError{code=_TIMED_OUT,val=-185,str="Failed to get metadata: Local: Timed out"})
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 133, in run
    self.setup_test()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 218, in setup_test
    self.test.setup()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/test.py", line 91, in setup
    self.setUp()
  File "/root/tests/rptest/tests/redpanda_test.py", line 99, in setUp
    self.redpanda.start()
  File "/root/tests/rptest/services/redpanda.py", line 1022, in start
    self.wait_for_membership(first_start=first_start)
  File "/root/tests/rptest/services/redpanda.py", line 937, in wait_for_membership
    wait_until(lambda: {n
  File "/usr/local/lib/python3.10/dist-packages/ducktape/utils/util.py", line 53, in wait_until
    raise e
  File "/usr/local/lib/python3.10/dist-packages/ducktape/utils/util.py", line 44, in wait_until
    if condition():
  File "/root/tests/rptest/services/redpanda.py", line 937, in <lambda>
    wait_until(lambda: {n
  File "/root/tests/rptest/services/redpanda.py", line 939, in <setcomp>
    if self.registered(n)} == expected,
  File "/root/tests/rptest/services/redpanda.py", line 2209, in registered
    brokers = client.brokers()
  File "/root/tests/rptest/clients/python_librdkafka.py", line 33, in brokers
    return client.list_topics(timeout=10).brokers
cimpl.KafkaException: KafkaError{code=_TIMED_OUT,val=-185,str="Failed to get metadata: Local: Timed out"}

[dlex]

Egress TP limit is set by the test to 64 B/s. Controller log leader gets this sequence of KAPI requests:

• api_versions
• metadata
• metadata (ok so far)
• (another connection established)
• api_versions (throttle requested: 3.6s)
• metadata (throttle enforced: 3.6s, requested: 12.6s)
• metadata (throttle enforced: 12.6s)

12.6s is too much for the client and it times out the metadata request.

As a mitigation, I will double the minimum egress TP limit.

[mmaslankaprv]

Happened again here:
https://buildkite.com/redpanda/redpanda/builds/23428#01865fa8-a772-45fb-bd38-b9208d759b43

[dlex]

Happened again in CDT: https://buildkite.com/redpanda/vtools/builds/7321#0187c490-ccb8-4b5c-89bc-a3d4ebd3ee93

128 B/s still seems to be on the edge, needs a bump to 256 B/s

[travisdowns]

@BenPope - this is still happening and as you see there has been a history of just bumping the limit which hasn't panned out (yet?).

As you are planning big changes in this area I figure this failure may be obsoleted by them, so looking into it is probably fruitless in light of that. I guess after you changes go in and this hasn't happened for a while we can simply close it.

WDYT?

[michael-redpanda]

I'm going to mark this as sev/low and point at the epic https://github.com/redpanda-data/core-internal/issues/917

[BenPope]

@BenPope - this is still happening and as you see there has been a history of just bumping the limit which hasn't panned out (yet?).

As you are planning big changes in this area I figure this failure may be obsoleted by them, so looking into it is probably fruitless in light of that. I guess after you changes go in and this hasn't happened for a while we can simply close it.

WDYT?

This makes sense to me.

[travisdowns]

Closing this as discussed above, cc @BenPope .

This is less good than more recent issues as the link in the OP does not exist (> 1 y/o so it is cleaned up on the BK side), so I'm going to unduplicate #16508 so that one can be re-opened if this is still failing.