CI Failure (Unable to attain target rate in time) in FullDiskTest.test_full_disk_no_produce

[michael-redpanda]

https://buildkite.com/redpanda/redpanda/builds/31036#0188a907-224e-4636-b364-8e7c76dfa7c4

Module: rptest.tests.full_disk_test
Class:  FullDiskTest
Method: test_full_disk_no_produce

test_id:    rptest.tests.full_disk_test.FullDiskTest.test_full_disk_no_produce
status:     FAIL
run time:   1 minute 58.114 seconds


    RuntimeError('Unable to attain target rate in time.')
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 135, in run
    data = self.run_test()
  File "/usr/local/lib/python3.10/dist-packages/ducktape/tests/runner_client.py", line 227, in run_test
    return self.test_context.function(self.test)
  File "/root/tests/rptest/services/cluster.py", line 79, in wrapped
    r = f(self, *args, **kwargs)
  File "/root/tests/rptest/tests/full_disk_test.py", line 248, in test_full_disk_no_produce
    self.bytes_prod.expect_rate(not_producing)
  File "/root/tests/rptest/utils/expect_rate.py", line 111, in expect_rate
    raise RuntimeError("Unable to attain target rate in time.")
RuntimeError: Unable to attain target rate in time.

[ztlpn]

https://buildkite.com/redpanda/redpanda/builds/31153#0188b34c-f100-4ad4-a273-9d44441b5a78

[michael-redpanda]

https://buildkite.com/redpanda/redpanda/builds/31297#0188bac0-d13a-450a-badf-77ab2a224415

[jcsp]

https://buildkite.com/redpanda/redpanda/builds/32508#018924b5-3427-4d63-b300-d891023b0536

[abhijat]

seen in https://buildkite.com/redpanda/redpanda/builds/32924#01894414-ee9d-482c-a943-bc242766cae4

https://ci-artifacts.dev.vectorized.cloud/redpanda/32924/01894414-ee9d-482c-a943-bc242766cae4/vbuild/ducktape/results/2023-07-11--001/FullDiskTest/test_full_disk_no_produce/45/

[piyushredpanda]

I wonder if @dotnwat's changes are impacting this?

[dotnwat]

I wonder if @dotnwat's changes are impacting this?

I see that the condition upon which this test fails is calculated based on metrics related to produce rate. IIRC we recently saw a bug in fetch latency metrics. Could there be something going on with produce as well?

However, I can imagine one thing from my work that might be interfering with this test which I'll take a look at.

[andijcr]

#12023 A longer timeout only in debug mode helps, so at least it's nothing getting stuck

[rystsov]

https://buildkite.com/redpanda/redpanda/builds/34840

[andijcr]

A quick test run seems to confirm that the test fails on 23.2.x and not on HEAD. #12953 #12954
A redpanda PR that wasn't backported?

[andijcr]

closing since the backport was merged