CI Failure (p99 spikes over threshold) in OMBValidationTest.test_max_partitions

[vbotbuildovich]

https://buildkite.com/redpanda/vtools/builds/13711

Module: rptest.redpanda_cloud_tests.omb_validation_test
Class: OMBValidationTest
Method: test_max_partitions

test_id:    OMBValidationTest.test_max_partitions
status:     FAIL
run time:   787.923 seconds

AssertionError("['Metric aggregatedEndToEndLatency99pct, value 131.356, Expected to be <= 120.0, check failed.', 'Metric aggregatedEndToEndLatency999pct, value 310.797, Expected to be <= 200.0, check failed.']")
Traceback (most recent call last):
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 184, in _do_run
    data = self.run_test()
  File "/opt/.ducktape-venv/lib/python3.10/site-packages/ducktape/tests/runner_client.py", line 276, in run_test
    return self.test_context.function(self.test)
  File "/home/ubuntu/redpanda/tests/rptest/services/cluster.py", line 103, in wrapped
    r = f(self, *args, **kwargs)
  File "/home/ubuntu/redpanda/tests/rptest/redpanda_cloud_tests/omb_validation_test.py", line 458, in test_max_partitions
    benchmark.check_succeed()
  File "/home/ubuntu/redpanda/tests/rptest/services/openmessaging_benchmark.py", line 384, in check_succeed
    OMBSampleConfigurations.validate_metrics(self._metrics,
  File "/home/ubuntu/redpanda/tests/rptest/services/openmessaging_benchmark_configs.py", line 123, in validate_metrics
    assert is_valid, str(results)
AssertionError: ['Metric aggregatedEndToEndLatency99pct, value 131.356, Expected to be <= 120.0, check failed.', 'Metric aggregatedEndToEndLatency999pct, value 310.797, Expected to be <= 200.0, check failed.']

JIRA Link: CORE-2959

[rpdevmp]

There are 2 readings in a row above allowed:

170.727, 370.477,

"endToEndLatency99pct" : [ 43.147, 51.325, 47.856, 46.15, 42.678, 44.578, 45.264, 43.281, 46.71, 45.613, 44.348, 45.829, 44.663, 44.317, 48.659, 45.526, 170.727, 370.477, 43.558, 49.542, 49.821, 47.037, 51.814, 46.226, 44.755, 47.846, 44.819, 44.503, 42.481, 48.834 ],

209.434, 456.771

"endToEndLatency999pct" : [ 54.919, 157.728, 61.438, 59.856, 56.24, 55.506, 58.216, 57.755, 67.128, 132.129, 56.255, 58.183, 57.662, 53.804, 59.724, 56.227, 209.434, 456.771, 54.495, 64.736, 64.122, 122.681, 77.082, 60.931, 55.851, 60.86, 66.105, 60.409, 52.206, 62.648 ],

[vbotbuildovich]

*https://buildkite.com/redpanda/vtools/builds/13741

[vbotbuildovich]

*https://buildkite.com/redpanda/vtools/builds/14516

[vbotbuildovich]

*https://buildkite.com/redpanda/vtools/builds/14618
*https://buildkite.com/redpanda/vtools/builds/14622

[vbotbuildovich]

*https://buildkite.com/redpanda/vtools/builds/14862

[travisdowns]

@rpdevmp - but are they disk spikes? If they fit the pattern of temporary blips in disk performance then we should still treat it as a spike, e.g., maybe relax the rule about only 1 in a row.

[vbotbuildovich]

*https://buildkite.com/redpanda/vtools/builds/15506
*https://buildkite.com/redpanda/vtools/builds/15544

[vbotbuildovich]

*https://buildkite.com/redpanda/vtools/builds/15673

[vbotbuildovich]

*https://buildkite.com/redpanda/vtools/builds/15781

[vbotbuildovich]

*https://buildkite.com/redpanda/vtools/builds/15886

[vbotbuildovich]

*https://buildkite.com/redpanda/vtools/builds/16111

[vbotbuildovich]

*https://buildkite.com/redpanda/vtools/builds/16378

[vbotbuildovich]

*https://buildkite.com/redpanda/vtools/builds/16404

[piyushredpanda]

Closing older-bot-filed CI issues as we transition to a more reliable system.