Stabilize OMB regression tests #16159
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Latency and throughput validators are derived from runs of the test over a two week period.
End to end latency is effected by build a consumer backlog making it variable and higher than expected. Because of this it isn't a good signal for regression.
|
new failures in https://buildkite.com/redpanda/redpanda/builds/43928#018d2046-7cb3-4e52-971f-ff391e1b67c0: |
|
ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/43928#018d2058-260c-4440-9856-0384baddbe11 ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/43928#018d2058-2609-4c37-91d7-690de988b13c |
StephanDollberg
approved these changes
Jan 19, 2024
| self._ctx, self.redpanda, "ACK_ALL_GROUP_LINGER_1MS", | ||
| (workload, OMBSampleConfigurations.UNIT_TEST_LATENCY_VALIDATOR)) | ||
| benchmark = OpenMessagingBenchmark(self._ctx, self.redpanda, | ||
| "ACK_ALL_GROUP_LINGER_1MS", |
There was a problem hiding this comment.
Did we ever check what effective batch size this results in? This seems more like a many partition test than a small batch size one. I'd also be surprised if the client machines can handle 200 producers with 20k partitions. I expect we get lots of client induced latency here.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changes and removes validation from the
small_batches_testandts_read_openmessaging_testto remove more false negatives and therefore making failures a more accurate signal for regression.Backports Required
Release Notes