tests: Use multiple topics in OMBValidationTest.test_max_partitions #16000
Conversation
| MAX_PARTITIONS_PER_TOPIC = 5000 | ||
| topics = math.ceil(tier_limits.max_partition_count / MAX_PARTITIONS_PER_TOPIC) | ||
|
|
||
| partitions_per_topic = tier_limits.max_partition_count // topics |
There was a problem hiding this comment.
nit: I think you could do a ciel(... / topics) here as above instead of // so we are conservative with the rounding: if not exact we will test slightly more than the advertised number rather than slightly less.
There was a problem hiding this comment.
Yep makes sense, amended.
StephanDollberg
force-pushed
the
stephan/omb-validation-max-part-multiple-topics
branch
from
1b61a13
to
9e6fe57
Compare
|
new failures in https://buildkite.com/redpanda/redpanda/builds/43594#018ceebf-09c3-49b2-90d6-cbb84c463174: new failures in https://buildkite.com/redpanda/redpanda/builds/43603#018cef30-8c94-4307-bb82-119903587126: new failures in https://buildkite.com/redpanda/redpanda/builds/43603#018cef30-8c9e-429b-95e6-a22815756a73: new failures in https://buildkite.com/redpanda/redpanda/builds/43603#018cef30-8c98-4137-876a-9c810e8276b1: new failures in https://buildkite.com/redpanda/redpanda/builds/43603#018cef59-05b6-4c6e-be89-6010edea1167: new failures in https://buildkite.com/redpanda/redpanda/builds/43603#018cef59-05b3-4c89-81d2-0a09216476c1: new failures in https://buildkite.com/redpanda/redpanda/builds/43603#018cef59-05b0-44c6-bf8f-6864f88acb21: |
Many kafka client libraries (java, librdkafka) perform poorly when producing to very high partition topics. This leads to one thread not being fast enough to produce at a reasonable batch size. In a tier 6 test we were seeing batch size slowly growing all the way to 85K. This isn't great because it means we are not running the test in a stable state and this becomes more a client benchmark. Further lower batch sizes are actually more demanding for RP itself. To avoid this problem we split the total partition count up across multiple topics and limit each to a max of 5000. Each producer will only produce to one topic and hence it's easier on them. This is similar to a pattern that we have already seen employed by customers with similar high partition counts. With this change batch size stays stable at 10K from the beginning of the test. Total producer/consumer count might be slightly different because of rounding but that shouldn't be an issue as the calculation for the optimal producer count is fairly conservative already.
StephanDollberg
force-pushed
the
stephan/omb-validation-max-part-multiple-topics
branch
from
9e6fe57
to
e89fea7
Compare
|
Linted |
|
Those failures are all unrelated to this test (caused by adding the 24.1 tag). |
piyushredpanda
deleted the
stephan/omb-validation-max-part-multiple-topics
branch
Many kafka client libraries (java, librdkafka) perform poorly when
producing to very high partition topics. This leads to one thread not
being fast enough to produce at a reasonable batch size.
In a tier 6 test we were seeing batch size slowly growing all the way to
85K. This isn't great because it means we are not running the test in a
stable state and this becomes more a client benchmark. Further lower
batch sizes are actually more demanding for RP itself.
To avoid this problem we split the total partition count up across
multiple topics and limit each to a max of 5000. Each producer will only
produce to one topic and hence it's easier on them.
This is similar to a pattern that we have already seen employed by
customers with similar high partition counts.
With this change batch size stays stable at 10K from the beginning of
the test. Total producer/consumer count might be slightly different
because of rounding but that shouldn't be an issue as the calculation
for the optimal producer count is fairly conservative already.
Backports Required
Release Notes