PESDLC-919 Constant data generation workload #16843
Conversation
savex
force-pushed
the
PESDLC-919-topics-constant-workload
branch
from
59a90e5
to
dfffcf1
Compare
savex
force-pushed
the
PESDLC-919-topics-constant-workload
branch
3 times, most recently
from
fcb1d3f
to
8f96bea
Compare
|
/ci-repeat 1 |
|
new failures in https://buildkite.com/redpanda/redpanda/builds/45647#018e0b57-b3d0-459d-99e1-6691f079f413: |
savex
force-pushed
the
PESDLC-919-topics-constant-workload
branch
2 times, most recently
from
3443039
to
532631b
Compare
|
ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/45758#018e15ae-a9f1-4f7f-b2dd-9a5d8c4f7145 ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/45758#018e15c0-0960-4b12-8f0a-266ab06deaac ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/45916#018e2112-3edd-49c1-be0a-c6db8cd04c08 |
ballard26
force-pushed
the
PESDLC-919-topics-constant-workload
branch
from
e269650
to
7675ec7
Compare
| self._max_record_size = max_record_size | ||
| self._keys = keys | ||
| self._unique_topics = unique_topics | ||
| self._unique_topics = unique_groups |
There was a problem hiding this comment.
Will note and correct in the first PR to follow.
| Iterate. | ||
| """ | ||
| # Prepare librdkafka python client | ||
| kclient = PythonLibrdkafka(self.redpanda) |
There was a problem hiding this comment.
As I mentioned in the past I don't think this is great because this will run on the ducktape monitoring/client node.
There was a problem hiding this comment.
Yes, and I had an idea to implement it inside remote script additions. But that is really a lot of effort at this point and we are already really tight on schedule. So this is merely a minimal load generated for the topic creation script.
| nodes=[n]) for n in nodes | ||
| ] | ||
| stddev = numpy.std(leaders_per_node) | ||
| error = stddev / (self._get_partition_count() / len(nodes)) |
There was a problem hiding this comment.
similar code exists in the MPT already.
There was a problem hiding this comment.
That code is operated on a per-topic basis and will run very long for 40k. This one is faster for our cause.
|
Approved as per slack comment |
Have a universal method that will provide constant data generation with configurable parameters using scale profiles
Backports Required
Release Notes