rptest: Create multi-node test with 1k topics and >10M events #16375

savex · 2024-01-30T23:23:52Z

Goal is to generate as many topics as possible until RP starts to fail.
Flink workload is used with auto-sized task managers and flexible failure rate settings.

Workloads should generate low to low-moderate data to random topics. Data rate should be controllable.

Milestones

1k tops

Fixes: redpanda-data/devprod#1013

Backports Required

Release Notes

none

emaxerrno · 2024-01-31T22:37:53Z

why not 10k or 100k, too slow of a test? curious.

savex · 2024-01-31T23:28:25Z

It is possible. The test becomes very complex and there is too many moving parts to create it whole from scratch.
Right now there is a few bottlenecks to consider

Slow topic creation. Solvable.
Workloads in flink has to be isolated due to Flink's auto-parallelizing nature. Takes time to build and debug. And it will be complicated to support.
These tests have more of a stress commit_tx swarming nature (see here for latest details) not huge amount of topics.

On the other hand, to my humble opinion, E2E tests should simulate real-like traffic as much as possible and it is not a swarm like approach, but steady random events with variations in sizing and delays. Like, 10k topics onboard, but active amount is about 30% at max with random active topics each timeframe.

vbotbuildovich · 2024-02-05T18:20:56Z

new failures in https://buildkite.com/redpanda/redpanda/builds/44703#018d7a36-570f-436a-86a9-64d878b2cc42:

"rptest.tests.partition_move_interruption_test.PartitionMoveInterruption.test_cancelling_partition_move.replication_factor=3.unclean_abort=True.recovery=no_recovery.compacted=False"
"rptest.tests.topic_creation_test.TopicRecreateTest.test_topic_recreation_while_producing.workload=IDEMPOTENT.cleanup_policy=delete"

new failures in https://buildkite.com/redpanda/redpanda/builds/45171#018dc95f-3d66-472d-9127-5b654fade6df:

"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"

new failures in https://buildkite.com/redpanda/redpanda/builds/45171#018dc95f-3d67-4773-b236-ae9728911652:

"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"

new failures in https://buildkite.com/redpanda/redpanda/builds/45171#018dc95f-3d63-478e-880c-7bb336aabdb9:

"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"

new failures in https://buildkite.com/redpanda/redpanda/builds/45171#018dc95f-3d64-4031-9543-1b64e0756e06:

"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"

new failures in https://buildkite.com/redpanda/redpanda/builds/45306#018dd73f-27c6-4aaf-9759-796aae8b8a1d:

"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"

new failures in https://buildkite.com/redpanda/redpanda/builds/45306#018dd73f-27c6-4b4c-be77-1a0384655e33:

"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"

new failures in https://buildkite.com/redpanda/redpanda/builds/45306#018dd73f-27c1-4a07-94eb-453a926d5cb1:

"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"

new failures in https://buildkite.com/redpanda/redpanda/builds/45306#018dd73f-27c2-4039-bfee-444271e5478d:

"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"
"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"

new failures in https://buildkite.com/redpanda/redpanda/builds/45831#018e1b11-1c41-4dcf-967b-1d0d74c99cca:

"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"

new failures in https://buildkite.com/redpanda/redpanda/builds/45831#018e1b11-1c45-44a6-b37f-77aa1d0047b0:

"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"

new failures in https://buildkite.com/redpanda/redpanda/builds/46141#018e39d6-c0e3-4840-bf1a-b93e3ad78139:

"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"

new failures in https://buildkite.com/redpanda/redpanda/builds/46141#018e39d6-c0e6-46ee-9037-afcafa33c669:

"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"

new failures in https://buildkite.com/redpanda/redpanda/builds/46141#018e39e4-7484-4623-86b2-d7459adb6e2a:

"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=False"

new failures in https://buildkite.com/redpanda/redpanda/builds/46141#018e39e4-88f6-450e-8c19-544a464e52b1:

"rptest.e2e_tests.flink_scale_test.FlinkScaleTests.test_transactions_scale_single_node.unique_topics=True"

vbotbuildovich · 2024-02-05T18:24:50Z

ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/44703#018d7a36-570f-436a-86a9-64d878b2cc42

ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/45254#018dd2ae-3954-47d6-a761-711e2d744ebd

ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/46196#018e3dbb-e273-491d-97b9-b5ce346c792e

ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/46646#018e68b0-56f5-4059-b746-9c2225ee3f70

ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/46646#018e68b0-56ec-47a2-aa6b-e7a5bc4f23d9

savex · 2024-02-08T14:26:13Z

@emaxerrno, we reached 65k here when using 6xlarge: #16463
With the use of 12xlarge, we reached 106k, after that multiple nodes out of 9 crashing. That's yet to be investigated. Focus at hand make sure that those topics can support at least low bandwidth traffic.

bharathv · 2024-02-21T01:02:10Z

/ci-repeat 2
skip-unit
dt-repeat=30
tests/rptest/e2e_tests/flink_scale_test.py

bharathv · 2024-02-21T16:59:45Z

@savex can you check the failures please? Is test_transactions_scale_swarm should it be a scale_test?

savex · 2024-02-21T20:42:19Z

This is a part of e2e activity that would simulate some real-like client activities. I would not want to mix them up with our internal scale tests.

bharathv · 2024-02-21T22:56:12Z

not sure I follow, the test is timing out (not finishing in 30 mins or less), so not a good fit for e2e or maybe there is a bug.

savex · 2024-02-23T17:17:32Z

/ci-repeat 2
skip-unit
dt-repeat=30
tests/rptest/e2e_tests/flink_scale_test.py

savex · 2024-03-07T21:42:05Z

/ci-repeat 1
skip-unit
tests/rptest/e2e_tests/flink_scale_test.py

savex · 2024-03-13T20:41:28Z

/ci-repeat 1
skip-unit
tests/rptest/e2e_tests/flink_scale_test.py

savex · 2024-03-14T15:24:15Z

Updated with skips for debug version of RP as it would not offer required performance.

savex · 2024-03-14T18:12:53Z

/ci-repeat 2
skip-unit
dt-repeat=30
tests/rptest/e2e_tests/flink_scale_test.py

savex · 2024-03-15T18:20:45Z

/ci-repeat 2
skip-unit
dt-repeat=30
tests/rptest/e2e_tests/flink_scale_test.py

tests/rptest/e2e_tests/flink_scale_test.py

bharathv · 2024-03-22T17:43:19Z

tests/rptest/e2e_tests/flink_scale_test.py

+            target_total_events = 500 * 1024
+        else:
+            # Prepare topics for EC2
+            # Total number of workloads would be 2 x 5 = 10


Is this comment accurate?

total flink nodes = 5
workloads_per_node = 4

total = 20?

Yes, this should be updated. Thanks

total workloads 4 * 5 = 20
total topics 4 * 5 * 25 = 500

tests/rptest/e2e_tests/flink_scale_test.py

To test multi-node transactions at scale (>10M) multi-node test is created. Also, single node test modified to reuse similar code Also, updated transaction rate validation formulas since there should be more clear validation than used before with explanation on what it going on.

Treat CREATED and SCHEDULED as active statuses. In case of docker, job manager is slower and this will cause flink.wait() to function properly Also, flink_scale tests would not work on debug version of RP

savex · 2024-03-22T23:08:34Z

/ci-repeat 2
skip-unit
dt-repeat=30
tests/rptest/e2e_tests/flink_scale_test.py

savex self-assigned this Jan 30, 2024

savex force-pushed the rp-1013-topic-swarm-test branch from fcdab33 to 6988e17 Compare January 31, 2024 22:20

savex changed the title ~~Topic swarm test~~ rptest: Create multi-node test with 1k topics and >10M events Jan 31, 2024

savex force-pushed the rp-1013-topic-swarm-test branch 3 times, most recently from 0484be9 to 1c50027 Compare February 1, 2024 19:33

savex requested a review from bharathv February 1, 2024 19:33

savex marked this pull request as ready for review February 1, 2024 19:33

savex force-pushed the rp-1013-topic-swarm-test branch from 1c50027 to 0973e45 Compare February 5, 2024 16:06

savex force-pushed the rp-1013-topic-swarm-test branch from 0baaf0b to 0118ef5 Compare February 22, 2024 17:47

savex force-pushed the rp-1013-topic-swarm-test branch from 4de69a8 to 977f9ab Compare March 13, 2024 19:33

savex force-pushed the rp-1013-topic-swarm-test branch from 977f9ab to 123e512 Compare March 14, 2024 15:18

savex force-pushed the rp-1013-topic-swarm-test branch from 123e512 to 3f5abbe Compare March 15, 2024 18:19

bharathv reviewed Mar 22, 2024

View reviewed changes

savex force-pushed the rp-1013-topic-swarm-test branch from 3f5abbe to a386410 Compare March 22, 2024 23:01

savex added 2 commits March 22, 2024 18:06

rptest: Update single_node test

6abb710

rptest: Update flink job status handling

743fc35

Treat CREATED and SCHEDULED as active statuses. In case of docker, job manager is slower and this will cause flink.wait() to function properly Also, flink_scale tests would not work on debug version of RP

savex force-pushed the rp-1013-topic-swarm-test branch from a386410 to 743fc35 Compare March 22, 2024 23:06

savex requested a review from bharathv March 25, 2024 13:07

bharathv approved these changes Mar 26, 2024

View reviewed changes

bharathv merged commit 7273643 into dev Mar 26, 2024
16 checks passed

bharathv deleted the rp-1013-topic-swarm-test branch March 26, 2024 02:58

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

rptest: Create multi-node test with 1k topics and >10M events #16375

rptest: Create multi-node test with 1k topics and >10M events #16375

savex commented Jan 30, 2024 •

edited

emaxerrno commented Jan 31, 2024

savex commented Jan 31, 2024

vbotbuildovich commented Feb 5, 2024 •

edited

vbotbuildovich commented Feb 5, 2024 •

edited

savex commented Feb 8, 2024

bharathv commented Feb 21, 2024

bharathv commented Feb 21, 2024

savex commented Feb 21, 2024

bharathv commented Feb 21, 2024

savex commented Feb 23, 2024

savex commented Mar 7, 2024

savex commented Mar 13, 2024

savex commented Mar 14, 2024

savex commented Mar 14, 2024

savex commented Mar 15, 2024

bharathv Mar 22, 2024

savex Mar 22, 2024

savex Mar 22, 2024

savex commented Mar 22, 2024

rptest: Create multi-node test with 1k topics and >10M events #16375

rptest: Create multi-node test with 1k topics and >10M events #16375

Conversation

savex commented Jan 30, 2024 • edited

Backports Required

Release Notes

emaxerrno commented Jan 31, 2024

savex commented Jan 31, 2024

vbotbuildovich commented Feb 5, 2024 • edited

vbotbuildovich commented Feb 5, 2024 • edited

savex commented Feb 8, 2024

bharathv commented Feb 21, 2024

bharathv commented Feb 21, 2024

savex commented Feb 21, 2024

bharathv commented Feb 21, 2024

savex commented Feb 23, 2024

savex commented Mar 7, 2024

savex commented Mar 13, 2024

savex commented Mar 14, 2024

savex commented Mar 14, 2024

savex commented Mar 15, 2024

bharathv Mar 22, 2024

Choose a reason for hiding this comment

savex Mar 22, 2024

Choose a reason for hiding this comment

savex Mar 22, 2024

Choose a reason for hiding this comment

savex commented Mar 22, 2024

savex commented Jan 30, 2024 •

edited

vbotbuildovich commented Feb 5, 2024 •

edited

vbotbuildovich commented Feb 5, 2024 •

edited