Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

KAFKA-9010: Add PartitionGrower, Support partition requery in ProduceBench test. #7487

Closed
wants to merge 1 commit into from

Conversation

scott-hendricks
Copy link
Contributor

This creates a test in trogdor to grow a topic at a specified interval by a specified number of partitions.

This test also introduces the ability for the ProduceBench test workload to requery the partitions of the active topics at an interval, if so desired. If no query interval is specified, the default behaviour of ProduceBenchWorker remains unchanged.

Copy link
Contributor

@stanislavkozlovski stanislavkozlovski left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have been thinking about the PartitionGrower and spec bloat in general.

There are many cases where it would be useful to run workers that do one-off things (create topics, delete topics, add configs) and all of those operations are AdminClient-based.

My suggestion is we invest the time now and come up with a generic AdminClientSpec/Worker that would lay the foundation for one-off tasks. As it is right now, we would have it support growing the partitions of a topic.
If we do not go this route, we will likely need to introduce a different spec for each different one-off task. This will duplicate a lot of logic and we will ultimately end up with more boilerplate, less readable code and a harder to use API

@scott-hendricks
Copy link
Contributor Author

I have addressed all code comments. As for the AdminClientSpec/Worker classes Stan mentioned, I would argue that would just add complexity. If anything I would think we may want to create a new folder for these types of tests if they become popular, but for now I don't see why they shouldn't be treated like any other Trogdor test.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
3 participants