Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow ksql-datagen's msgRate to have slower cadences. #10351

Open
pavel-kalmykov opened this issue May 27, 2024 · 0 comments
Open

Allow ksql-datagen's msgRate to have slower cadences. #10351

pavel-kalmykov opened this issue May 27, 2024 · 0 comments

Comments

@pavel-kalmykov
Copy link

Is your feature request related to a problem? Please describe.
When running ksql-datagen with the msgRate parameter using a decimal number, I get the following error:

Invalid integer value '0.2'; must be a valid base 10 integer
usage: DataGen 
[help] 
[bootstrap-server=<kafka bootstrap server(s)> (defaults to localhost:9092)] 
[quickstart=<quickstart preset> (case-insensitive; one of 'orders', 'users', or 'pageviews')] 
schema=<avro schema file> 
[schemaRegistryUrl=<url for Confluent Schema Registry> (defaults to http://localhost:8081)] 
key-format=<message key format> (case-insensitive; one of 'avro', 'json', 'kafka' or 'delimited') 
value-format=<message value format> (case-insensitive; one of 'avro', 'json' or 'delimited') 
valueDelimiter=<message value delimiter> (case-insensitive; delimiter to use when using delimited value format. Must be a single character or special values 'SPACE' or 'TAB'. Defaults to ',')
topic=<kafka topic name> 
key=<name of key column> 
timestamp=<name of timestamp column> 
[iterations=<number of rows> (if no value is specified, datagen will produce indefinitely)] 
[propertiesFile=<file specifying Kafka client properties>] 
[nThreads=<number of producer threads to start>] 
[msgRate=<rate to produce in msgs/second>] 
[printRows=<true|false>]

This validation, of course, makes it obvious that the parameter's type must be an integer. Because of this validation, the message cadence cannot go below 1 msg/sec.

Describe the solution you'd like
By allowing decimal numbers, we could have a slower cadence. For instance, if I wanted to generate one message every 10 seconds, I would set msgRate to 0.1.

Describe alternatives you've considered
Another possibility would be to allow specifying a time unit (seconds, milliseconds, s, ms...). Both proposals are not mutually exclusive.

Additional context
I am aware that the maxInterval parameter was deprecated in favour of this one; however, the former had milliseconds as a unit, making it feasible to configure a 1msg/10secs cadence, and forcing the usage of msgRate (with seconds as its unit) comes across as a regression. Additionally, having this tool parameter's types for the help command and the site's documentation would be more than helpful.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant