KAFKA-7667: Synchronous send support for kafka performance producer java application #5939
Conversation
|
Asynchronous_send_summary_report.txt added files, capturing the summary report at the end of producer run. |
mrsrinivas
left a comment
mrsrinivas
left a comment
There was a problem hiding this comment.
Thanks for the PR!!!
Will it be possible to continue using existing formatting?
Code seem indented with more spaces.
|
retest this please. |
|
@kaushiksrinivas Thanks for the PR. Since ProducerPerformance tool is considered as public interface, we need to write small KIP. https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals |
| percs[3]); | ||
| System.out.printf( | ||
| "%d records sent, %f records/sec (%.2f MB/sec), %.2f ms avg latency, %.2f ms max latency, %d ms 50th, %d ms 95th, %d ms 99th, %d ms 99.9th.%n", | ||
| count, recsPerSec, mbPerSec, totalLatency / (double) count, (double) maxLatency, percs[0], percs[1], |
There was a problem hiding this comment.
nit: one line per parameter for method call when statement is long acceptable.
| .dest("transactionDurationMs") | ||
| .setDefault(0L) | ||
| .help("The max age of each transaction. The commitTransaction will be called after this time has elapsed. Transactions are only enabled if this value is positive."); | ||
| parser.addArgument("--topic").action(store()).required(true).type(String.class).metavar("TOPIC") |
There was a problem hiding this comment.
nit: same formatting would have been nice for all places with parser.
|
This PR is being marked as stale since it has not had any activity in 90 days. If you If you are having difficulty finding a reviewer, please reach out on the [mailing list](https://kafka.apache.org/contact). If this PR is no longer valid or desired, please feel free to close it. If no activity occurs in the next 30 days, it will be automatically closed. |
|
This PR has been closed since it has not had any activity in 120 days. If you feel like this |
3 participants
ProducerPerformance java application now supports synchronous blocking sends to produce messages to kafka brokers.
Till now, performance producer application supported only asynchronous sends with a callback function registered.
this would be enough when trying to load the kafka brokers without bothering much on the replication & producer responses.
But when kafka brokers replication capacity has to be loaded, asynchronous sends would not suffice.
So synchronous blocking send calls are now supported with all the existing metrics calculation of producer performance.
A boolean command line flag --synchronous-send has been added to enable/disable this feature.
Blocking .get() calls are now done upon each send, and as of this commit ResultMetadata is not being used or logged.
A new function has been introduced in the Stats class to update and get the iteration counter.
Tests done:
Comparison tests were done with and without synchronous sends.
below were the configurations of kafka cluster:
No of brokers: 3
CPU per broker: 10 cores
Memory : 32GB
Topic partitions : 1
replication factor : 3
min.insync.replicas: 1 (default).
producer acks level : 1(default), response after only leader write.
Similar configurations were given for performance producer with & without synchronous sends enabled.
It was observed that, synchronous producer was producing records approximately 5 times slower than asynchronous producer
but with guarantee of replication before subsequent sends.
So this clearly indicates the impact of replication in brokers. And these tests would help optimise resources in
kafka broker from replication performance point of view.
Attached results snapshot of performance producer run with/without synchronous send flag enabled.