SyncProducer performance

[victorlcm]

Hi 👋

I'm currently using this extension in the company I work and we're having some issues regarding the performance of sending the messages to our Kafka Broker. In order to speed up things, me and @knmsk are currently working on a full rewrite to match the new Output struct in k6 repository and also changing the SyncProducer to AsyncProducer.

The Idea is to improve throughput by using the AsyncProducer, batching messages and running in go routines. Also, we think that we should include an option to wait for acks from Kafka or not. This should be a choice for the users if they're willing to lose some messages to gain performance

Please let us know what you think 😄

[mstoykov]

this output is already matching the new Output

xk6-output-kafka/pkg/kafka/collector.go

Line 46 in 17c4400

var _ output.Output = new(Collector)

.

Other than that I guess this can be beneficial, I just don't know how much, but if you are willing to do it - go for it :)

[victorlcm]

@mstoykov yes, sorry, what I meant was to use the helpers like the periodic flusher 😄

[mstoykov]

I am closing this as #10 changed the extension to use the asyncproducer