Q: What is the intended design for fast producers ?

[buixor]

Hello,
disclaimer : I'm pretty new to both go and kafka

While playing around with kafka-go, I noticed that the WriteMessages() is blocking until it finished delivering messages. Is the expected usage to manage a set of goroutines to ensure fast throughtput, or is there a method I missed in the API to do that ?

Thanks in advance,

[achille-roussel]

The writer configuration has an "Async" boolean property which makes WriteMessages a non-blocking operation see https://godoc.org/github.com/segmentio/kafka-go#WriterConfig.

Keep in mind that in this mode there is no way to know whether the write was successful, if the kafka servers are unreachable or the program exits before the async send happened the messages would get lost.

Your idea of using multiple producer goroutines with blocking WriteMessages calls would also work, and you'd receive an error if producing the messages fail.

[buixor]

Thanks for the prompt answer !

I will thus stick with goroutines :)