Document the working of QueueCapacity/BatchSize/BatchTimeout

[dgregoire]

I've been using the Writer with a deadlined Context and I'm unsure how to configure it to allow batching while not blocking for the whole deadline duration.

If I set a BatchSize to 1, all is well but the load on the brokers and the network traffic is considerable. It's greatly reduced by setting BatchSize to another number - like 2, however WriteMessages blocks for the duration of the deadline (300ms here)

I believe my understanding of the documentation is wrong?

Unless the writer was configured to write messages asynchronously, the method blocks until all messages have been written, or until the maximum number of attempts was reached.

There's about 1.3k WriteMessages per second in this test, with BatchSize set to 2 and RequiredAcks set to 1.

[achille-roussel]

@dgregoire are you able to share which version of Kafka you had done these tests against, as well as the number of brokers and partitions in the topic? The writer definitely shouldn't block until the deadline is reached, because we have tests that verify its behavior.

If it timed out due to reaching the context deadline you should be getting en error returned from the (*Writer).WriteMessages call, is this what you observed? (seems like it from your screenshot but I just wanted to confirm)

A batch size of 2 is quite small, I would expect this to put significant load on the Kafka brokers, which means some requests could timeout if there is concurrency involved and you hit hardware limits on the underlying hardware (like disk bandwidth, IOPS, CPU, etc...).

[dgregoire]

@achille-roussel Kafka 1.0.1 with 3 brokers, 12 partitions / topic.

The specific use case is:

• default batch size of 5/2
• context with timeout in 100/200/500ms
• WriteMessages() with 1 kafka.Message

In all combinaisons I tested, the WriteMessages function locks until the context is deadline exceeded and the function returns that error, unless the configured batch size is 1.

I suspected this might happen when using send messages with a number of message != batch size but never proved that theory, what do you think?