Add producing batched messages

[Gvain]

To improve throughput, i think we should add producing batched messages as rd_kafka_consume_batch() do.
And also producing batched messages is supported by JAVA API.

[edenhill]

Good idea!

It should probably take an array of rd_kafka_message_t allowing per-message key, opaques, etc, to be set. This would even allow for per-message partitioning, if that is of interest, even though that would slow things down - so maybe that should be unsupported.

int rd_kafka_produce_batch (rd_kafka_topic_t *rkt, [int32_t partition,] rd_kafka_message_t *rkmessages, int message_cnt, int msgflags);
Returns the number of messages accepted, unaccepted messages will have their 'rkmessages[N].err` value set accordingly.
It would be nice to have per-message msgflags (COPY, FREE) but that would require a whole new struct (since the current ones cant be changed due to ABI restrictions), but this could be left as a FIXME for the API v2 work.

dr_cb, if set, will be called for each message, not for the whole batch.

Sounds good?

[Gvain]

Sounds good.
One more thing. If i set "batch.num.messages", would this have some what the same effect with produce batched messages ?

[edenhill]

Internally librdkafka already batches messages sent to the broker, and that is what batch.num.messages controls: how many messages to grab from internal queue and send to broker in one Produce request.

The new produce_batch() API is not related to batch.num.messages, it will simply allow adding multiple messages to the internal queues in one call (and one lock!), which will be a win performance wise.

[Gvain]

Get it. Thanks a lot.

[edenhill]

There you go, rd_kafka_produce_batch().
See rdkafka.h for documentation and tests/0011-produce_batch.c for a usage example.

Please verify this on your end.

[edenhill]

Hey @Gvain, have you had time to test the new batch call yet?

[Gvain]

@edenhill I am so sorry for the late reply ! The notification email was wrongly ignored by a newly start up filter. The new batch call seems good. Thank you.

[kparadkar]

Do you have any performance numbers on use of rd_kafka_produce_batch vs rd_kafka_produce? Does it give significant increment in the throughput?

[edenhill]

@kparadkar Expirements have shown that there isn't typically much to gain from using the produce_batch() API.
The improvements it provides over the standard produce() API is mainly when you are producing a large number of messages to the same partition since it will cut down on the number of lock acquisitions from one per message to one per batch.

But, as is always the case with performance measurements, you will need to do your own using your own environment and usage.

[archanapujar]

How does rd_kafka_produce_batch() handle NULL topic handle. Will it error out with something like EINVAL?

[edenhill]

@archanapujar NULL topic handle is not supported and the produce_batch() call will crash.
It is up to the application to provide valid function arguments.

[archanapujar]

@edenhill Thanks for the quick response! Does this behavior hold good for all the supported apis?

[edenhill]

Generally yes