Support for rd_kafka_produce_batch

[cmccarthy1]

Internal Feature Request

Is your feature request related to a problem? Please describe.
Within the librdkafka C api there appears to be functionality to allow the batched send/receive of data. In many use cases this is preferable to the continuous consumption/sending of data.

Describe the solution you'd like
Provide a logical mechanism for q to expose the use of some of the below librdkafka C functions

• rd_kafka_produce_batch

Describe alternatives you've considered
There isn't currently an alternative mechanism within the structure of the interface to provide this functionality

Additional resource

• librdkafka documentation for produce_batch

[sshanks-kx]

confluentinc/librdkafka#112

[sshanks-kx]

Initial code done, but rd_kafka_produce_batch isn't in older librdkafka packages (build fails on travis cos librdkafka too old).
Find release it appeared to see if it may be possible to add condition to add when build with newer release.

[sshanks-kx]

Added in librdkafka release 0.8.4 https://github.com/edenhill/librdkafka/releases/tag/0.8.4

[sshanks-kx]

Looks like batch supported added in 0.8.4 but didn't support RD_KAFKA_MSG_F_PARTITION at this stage …
confluentinc/librdkafka@b8d0b84
Looks like 0.11.4

[sshanks-kx]

Support for batch publish.

Example use:

/ send 2 msgs to any partition for topic, use default key generation
 .kfk.BatchPub[;.kfk.PARTITION_UA;batchMsg;""]each(topic1;topic2);
/ send 2 msgs to partition 1 for topic, use default key generation
 .kfk.BatchPub[;1i;batchMsg;""]each(topic1;topic2);
/ send 2 msgs. First to partition 0 and second to partition 1. Use keys provided for each msg.
.kfk.BatchPub[;(0i;1i);batchMsg;batchKeys]each(topic1;topic2);

* @param x Topic Index (prev created)
 * @param y Partition to use for all message (int) or partition per message (list of ints)
 * @param z Payload for all messages (mixed list containing either bytes or string).
 * @param r Key. Empty string to use auto key for all messages, or key per message (mixed list containing either bytes or string)
 * @returns Integer list with a status for each send message (zero indicates success) 
 *          reference: https://github.com/edenhill/librdkafka/blob/master/src/rdkafka.h (rd_kafka_resp_err_t)

Requires librdkafka be release 0.11.4 and above.
If older, function still exists but returns an error saying its unsupported.