Safely consuming batch of messages

[arnaudbriche]

Hi,

I'm new to kafka so I might miss something; but here's what I want and think I cannot achieve with the Consumer type.
I'd like to be able to fetch 100 messages, process them, and only then notify Zookeeper that it can move my consumer group offset. This would allow my process to fail in the middle of a batch processing safely: so that the next time I start it, I can start processing the same batch again.
Is this possible with the current state of the library ?

[eapache]

The Consumer type doesn't support moving consumer group offsets at all, because at the time I tried to implement it that part of Kafka was broken anyways so I gave up (presumably it has since been fixed, I haven't checked).

You can achieve what you want by using a Consumer, and counting the messages you process. Every 100 messages, send an OffsetCommitRequest to Kafka with the appropriate data.

[arnaudbriche]

Ok. Then, should I instanciate a Broker by myself to emit this OffsetCommitRequest, or can I get one from the Consumer object ?

[eapache]

You should get the broker by calling Leader on the Client you used to instantiate the Consumer.

[arnaudbriche]

Thx eapache.

[sanchitlohia-ovo]

@eapache How can I reread the same batch again