Checkpointing using the __consumer_offsets topic.

[chinmaychandak]

I see that whenever a consumer consumes messages from a topic, the latest metadata and offsets for this are updated in the __consumer_offsets topic.

I wanted to know if there exists an efficient way today to retrieve the partitions (& therefore offsets) specific to a group.id+topic from this __consumer_offsets topic. Can someone please help me with this?

I was thinking of implementing checkpointing conforming to Spark Streaming in streamz. But since checkpointing requires creation of checkpoint (partition:offset dictionary) text-files after the messages have been processed, I was thinking of using the already existing information in __consumer_offsets to do this.

[rnpridgeon]

Closest thing we have today is the Consumer.commited() api. This will be bound to the group the consumer is a member of however.

https://docs.confluent.io/current/clients/confluent-kafka-python/#confluent_kafka.Consumer.committed

I suspect instead you are looking to something more similar to the functionality provided by the java admin client.

https://kafka.apache.org/20/javadoc/org/apache/kafka/clients/admin/AdminClient.html#listConsumerGroupOffsets-java.lang.String-

[chinmaychandak]

Thanks @rnpridgeon! I went ahead and implemented it using committed().