expiration timeout for committed offsets

[FDePourcq]

Context: the kafka server will keep a committed offset by default for a duration of offsets.retention.minutes .
The protocol allows the clients to specify a RetentionTime per OffsetCommitRequest
https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol
Currently librdkafka does not offer a way to specify this RetentionTime to the user.

Feature: Somehow offer a way to configure the RetentionTime be it globally be it per topic be it per partition, or even as a parameter to rd_kafka_commit.

[edenhill]

The most flexible approach would be for an argument to commit...(), but that's also the biggest change.

Would you be fine with a global configuration property?

[FDePourcq]

Oh hoi! Good to hear from you! Though I'm not pushing for this feature,
It is up to you to decide how much code (rd_kafka_op_s, rd_kafka_commit0, rd_kafka_commit) you want to defile with that additional int64 :)

It would only be a problem for processes with multiple consumers and each of those consumers having different stringent requirements on commit-offset-retention. Looks as if a global configuration option would be fine for the coming years.

[edenhill]

Great! (The global config is per consumer instance though, not per process)

[edenhill]

The per-commit timestamp/timeout has been removed from later versions of the protocol.