Consumer offset management

[eapache]

@burke @airolson @tobi @fw42

This doesn't actually work with the Kafka 0.8 beta yet. It seems to misinterpret the OffsetCommit request messages somehow and ends up underflowing a buffer and disconnecting.

However, I've double-checked the bytes-on-the-wire with Wireshark and the code does produce the correct message according to the spec, so that's their bug (the python client has the same problem).

Fixes issue #2.

[tobi]

Can you file a ticket with Kafka so that they are aware?

• tobi
  CEO Shopify

On Tue, Jul 30, 2013 at 2:41 PM, Evan Huus notifications@github.com wrote:

@burke https://github.com/burke @airolson https://github.com/airolson
@tobi https://github.com/tobi @fw42 https://github.com/fw42

This doesn't actually work with the Kafka 0.8 beta yet. It seems to
misinterpret the OffsetCommit request messages somehow and ends up
underflowing a buffer and disconnecting.

However, I've double-checked the bytes-on-the-wire with Wireshark and the
code does produce the correct message according to the spec, so that's
their bug (the python client has the same problem).

Fixes issue #2 #2.

You can merge this Pull Request by running

git pull https://github.com/Shopify/sarama consumer-offset-management

Or view, comment on, or merge it at:

#3
Commit Summary

• Add a GetBlock helper to the OffsetFetchResponse
• Fetch the offset when starting a consumer
• Have OffsetCommitResponse hold pointers to errors
• Tweaks to fetchOffset
• Implement Consumer.Commit, Fixes Offset Fetch/Commit Support #2.

File Changes

• M kafka/consumer.gohttps://github.com/Consumer offset management #3/files#diff-0(160)
• M kafka/consumer_test.gohttps://github.com/Consumer offset management #3/files#diff-1(32)
• M protocol/offset_commit_response.gohttps://github.com/Consumer offset management #3/files#diff-2(21)
• M protocol/offset_commit_response_test.gohttps://github.com/Consumer offset management #3/files#diff-3(3)
• M protocol/offset_fetch_response.gohttps://github.com/Consumer offset management #3/files#diff-4(12)

Patch Links:

• https://github.com/Shopify/sarama/pull/3.patch
• https://github.com/Shopify/sarama/pull/3.diff

[eapache]

According to dpkp/kafka-python#18 it may already be fixed in Kafka trunk, I will try that and see.

[eapache]

So according to https://issues.apache.org/jira/browse/KAFKA-852 "The Offset fetch/commit APIs are not slated for the 0.8 release". They're only going to be in 0.8.1? I'm confused...

[eapache]

Ticket filed with Kafka, maybe we'll get some answers: https://issues.apache.org/jira/browse/KAFKA-993

[eapache]

Needs redoing after the refactor.