Initial support for consumer coordinator

[toddpalino]

I've added basic support for the consumer coordinator in the Kafka broker, specifically for fetching offsets from the brokers that have been committed to Kafka, as opposed to Zookeeper. Internally, we use this library for some API and monitoring work, and one of the functions we need is to retrieve offsets that have been committed using kafka storage.

I've added a client request method for sending consumer metadata requests, along with the appropriate encoder and decoder routines. I have also added a _send_consumer_aware_request method which uses the consumer metadata information to find the coordinator for the specified consumer group and send requests to that broker. These parts are the basic support framework for communicating with the consumer coordinator.

In addition, to support fetching kafka-committed offsets without breaking the existing zookeeper-committed offsets support, I have added a send_offset_fetch_request_kafka method to the client. This allows the caller to perform the offset fetch request using version 1 of the protocol (which is identical on the wire to version 0, but is for kafka-committed offsets). Because the broker is using different request version numbers for Zookeeper and Kafka committed offset fetches, I believe this is the best way to support both in parallel.

Moving forwards, support should be added for offset commit requests to use Kafka storage as well, potentially with the same model.

[dpkp]

Todd -- are you still interested in merging this? I was planning to update the code for 0.9 consumer groups, but had planned to re-write the entire protocol stack in doing so (some initial work in the 0.9 branch). I'll probably push a patch release on the current implementation. We could include this patch if you think it is worth it. Then we'll bump to 0.10 for the larger changes and at that point I think this functionality would be covered by the larger consumer group support.

[toddpalino]

I think it still makes sense to merge it in. As it currently stands, the implementation for the requests that require a consumer coordinator is not working properly, so definitely if you're going to push a patch before doing a rewrite. We've been using this patch internally for monitoring for some time now, and it would be good if we can get back on a release version.

[dpkp]

Ok -- merge it is.

The API names did change in 0.9 from ConsumerMetadata* to GroupCoordinator* , but nothing changed on-the-wire. I expect some variable/constant/function names will get updated in 0.9 support.