upgrade to kafka 9

[pdeva]

Reasons here:
http://www.confluent.io/blog/290-reasons-to-upgrade-to-apache-kafka-0.9.0.0

[sixtus]

even more interesting:

This release introduces beta support for the newly redesigned consumer client. At a high level, the
primary difference in the new consumer is that it removes the distinction between the “high-level”
ZooKeeper-based consumer and the “low-level” SimpleConsumer APIs, and instead offers a unified
consumer API.

The new consumer allows the use of the group management facility (like the older high-level
consumer) while still offering better control over offset commits at the partition level (like the older low
level consumer). It offers pluggable partition assignment amongst the members of a consumer group
and ships with several assignment strategies. This completes a series of projects done in the last few
years to fully decouple Kafka clients from Zookeeper, thus entirely removing the consumer client’s
dependency on ZooKeeper.

i.e. we finally can get a nice kafka-rt pipeline :)

[gianm]

We are looking to update our consumer as part of #2220 (there's some code in there using the new consumer already).

Most of the other benefits can be gotten by just upgrading the Kafka brokers and don't need client side changes.

[gianm]

The new Kafka indexing service is built with the 0.9 client. It also uses the new consumer and we totally do all that nice stuff with controlling offset commits ourselves and not using Kafka's ZooKeeper.