Kafka 0.10.0.0

[justizin]

Pulled changes from moonkev/kafka-mesos@acc99a2 pursuant to #214 into a branch, merged with recent upstream changes, and verified build.

[justizin]

Some failing tests here, I'll see if I can sort them:

ly.stealth.mesos.kafka.CliTest > topic_list FAILED
kafka.common.BrokerNotAvailableException at CliTest.scala:273

ly.stealth.mesos.kafka.CliTest > topic_add FAILED
ly.stealth.mesos.kafka.Cli$Error at CliTest.scala:372

ly.stealth.mesos.kafka.CliTest > topic_rebalance FAILED
kafka.common.BrokerNotAvailableException at CliTest.scala:331

ly.stealth.mesos.kafka.CliTest > topic_update FAILED
kafka.common.BrokerNotAvailableException at CliTest.scala:311

ly.stealth.mesos.kafka.ExprTest > expandTopics FAILED
kafka.common.BrokerNotAvailableException at ExprTest.scala:111

ly.stealth.mesos.kafka.HttpServerTest > topic_list FAILED
kafka.common.BrokerNotAvailableException at HttpServerTest.scala:267

ly.stealth.mesos.kafka.HttpServerTest > topic_add FAILED
java.io.IOException at HttpServerTest.scala:284

ly.stealth.mesos.kafka.HttpServerTest > topic_rebalance FAILED
kafka.common.BrokerNotAvailableException at HttpServerTest.scala:330

ly.stealth.mesos.kafka.HttpServerTest > topic_update FAILED
kafka.common.BrokerNotAvailableException at HttpServerTest.scala:309

ly.stealth.mesos.kafka.RebalancerTest > start FAILED
kafka.common.BrokerNotAvailableException at RebalancerTest.scala:58

ly.stealth.mesos.kafka.TopicsTest > fairAssignment FAILED
kafka.common.BrokerNotAvailableException at TopicsTest.scala:44

ly.stealth.mesos.kafka.TopicsTest > addTopic FAILED
kafka.common.BrokerNotAvailableException at TopicsTest.scala:53

ly.stealth.mesos.kafka.TopicsTest > updateTopic FAILED
kafka.common.BrokerNotAvailableException at TopicsTest.scala:70

ly.stealth.mesos.kafka.TopicsTest > getTopics FAILED
kafka.common.BrokerNotAvailableException at TopicsTest.scala:37

ly.stealth.mesos.kafka.TopicsTest > getTopic FAILED
kafka.common.BrokerNotAvailableException at TopicsTest.scala:29

92 tests completed, 15 failed
:test FAILED

FAILURE: Build failed with an exception.

• What went wrong:
  Execution failed for task ':test'.

  There were failing tests. See the report at: file:///Users/juryan/kafka/mesos-kafka/out/gradle/reports/tests/index.html

• Try:
  Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

BUILD FAILED

Total time: 29 mins 0.847 secs

[akirillov]

Hi guys, if it might help, you can check out my PR to migrate to Kafka 0.10. There's a change in ZkUtils which wraps most of ZkClient functionality so it's not needed to use the raw client in most of the cases. Here's the PR https://github.com/datastrophic/kafka/pull/1/files I run this scheduler in production so if it helps feel free to use it. Or maybe should I create a PR to this repo?

[joestein]

You shouldn't need to build to a later version of Kafka doing so could break backwards compatibility which is why we have always built for the lowest Kafka version orgs still use in production. I haven't tried a 0.10 broker with mesos/kafka yet but it should work and if not then it would best to introduce fixes where 0.8 && 0.9 isn't broken for folks. Just rushing upgrades without backwards compatibility isn't a stable release process for folks that aren't yet ready to upgrade Kafka version just because some folks are. Its a balance that should be doable (and has been so far) via the code changes so I encourage folks to try to PR according to the community as a whole, thnx.... granted this may not break 0.8 and 0.9 users but it seems like it would but haven't tried it, just trying to make sure everyone gets spoken for. thnx.

[steveniemitz]

It seems like the ZkUtils change was the only big thing in 0.10.0. How about instead of directly using/importing it from Kafka we make our own wrapper interface, with 2 implementations, one for < 0.10.0 and one for 0.10.0? That should make it much easier to support both versions.

[justizin]

@joestein the java API has changed between 0.9 and 0.10, and kafka-mesos needs to use the client library, Ithink @steveniemitz solution is ideal and might put some energy into getting this to work. Maybe @akirillov is also interested?

[steveniemitz]

I actually just made the changes needed to work with 0.10 to my branch. I'll update the pull request here, but you can check out tc-dc@02a000c too.

[gondor]

Any progress on this? Would love to be able to upgrade

[steveniemitz]

https://github.com/tc-dc/kafka-mesos/tree/tcdc is what we've been running (in production) with 0.10.1.1

[gondor]

@steveniemitz awesome, thank you. What's the best way to upgrade the framework/brokers in the cluster or will we loose data?

[steveniemitz]

So, in theory you should be able to upgrade to my branch without any issues. The only reason I say in theory is because I've only tested incremental updates, never from master -> my current branch. However, the saved cluster state is 100% compatible (although I still recommend backing it up first). If you have a test cluster I'd recommend testing there before rolling out to production.

Also don't forget kafka has its own guide on how to upgrade versions depending on what version you're going from -> to.

[steveniemitz]

#260 should address this, which I'll merge soon