Initial stab at Zookeeper-managed topic consumer. #1
Conversation
|
My Go knowledge is far too basic to offer any real help on that, but this looks / sounds awesome to me! 👍 |
|
@manygrams You should check out |
| for { | ||
| event, ok := <-stream | ||
| if !ok { | ||
| log.Println("Consumer is done") |
There was a problem hiding this comment.
Is the consumer ever done and it's not an error? I'm just thinking this should exit with a non-zero exit code?
There was a problem hiding this comment.
consumer.Close() above will end up closing the stream channel. So by pressing ctrl+c.
|
|
||
| topicChannel := make(chan *Event) | ||
|
|
||
| go func() { |
There was a problem hiding this comment.
this func is sufficiently large I would prefer it be named
Initial stab at Zookeeper-managed topic consumer.
|
Going to merge this so we can start using it. It will probably need some more work over time once we start using it for stuff |
|
Can we pair on using this from my Rails app? |
|
Yes! On May 26, 2014, at 12:20 PM, Nick Evans notifications@github.com wrote:
|
This needs some work, but is ready for a first look. This implements the zookeeper logic to have a nice consumer group:
This is heavily based on https://github.com/bsm/sarama/tree/master/cluster, but large parts are rewritten because it wasn't working for me. The Zookeeper-backed consumer group management should be compatible with the JVM library, as documented here: https://cwiki.apache.org/confluence/display/KAFKA/Offset+Management
TODO
@jnormore @berkcaputcu @manygrams /cc @eapache @bsm