Apache Kafka Client Library for Go
Clone or download
serejja Merge pull request #196 from miketonks/master
UPDATED: producer.Close() no longer accepts input parameters
Latest commit 95d107b May 13, 2016
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
Godeps Move syslog producer to elodina/syslog-kafka Feb 10, 2016
avro Fix logline package Jan 26, 2016
config Fixed file permissions. Jul 28, 2015
docs Preparations for ownership transfer Jan 26, 2016
mirrormaker Change underlying producer in mirror maker to Siesta Jan 28, 2016
swtich Preparations for ownership transfer Jan 26, 2016
vagrant Preparations for ownership transfer Jan 26, 2016
.gitignore make broker list command line to minimize the need for producer prope… Jan 16, 2015
.travis.yml Changed to tools/godep package manager Jan 25, 2016
Dockerfile Preparations for ownership transfer Jan 26, 2016
Dockerfile.mirrormaker Preparations for ownership transfer Jan 26, 2016
LICENSE Fixed file permissions. Jul 28, 2015
README.md update readme Jan 27, 2016
Vagrantfile Fixed file permissions. Jul 28, 2015
api.go Fixed file permissions. Jul 28, 2015
consumer.go Stop Ticker not to leak it Mar 9, 2016
consumer_config.go Removed Sarama low level client Jan 27, 2016
consumer_test.go Rename ZookeeperTimeout to ZookeeperSessionTimeout Mar 10, 2016
emitters_test.go siesta producer new repo Jan 29, 2016
encoder_decoder.go Fix test Jan 29, 2016
fetcher.go resolve merge conflicts Nov 5, 2015
filter.go Fixed file permissions. Jul 28, 2015
log_emitters.go producer.Close() no longer accepts input parameters May 12, 2016
logger.go Add possibility to skip corrupted messages, fix broken mirrormaker Sep 10, 2015
logline.avsc Fixed file permissions. Jul 28, 2015
low_level_client.go Fix collector function. Feb 9, 2016
message_buffer.go Fixed file permissions. Jul 28, 2015
message_buffer_test.go Added highwatermark offset field to Message struct Sep 25, 2015
metrics.go bugfixes Sep 25, 2015
metrics_emitters.go siesta producer new repo Jan 29, 2016
mirror_maker.go UPDATED: producer.Close() no longer accepts input parameters May 12, 2016
mirror_maker_test.go Fix test Jan 29, 2016
partition_assignment.go Various optimizations and fixes Oct 19, 2015
partition_assignment_test.go Fixed file permissions. Jul 28, 2015
pool.go Various optimizations and fixes Oct 19, 2015
run-tests.sh Preparations for ownership transfer Jan 26, 2016
structs.go Change underlying producer in mirror maker to Siesta Jan 28, 2016
testing_utils.go producer.Close() no longer accepts input parameters May 12, 2016
topics.go Fixed file permissions. Jul 28, 2015
utils.go Migrate to 'cfg' mirror. Feb 9, 2016
workers.go resolve merge conflicts Nov 5, 2015
workers_test.go fix broken workers test Sep 17, 2015
zk_coordinator.go Merge pull request #191 from yudai/fix_zk_connection_recovery Mar 11, 2016
zk_coordinator_test.go Fixed file permissions. Jul 28, 2015

README.md

Go Kafka Client

The Apache Kafka Client Library for Go is sponsored by [CrowdStrike] (http://www.crowdstrike.com/) and [Elodina Inc] (http://elodina.net)

Build Status

Ideas and goals behind the Go Kafka Client:

1) Partition Ownership

We decided on implementing multiple strategies for this including static assignment. The concept of re-balancing is preserved but now there are a few different strategies to re-balancing and they can run at different times depending on what is going on (like a blue/green deploy is happening). For more on blue/green deployments check out this video.

2) Fetch Management

This is what “fills up the reservoir” as I like to call it so the processing (either sequential or in batch) will always have data if there is data for it to have without making a network hop. The fetcher has to stay ahead here keeping the processing tap full (or if empty that is controlled) pulling the data for the Kafka partition(s) it is owning.

3) Work Management

For the Go consumer we currently only support “fan out” using go routines and channels. If you have ever used go this will be familiar to you if not you should drop everything and learn Go.

4) Offset Management

Our offset management is based on a per batch basis with each highest offset from the batch committed on a per partition basis.

Prerequisites:

  1. Install Golang http://golang.org/doc/install
  2. Make sure env variables GOPATH and GOROOT exist and point to correct places
  3. Install godep https://github.com/tools/godep
  4. mkdir -p $GOPATH/src/github.com/elodina && cd $GOPATH/src/github.com/elodina
  5. git clone https://github.com/elodina/go_kafka_client.git && cd go_kafka_client
  6. godep restore

Optional (for all tests to work):

  1. Install Docker https://docs.docker.com/installation/#installation
  2. cd $GOPATH/src/github.com/elodina/go_kafka_client
  3. Build docker image: docker build -t elodina/go_kafka_client .
  4. docker run -v $(pwd):/go_kafka_client elodina/go_kafka_client

After this is done you're ready to write some code!

For email support https://groups.google.com/forum/#!forum/kafka-clients

Related docs:

  1. Offset Storage configuration.
  2. Log and metrics emitters.