burrowx - kafka offset lag monitor,stored by influxdb
A simple, lightweight kafka offset monitor, currently metrics stored by influxdb. Motivated by Burrow, but much faster and cleaner and more stable. burrowx is good integration with influxdb and grafana.
By release binary
Just click the release links
By source build
$ go get github.com/sundy-li/burrowx $ cd $GOPATH/src/github.com/sundy-li/burrowx $ go build && go install
## new workspace for burrowx mkdir -p /data/app/burrowx ## cd to the workspace cd /data/app/burrowx ## cp the files to here cp $GOPATH/bin/burrowx ./ cp -rf $GOPATH/src/github.com/sundy-li/burrowx/config ./ ## modify server.json file config and run it ./burrowx
A Docker file is available which builds this project on top of an Alpine Linux image.
- Create your desired configuration files (server.yaml, logging.xml)
docker build -t burrowx .It will include the server.yaml and logging.xml and automatically start service.
- Run the container on your favourite container platform
Test the data
Create a new test topic
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 8 --topic test_burrowx_topic
Produce the data to the topic
for i in `seq 1 10000`;do echo "33" | bin/kafka-console-producer.sh --topic test_burrowx_topic --broker-list localhost:9092 ; sleep 1; done
Create a consumer to consume the data
$ pip install kafka
from kafka import KafkaConsumer consumer = KafkaConsumer('test_burrowx_topic', group_id='my_group2') for msg in consumer: print(msg) print("end")
Then you will find the data in the influxdb database
- Light weight and extremely simple to use, metrics are stored in influxdb, and could be easily viewed on grafana
- Only support kafka version >= 0.9.X, which stores the consumer offsets in the topic
__consumer_offsets,if you are using kafka 0.8.X, try my previous repo
- Base on topic,partitions of a topic are merged into topic