This is an application to track the offsets for all the consumer groups in a Kafka cluster. This app will collect offset information for both a regular Kafka consumer and also Storm-Kafka Spout. The app provides a rest endpoint that can be called on-demand to get this information. You can also configure the app to push the offset metrics at a regular interval to a StatsD server, so that you can use a metric collection system like InfluxDB/Grafana to collect the information from StatsD and monitor.
Java 1.7 Maven 3
Installation (install commands)
You will have the kafka-monitoring-tool-0.0.1.jar created in the “target” directory.
java -jar target/kafka-monitoring-tool-0.0.1.jar server
By default the application assumes the zookeeper is running localhost on port 2181. If you need to provide a zookeeper host, pass it as a jvm parameter like this:
java -Ddw.zookeeperUrls=ZKHOST:ZKPORT,ZKHOST:ZKPORT,ZKHOST:PORT -jar kafka-monitoring-tool-0.0.1.jar server
Once the server is up, run the following command from localhost to get the information in a json format. curl -X GET http://localhost:8080/kafka/offset
Configuration (config commands)
The application can also be passed a yml file as a configuration file, instead of passing the zookeeper urls on the command line. Default.yml file is available in the project. The way you start the project with yml file is like this:
java -jar kafka-monitoring-tool-0.0.1.jar server default.yml
There are few Query Params you can pass to the API to get specific results. Examples are:
If you would like the output in a HTML format instead of Json format, try this: http://localhost:8080/kafka/offset?outputType=html
If you would like to get only the offsets from a regular consumers, try with the consumerType=regular parameter, example: http://localhost:8080/kafka/offset?consumerType=regular
If you would only like the storm spout consumers, try this: http://localhost:8080/kafka/offset?consumerType=spout
If you would like to push the stats to a StatsD server, you can configure the service to do so. You just need to enable the the pushToStatsD property. By default the application assumes the statsD is listening on the localhost at port 8125.
java -Ddw.zookeeperUrls=ZKHOST:ZKPORT,ZKHOST:ZKPORT,ZKHOST:PORT -Ddw.pushToStatsD=true -jar kafka-monitoring-0.1.0.jar server
By default the metrics are collected and pushed every 60 seconds. If you want to change the frequency, set this property: java -Ddw.zookeeperUrls=ZKHOST:ZKPORT,ZKHOST:ZKPORT,ZKHOST:PORT -Ddw.pushToStatsD=true —Ddw.refreshSeconds=60 -jar kafka-monitoring-tool-0.0.1.jar server
To push metrics to statsd server listening on a different host on a different port, set the following parameters: java -Ddw.zookeeperUrls=ZKHOST:ZKPORT,ZKHOST:ZKPORT,ZKHOST:PORT -Ddw.pushToStatsD=true —Ddw.refreshSeconds=60 -Ddw.statsDHost=locahost —Ddw.statsDPort=8125 -Ddw.statsDPrefix=kafka-monitor -jar kafka-monitoring-tool-0.0.1.jar server
You can also configure all these properties in a yml config file and pass in the config file to the app, like java -jar kafka-monitoring-tool-0.0.1.jar server default.yml