Metrics produced to Kafka and consumers for monitoring them
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
codahale/src
config
gradle/wrapper
graphite/src/main/java/ly/stealth/graphite
metrics-test/src/main/scala/ly/stealth/psutil
psutil-entities/src/main/scala/ly/stealth/psutil/entities
psutil-statsd/src/main/scala/ly/stealth/kafka/metrics/statsd
psutil/src/main/python
riemann/src
statsd/src/main/scala/ly/stealth/metrics/kafka/statsd
vagrant
yammer/src/main/java/ly/stealth/kafka/metrics
.gitignore
LICENSE
README.md
Vagrantfile
bootstrap.sh
build.gradle
gradle.properties
gradlew
gradlew.bat
settings.gradle
shutdown.sh

README.md

Kafka CodaHale Metrics Reporter + Riemann metrics consumer

The goals of this repo is to have an end to end working environment to provide the ability for systems (applications and infrastructure) to send their metrics/sensor data to Kafka and then to report on that data for both alerts and charts.

More about this project in this blog post http://allthingshadoop.com/2014/04/18/metrics-kafka/.

Quick up and running

Use Vagrant to get up and running.

  1. Install Docker http://docs.docker.com/installation/
  2. Install Virtual Box https://www.virtualbox.org/

In the main metrics-kafka folder

  1. sudo ./bootstrap.sh
  2. ./gradlew test
  3. sudo ./shutdown.sh

once this is done

  • Zookeeper will be running localhost:2181
  • Broker 1 on localhost:9092
  • Riemann on localhost:5555
  • Riemann dash on localhost:4567
  • All the tests in metrics/src/test/java/* and riemann/src/test/scala/* should pass

You can access the brokers, zookeeper, riemann and riemann-dash by their IP from your local without having to go into vm.

e.g.

bin/kafka-console-producer.sh --broker-list localhost:9092 --topic

bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic --from-beginning

http://192.168.86.5:4567/#Riemann ./gradlew :psutil:installDependencies ./gradlew :metrics-test:run

Kafka Yammer Metrics reporting

In order to assemble jar for metrics yammer do the following steps:

  1. ./gradlew :yammer:jar
  2. put the jar from metrics-yammer/build/libs to libs dir in root kafka folder
  3. add "kafka.metrics.reporters=ly.stealth.kafka.metrics.KafkaBrokerReporter" to config/server.properties in kafka root folder