Skip to content

Docker based Storm cluster with running Graphite for metrics reporting + Grafana Dashboard for visualization. Example kafka service as datasource included.

Notifications You must be signed in to change notification settings

jfr2102/storm-cluster

Repository files navigation

Storm-cluster

Storm docker cluster using docker-compose. Including a Prometheus/Graphite container for Storms Metrics Reporting API v2 and Grafana for metrics visualization. Docker-compose/stack also includes Kafka service as example testdata source.

Based on offical Apache Storm docker image , Graphite docker image / Prometheus docker image, Grafana docker image, wurstmeister Apache Kafka image and cAdvisor.

Configuration:

  • For swarm deploy set up docker swarm and set labels / edit placement labels accordingly in docker-stack.yml
  • Adding configs to storm.yaml file overwrites the default values (for version 2.2.0).
  • configure the used metric exporter and potential metric filter regex expressions according to Prometheus/Graphite usage in storm.yaml
  • to collect host metrics with the included Prometheus service install and start Prometheus Node exporter on hosts
  • If graphite is used set cAdvisor enviroment variable to use statsd storage driver of the graphite service:
environment: 
- storage_driver=statsd
- storage_driver_host=graphite:8125

Usage:

(Re)build and deploy:

$ docker-compose -f docker-stack.yml build
$ docker stack deploy --compose-file docker-stack.yml storm-cluster

or utilize scripts in /buildscripts

Get a cmdline on nimbus / supervisor (on the corresponding machine):

$ ./nimbus.sh

Run topology on nimbus (with shell on nimbus):

$ storm jar /topology.jar MainTopology Mytopology
Alternatively:
  • configure enviroment variables (TOPOLOGY_JAR, TOPOLOGY_MAIN, TOPOLOGY_Name) in Dockerfile

  • on nimbus shell:

  /./deploy_topology.sh

Write in Kafka example topic

  • get shell on kafka:
  ./kafka.sh
  java -jar kafka-generator-1.0-SNAPSHOT-jar-with-dependencies.jar
  • the example messages generated by this kafka-generator are compatible to the provided example topology

Accessing via Browser on host:

When using swarm deployment, services are available via all host IPs. (see: ingress)

Storm UI: http://host:8080

Prometheus Dashboard: http://host:9090

Grafana Dashboard: http://host:3000 (userdetails: admin/admin)

About

Docker based Storm cluster with running Graphite for metrics reporting + Grafana Dashboard for visualization. Example kafka service as datasource included.

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published