Monitoring with Prometheus Demo
Clone or download
Latest commit 79604a2 Sep 23, 2016
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
alert-console Add alerting Sep 21, 2016
alertmanager Use private container ports for sending alerts Sep 21, 2016
prometheus Remove description from rule Sep 22, 2016
voting-app Add alerting Sep 21, 2016
voting-generator Add alerting Sep 21, 2016
.gitignore Docker compose file Aug 3, 2016
Dockerfile.build Add alerting Sep 21, 2016
Makefile Add alerting Sep 21, 2016
README.md Update README Sep 23, 2016
docker-compose.yml Fix target groups volume persisting old configuration Sep 21, 2016

README.md

Monitoring with Prometheus Demo

This project demonstrates the possibilities of monitoring with Prometheus.

Overview

This project contains the following components:

  • Prometheus
  • Prommer for custom target discovery
  • Alert Manager for managing alerts
  • Grafana for metric and analytic dashboards
  • cAdvisor for exposing container and host metrics
  • A voting app with a RESTish API which exposes custom metrics for the votes
  • A voting generator app that generates votes
  • An alert console that displays the alerts

Running This Project

Prerequisites

Building and running

The project can be completely built and run with the default make command.

$ make

All build steps are containerized with Docker. After the build process, all services are started with Docker Compose.

To stop all services, use the stop command:

$ make stop

To clean the project, use the clean command. This will remove all built binaries and images. Note that this will not remove any third-party images (like cAdvisor) that were pulled in.

$ make clean

Prometheus Server With Custom Service Discovery

The Prometheus server is configured to use Prommer as service discovery. Prommer listens to the Docker events stream and updates the target groups configuration of Prometheus.

Voting App

For each vote, the application will increment the counter of the Prometheus metric votes_amount_total. The label vote is added and the value is set to the name of the vote.

Metrics

  • votes_amount_total{name=<name>} - Counter that returns the total amount of votes

Endpoints

  • POST /api/vote - Make a vote
  • GET /api/results - Fetch results
  • GET /metrics - Returns the Prometheus metrics
$ curl -X POST -H 'Content-Type: application/json' 
-d '{"name":"prometheus"}' \
http://$(docker-compose port voting-app 8080)/api/vote

{
  "prometheus": {
    "count": 1
  }
}

$ curl -X GET http://$(docker-compose port voting-app 8080)/api/results

{
  "prometheus": {
    "count": 1
  },
  "go": {
    "count": 1
  }
}

Voting Generator

The voting-generator app, as the name suggests, generates votes.

Pass in the -vote flag for which thing it should vote for (e.g. -vote=dog).

The docker-compose.yml file contains two different voting generator services: vote-cats and vote-dogs. If you wish to generate more votes, you can scale them up with docker-compose scale:

$ docker-compose scale vote-dogs=3 vote-cats=2

Alert Console

The alert console is an application that consumes the alerts sent by the Alert Manager. It is configured as a webhook receiver in the Alert Manager. When it receives an alert, it logs the JSON body to the console.

You can view the logs of the alert console using docker-compose logs:

$ docker-compose logs --tail="all" -f alert-console