Example Prometheus

This is a sample setup of prometheus. It can be used to test all the typical functionalities provided by it. It is inspired by this repository.

Running it

docker-compose is enough to run everything, simply by doing:

docker-compose up --build

You can access the different components:

• prometheus
• grafana (user: root / pwd: root)
• alert-manager
• node-exporter
• cadvisor
• blackbox-exporter

NOTE: This setup is not production ready. There is no network distinction and all services are exposed.

Restarting prometheus

When changing the configuration, you can restart prometheus through this command

curl -X POST http://localhost:9090/-/reload

Similarly, for the alert-manager:

curl -X POST http://localhost:9090/-/reload

Metrics scraped

There are a bunch of sources configured:

• prometheus

Prometheus scrapes itself

• alert-manager

Scrapes data about generated alerts. Such as alertmanager_alerts.

• node-exporter

Scrapes data about the underlying host. Such as node_cpu_seconds_total.

• cadvisor

Scrapes data about running containers. Such as container_cpu_usage_seconds_total.

• black box exporter

Checks whether a list of defined target urls is online or not, sort of like a homegrown freshping. Note that the actual targets are verified by the probe_success metric.

• different jvm based applications (cookery2, echo)

Scrapes metrics provided by the application or the jvm, such as jvm_threads_states_threads.

Sample metrics

The number of samples ingested by prometheus per second:

• rate(prometheus_tsdb_head_samples_appended_total[5m])

A rate incorporating requests sent by prometheus, and a second one aggregated:

• rate(prometheus_http_requests_total{code="200", job="prometheus"}[5m])
• sum(rate(prometheus_http_requests_total{code="200", job="prometheus"}[5m]))

Aggregated heap consumption for JVM applications

sum by(instance)(sum_over_time(jvm_memory_used_bytes{area="heap"}[1h]))

Probes set up by the blackbox exporter that are failing

probe_success == 0

Grafana

The grafana instance comes with some dashboards preinstalled as a starting point.

Alert Manager

The alert manager comes preconfigured with a bunch of alerts. You can simulate a failure that will trigger an alert by doing

docker-compose stop echo

Critical alerts are sent to a custom receiver. To see the alerts, you can check its logs:

docker logs receiver