Kamila's monitoring toolkit. Revolves around Prometheus.
Quickly set up a monitoring server
Note: I made this for DataCenterLight. Check it out!
cdist is an interesting take on configuration management developed at ungleich. It is simple, versatile, and has no dependencies on the target system. In fact, it is a bit too simple compared to other, far more sophisticated configuration management solutions -- and that's precisely why I like it.
I have created several cdist types useful for installing a Prometheus monitoring server, as well as many exporters:
High-level types for Prometheus and friends
__prometheus_server: Installs and configures Prometheus server. Yay!
__prometheus_alertmanager: Installs and configures Prometheus Alertmanager. Yay too!
__grafana_dashboard: Installs and configures Grafana. Yay three!
Extras (use as needed)
Daemontools are awesome. In line with the Unix philosophy, they do one thing and do it well. In this case they make sure that services run. Without the messy parts!
__daemontools: Installs daemontools or your favorite derivative, plus an init script in case your package doesn't come with one.
__daemontools_service: Creates a service directory. I gladly use this for everything that doesn't come with a good init script, such as some exporters. It's so simple!
Installing golang and go packages
Most distros provide a reasonably new version of Prometheus and the basic exporters (worst case you'll have to use the
--install-from-backports with my
__prometheus_server). This is the recommended way. However, if you need something more, these types let you install a recent version of golang and go packages. Note that I am currently not using these in production. I am listing them here anyway for completeness.
__golang_from_vendor: Installs golang from https://golang.org. This makes it easy to install any version of golang, including those that your old and obsolete distro doesn't have.
__go_get: Easy installation of golang packages straight from online repos. Useful for installing a newer version of Prometheus or various exporters.
Put this in you cdist manifest, edit to your needs, and rejoice!
__prometheus_server \ --install-from-backports \ --config "$__manifest/files/prometheus.yml" \ --retention-days 14 \ --storage-path /data/prometheus \ --rule-files "$__manifest/files/*.rules" __prometheus_alertmanager \ --install-from-backports \ --config "$__manifest/files/alertmanager.yml" \ --storage-path /data/alertmanager __grafana_dashboard
A cdist lightning-quick-start
git clone https://github.com/ungleich/cdist.git cd cdist $EDITOR cdist/conf/manifest/init bin/cdist config -vv <hostname>
Read more at cdist quickstart.
Questions, ideas, and pull requests are welcome!
Exporters for monitoring software that doesn't have built-in instrumentation
- ttn2prom: Trivial Python webserver that accepts webhooks (i.e. HTTP requests), processes the data in them, and exports the stuff in the Prometheus format. Very easy to adapt. This version exports data from some LoRaWan temperature sensors through webhooks from The Things Network.
- opennebula exporter: Exports useful metrics for OpenNebula. For now it is an evil hack, but it will get better with time.
- OpenNebula integration test: tests VM creation and connectivity, and exports the results in a Prometheus-compatible format.
Useful alerts collection