prometheus metrics support

[appliedprivacy]

Hi,
it would be great to see prometheus metrics support directly in unbound,
this would make 3rd party exporter tools with varying quality unnecessary and the exporte would always be compatible with unbound since it is directly integrated. NLnetLabs appears to agree that prometheus makes sense since other NLnetLabs projects incorporate it already (like routinator).

thanks!

[wcawijngaards]

There already seems to be a project in https://github.com/svartalf/unbound-telemetry that does this.

[wcawijngaards]

In the commit there is the file https://github.com/NLnetLabs/unbound/blob/master/contrib/metrics.awk . You could use this file with eg. unbound-control stats | awk -f metrics.awk and that produces Prometheus format output. The graphs are like what contrib/unbound_munin_ produces. I have not tested it in prometheus, but it may be helpful to output the unbound statistics into Prometheus. Likely needs some grafana config too, eg. type stacked for the histogram.

[appliedprivacy]

Thanks for your feedback.
The feature request was specifically for unbound itself to avoid having to use third party tools like
https://github.com/kumina/unbound_exporter

The issue with the awk approach is that the data is generated asynchronously, meaning that the data does not represent unbound's state when prometheus came along end fetched it.

[wcawijngaards]

What is the awk issue? Asynchronous? Do you want cumulative numbers or something? The awk script runs very quick, so it does not delay the measurement in that sense.

[appliedprivacy]

Maybe I misunderstood, but I assumed the awk command runs at a fixed interval (cronjob) and writes the output into a file served by a webserver (instead of running when prometheus asks for it).

We will test with the noreset version of stats
unbound-control stats_noreset | awk -f metrics.awk

Thanks!

[jsha]

At Let's Encrypt we deploy Unbound and unbound_exporter. We also semi-recently took over maintenance of the prometheus unbound_exporter tool: https://github.com/letsencrypt/unbound_exporter.

Some reasons we would prefer to see Prometheus metrics exported directly by Unbound:

• Right now the stats privilege is conflated with general control privilege, but it doesn't have to be. Unbound's default notion of retrieving stats also resets them, but in Prometheus stats are never reset, so allowing a machine to fetch stats is a low-privilege operation that can be separated from the control privilege. In our environment I think we would turn off unbound-control if we could get stats without it.
• Managing configuration of two components on a host that talk to each other, along with a control channel, is more complex than managing a single binary. This is particularly true when using containers, which typically assume they are responsible for a single process. It's possible to get around that assumption but it makes the deployment story more complex.
• We have to maintain a build target and deployment code for unbound_exporter in each of our environments, as well as Unbound, and keep both up to date.
• In general, it would be nice to be able to deprecate unbound_exporter.

Also, to clarify: Prometheus stats are generally fetched over the network, via an HTTP GET. So the awk script only solves a part of the problem. What we would really like is an option for Unbound to serve stats in Prometheus format via HTTP.

[james-stevens]

And while you're at it, would be nice to have prometheus metrics directly out of nsd as well ;)

[james-stevens]

Maybe I misunderstood, but I assumed the awk command runs at a fixed interval (cronjob) and writes the output into a file served by a webserver (instead of running when prometheus asks for it).

We will test with the noreset version of stats unbound-control stats_noreset | awk -f metrics.awk

Thanks!

prob the easiest way to use it, to integrate with prometheus, would be to run it from xinetd (or similar)

But, TBH, we just use the LetsEncrypt exporter - it does a fine job.

[jirutka]

I like the AWK approach. unbound_exporter by Let’s Encrypt is a ~10 MiB bloat which is awful for something that can be easily replaced with a simple AWK script and a tiny HTTP server with CGI support and a fraction of RAM usage.

The problem is that metrics.awk filter uses a different metrics structure than unbound_exporter, so it cannot be used with existing Grafana dashboards.

So I completely rewrote the AWK filter to produce the same metrics as unbound_exporter. I just omitted the threads label because I think it’s unnecessary – it just increases the amount of data.

You can find it at https://github.com/jirutka/unbound-exporter-cgi/. It’s wrapped in a simple CGI script that can be served by any HTTP server with CGI support. It makes the most sense to use a tiny HTTP server such as thttpd, busybox httpd or OpenWrt uhttpd.

[madnuttah]

I solved this with a simple modified healthcheck script shipping the statistics to Zabbix in our Unbound Docker image's unbound-docker-stats-addon.