status: allow exporting ConnectivityCheck results over HTTP

[gnarula]

Adds a command that initiates ConnectivityCheck every interval
and serves the results in the form of prometheus metrics.

[ineiti]

I think I like that approach very much: do I correctly understand that you put the charge of fetching the status and sending it to prometheus on the status binary?

In our current systems, this would work best as a one-shot thing: call status serve once every 5 minutes. But YMMV...

[pierluca]

I think I like that approach very much: do I correctly understand that you put the charge of fetching the status and sending it to prometheus on the status binary?

In our current systems, this would work best as a one-shot thing: call status serve once every 5 minutes. But YMMV...

This is not how prometheus works. Prometheus works by scraping various systems and letting them expose their status.
I.e. Prometheus just needs a few addresses to scrape, but every service exposes metrics that are relevant for itself.
As a matter of fact, "sending to prometheus" is considered an anti-pattern and only recommended for ephemereal / batch jobs ( https://prometheus.io/docs/practices/pushing/ )

As such, this starts a webserver that exposes the metrics on a webpage.

[pierluca]

LGTM, minor remarks to improve our debugging capabilities

[ineiti]

Prometheus works by scraping various systems and letting them expose their status.

Hmm - true. I was still thinking about Grafana. You'll have to explain to me sometimes why a logger that calls the service to be logged is a good idea.

Getting back to my previous idea - shouldn't this be implemented using https://github.com/dedis/onet/blob/992a708c6c664b744ec67d8ea6f9b0181c16d166/processor.go#L192 ? Then you wouldn't even have to start yet another process, but could let the nodes serve Prometheus. You would have to add a rate-limitation, but other than that, I would prefer that...

[pierluca]

Prometheus works by scraping various systems and letting them expose their status.

Hmm - true. I was still thinking about Grafana. You'll have to explain to me sometimes why a logger that calls the service to be logged is a good idea.

Getting back to my previous idea - shouldn't this be implemented using https://github.com/dedis/onet/blob/992a708c6c664b744ec67d8ea6f9b0181c16d166/processor.go#L192 ? Then you wouldn't even have to start yet another process, but could let the nodes serve Prometheus. You would have to add a rate-limitation, but other than that, I would prefer that...

Grafana just does visualisation. I assume you're thinking about Graphite.
Prometheus is explicitly NOT about logging, and "why scraping"... I could talk about for hours :)
Happy to have that conversation around coffee :)

[gnarula]

Getting back to my previous idea - shouldn't this be implemented using https://github.com/dedis/onet/blob/992a708c6c664b744ec67d8ea6f9b0181c16d166/processor.go#L192 ? Then you wouldn't even have to start yet another process, but could let the nodes serve Prometheus. You would have to add a rate-limitation, but other than that, I would prefer that...

I think the limitation there is the REST Handler only supports JSON responses but Prometheus expects a text format in the response.

Re: rate-limitation, I think the metrics would only be consumed by the conode internally and I'd assume the HTTP port would be firewalled to allow connections only from trusted sources (i.e. the prometheus server).

[pierluca]

There's no need for rate limiting : Prometheus metrics are never meant to be "computed" on HTTP request, merely displayed.
We're following this pattern : we check the connectivity regularly, display it on request.
The cost of displaying the metric, i.e. a template with some data, is minimal.
Worst case we can cache the metric page for optimisation purposes, but that's an unlikely scenario.

[pierluca]

LGTM. Awesome :)