Prometheus monitoring

[nickray]

I would like to see a Prometheus monitoring endpoint available (basic example). The use case is for running a Cosmos Validator, where escrowed value is at risk of improper operations.

Performance considerations permitting (compile time option?), I assume integrating the client library, exposing the endpoint, and adding basic metrics (such as latest_block which are currently available at the 46657/status endpoint) should be fairly straight forward. From there, the actual metrics could be added on a pull request basis, or Tendermint users could patch in their own (possibly experimental) metrics.

The old tm-monitor repository implies that this has been considered before. What was the outcome? In my opinion, Tendermint should expose the information necessary to monitor its operation and state changes, but not be (officially) providing the tools to monitor them, as project clients may have varying requirements. Moreover, if there are multiple possibilities of implementing a solution, a (semi)standard should be chosen instead of a homegrown solution.

With the Christmas holiday coming up, I could possibly contribute pull requests. I have professional experience with C++ and Python, however Go and this codebase are new to me.

Looking forward to the discussion!

[melekes]

It would be great for Tendermint to report metrics. Would you be interested in writing an input plugin for https://github.com/influxdata/telegraf? It will probably require some work on the TM side too (API).

P.S. tm-monitor now lives here: https://github.com/tendermint/tools/tree/master/tm-monitor

[melekes]

Other option is to use https://github.com/go-kit/kit/tree/master/metrics.

go-kit does require to use one of the adapters:

var dur metrics.Histogram = prometheus.NewSummaryFrom(stdprometheus.SummaryOpts{
        Namespace: "myservice",
        Subsystem: "api",
        Name:     "request_duration_seconds",
        Help:     "Total time spent serving requests.",
    }, []string{})

not sure we want to mirror these adapter with settings in TM (given it may be faster to send directly opposite to using http client)

I am open to other solutions as well.

[ebuchman]

Do we need to change anything in Tendermint to support this ?

[nickray]

I apologize for not having sent any pull requests regarding this, and probably will not. It's an architectural decision on your side I think.

As supporting evidence, that go-kit/metrics itself states "It's primarily designed to help you get started with good and robust instrumentation, and to help you migrate from a less-capable system like Graphite to a more-capable system like Prometheus. If your organization has already standardized on an instrumentation system like Prometheus, and has no plans to change, it may make sense to use that system's instrumentation library directly."

What would have to changed AFAIK is

• adding an endpoint for scraping the metrics (configurable whether/where it's exposed)
• linking in the https://github.com/prometheus/client_golang library
• starting to expose relevant metrics (this can be minimal at first, and ideally someone running a node can easily extend those metrics by hacking the code)

I kind of disagree with the "post-launch" target, cause I wouldn't want to see the launch fail due to people not being able to monitor their validators properly :)

[xla]

Do we need to change anything in Tendermint to support this ?

While integrating the prometheus client library and exposing the corresponding endpoint for Prometheus to scrape is relatively straight forward I assume that the gathering of the metrics themselves will be the challenge.

@ebuchman @melekes Would a reasonable approach be to create a new service which hooks into the nodes eventbus to listen to significant events and observes timings and other information through that?

[t-bast]

Note that something that could be considered as well for more powerful monitoring would be integrating a tracing system. It's a bit more opinionated though, but I think it has a lot of value for HA production systems.
I'd love to see an integration with OpenCensus (or an OpenTracing implementation like Jaeger). Even if at first it simply propagates the context in ABCI queries (without adding its own events), that would already be a very big win.
That would be an addition to Prometheus metrics, but the metrics could be derived from traces automatically (that's what both OpenCensus and Jaeger do).
I'm trying to setup OpenCensus tracing on our platform and the fact that Tendermint sits in the middle and doesn't support it makes the traces a lot less useful.
It requires quite a good architectural knowledge of the Tendermint codebase to add though, so it will probably be quite tough to do as an external contribution :(

[xla]

@t-bast While both are in the realm of observability they are not interchangable. Could you open a new issue where we can discuss the merrits and feasibilities of tracing for tendermint?

[t-bast]

Thanks, done: #1342

[melekes]

Don't think we want to commit to any specific instrument like Prometheus. Our users should be free to choose whenever they want - statsd, prometheus, etc. I like go-kit metrics package. It supports multiple backends and provides clean API (https://github.com/go-kit/kit/tree/master/metrics). opencensus looks cool. It comes with a benefit of tracing, which may be implemented later at some point. But it's API looks a bit weird with views (https://opencensus.io/go/index.html).

[t-bast]

As long as it's possible to build and plug into Tendermint an exporter to convert the metrics to the format of our choice (be it Prometheus, OpenCensus or anything else) we'd be happy ;)