Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Prometheus observability of last signed height per validator #3773

Open
mdyring opened this issue Jul 4, 2019 · 4 comments

Comments

@mdyring
Copy link
Contributor

commented Jul 4, 2019

We'd like to observe individual last signed height for validators via Prometheus by requesting a tendermint_consensus_last_signed_height metric with the following labels:

  • address (example: "E33032D801DA4D124FBF696DEF7832E415EA74EE")
  • operator (example: "cosmosvaloper1lc4jjw8zveuuj3njhy70zl23vgmpds2xg830hn")
  • chain_id (example: cosmoshub-2)

As some might object to the cardinality of this (100+ validators on cosmoshub-2), a config option to enable this could be beneficial as well. Request is related to per-validator voting power in #3083, so a common toggle for per-validator observability would be suitable.

Thanks for your consideration. :-)

@ebuchman

This comment has been minimized.

Copy link
Contributor

commented Jul 12, 2019

Note operator isn't a concept native to Tendermint. Tendermint could provide the validator's consensus pubkey, but the operator is a Cosmos-SDK concept. I wonder if the SDK is a better place for per-validator metrics?

@mdyring

This comment has been minimized.

Copy link
Contributor Author

commented Jul 21, 2019

Hm, seems nice to provide for any Tendermint-based chain I think it makes sense to provide it here.

Providing address, consensus pubkey and chain-id labels would be neat!

@mdyring

This comment has been minimized.

Copy link
Contributor Author

commented Jul 21, 2019

On second thought, would be messy to have both Tendermint and cosmos-sdk endpoints for Prometheus. I suspect cosmos-sdk is doing all the Prometheus-related work today, so if that it the case it might make better sense to do it from there. @jackzampolin any thoughts?

@mdyring

This comment has been minimized.

Copy link
Contributor Author

commented Jul 31, 2019

This metric is useful for detecting external signing errors such as this.

Currently we're using our own tool for monitoring the last signed height, but would be nice to offer out of the box IMHO.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.