Cluster monitoring metrics

[unknown321]

Currently (v1.0.2), /state endpoint doesn't provide enough information about real state of the cluster:

• List of nodes is always the same, even if some nodes have lost connection to the cluster - there is no node state
• Lack of synchronization status (ie number of last transaction on each node)
• No simple message with cluster status, for example: green when all nodes are online, yellow when one node is down and red when 2 nodes are down
• Maybe some info about RAFT leases, not sure about it
• Something else..?

[manishrjain]

/state isn't meant to know about the connectivity issues for each server. Each server sees the world, based on its connectivity to it. For e.g., server A can see server B and C, but it's possible that B can only see C, and not A. And C might not be able to talk to any other server. Zero can't keep track of who sees what.

However, what we could do is to export the connectivity metrics to Open Census. That way, a user can see them live via Grafana or something.

[srfrog]

Closing, this is being handled in #2257