Improve validator monitor reporting

[potuz]

With the intuit of deprecating the current validator reporting metrics and move them to the validator monitor service within the beacon-chain, it would be good to extend the number of metrics exported and logged so that they cover all of the current ones in the validator metrics.

[james-prysm]

I'd like to help on this if there are more details on what needs to be done.
i'm not sure where the validator reporting metrics are and whether or not I'm looking at the correct thing in the beacon-chain ( beaconchain/ monitor/metrics.go?)

[james-prysm]

Will start after Eth denver by asking some individuals on their use of grafana stats from the validator, and look at what metrics can be migrated to ( beaconchain/ monitor/metrics.go?)

[abdulrabbani00]

@potuz and @james-prysm - I can get started on the work here, I'll browse around the code base and get back to yall with any questions!

[abdulrabbani00]

Related - #10983

[abdulrabbani00]

@potuz - A few questions/clarifications:

1. Could you help me understand why the metrics need to be migrated in the first place? It might give me some insight on how to proceed with migrating the metrics.
2. I assume since we are going to capture the metrics in beacon-chain/, the value of those metrics should be capturable within its packages.
3. I noticed that most of the metrics for the validators are captured in: validator/client/metrics.go. Would you like the instantiation of these metrics migrated to beacon-chain/monitor/metrics.go, and for the values to be captured within the beacon-chain/ packages?

If my assumption for point 3 is correct, then I think I know how I need to proceed.

I would also imagine that there would be a 1 <> 1 correlation for the metrics within the two packages. As I migrate the metrics, they should be equal to each other.

Let me know what you think.

[potuz]

@abdulrabbani00 Most of the metrics in the validator client (if not all) are already exposed by the monitor and they are already captured. So most of 1,2,3 is already done for these metrics. The issue here is to identify other metrics that may be useful to users and implement them within the monitor package. V3 is a good time to deprecate the validator metrics and only ship them in the monitor package.

[abdulrabbani00]

@potuz - That helps clear things up for me. I will work on adding new useful metrics to the monitor package. I'll also try to parse through the validator package to deprecate metrics, but I will prioritize adding metrics instead of deprecating metrics.

[abdulrabbani00]

I was considering the sort of metrics that might be useful for users that are not currently present. A few metrics I was capturing in my application were the following, and I wanted to see if they would be useful for validators to know:

• knownGaps - Keep track of any gaps that might have occurred when head tracking. For instance, if the last slot we saw was x, and the next slot we see is x+2 instead of x+1, then we know we have a gap. Not sure if this is something that can actually occur, or if its even worth capturing.
• reorgs - Simply capture the number of reorgs that have occurred.
• skipped - Keep track of the number of slots that have been skipped.

Let me know if any of these seem useful. Also if you know of any metrics that users want/need, I would be more than happy to work on those. Currently, I am just looking for some inspiration. @potuz @james-prysm

[potuz]

reorgs - Simply capture the number of reorgs that have occurred.

These metrics you mention are fine, particularly this one may be nice to have in the beacon node. But they are out of scope for this issue as this does not pertain the validator monitor. Metrics that should be added for this issue should be typically indexed by validator index tracked or aggregates among the tracked validators.