Display "BeaconNode far behind" on grafana

[xenowits]

🎯 Problem to be solved

Beacon node for some nodes may lag a lot behind the head slot. We should alert this to the node operator on the grafana Healthy panel.

🛠️ Proposed solution

Add another const to app/metrics.go:

// readyzBeaconNodeFarBehind indicates that readyz is returning 500s since the Beacon Node is too far behind the head slot.
readyzBeaconNodeFarBehind = 8

Add the following logic to app/monitoring.go:

if beaconNodeDown {}
else if beaconNodeSyncing {}
else if beaconNodeZeroPeers {}
# The following is a new addition
else if beaconNodeFarBehind {  readyzGauge.Set(readyzBeaconNodeFarBehind) }
...

BN far behind is TRUE if beacon_node_sync_distance > 320 slots. This is a fair threshold as 320 slots correspond to 10 epochs which is around 60 minutes.

So, it should potentially follow this

if beacon_node_not_syncing AND beacon_node_sync_distance > 320 slots { "BN far behind" }

If the panel shows "BeaconNode far behind" means that the operators need to check their Beacon Node for potential syncing issues.

🧪 Tests

• Tested by new automated unit tests