investigate issues with beacon node load balancing [3D]

[rolfyone]

No description provided.

[rolfyone]

• validator node using a remote beacon node on a service provider, seem to periodically have issues with lower attestation performance

one avenue would be to have a prater or pyrmont node be validator only and using a remote beacon node provider

[rolfyone]

• how do we measure performance of a validator when we dont own the beacon node?
• validator client metrics

[rolfyone]

As part of the investigation, disabled early attestation production on a validator node on prater that used a remote beacon node.

Xvalidators-early-attestations-enabled: False

The outcome was that attestation inclusion rate much more closely tracked with expectations.

It seems fairly clear that this configuration option needs to be publicised for use in this kind of configuration, so that other validator clients can gain similar improvements.

I created a limited dashboard to monitor using grafana:

• local validator count
• published attestations, blocks, aggregations
• scheduled blocks, attestations (eg. if you're about to get to propose a block, you might not want to restart)
• current epoch (will be time based epoch calculated by validator client)
• invalid events received on the event stream
• event stream disconnections
• cpu / mem/GC stats

It'd be helpful to get some context of the beacon node to allow things such as

• attestation effectiveness for validators I'm running
• head slot of the beacon node (could sign up for head events and put a metric on that easily)
• sync status of the beacon node

[rolfyone]

raised #4225 also. closing spike.