Load balance resolved bootnodes

[corverroos]

Problem to be solved

With public testnet approaching and with direct network connections via NAT traversal being a problem, we are going to rely on Obol hosted bootnodes to act as circuit relays for cluster connectivity (for charon run and charon dkg).

Problems:

• We currently only have a single bootnode, and this will not scale to support hundreds of nodes and thousands of relay connections.
• Clusters must also use the same bootnode instance.
• Bootnodes must be long lived, since charon doesn't support dynamic bootnodes or recycling of bootnodes.
• Bootnodes need to be able to handle a large amount of relay connections.

We therefore need a scalable bootnode solution that would support hundreds of long lived cluster for our public testnet.

Proposed solution

Charon resolves bootnodes on startup. It polls a static configured HTTP (using DNS) endpoint (http://bootnode.gcp.obol.tech:3640/enr) on startup which returns a bootnode ENR. That ENR contains the publickey and public addresses (TCP and UDP) of the bootnode which we then provide to discv5 (UDP) and libp2p (TCP) which is then used for the rest of the lifetime of the instance.

If we deploy a number of bootnode instances (e.g. 64), which are all publicly available on their own and we deploy a loadbalancer using sticky header routing then clusters will resolve 1-of-64 bootnodes.

• Deploy sufficient bootnodes (since we cannot change the number since that will result in previously-resolved bootnodes to mismatch subsequently-resolved bootnodes).
• Each bootnode should be long-loved. Since its ENR should be static for the lifetime of a cluster. Suggest using static-sets to achieve ENR consistency across restarts/deploys.
• If 64 small instances are insufficient to handle the load, we can (to a limited extent) vertically scale by increasing the instance size.
• Deploy a HTTP header based sticky load balancer at http://bootnode.gcp.obol.tech:3640/enr.
• Charon will include a Charon-Cluster:<lock_hash> header as part of bootnode ENR resultion HTTP call.
• All charon nodes of the same cluster will therefore resolve the same long lived bootnode.

Out of Scope

We only support load balancing at ENR resolution time. We do not support dynamic rolling bootnode load balancing.

[corverroos]

Closing this for now since the charon part is done. There is another ticket on ObolNetwork/obol-infrastructure#50 for the infra side of things.