p2p: robust intra-cluster broadcasting

[corverroos]

Problem to be solved

DutyAttester and DutyProposer doesn't work in a charon cluster with one or more nodes being unavailable/down.

This is due to parsigex (and probably other libp2p components) synchronously and sequentially sending a message to all other peers, one after the other. If sending to one node errors, then the error is caught and sending to the next node is continued. This is fine. But networking is IO, and IO can block. So sending to a down node, can and does result in the call to send blocking for multiple seconds. When the duty times out during this period, the context is cancelled and we log "partial send success" without sending to all nodes.

The other issue is that we calculate "success count" incorrectly, so it looks like we sent all available nodes. But we actually only sent to a few other nodes, then blocked, then exitted.

A related problem is warning/error log storm generated when a single node is down. In large clusters, when multiple nodes are down, these warnings will spam the logs.

Proposed solution

Introduce a "libp2p-sender", a thing that solves both these problems (name TBD):

• Async sending: components like parsigex uses libp2p-sender to send async messages. sender.SendAsync(ctx, targetPeer, protocol, msg)
• This will prevent sends blocking other sends when a node is down and IO blocks.
• Warn log filtering: It doesn't log warnings when sending to a known down node fails.
• It keeps track of higher level networking metrics, and logs summaries and state changes only.
• Wire this thing into all components doing p2p sending.

[corverroos]

Out of scope (follow up PR): Introduce a Broadcast method to the p2psender so each component doesn't need to duplicate broadcasting logic.