Improve telemetry of bandwidth usage

[lasarojc]

Target audience

Operators and consensus developers.

Problem definition

Currently we rely on a small set of metrics, from a small set of setups, to determine how bandwidth is used.
We need to gather more data on how the bandwidth is being used in real world scenarios.
The data gathered and the method to do so should be well understood by operators, should not disclose any information the validators wouldn't like to share, and should not burden operators.

Upsides

• Develop fine grain understanding of data/metadata traffic on the network in in CometBFT.
• The approaches developed could later be expanded to collect other kinds of information, for example, storage usage.

Downsides

• Validators need to vet information before sharing with the team, which adds to their work.
• Code developed may end up being used only in test environments.

Tasks

Goals (1w)

Beta Give feedback

1. Define metrics to improve and to capture to determine bandwidth usage
2. Collect real-world samples for various versions of CometBFT
3. Plot bandwidth usage at a given point in time for test setups (Implemented as a directed weighted graph with metrics on the edges; Collects data at the end of a run from a Prometheus node; Implemented as part of Add full support for e2e tests in fast-prototyping mode #1085)

Stretch goals

Beta Give feedback

1. Provide guidelines to validators for collecting and sharing bandwidth usage with the CometBFT team

Long term goals

Beta Give feedback

1. Provide real-time visualization of bandwidth usage of metrics

Definition of done

• Metrics to be captured are defined.
• Running an experiment in the e2e testbed, or over a single administrative domain in a real-world deployment details bandwidth usage.
• Details come as a graph where a link between two nodes is weighted by the amount of information that transited among.
• This information is broken down wrt. the reactors (mempool, consensus, evidence, etc.).
• If time allows, this information is updated in real time.
• Jointly with this, we collect samples from real-world deployments to gather additional information about bandwidth usage.
• To this end, we provide validators with clear guidelines to collect such information and share it with us.

[lasarojc]

For bandwidth, the obvious candidates are

• already_received_txs, peer_receive_bytes_total, peer_send_bytes_total (ratios).
• bther metrics about duplicate txs could be needed.

For latency, we can use the script to extract latencies in the QA process

• a shorter goal is not to substantially change the current latency distribution
• We may want also to know: the number of connected peers, the internal state of the mempool reactor (e.g. set of senders for each tx)