Expand the QA to include a performance benchmark

[lasarojc]

Feature Request

Summary

The QA process should be extended with a performance benchmark

Problem Definition

The QA report does not provide much information on the performance of the system.
It should be extended to include reproducible tests with

• fixed and varied topologies
• different sizes of proposals
• different sizes of vote extensions

Proposal

• Predefine large topologies and workloads, including different transaction, vote extension and proposer set sizes.
• Run the tests multiple times on these topologies.
• Determine latency and throughput at different percentiles for comparison with previous and past runs.

[lasarojc]

Going into the solution space, after a brief discussion it seems that a quick and dirty way of fixing topologies in our large topology tests would be

• define a topology
• generate the configuration to include neighbors in the topology as persistentPeers
• set the maxOutgoingConnections to len(persistentPeers) OR turn the PEX off

In case o timeouts, this settings will recreate the same connections, but timeouts are more of a concern on small k8 networks.

[sergio-mena]

Thanks for this issue. I just hooked it to #42, which tracks all QA-related work that is (or will be) planned

[cason]

In case o timeouts, this settings will recreate the same connections, but timeouts are more of a concern on small k8 networks.

Actually, in case of disconnection from a persistent peer, new connection attempts are made, regardless of the reason for the disconnection.

The only exception here is when seed nodes disconnect from peers after crawling their addresses or providing them a fresh list of address. Recall, please, to not set seed nodes as persistent peers.

[cason]

different sizes of proposals

You probably mean here different block sizes. Proposals, meaning here the message and type, are fixed size.

[cason]

An important output of this approach, in my opinion, is to understand whether or how much the topology impacts on performance. We need to establish whether we can compare performance results obtained in similar setups (size system size and workload) but having different topologies. While it is clear that different topologies in real-world WAN setups probably lead to different performance, it is not clear whether in a restricted/LAN/same-region setup the impact of the topology is really relevant.