Testing Methodology

This work started last year with Eth Research.

Within the following test series, we will be testing libp2p's go implementation of gossipsub.

Additional testing considerations can be viewed within the following document. Please feel free to provide any feedback or commentary and contribute to these efforts however you see fit. Please reference this document for further details pertaining to the test plan.

Overview

The network is segmented into X number of shards. Every ~10 minutes, validators are randomly assigned to a shard, so the stress point is observing and testing the ability of validators to subscribe to new topics and send/receive messages pertaining to this new topic in an adequate amount of time.

Test Utility

• Number of nodes: <100 (more if necessary)
• Bandwidth: 1G (standard, up to 10G if necessary) can be configured and assigned to each individual node.
• Latency: >1 second of network latency can be applied to each node’s individual link.
• Data aggregation and visualization

Test Scenarios

• Observe and measure performance under the presence of various network conditions.
  • Latency between nodes:
    • What is the maximum amount of network latency each individual node can tolerate before performance begins to degrade?
    • What are the security implications of high degrees of latency?
    • Are there any other unforeseen issues which may arise from network conditions for which we can’t accommodate via traditional code?
  • Latency between shards.
  • Intermittent blackout conditions
  • High degrees of packet loss
  • Bandwidth constraints (various bandwidth sizes)
• Introduce new nodes to network:
  • Add/remove nodes at random.
  • Add/remove nodes at set intervals.
  • Introduce a high volume of nodes simultaneously.
• Partition tolerance
  • Prevent segments of nodes from communicating with one another.
• Measure the performance of sending/receiving messages within set time periods and repeat for N epoches.
• Observe process of nodes joining and leaving shards
  • Subscribing to shards
  • Connecting to other nodes within shard
  • Synchronize collations and ShardPreferenceTable
  • Unsubscribing from shards

Need to Define

• Configuration specifications for relevant test cases to create templates which allow for quicker test automation.
• Code which should be tested.
• Preliminary testing methodology should be established based on everyone's input.
  • We can make adjustments to this methodology based on the results of each test case.
  • It's generally best (in my experience) to create a high-level overview which provides a more granular definition of the first three test cases and then make adjustments to each subsequent test series based on the results of those three.

Other Notes

• This document acts as a high-level overview to communicate potential test scenarios based on our initial assumptions of the existing codebase. It is meant to act as a starting point to guide the development of a preliminary test series.
• Although tests will be run locally within our lab, access to the test network can be granted to appropriate third-parties for the sake of due diligence, validation, or other purposes as deemed necessary.
• Network statistics and performance data dashboard can be assigned a public IP to allow for public access.
• Raw and formatted data will be shared within appropriate repos.
• Please voice any suggestions, comments, or concerns in this thread and feel free to contact me on Gitter.