CloudSim Network Performance Testing
This work is intended to help diagnose CloudSim performance irregularities which occurred during the DARPA SubT Challenge Urban Circuit event.
A likely cause of these irregularities was a network-related issue, so several test cases were created to emulate network delays, bandwidth limits, and packet loss.
As well, test cases were created to emulate a theoretical event in which a large number of UDP multicast Ignition Transport messages are received by each infrastructure container, saturating their ability to send/receive messages, and thereby causing a failure.
Results of this testing have been shared with the DARPA SubT Challenge team in the hope that it will help to resolve these irregularities in the future.
For more information about the DARPA SubT Challenge, please see https://www.subtchallenge.com/.
Internal Docker Containers
advertisetopics: Build with build_advertisetopics.sh. Allows many concurrent Ignition Transport topics to be advertised, in order to cause saturation of any listening nodes. Build this before running the Ignition Transport related tests.
docker run -it advertisetopics advertisetopics <num_nodes> <topics_per_node>
docker run -it getalltopics getalltopics list
docker run -it getalltopics_ros
External Docker Containers
osrf/subt-virtual-testbed:cloudsim_sim_latest: This is the Gazebo simulation container from Open Robotics for running the simulation. It communicates using Ignition Transport with the bridge containers.
osrf/subt-virtual-testbed:cloudsim_bridge_latest: This is the bridge container from Open Robotics. It communicates using Ignition Transport with the simulation container, and using ROS messages with a competitor solution container.
osrf/subt-virtual-testbed:subt_solution_latest: This is a sample solution container from Open Robotics, capable of moving a robot to the world entrance.
The test_cases directory contains several test cases, each of which has a docker-compose.yml file defining the test setup. A common test_config.env file in the root directory defines common test parameters.
New test cases can be added simply by creating a new directory in test_cases with a
Running Test Cases
First, setup test_config.env with the common test setup. For testing the newest version of the CloudSim containers, you can define:
For debugging issues that occurred in February, these were instead set to:
solutionImage can be set to any valid solution container, including
After defining the common configuration, run
run_test.sh <test_case> to run
one of the defined test cases, or just
run_test.sh to see a list of test
cases and descriptions.
To stop a test case from running, use
Note: be careful to monitor system memory while the tests are running, because certain test cases can cause a memory leak which may freeze up the host machine.
Test Case Descriptions
Test with 100 Mbps bandwidth.
Test with 100 Mbps bandwidth and 5ms delay, variance 1ms, packet loss 0.1%.
Test with 1 Gbps bandwidth.
Test with 1 Gbps bandwidth and 5ms delay, variance 1ms, packet loss 0.1%.
Test with 100 ms delay, 10 ms variance.
Test with 10 ms delay, 1 ms variance.
Test with 10 ms delay, 1 ms variance, 0.1% packet loss.
Test with 10 ms delay, 1 ms variance, 1% packet loss.
Test with 10 ms delay, 1 ms variance, 5% packet loss.
Test with 10 ms delay, 1 ms variance, 0.1% reordering.
Test with 10 ms delay, 1 ms variance, 1% reordering.
Test with 10 ms delay, 5 ms variance.
Test with 10 ms delay, 5 ms variance, 0.1% packet loss.
Test with 10 ms delay, 5 ms variance, 1% packet loss.
Test with 10 ms delay, 5 ms variance, 5% packet loss.
Test with 10 ms delay, 5 ms variance, 0.1% reordering.
Test with 10 ms delay, 5 ms variance, 1% reordering.
Ignition Transport Saturation
Test with ignition transport message saturation. Two containers have 100 nodes advertising 100 topics per node, and one container has 50 nodes advertising with 100 topics per node.
Test with ignition transport message saturation. Two containers have 100 nodes advertising 100 topics per node, and one container has 90 nodes advertising with 100 topics per node.
Test with ignition transport message saturation. Three containers have 150 nodes advertising 100 topics per node.
Test with ignition transport message saturation. Ten containers have 50 nodes advertising 10 topics per node.
Test with ignition transport message saturation. 20 containers have 125 nodes advertising 1 topic per node.
Simple routing test with the
netsim docker container.