[E] Parallelizable system tests #90
Description
Tasks
- Create Dockerfile to run system_tests.sh script
- Rewrite system_tests.sh to include parallel option, e.g. -t
- Write listener script that monitors the progress of each docker container, copies the logs from the containers using
docker cpwhen they're finished, and uses these logs to report the test results.
Description
Currently, the system tests take quite a long time to run. However, parallelizing them is not trivial since the network setup only contains two private networks, and hence only two NAT configurations at a time. Extending the amount of private networks would be possible, but the resulting multiple connections to the eduP2P control and relay servers would make the system test logs harder to read.
My idea to parallelize the system tests is to use Docker containers to simulate each 'thread'. If the system tests are run in parallel, the script will still be called from the host. However, instead of actually executing the tests, the host will divide the run_system_test commands among files (one file per thread) in a round-robin manner. The docker containers can then execute the system_tests.sh script with the -f option on their own file.
After dividing the system tests, the host should run a listener script that monitors the progress of the Docker containers. When all docker containers are finished, the host can copy the logs from the docker containers back to the host, recreate the test results from the logs, and write them to the terminal.
With parallel system tests, the user must take into account the possibility that any failing tests may be caused by their machine not having enough computing power to run all the system tests in parallel without dropping packets. This feature should not be used for performance tests, since it is known that the test suite's performance tests results heavily depend on the amount of CPU cores.