Python infrastructure for integration tests

[teknico]

As recently discussed on the team, and approved by upstream, we are going to write integration tests in Python, in addition to shell scripts.

Python version

We are going to use Python 3, not Python 2.

Python 3.0 was released in 2008, and its current version is 3.6.

Python 2.7.0, the last v. 2 minor version, was released in 2010, and its end of life is in 2020.

The migration of the ecosystem to Python 3 is almost complete, and we will not be needing additional libraries anyway (with one exception, see below).

The Python 3 interpreter has been available for years on NixOS, on Ubuntu, and on most other platforms.

Both Ubuntu 14.04 and NixOS include at least Python 3.4, therefore we will be able to use the enum and pathlib libraries introduced in v. 3.4.

Also, implicit namespace packages (introduced in Python 3.3) will avoid unsightly __init__.py files all over the place.

Last but not least, Python 3 is better overall. :-)

Test framework

We are going to use the standard unittest library. Other testing frameworks are available, but they don't add enough value for our use case to justify adding a dependency.

Subprocess handling

The standard subprocess library is not easy to use, and feature incomplete.

We are going to use the sh library, which handles gracefully most of our use cases with a clear and terse syntax. It has been around for years and is implemented by one single file that we are adding to our repo, to avoid having dependencies other than the Python 3 interpreter.

Code structure

Integration tests are run by the make test command, which calls all selftest.sh shell scripts in the codebase. We will have a single entry point for all lwAftr Python tests, located in the src/program/lwaftr/tests/selftest.sh script.

A single entry point reduces setup code duplication and avoids the overhead of starting up the Python interpreter multiple times. We will still be able to run tests in specific subdirectories.

Furthermore, Python integration tests will be confined to the src/program/lwaftr/tests/, to ease human and automated discoverability.

Security

Snabb commands executed in integration tests need root privileges. However, tests as a whole do not have to be run under sudo anymore: individual commands are run under sudo as needed.

The user under which tests are executed still has to be added to /etc/sudoers with a NOPASSWD clause, to enable unattended execution.

[teknico]

Hrmpf. Spoke too soon with regard to not having to pepper all test directories with __init__.py files: apparently recursive test discovery in namespace packages is currently broken. So peppering it is, for now.

[teknico]

This is now in place with #750, #751 and subsequent changes (notably #763).

[kbara]

Fantastic, thank you!