README.md

Testsuite for SCS standards

The tool scs-compliance-check.py parses a compliance definition file and executes the test executables referenced in there for the specified layer (iaas or kaas).

Local execution (Linux, BSD, ...)

On your Linux machine, please ensure you have installed the testsuite dependencies from requirements.txt. We recommended using Python >= 3.10 and to install the requirements into a virtualenv as follows:

python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

With a cloud environment configured in your ~/.config/openstack/clouds.yaml and secure.yaml, then run

./scs-compliance-check.py scs-compatible.yaml iaas --os-cloud CLOUDNAME

Replace CLOUDNAME with the name of your cloud environment as specified in clouds.yaml.

The exit code indicates success (0) or failure (!= 0). You can also request a YAML report using the option -o OUTPUT.yaml

Usage information (help output)

Usage: scs-compliance-check.py [options] compliance-spec.yaml layer [layer [layer]]
Options: -v/--verbose: More verbose output
 -q/--quiet: Don't output anything but errors
 -s/--single-layer: Don't perform required checks for dependant layers
 -d/--date YYYY-MM-DD: Check standards valid on specified date instead of today
 -V/--version VERS: Force version VERS of the standard (instead of deriving from date)
 -c/--os-cloud CLOUD: Use specified cloud env (instead of OS_CLOUD env var)
 -o/--output path: Generate yaml report of compliance check under given path

Testing in docker containers

Build a docker container

docker build --tag scs-compliance-check .

Run tests in a docker container

docker run -it --env OS_CLOUD=CLOUDNAME -v ~/.config/openstack:/root/.config/openstack:ro scs-compliance-check

The Docker entrypoint uses scs-compatible-iaas.yaml on the iaas layer by default. You can use an alternative spec file by simply appending it to the above call, e.g.

docker run -it --env OS_CLOUD=CLOUDNAME -v ~/.config/openstack:/root/.config/openstack:ro scs-compliance-check my-own-certification.yaml iaas

Debugging

docker run -it --env OS_CLOUD=CLOUDNAME -v ~/.config/openstack:/root/.config/openstack:ro --entrypoint /bin/bash scs-compliance-check

Information for developers

Unit and regression tests

Some of the conformance tests scripts are themselves tested with unit tests.

To run them, first ensure that you have installed the unit test dependencies in addition to the main dependencies (inside your virtualenv as described above):

pip install -r test-requirements.txt

Now you can run the unit tests with pytest:

# Option A: let pytest discover and run all unit tests (**/*_test.py)
pytest

# Option B: run only a subset of the tests
pytest kaas/k8s-version-policy/k8s_version_policy_test.py

# Option C: produce a HTML code coverage report and open it
pytest --cov --cov-report=html
xdg-open htmldoc/index.html

You are encouraged to cover new conformance tests with unit tests! We run the tests on a regular basis in our GitHub workflows.

Maintaining the Python dependencies

We list our main Python dependencies in requirements.in. Additionally, we list unit tests dependencies in test-requirements.in. The *.in files are fed to pip-compile to produce corresponding *.txt files that contain an exact, version-pinned listing of all dependencies, including transitive ones.

pip-compile can be installed via pip install pip-tools. It needs to be run in two cases:

1. You modified an *.in file: run pip-compile <INFILE>. For example:

   pip-compile test-requirements.in

2. You want to bump the pinned dependencies: add the --upgrade flag to the pip-compile invocation. For example:

   pip-compile --upgrade requirements.in

Note: The Python version used for running pip-compile should be consistent. The currently used version is documented in the header of the requirements.txt. It should match the version used in the Docker image (see Dockerfile) and in our GitHub workflows (lint-python.yml and test-python.yml in .github/workflows).