Running Netdev CI tests locally

This page describes which tests are being executed by the netdev CI (aka NIPA), and how to reproduce them locally.

Important

➡️ It is highly recommended to executed the same tests before any submissions, if you are able to, not to give more work to the maintainers and reviewers

Various tests are executed by the CI. There are currently 3 categories:

LLM reviews per patch: Sashiko and Clashiko.
Static tests per patch: builds in different conditions, including the doc, and static analytic checks like checkpatch.pl, Python and shell scripts related ones, etc.
Runtime tests: executed in VMs: kselftests, KUnit, etc.

Status pages

Various webpages can help to find the current status for the runtime tests:

There are 3 runner types which can be distinguished by looking at the "Remote":

runners called metal-${name} are netdev runners running with a normal / full performance kernel;
runners called metal-${name}-dbg are netdev runners running with debug kernels;
runners without metal- in the name are external, run by other teams / companies and reported to the system.
(runners using dedicated hardware are not coverred by this page, please check this page instead)

Note

Due to the very high amount of traffic on the netdev mailing list, runtime tests are not executed separately for each patch, but executed using all pending patches on top of net-next merged with net, plus a couple of local patches. Git branches are exported on GitHub, and the content can be seen by clicking on the branch name on the status page.

LLM reviews

Sashiko can be easily executed locally, with different models. The project's README explains how to install and use it. In short, it is supposed to be easy: configure the model to use, run sashiko init, then sashiko review. That's it.

Clashiko is also using Sashiko, but running with multiple models, doing cross reviews.

Static tests

The netdev CI executes a bunch of scripts from this repo. An easy way to reproduce this is to use the ingest_mdir.py script with your patches:

export your patches, e.g. using b4 prep/send
switch to a branch on top of net-next (or net)
execute ingest_mdir.py

Example:

## Export patches
$ cd $linux
$ b4 prep --set-prefixes 'net-next'  ## or use `git format-patch` with `--subject-prefix="PATCH net-next"`
$ b4 send -o /tmp/my-series          ## or use `git format-patch -o /tmp/my-series/ (...)`

## New branch on top of net-next (or net)
$ git switch -c test netdev-next/main

## Execute the tests: be patient!
$ cd $nipa
$ ./ingest_mdir.py --mdir /tmp/my-series --tree $linux

The last step can be executed from a container, not to have to install all dependences manually:

$ cd $nipa
$ ./docker/build/run.sh --pull ./ingest_mdir.py \
        --mdir /tmp/my-series --tree $linux --result-dir out

Note

Images can be built locally using ./docker/build.sh script.
--pull is used to always download the latest image.
Docker is being used here, but podman can be used instead.

KUnit

KUnit instructions can be used here, but in short, you can execute:

$ cd $linux
$ ./tools/testing/kunit/kunit.py run --alltests --arch=x86_64

Or using the container:

$ cd $linux
$ $nipa/docker/selftests/run.sh --pull \
        ./tools/testing/kunit/kunit.py run --alltests --arch=x86_64

Selftests

This part describes how selftests are executed in the netdev CI. It's not the one and only way the tests can be run!

Environment

Virtual tests are executed in Fedora, latest version, with a bunch of dependences either from the official repos, or using some specific versions built from source. Tests are executed in light VMs with the help of virtme-ng which is used to build the kernel and run the tests.

Tip

The recommended way to execute these tests locally is to use our container images. This can be done by prefixing each vng/make command below with:

either: ${nipa}/docker/selftests/run.sh --pull

or the full Docker or Podman command, e.g.:

$ mkdir -p .ccache
$ docker run \
    -v .ccache:/home/nipa/.ccache:rw \
    -v "${PWD}:${PWD}:rw" \
    -w "${PWD}" \
    -u "${RUID:-$(id -u)}:${RGID:-$(id -g)}" \
    --group-add "$(grep "^kvm:" /etc/group | cut -d: -f3)" \
    --rm \
    -i -t \
    --privileged \
    --pull always \
    ghcr.io/linux-netdev/nipa-selftests:latest \
    ## the rest of the command here

An alias can be used not to have to re-type this long command

TL;DR

To build and execute all net selftests:

cd $linux
alias run="${nipa}/docker/selftests/run.sh"             ## or the full docker/podman command
docker pull ghcr.io/linux-netdev/nipa-selftests:latest  ## to get the latest version once

TARGETS=(net net/af_unix net/forwarding net/hsr net/mptcp net/netfilter net/openvswitch net/ovpn net/packetdrill net/ppp net/rds net/tcp_ao nci)
for target in "${TARGETS[@]}"; do
    rm -f .config
    run vng --build --force --config "${config}" -v
    run make headers
    run make -C tools/testing/selftests TARGETS="${target}"
    run sudo vng -v --run . --user root -a mitigations=off --rw --cpus 4 --network loop -- \
        make -C tools/testing/selftests TARGETS="${target}" run_tests
done

Feel free to adapt the TARGETS array depending on the modifications you did, e.g. by executing tests from the drivers side as well: drivers/net drivers/net/bonding drivers/net/hw drivers/net/netconsole drivers/net/netdevsim drivers/net/team drivers/net/virtio_net

How to build

Here are more detailed explanations ⬇️

Building the kernel

Kernels are built with just the relevant options enabled, for instance for net selftests:

vng --build --config tools/testing/selftests/net/config

and for forwarding selftests we'd use the forwarding config:

vng --build --config tools/testing/selftests/net/forwarding/config

In other words the runner for net will not have the options for forwarding enabled. Test authors must ensure that all relevant options are listed in the config file local to the test!

The -dbg runners get extra debug options:

vng --build  \
        --config tools/testing/selftests/net/forwarding/config \
        --config kernel/configs/debug.config

Tip

vng is unreliable at detecting when kernels need to be rebuilt so it's a good idea to run make mrproper before the build to delete old builds. Note that make mrproper will remove a lot of artifacts, including your old configs!

Important

When using vng --config <config file>, be aware that it uses your local .config as a starting point. If you're introducing a new test or dependency, ensure you don't have a pre-existing .config file in your working directory. This allows you to test against the kernel's default configuration without any local modifications interfering.

Building the tests

Selftests binaries are built separately, e.g. for forwarding:

make -C tools/testing/selftests/ TARGETS=net/forwarding

Tip

Building selftests is prone to failing as it tries to use distro headers by default. If compilation fails, generate uAPI headers: make headers, and try again.

How to run

vng is used to start a VM with the freshly built kernel, and the same file system as the host (or the container). It is possible to pass the command to execute, e.g. here to execute the tests from tools/testing/selftests/net/forwarding:

vng -v --run . --user root -a mitigations=off --cpus 4 -- make -C tools/testing/selftests TARGETS=net/forwarding run_tests

Tip

Executors can also run tests one by one:

vng -v --run . --user root --cpus 4 -- \
        make -C tools/testing/selftests TARGETS=net TEST_PROGS=pmtu.sh TEST_GEN_PROGS="" run_tests

Refer to the kselftest doc for more details about running a subset of selftests and more.

Dealing with slow runners in performance/latency tests

The -dbg runners (and possibly other external runners) export this KSFT_MACHINE_SLOW=yes environment variable if the runner's performance is "slow". This can be used by performance and timing tests to avoid returning failures. The tests are expected to still execute the steps (for pure code coverage) but ignore not meeting performance goals.

Tips

Improving performances

virtme-ng will use virtiofs if virtiofsd is available on the host. virtiofs apparently performs much better than 9p. Please note that 9p will be used for directories given to --rwdir and --rodir options: best not to use it with the whole kernel source tree. It might then be better to use --rw or --overlay-rwdir instead. virtiofs is available in the nipa-selftests container.

When virtme-ng is launched from a container, make sure it can access /dev/kvm, not to fall back to the tcg backend, which is slower. For example if you use Docker, an easy way to get KVM support is to use the --privileged option with docker run.

Reproducing unstable tests

We found that following changes increase test flakiness and thus can help reproduce rare failures:

Increase the number of CPUs by modifying --cpus $number in the vng command.
Disable KVM support by passing --disable-kvm to the vng command.
Don't disable the mitigations by not passing -a mitigations=off to the vng command
Generate some noises on the host: stress-ng --cpu $(nproc) --iomix $(nproc) --vm $(nproc) --vm-bytes 1G --timeout 60m
Reduce the priority of the VM: sudo renice -n 20 -p $(pidof qemu-system-x86_64) (from docker exec to modify VMs from a running container)

If things are not working

Skip the -v option if you don't want to see kernel logs.
If QEMU fails to start, try using vng with --disable-microvm.
Some tests try to write into the PWD and virtme-ng gives by default RO access to the host filesystem. You can either use OverlayFS with --overlay-rwdir tools/testing/selftests/net/, or a direct write access with 9p (might be slow if IO is important) using --rwdir tools/testing/selftests/net/ or put the whole host filesystem in RW access with --rw (e.g. if launched from a container).
Some tests use kernel modules loaded at run-time and the virtme environment uses the host modprobe configuration. Local configuration, e.g. module blacklist, can cause tests failure. To start virtme with an empty modprobe configuration use: mkdir modprobe.d; vng --rodir /etc/modprobe.d=modprobe.d #... other options

Navigation Menu

Uh oh!

Running Netdev CI tests locally

Running Netdev CI tests locally

Status pages

LLM reviews

Static tests

KUnit

Selftests

Environment

TL;DR

How to build

Building the kernel

Building the tests

How to run

Dealing with slow runners in performance/latency tests

Tips

Improving performances

Reproducing unstable tests

If things are not working

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Clone this wiki locally