Netdev CI system
High level flow of the full system:
The design is supposed to be very loosely connected. All the blue boxes communicate with other blue boxes thru JSON files fetched over HTTPs. The purple interface boxes correspond to the JSON formats consumed and produced by executors (the services actually running tests). Since we want to give people the ability to run their own tests in whatever environment suits them best, the implementation of the executor is entirely up to whoever hosts it. These interfaces will not change in a backward-incompatible way.
Branch constructor creates git branches and instructs the rest of the pipeline to test them. It maintains a manifest file (branches.json) and a set of files describing particular branches (branches-info.json). The pipeline only consumes the manifest, the info files are consumed only by the Status updater to map branches back to patchwork patches.
The manifest (branches.json) is an array of objects with following fields:
| Field | Format | Description |
|---|---|---|
| branch | String | Name of the branch tested. Unique ID of the branch. |
| url | String | Git URL for fetching the branch. |
| date | String | ISO formatted date (UTC). When the branch was created. |
| base | String | Hash of the base commit, on top of which changes to be tested are applied. |
Branches are deleted a month after creation.
References:
- Code: https://github.com/linux-netdev/nipa/blob/master/trivial-net-brancher.py
- Repo where branches to test are pushed: https://github.com/linux-netdev/testing
- JSON manifest of branches: https://netdev.bots.linux.dev/static/nipa/branches.json
Executors periodically check branch manifest and kick off tests if new branches were created. It’s possible to also hook executors to GitHub actions. There should be no filtering whether tests should run for a given branch (any code exercised by the tests has been changed), to simplify analysis and reasoning about reliability of the infra (if the tests are not reported - it’s a bug).
References:
- NIPA’s basic executor running a bash script: exec.py
- NIPA’s executor for running kselftests: vmksft-p.py
Important: keep in mind that NIPA executors are just samples, the only real requirement is that the executor produces a correct results JSON.
Static / hand edited file listing all remote sites. Its primary purpose is to inform the Collector where to fetch results from. Collector uses it to poll appropriate sources of results, it also times out the search if the results do not arrive within 8 hours.
Results are the central component of the design. Basic remote site should host a manifest file (results.json) listing all runs and URLs for full results from those runs. The manifest is an array of objects with following fields:
| Field | Format | Description |
|---|---|---|
| executor | String | Name of the setup running the test, should include enough information to identify the entity (vendor) running the test. Used for grouping to exclude unreliable executors from final verdict. |
| branch | String | Name of the branch tested. Used to ID the code which was tested. |
| url | String | URL of the results for this run. null if tests have not finished. |
When a remote site ingests a new branch it should spawn entries for all executors in its results manifest leaving the URL as null. Collector will wait until all URLs are populated before declaring remote as done with a branch. Note that Remote DB only lists sites, not executors, so Collector does not know about all executors upfront. This allows sites to create new executors without updating Remote DB.
Each run of tests on an executor generates a separate results file (linked in the manifest as URL).
Results file contains:
| Filed | Format | Description | |
|---|---|---|---|
| executor | See doc in manifest. | ||
| branch | See doc in manifest. | ||
| start | String | ISO formatted date (UTC). When execution started. | |
| end | String | ISO formatted date (UTC). When execution finished. | |
| link | String | (Optional) URL to detailed outputs for the entire executor run (including executor setup and infrastructure logs). | |
| results | test | String | Name of the test. |
| results | group | String | Group of tests, used to collapse outputs in UI, and potentially selectively run tests based on what changed in the tree. |
| results | String | ||
| results | result | String | pass / fail / skip. |
| results | link | String | URL to where the detailed outputs can be viewed. Could be the same for all tests if the executor generates a single combined output for all tests. Splitting up results per test is preferred. |
| results | crashes | String | (Optional) Array of strings. Fingerprints of crashes which happened during the test. |
The contents of the manifest and result files should be rotated monthly.
References:
- Example results manifest file: https://netdev-3.bots.linux.dev/vmksft-net-dbg/jsons/results.json
- Example results files: https://netdev-3.bots.linux.dev/vmksft-net-dbg/jsons/
Collector fetches results from remote sites periodically and populates a local database.
Periodically reads the results collected by Collector and reports results for individual patches to patchwork. Status updater performs reverse mapping of branch to list of patches it contained. One patch can be in multiple branches (in fact it's present in all branches until discarded from PW or merged). The status in patchwork will be the "best" result over all branches.