diff --git a/docs/Development/Building/Dependencies.md b/docs/Development/Building/Dependencies.md index 5c41ff8a9..d1fc9d00a 100644 --- a/docs/Development/Building/Dependencies.md +++ b/docs/Development/Building/Dependencies.md @@ -21,6 +21,29 @@ Trident's dependencies. sudo apt install build-essential pkg-config libssl-dev libclang-dev protobuf-compiler ca-certificates unzip ``` + :::warning protobuf-compiler version + Building Go tools that use gRPC (e.g., `netlaunch`, `storm-trident`) requires + `protoc` 3.15+ for proto3 optional field support. On Ubuntu 22.04 and earlier, + the apt `protobuf-compiler` package is too old (3.12). Install a newer version + manually (example shown for x86\_64; adjust the archive name for other + architectures): + + ```bash + PROTOC_VERSION=28.3 + PROTOC_ARCH=linux-x86_64 # use linux-aarch_64 for arm64 + curl -sL https://github.com/protocolbuffers/protobuf/releases/download/v${PROTOC_VERSION}/protoc-${PROTOC_VERSION}-${PROTOC_ARCH}.zip -o /tmp/protoc.zip + sudo unzip -o /tmp/protoc.zip -d /usr/local + rm /tmp/protoc.zip + ``` + + You also need the Go protobuf plugins: + + ```bash + go install google.golang.org/protobuf/cmd/protoc-gen-go@latest + go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@latest + ``` + ::: + For RPM builds (run inside the Azure Linux build container, not on the Ubuntu host), additional packages are needed: `rpmdevtools`, `sed`, `perl`, and `selinux-policy-devel`. diff --git a/docs/Development/Building/Testing.md b/docs/Development/Building/Testing.md deleted file mode 100644 index 17adef0f6..000000000 --- a/docs/Development/Building/Testing.md +++ /dev/null @@ -1,55 +0,0 @@ ---- -sidebar_position: 3 ---- - -# Testing Trident - -## Code Checks - -To ensure code quality and consistency, run `make check`. This verifies -formatting (`cargo fmt --check`), runs `cargo check` with all features, and -then runs `clippy` with `-D warnings`: - -```bash -make check -``` - -## Unit Testing - -To run Trident's unit tests, you can use the following command: - -```bash -cargo test --all -``` - -or - -```bash -make test -``` - -## Functional Testing - -:::info NOTICE -Running these tests depends on internal resources that are not publicly -available yet. -::: - -Many operations in Trident cannot be tested with unit tests alone given the -nature of the operations (e.g., manipulating disks, RAID arrays, mounts, -filesystems, etc.). For this reason, we have a suite of functional tests that -can be run in a controlled virtual environment. These tests are run as part of -our CI/CD pipelines. - -The tests themselves are located in the Rust code under `cfg` -attributes: - -```rust -#[cfg(feature = "functional-test")] -mod functional_test { - // ... -} -``` - -You can read more about how functional tests work in -[Functional Tests](Functional-Tests.md). diff --git a/docs/Development/Contributing/Pull-Requests.md b/docs/Development/Contributing/Pull-Requests.md index 3bc07797e..02cc1d9f3 100644 --- a/docs/Development/Contributing/Pull-Requests.md +++ b/docs/Development/Contributing/Pull-Requests.md @@ -33,11 +33,11 @@ When making a PR you are attempting to merge, make sure that: - Code is formatted correctly. (`make format`) (See: [Formatting](./Formatting.md)) - The build works and cargo-clippy is happy. (`make check`) - API docs are up to date. (`make build-api-docs`) -- Unit tests pass. (`make test`) (See: [Testing](../Building/Testing.md)) +- Unit tests pass. (`make test`) (See: [Testing](../Testing/Testing.md)) - Functional tests pass. (`make functional-test`) - Local make targets work. - `make docker-build` -- Coverage is above the baseline. (See: [Coverage](../Building/Coverage.md)) +- Coverage is above the baseline. (See: [Coverage](../Testing/Coverage.md)) *Note: Running `make all` will perform these checks.* diff --git a/docs/Development/Building/Coverage.md b/docs/Development/Testing/Coverage.md similarity index 75% rename from docs/Development/Building/Coverage.md rename to docs/Development/Testing/Coverage.md index e6fddc06e..cf2fc01af 100644 --- a/docs/Development/Building/Coverage.md +++ b/docs/Development/Testing/Coverage.md @@ -11,7 +11,7 @@ gates ensure we maintain a high level of code coverage. ## Requirements -See [Code Coverage Dependencies](Dependencies.md#code-coverage-dependencies). +See [Code Coverage Dependencies](../Building/Dependencies.md#code-coverage-dependencies). ## Running Coverage diff --git a/docs/Development/Testing/E2E-Tests.md b/docs/Development/Testing/E2E-Tests.md new file mode 100644 index 000000000..c5e72a3d2 --- /dev/null +++ b/docs/Development/Testing/E2E-Tests.md @@ -0,0 +1,693 @@ +--- +sidebar_position: 6 +--- + +# E2E Tests + +E2E tests validate complete Trident install-and-update workflows using +`netlaunch` to boot a QEMU VM from an installer ISO, followed by pytest +validation of the resulting host state. They are defined by Host Configurations +and test selections in `tests/e2e_tests/trident_configurations/`. + +E2E tests are orchestrated by three systems: + +- **storm-trident** (`bin/storm-trident`): A Go-based test orchestrator built on + the [Storm](https://github.com/microsoft/storm) framework. It manages VM + lifecycle, runs `netlaunch` for clean installs, and coordinates multi-step + update scenarios. +- **pytest e2e suite** (`tests/e2e_tests/`): A Python test suite that validates + the host state after servicing operations (partitions, filesystems, boot + order, etc.) by connecting to the VM over SSH. +- **Test configurations** (`tests/e2e_tests/trident_configurations/`): A + declarative system that pairs Host Configurations with test selections. Each + configuration directory defines *what* to install (disk layout, filesystems, + features) and *which tests* to run against it, enabling the same pytest suite + to validate many different Trident deployment scenarios. + +For VM-image-based servicing and rollback tests that don't use netlaunch, see +[Servicing Tests](Servicing-Tests.md) and [Rollback Tests](Rollback-Tests.md). + +## COSI Image Types + +E2E tests validate three COSI image types, each testing a different bootloader +and integrity configuration: + +| Image | Output File | Bootloader | Integrity | Configurations | +|-------|------------|-----------|-----------|----------------| +| `trident-testimage` | `regular.cosi` | grub2 | None | base, encrypted-partition, encrypted-raid, encrypted-swap, extensions, health-checks-install, misc, raid-big, raid-mirrored, raid-resync-small, raid-small, simple, split | +| `trident-verity-testimage` | `verity.cosi` | grub2 | Root dm-verity | root-verity | +| `trident-usrverity-testimage` | `usrverity.cosi` | systemd-boot | `/usr` dm-verity (UKI) | combined, memory-constraint-combined, rerun, usr-verity, usr-verity-raid | + +**`regular.cosi`** — Standard grub2-based image with no integrity protection. +Uses `grub2-efi-binary-noprefix`, includes `trident-service` and +`tridentd.socket`. This is the baseline image for most E2E configurations. + +**`verity.cosi`** — Root filesystem is protected by dm-verity, making `/` +read-only. Uses grub2 with a separate `/var` partition and an `/etc` overlay +service for runtime state. Includes `veritysetup` and `dracut-overlayfs`. + +**`usrverity.cosi`** — The `/usr` filesystem is protected by dm-verity, with +a Unified Kernel Image (UKI) and systemd-boot as the bootloader. This is a +preview feature (`previewFeatures: uki`). Requires `ukify` on the build host. + +## Prerequisites + +- **Linux host** with root access +- **libvirt and QEMU** installed and configured (user must be in the `libvirt` + group) +- **Docker** (for building images with Image Customizer) +- **[oras](https://oras.land/)** CLI (for downloading base images from MCR) +- **Go 1.25+** (for building Go tools) +- **Rust** (latest stable, for building Trident) +- **Python 3.8+** with packages: + + ```bash + pip3 install fabric pyyaml pytest + ``` + +See [Dependencies](../Building/Dependencies.md) for full build dependency details including +protobuf compiler requirements. + +Unless otherwise noted, commands are run from the repository root. Pytest +commands are run from `tests/e2e_tests`. + +## Building Dependencies + +### 1. Build Trident + +```bash +make build +``` + +### 2. Build Go Tools + +```bash +# Build all Go tools at once +make go-tools + +# Or build individually: +make bin/netlaunch # Boots VM from ISO, serves config over HTTP +make bin/netlisten # Serves COSI images for A/B updates +make bin/storm-trident # E2E test orchestrator +make bin/virtdeploy # VM lifecycle management +make bin/isopatch # Injects files into ISOs +make bin/rcp-agent # Remote control plane agent +``` + +### 3. Build osmodifier + +```bash +make artifacts/osmodifier +``` + +### 4. Build Trident RPMs + +The test images include Trident packages built from your local tree. This step +builds the RPMs into `bin/RPMS/`, which `testimages.py` passes to Image +Customizer via `--rpm-source`: + +```bash +make bin/trident-rpms.tar.gz +``` + +This requires Docker and uses the Trident packaging Dockerfile to produce +RPMs from your compiled binary and osmodifier. + +### 5. Generate SSH Keys + +```bash +make artifacts/id_rsa +``` + +### 6. Download Base Image + +```bash +# Downloads baremetal.vhdx from MCR +./tests/images/testimages.py download-image baremetal +``` + +### 7. Build COSI Images + +Build the test COSI images that Trident will install and update. A/B updates +require two images with unique filesystem UUIDs — Trident rejects updates where +the new image matches the installed one. Use `--clones 2` to produce two images, +then rename them into `artifacts/test-image/`: + +```bash +mkdir -p artifacts/test-image + +# Build two clones (produces trident-testimage_0.cosi and trident-testimage_1.cosi) +sudo ./tests/images/testimages.py build trident-testimage \ + --output-dir ./artifacts/test-image --clones 2 + +# Rename clones to the filenames referenced by Host Configurations +mv artifacts/test-image/trident-testimage_0.cosi artifacts/test-image/regular.cosi +mv artifacts/test-image/trident-testimage_1.cosi artifacts/test-image/regular_v2.cosi +``` + +Repeat for other image types as needed: + +```bash +# Verity image (for root-verity configuration) +sudo ./tests/images/testimages.py build trident-verity-testimage \ + --output-dir ./artifacts/test-image --clones 2 +mv artifacts/test-image/trident-verity-testimage_0.cosi artifacts/test-image/verity.cosi +mv artifacts/test-image/trident-verity-testimage_1.cosi artifacts/test-image/verity_v2.cosi + +# UKI/usr-verity image (for usr-verity, combined configurations) +sudo ./tests/images/testimages.py build trident-usrverity-testimage \ + --output-dir ./artifacts/test-image --clones 2 +mv artifacts/test-image/trident-usrverity-testimage_0.cosi artifacts/test-image/usrverity.cosi +mv artifacts/test-image/trident-usrverity-testimage_1.cosi artifacts/test-image/usrverity_v2.cosi +``` + +The images use the Image Customizer container from +`mcr.microsoft.com/azurelinux/imagecustomizer:latest`. + +### 8. Extract Image Customizer + +The MOS ISO build uses the Image Customizer binary directly (not the +container). Extract it from the container image: + +```bash +mkdir -p artifacts +id=$(docker create mcr.microsoft.com/azurelinux/imagecustomizer:latest) +docker cp "$id:/usr/bin/imagecustomizer" artifacts/imagecustomizer +docker rm "$id" +chmod +x artifacts/imagecustomizer +``` + +### 9. Build the Installer ISO + +The management OS ISO is used by `netlaunch` to boot the VM and run Trident: + +```bash +make bin/trident-mos.iso +``` + +## Running a Clean Install + +### 1. Create the QEMU VM + +Use `virtdeploy` to create a VM with empty disks: + +```bash +sudo bin/virtdeploy create-one --disks 32,8 +``` + +This creates a QEMU/libvirt VM and writes `tools/vm-netlaunch.yaml` with the +VM UUID. The `--disks` flag specifies disk sizes in GB (here, 32 GB for the OS +disk and 8 GB for a secondary disk). + +### 2. Create a Host Configuration + +Use the starter configuration as a template: + +```bash +make starter-configuration +``` + +This copies `tests/e2e_tests/trident_configurations/simple/trident-config.yaml` +to `input/trident.yaml`. Edit it to add your SSH public key under +`os.users[0].sshPublicKeys`. + +Alternatively, copy a configuration directly from any test configuration +directory: + +```bash +mkdir -p input +cp tests/e2e_tests/trident_configurations//trident-config.yaml input/trident.yaml +``` + +The Host Configuration references image URLs like +`http://NETLAUNCH_HOST_ADDRESS/files/regular.cosi`. Netlaunch automatically +replaces `NETLAUNCH_HOST_ADDRESS` with its own `:` at runtime, +so the VM can reach the files served by netlaunch. + +### 3. Run netlaunch + +```bash +NETLAUNCH_PORT=4000 make run-netlaunch +``` + +:::important +Set `NETLAUNCH_PORT` to a fixed port (e.g., `4000`). A/B updates use +`netlisten` to serve update images, and it must listen on the **same port** as +netlaunch — the host configuration on the VM retains the original +`:` URL from the install. +::: + +Netlaunch will: + +- Validate the Host Configuration +- Symlink `tools/vm-netlaunch.yaml` to `input/netlaunch.yaml` +- Boot the QEMU VM from the installer ISO +- Serve `artifacts/test-image/` over HTTP (including COSI images) +- Patch the Host Configuration with the server address +- Stream Trident's install logs to the terminal + +Netlaunch blocks until the install completes and the VM reboots into the +installed OS. Do not stop it until you see the VM reboot. The VM IP address +is printed in the output (typically `192.168.242.2` for the default +`192.168.242.0/24` network). + +To use a custom ISO or config: + +```bash +NETLAUNCH_PORT=4000 NETLAUNCH_ISO=path/to/custom.iso TRIDENT_CONFIG=path/to/config.yaml make run-netlaunch +``` + +### 4. Watch the VM console (optional) + +In a separate terminal: + +```bash +make watch-virtdeploy +``` + +## Running E2E Validation (pytest) + +After an install or update, validate the host state with the pytest suite. +Run from the `tests/e2e_tests` directory: + +### After Clean Install + +```bash +cd tests/e2e_tests +python3 -u -m pytest -m daily --capture=no \ + -H \ + -R host \ + -C trident_configurations/ \ + -K ../../artifacts/id_rsa \ + -v +``` + +### After A/B Update + +The `-A` flag specifies which volume is active after the update. Volumes +alternate with each update: a clean install boots `volume-a`, the first A/B +update switches to `volume-b`, the next back to `volume-a`, and so on. + +```bash +cd tests/e2e_tests +python3 -u -m pytest -m daily --capture=no \ + -H \ + -R host \ + -C trident_configurations/ \ + -K ../../artifacts/id_rsa \ + -A \ + -v +``` + +Where `` is `volume-b` after the first update, `volume-a` after +the second, and so on. + +### Flags + +| Flag | Long Form | Description | Default | +|------|-----------|-------------|---------| +| `-m daily` | | Pytest marker filter — selects the `daily` test ring | (required) | +| `--capture=no` | | Disables output capture (avoids conflicts with fabric SSH) | `fd` | +| `-H` | `--host` | IP address or hostname of the target VM | (required) | +| `-R` | `--runtime-env` | Runtime environment: `host` or `container` | `host` | +| `-C` | `--configuration` | Path to configuration directory | (required) | +| `-K` | `--keypath` | Path to SSH private key | `tests/e2e_tests/helpers/key` | +| `-A` | `--ab-active-volume` | Active A/B volume: `volume-a` or `volume-b` | `volume-a` | +| `-S` | `--expected-host-status-state` | Expected Trident servicing state | `provisioned` | +| `-v` | `--verbose` | Verbose test output | off | + +:::note +The default key path (`tests/e2e_tests/helpers/key`) is not checked into the +repo. Always pass `-K` explicitly with your key path, or create a symlink at +the default location. +::: + +## Running an A/B Update + +After a successful clean install, perform an A/B update using `netlisten` to +serve the update image and `storm-trident` to orchestrate the update. + +### 1. Start netlisten + +In a separate terminal, start `netlisten` to serve the update images. It +**must** use the same port that netlaunch used during install: + +```bash +sudo bin/netlisten \ + -s artifacts/test-image \ + -p 4000 \ + -m trident-ab-update-metrics.jsonl \ + -b logstream-ab-update.log +``` + +### 2. Run storm-trident A/B update + +```bash +sudo bin/storm-trident helper ab-update -- \ + artifacts/id_rsa \ + \ + testing-user \ + host \ + -c /var/lib/trident/config.yaml \ + -v 2 \ + -s -f +``` + +The `ab-update` helper takes four positional arguments followed by flags: + +| Argument | Description | +|----------|-------------| +| `` | Path to the SSH private key | +| `` | IP address of the VM | +| `` | SSH user on the VM | +| `` | `host` or `container` | + +| Flag | Description | Default | +|------|-------------|---------| +| `-c` | Path to the Host Configuration file **on the VM** | (required) | +| `-v` | Version number for the update image (appended as `_v` suffix) | (required) | +| `-s` | Stage the A/B update | `false` | +| `-f` | Finalize the A/B update (triggers reboot) | `false` | +| `-p` | SSH port | `22` | +| `-t` | SSH connection timeout in seconds | `600` | + +**Success looks like:** + +``` +=== SUMMARY of storm-trident::helper::ab-update === + get-config...........: PASS + update-hc............: PASS + trigger-update.......: PASS + check-trident-service: PASS + check-diagnostics....: PASS +=== RESULT === +OK: passed: 5; total: 5 +``` + +After the update succeeds, run the [post-update pytest validation](#after-ab-update) +with `-A volume-b`. + +### Other storm-trident helpers + +```bash +bin/storm-trident helper manual-rollback -- ... +bin/storm-trident helper check-selinux -- ... +bin/storm-trident helper display-logs -- ... +bin/storm-trident helper rebuild-raid -- ... +``` + +Run `bin/storm-trident helper -- --help` to see available flags. + +## Cleanup + +Destroy the QEMU VM and network: + +```bash +sudo bin/virtdeploy clean +``` + +## Test Configurations + +Test configurations live in `tests/e2e_tests/trident_configurations/`. Each +subdirectory defines a complete test scenario — what to install and which tests +to run against it. + +### Directory Structure + +``` +tests/e2e_tests/trident_configurations/ +├── base/ +│ ├── trident-config.yaml # Host Configuration for Trident +│ └── test-selection.yaml # Which tests to run +├── combined/ +├── encrypted-partition/ +├── misc/ +├── simple/ +├── usr-verity/ +└── ... +``` + +Each configuration directory contains two files: + +- **`trident-config.yaml`**: The Host Configuration that Trident uses to + provision the system. Defines disk layout, partitions, filesystems, A/B update + volume pairs, users, SELinux mode, kernel parameters, and other OS settings. + Image URLs use the placeholder `NETLAUNCH_HOST_ADDRESS`, which netlaunch + replaces at runtime. +- **`test-selection.yaml`**: Declares which pytest test categories are + compatible with this configuration, with optional per-ring overrides to + add or remove tests at different pipeline stages. + +### Test Selection + +The `test-selection.yaml` file controls which tests run for a given +configuration. Each test file declares a pytest marker (e.g., `base`, +`encryption`, `verity`) via `pytestmark`, and the test selection's `compatible` +list references these markers. + +**Available test markers:** + +| Marker | Test File | What It Validates | +|--------|-----------|-------------------| +| `base` | `base_test.py` | Connection, partitions, users, UEFI boot entries | +| `encryption` | `encryption_test.py` | LUKS encrypted partitions and swap | +| `verity` | `verity_test.py` | dm-verity root or `/usr` integrity | +| `extensions` | `extensions_test.py` | System extension (sysext/confext) servicing | +| `rollback` | `rollback_test.py` | Health-check triggered rollback | +| `ab_update_staged` | `ab_update_staged_test.py` | Staged A/B update state | + +**Basic test selection** — list the compatible markers: + +```yaml +# tests/e2e_tests/trident_configurations/misc/test-selection.yaml +compatible: + - base +``` + +This selects all tests marked `@pytest.mark.base` (from `base_test.py`). + +**Multi-marker selection** — configurations that exercise multiple features +list all applicable markers: + +```yaml +# tests/e2e_tests/trident_configurations/combined/test-selection.yaml +compatible: + - base + - usr_verity + - encryption + - uki +``` + +**Per-ring overrides** — test selections can be refined for each pipeline ring. +Rings are cumulative (each inherits from the previous), so overrides only need +to specify differences: + +```yaml +compatible: + - marker1 + - marker2 + - marker3 +weekly: + remove: + - file2.py::function_name1 # Remove a specific test function +daily: + remove: + - marker3 # Remove an entire marker category +post_merge: + remove: + - marker2 + add: + - file2.py::function_name1 # Add back a specific test function +pullrequest: + remove: + - file2.py::function_name1 +``` + +Overrides support both marker names (affecting all tests with that marker) and +specific test functions using `file.py::function_name` syntax. + +### How Test Selection Works + +The `conftest.py` `pytest_collection_modifyitems` hook processes +`test-selection.yaml` at collection time: + +1. All markers listed in `compatible` are added to matching tests. +2. Ring-level overrides (`weekly`, `daily`, `post_merge`, `pullrequest`, + `validation`) are applied cumulatively — each ring inherits the test set from + the previous ring, then applies its own `add`/`remove` operations. +3. When pytest runs with `-m daily`, only tests that received the `daily` marker + through this process are selected. + +### Pipeline Scheduling + +The file `tests/e2e_tests/target-configurations.yaml` maps each configuration +to the hardware types, runtimes, and pipeline frequencies where it should run: + +```yaml +virtualMachine: + host: + pullrequest: # Runs on every PR + - base + - misc + - simple + - ... + post_merge: # Runs after merge to main + - base + - misc + - ... + daily: # Runs nightly + - base + - misc + - ... +bareMetal: + host: + daily: + - base + - ... +``` + +The `invert.py` script in `tools/storm/e2e/` transforms this file into +`configurations/configurations.yaml`, which storm-trident embeds at build time +for scenario discovery. + +### Configuration Summary + +| Configuration | Image | Key Features | +|--------------|-------|-------------| +| `base` | `regular.cosi` | Standard grub2 install, baseline validation | +| `simple` | `regular.cosi` | Minimal single-root partition layout | +| `misc` | `regular.cosi` | NTFS partition, kernel modules, extra services, kernel command line | +| `split` | `regular.cosi` | Separate `/boot` partition | +| `encrypted-partition` | `regular.cosi` | LUKS-encrypted root partition | +| `encrypted-raid` | `regular.cosi` | LUKS encryption with RAID | +| `encrypted-swap` | `regular.cosi` | LUKS-encrypted swap partition | +| `raid-small` | `regular.cosi` | RAID-1 mirrored root (small disks) | +| `raid-mirrored` | `regular.cosi` | RAID-1 mirrored root | +| `raid-big` | `regular.cosi` | RAID with large disks | +| `raid-resync-small` | `regular.cosi` | RAID resync behavior | +| `extensions` | `regular.cosi` | System extensions (sysext/confext) | +| `health-checks-install` | `regular.cosi` | Health-check and rollback on install | +| `root-verity` | `verity.cosi` | dm-verity protected root filesystem | +| `usr-verity` | `usrverity.cosi` | dm-verity `/usr`, UKI, systemd-boot | +| `combined` | `usrverity.cosi` | UKI + encryption + verity combined | +| `memory-constraint-combined` | `usrverity.cosi` | Combined features under memory pressure | +| `rerun` | `usrverity.cosi` | Re-run idempotency validation | +| `usr-verity-raid` | `usrverity.cosi` | UKI + verity with RAID | + +## Automated E2E Scenarios + +For automated multi-step scenarios (install → update → validate → rollback), +use storm-trident's scenario mode. All E2E scenarios use the Storm framework and +share the same underlying code. + +### Listing Scenarios + +```bash +# List all E2E scenarios +bin/storm-trident list scenarios -t e2e + +# Filter by hardware and runtime +bin/storm-trident list scenarios -t e2e -t vm # VM scenarios only +bin/storm-trident list scenarios -t e2e -t container # Container runtime only +``` + +### Scenario Naming + +All E2E scenarios follow the naming convention `_-`: + +- ``: Name of the host config (e.g., `base`, `simple`, `usrverity`) +- ``: `vm` (virtual machine) or `bm` (bare metal) +- ``: `host` (runs directly on the host) or `container` (runs inside a container) + +For example: `base_vm-host`, `combined_vm-container`. + +### Running a Scenario + +```bash +bin/storm-trident run -- +``` + +To see available parameters: + +```bash +bin/storm-trident run -- --help +``` + +Common parameters: + +| Flag | Description | +|------|-------------| +| `--iso` | Path to the installer ISO | +| `-i, --test-image-dir` | Directory containing test COSI images (default: `./artifacts/test-image`) | +| `--logstream-file` | File to write logstream to (default: `logstream-full.log`) | +| `--tracestream-file` | File to write tracestream to | +| `--signing-cert` | Path to certificate for VM EFI variables | +| `--dump-ssh-key` | Dump SSH private key to a file for debugging | +| `--vm-wait-for-login-timeout` | Timeout for VM login prompt | +| `--test-ring` | Test ring to filter test cases | + +### Test Rings + +E2E scenarios are organized into test rings that control how frequently they run. +The valid `--test-ring` values are: + +- **pr-e2e**: Run on every pull request (innermost ring) +- **ci**: Run after merge to main (post-merge) +- **pre**: Run during pre-release validation +- **full-validation**: Run for release validation (outermost ring) + +Rings are cumulative — all scenarios in inner rings also run when an outer ring +is executed. + +:::note +The `tests/e2e_tests/target-configurations.yaml` file uses pipeline-frequency +labels (`pullrequest`, `post_merge`, `daily`, `weekly`) which `invert.py` maps +to the ring constants above: `pullrequest` → `pr-e2e`, `post_merge` → `ci`, +`daily`/`weekly` → `full-validation`. +::: + +### How E2E Discovery Works + +E2E scenario discovery automatically finds all configured Host Configurations +and determines when each should run. The key components: + +- **Configuration definitions**: All Host Configurations live in + `tests/e2e_tests/trident_configurations/`, and the mapping of which + configurations run in which test rings is defined in + `tests/e2e_tests/target-configurations.yaml`. +- **Discovery function**: `DiscoverTridentScenarios` in + `tools/storm/e2e/discover.go` produces instances of `TridentE2EScenario` + (from `tools/storm/e2e/scenario/trident.go`) for each valid combination of + Host Configuration, hardware type, and runtime. +- **Go embed**: Discovery uses Go's `go:generate` and `go:embed` directives to + copy configurations into the binary. The `invert.py` script in + `tools/storm/e2e/` produces `configurations/configurations.yaml` with the + structure: + + ```yaml + : + : + : + ``` + +- **Special config parameters**: Configurations can be customized with YAML keys + defined in `TridentE2EHostConfigParams` (in `tools/storm/e2e/scenario/trident.go`), + such as `maxExpectedFailures` for configs that may have intermittent failures. + +### Matrix Generation in Pipelines + +The `e2e-matrix` script generates ADO pipeline job matrices from discovered +scenarios: + +```bash +bin/storm-trident script e2e-matrix pr-e2e +``` + +Variable names follow the pattern `TEST_MATRIX_E2E__` (e.g., +`TEST_MATRIX_E2E_VM_HOST`). + +### E2E Test Code + +All E2E test logic lives under `tools/storm/e2e/scenario/`. The main entry point +is `trident.go`, which contains the `TridentE2EScenario` struct implementing the +Storm Scenario interface. diff --git a/docs/Development/Building/Functional-Tests.md b/docs/Development/Testing/Functional-Tests.md similarity index 74% rename from docs/Development/Building/Functional-Tests.md rename to docs/Development/Testing/Functional-Tests.md index 44cb5cde8..f024155e9 100644 --- a/docs/Development/Building/Functional-Tests.md +++ b/docs/Development/Testing/Functional-Tests.md @@ -21,7 +21,90 @@ They are used to test: - Operations that manipulate disks, RAID arrays, mounts, filesystems, etc. - Operations that modify the OS in any way. -This document aims to explain the architecture of how functional tests work, and +## Prerequisites + +Functional tests run inside a libvirt/QEMU virtual machine. You need: + +- **Linux host** with root access (functional tests manipulate disks, mounts, + etc.) +- **libvirt and QEMU** installed and configured +- **Docker** (to run Image Customizer for building the test VM image) +- **[oras](https://oras.land/)** CLI (to download base images from MCR) +- **Go 1.24+** (to build `virtdeploy`) +- **Python 3.8+** with test packages: + + ```bash + pip3 install -r tests/functional_tests/requirements.txt + ``` + +## Building the Test VM Image + +The functional test VM image is an Azure Linux 3 QCOW2 image built with +[Image Customizer](https://github.com/microsoft/azure-linux-image-tools). The +build uses a container from MCR (`mcr.microsoft.com/azurelinux/imagecustomizer:latest`) +and a base image also from MCR. + +1. **Download the base image:** + + ```bash + # Downloads baremetal.vhdx from mcr.microsoft.com/azurelinux/3.0/image/baremetal:latest + ./tests/images/testimages.py download-image baremetal + ``` + +2. **Build the functional test image:** + + ```bash + sudo ./tests/images/testimages.py build trident-functest --output-dir ./artifacts + ``` + + This produces `artifacts/trident-functest.qcow2`. The image configuration is + defined in `tests/images/trident-functest/base/baseimg.yaml`. + +## Building Test Dependencies + +These dependencies are built automatically by `make functional-test`, but you +can build them individually if needed: + +```bash +# Build virtdeploy (VM management tool) +make bin/virtdeploy + +# Build osmodifier +make artifacts/osmodifier + +# Build the functional test binaries with code coverage instrumentation +make build-functional-test-cc + +# Generate the test manifest (ft.json) +make generate-functional-test-manifest +``` + +## Running the Tests + +Run the full functional test suite: + +```bash +make functional-test +``` + +This will create a VM using `virtdeploy`, upload the test binaries, and run all +tests via pytest. + +To rerun tests on an already-running VM (faster iteration): + +```bash +make patch-functional-test +``` + +To run a subset of tests, use the `FILTER` variable: + +```bash +make functional-test FILTER="custom/test_trident_e2e.py -k test_name" +``` + +## Architecture + +This section explains the architecture of how functional tests work and how they are implemented. ## Native Python Tests diff --git a/docs/Development/Testing/Rollback-Tests.md b/docs/Development/Testing/Rollback-Tests.md new file mode 100644 index 000000000..4ca55f479 --- /dev/null +++ b/docs/Development/Testing/Rollback-Tests.md @@ -0,0 +1,203 @@ +--- +sidebar_position: 8 +--- + +# Rollback Tests + +Rollback tests validate manual rollback and runtime update workflows using +`storm-trident run rollback`. Like [servicing tests](Servicing-Tests.md), +they start from a pre-built VM image defined in +`tests/images/trident-vm-testimage/`, but focus specifically on the rollback +chain: A/B updates, runtime updates (sysexts and netplan), and rolling each +back in sequence. + +## VM Image Types + +The rollback scenario is typically run with verity-enabled images that support +extensions: + +| Image | Bootloader | Integrity | UKI | Extensions | `--uki` flag | +|-------|-----------|-----------|-----|------------|--------------| +| `trident-vm-usr-verity-testimage` | systemd-boot | `/usr` verity | Yes | Sysexts supported | Required | +| `trident-vm-grub-verity-testimage` | grub2 | Root verity | No | Not supported | Not needed | + +All image configs live under `tests/images/trident-vm-testimage/base/`. The +base image is `qemu_guest` (see +[Servicing Tests — Create the qemu\_guest Base Image](Servicing-Tests.md#4-create-the-qemu_guest-base-image) +for how to create it from the publicly available `baremetal` image on MCR). + +**`trident-vm-usr-verity-testimage`** (recommended) uses systemd-boot with a +Unified Kernel Image (UKI) and `/usr` dm-verity. This is the only image type +that supports full extension testing. When using this image, you **must** pass +`--uki` to `storm-trident run rollback` so that the `prepare-qcow2` step +generates the correct Image Customizer config with `os.uki.mode: passthrough`. + +**`trident-vm-grub-verity-testimage`** uses grub2 with root dm-verity. It does +not support extensions or runtime updates, so you must pass +`--skip-extension-testing`, `--skip-runtime-updates`, and +`--skip-netplan-runtime-testing` when using this image type. + +## What It Tests + +The full rollback scenario (`trident-vm-usr-verity-testimage`) exercises a +multi-step update-and-rollback sequence: + +1. Start a VM with sysext extension v1 +2. Verify extension is active, active volume is A +3. Run an A/B update with sysext extension v2 and new netplan +4. Verify extension is v2, netplan is correct, active volume is B +5. Run a runtime update with sysext extension v3 and new netplan +6. Verify extension is v3, active volume is still B +7. Run a runtime update removing sysext and netplan +8. Verify extension and netplan are gone +9. Roll back runtime update #2 → verify v3 is restored +10. Roll back runtime update #1 → verify v2 is restored +11. Roll back A/B update → verify v1, active volume is A + +When using `trident-vm-grub-verity-testimage`, steps 1–2 and 5–10 are skipped +(no extensions, no runtime updates, no runtime rollbacks). The test exercises +only A/B update and A/B rollback: + +1. Start a VM (no extensions) +2. Run an A/B update +3. Verify active volume is B +4. Roll back A/B update → verify active volume is A + +## Prerequisites + +- **Linux host** with root access +- **libvirt and QEMU** installed and configured +- **Docker** (for building images with Image Customizer) +- **[oras](https://oras.land/)** CLI (for downloading base images from MCR) +- **Go 1.24+** (for building Go tools) +- **Rust** (latest stable, for building Trident) + +See [Dependencies](../Building/Dependencies.md) for full build dependency details. + +## Building Dependencies + +### 1. Build Trident and RPMs + +```bash +make build +make bin/trident-rpms.tar.gz +``` + +### 2. Build Go Tools + +```bash +make bin/storm-trident +make bin/netlisten +``` + +### 3. Generate SSH Keys + +```bash +make artifacts/id_rsa +``` + +### 4. Create the qemu\_guest Base Image + +Follow the instructions in +[Servicing Tests — Create the qemu\_guest Base Image](Servicing-Tests.md#4-create-the-qemu_guest-base-image) +to download the `baremetal` image from MCR and convert it to `qemu_guest.vhdx` +using Image Customizer. + +### 5. Build Extension Images + +The rollback chain tests three versions of sysext extensions: + +```bash +mkdir -p artifacts +pushd ./artifacts +../bin/storm-trident script build-extension-images --build-sysexts --num-clones 3 +popd +``` + +### 6. Build the VM Image + +Choose an image type and build both COSI and QCOW2: + +```bash +# For UKI with usr verity (recommended — full extension testing): +TEST_IMAGE_NAME="trident-vm-usr-verity-testimage" + +# Alternative: grub with root verity (no extension testing): +# TEST_IMAGE_NAME="trident-vm-grub-verity-testimage" + +# Clean any previous test images +sudo rm -f artifacts/trident-vm-*-testimage.qcow2 artifacts/trident-vm-*-testimage.cosi + +# Build the COSI and QCOW2 +sudo ./tests/images/testimages.py build $TEST_IMAGE_NAME --output-dir ./artifacts +make artifacts/$TEST_IMAGE_NAME.qcow2 +``` + +## Running the Rollback Scenario + +The rollback scenario requires root access for VM creation via `virt-install`: + +```bash +# For UKI images (trident-vm-usr-verity-testimage): +sudo ./bin/storm-trident run rollback -w --verbose \ + --artifacts-dir ./artifacts/ \ + --output-path /tmp/rollback-output \ + --platform qemu \ + --ssh-private-key-path ./artifacts/id_rsa \ + --uki +``` + +```bash +# For grub-verity images (trident-vm-grub-verity-testimage): +sudo ./bin/storm-trident run rollback -w --verbose \ + --artifacts-dir ./artifacts/ \ + --output-path /tmp/rollback-output \ + --platform qemu \ + --ssh-private-key-path ./artifacts/id_rsa \ + --skip-extension-testing \ + --skip-runtime-updates \ + --skip-netplan-runtime-testing +``` + +:::note +The `--uki` flag is required for UKI-based images (e.g., +`trident-vm-usr-verity-testimage`). Without it, the `prepare-qcow2` step will +fail because Image Customizer requires explicit UKI handling (`os.uki.mode`) +when the base image contains Unified Kernel Images. Do not use `--uki` with +grub-based images. +::: + +### Flags + +| Flag | Description | Default | +|------|-------------|---------| +| `--artifacts-dir` | Directory containing VM images, COSI, and extensions | `/tmp` | +| `--output-path` | Output directory for logs | `./output` | +| `--platform` | `qemu` or `azure` | `qemu` | +| `--ssh-private-key-path` | Path to SSH private key | `~/.ssh/id_rsa` | +| `--uki` | Image uses UKI (adds `os.uki.mode: passthrough` to IC config) | `false` | +| `--verbose` | Enable verbose logging | `false` | +| `--skip-runtime-updates` | Skip runtime update testing | `false` | +| `--skip-manual-rollbacks` | Skip manual rollback testing | `false` | +| `--skip-extension-testing` | Skip extension (sysext) testing | `false` | +| `--skip-netplan-runtime-testing` | Skip netplan runtime update testing | `false` | +| `--force-cleanup` | Force VM cleanup on exit | `false` | +| `--test-case-to-run` | Run a specific test case only | `all` | + +### Test Cases + +The rollback scenario runs these test cases in order: + +1. **prepare-qcow2** — Modifies the QCOW2 using Image Customizer to inject the + v1 sysext extension and enable `systemd-sysext`. For UKI images, adds + `os.uki.mode: passthrough` to preserve existing UKIs. +2. **deploy-vm** — Creates and boots a QEMU VM from the prepared QCOW2 +3. **check-deployment** — Verifies the VM booted successfully and is accessible + via SSH +4. **multi-rollback** — Runs the full update → runtime update → rollback chain + described in [What It Tests](#what-it-tests) +5. **skip-to-ab-rollback** — Tests skipping runtime rollbacks and going directly + to A/B rollback +6. **split-rollback** — Tests split-phase (stage then finalize) rollback +7. **collect-logs** — Fetches Trident logs from the VM via SSH +8. **cleanup-vm** — Destroys the QEMU VM diff --git a/docs/Development/Testing/Servicing-Tests.md b/docs/Development/Testing/Servicing-Tests.md new file mode 100644 index 000000000..af1eb33b9 --- /dev/null +++ b/docs/Development/Testing/Servicing-Tests.md @@ -0,0 +1,196 @@ +--- +sidebar_position: 7 +--- + +# Servicing Tests + +Servicing tests validate multi-update workflows on pre-built VM images using +`storm-trident run servicing`. Unlike [E2E tests](E2E-Tests.md) which use +`netlaunch` and an installer ISO for initial provisioning, servicing tests start +from a QCOW2 image that already has Trident and an OS installed, then run +repeated A/B updates with optional rollback testing. + +The VM images are defined in `tests/images/trident-vm-testimage/` and built +with Image Customizer from a `qemu_guest` base image downloaded from MCR. + +## VM Image Types + +The servicing scenario expects a QCOW2 image matching the pattern +`trident-vm-*-testimage.qcow2` in the artifacts directory. The pipeline-tested +image types are: + +| Image | Bootloader | Integrity | UKI | Config File | +|-------|-----------|-----------|-----|-------------| +| `trident-vm-grub-verity-testimage` | grub2 | Root verity | No | `updateimg-grub-verity.yaml` | +| `trident-vm-usr-verity-testimage` | systemd-boot | `/usr` verity | Yes | `baseimg-usr-verity.yaml` | + +All image configs live under `tests/images/trident-vm-testimage/base/`. The +base image is `qemu_guest` (see [step 4](#4-download-the-qemu_guest-base-image) +for how to obtain it). + +**`trident-vm-grub-verity-testimage`** uses grub2 with root dm-verity. The root +filesystem is read-only and integrity-protected, with `/var` on a separate +partition and an `/etc` overlay service for runtime state. It uses the +`updateimg-grub-verity.yaml` config which includes SSH access, network +configuration, and sudoers for the test user. + +**`trident-vm-usr-verity-testimage`** uses systemd-boot with a Unified Kernel +Image (UKI) and `/usr` dm-verity. This is a preview feature +(`previewFeatures: uki`) that requires `ukify` on the build host. It uses the +`baseimg-usr-verity.yaml` config which defines the full runtime layout. + +### COSI Update Images + +During the update loop, the servicing scenario serves COSI files over HTTP +using `netlisten`. It expects COSI files in two directories within the +artifacts dir: + +- `/update-a/` — COSI image served on port 8000 (configurable + via `--update-port-a`) +- `/update-b/` — COSI image served on port 8001 (configurable + via `--update-port-b`) + +The update loop alternates between these two images across iterations. + +## Prerequisites + +- **Linux host** with root access +- **libvirt and QEMU** installed and configured +- **Docker** (for building images with Image Customizer) +- **[oras](https://oras.land/)** CLI (for downloading base images from MCR) +- **Go 1.24+** (for building Go tools) +- **Rust** (latest stable, for building Trident) + +See [Dependencies](../Building/Dependencies.md) for full build dependency details. + +## Building Dependencies + +### 1. Build Trident and RPMs + +```bash +make build +make bin/trident-rpms.tar.gz +``` + +### 2. Build Go Tools + +```bash +make bin/storm-trident +make bin/netlisten +``` + +### 3. Generate SSH Keys + +```bash +make artifacts/id_rsa +``` + +### 4. Download the qemu\_guest Base Image + +The VM test images are built from a `qemu_guest` base image. Download the +`minimal-os` image from MCR and rename it to `qemu_guest.vhdx`: + +```bash +mkdir -p artifacts +oras pull mcr.microsoft.com/azurelinux/3.0/image/minimal-os:latest \ + --output ./artifacts --platform linux/amd64 +mv artifacts/image.vhdx artifacts/qemu_guest.vhdx +``` + +The `minimal-os` image is a publicly available Azure Linux 3.0 base image that +works as a drop-in `qemu_guest` replacement. Image Customizer installs all +required packages (including VirtIO drivers and SSH) during the VM image build +step, so the base image's pre-installed package set does not matter. + +:::tip Internal shortcut +If you have access to the internal Azure DevOps artifacts feed, the Makefile +downloads `qemu_guest` automatically (requires `az login`). You can also +download it directly: `./tests/images/testimages.py download-image qemu_guest` +::: + +### 5. Build the VM Image + +Choose an image type and build the QCOW2: + +```bash +# For grub with root verity: +make artifacts/trident-vm-grub-verity-testimage.qcow2 + +# For UKI with usr verity: +make artifacts/trident-vm-usr-verity-testimage.qcow2 +``` + +### 6. Prepare Update Images + +The update loop needs two distinct COSI images — Trident rejects updates where +the new image has the same filesystem UUIDs as the currently installed one. +Use `--clones 2` to build two images with unique UUIDs, then rename them to +the same filename in each directory (the update loop picks the filename from +`update-a` and applies it to both update configs): + +```bash +mkdir -p artifacts/update-a artifacts/update-b + +# Build two clones of the COSI image (produces _0 and _1 suffixed files) +sudo ./tests/images/testimages.py build trident-vm-grub-verity-testimage \ + --output-dir ./artifacts --clones 2 + +# Move the clones into the update directories with the same filename +mv artifacts/trident-vm-grub-verity-testimage_0.cosi \ + artifacts/update-a/trident-vm-grub-verity-testimage.cosi +mv artifacts/trident-vm-grub-verity-testimage_1.cosi \ + artifacts/update-b/trident-vm-grub-verity-testimage.cosi +``` + +:::note +Both the QCOW2 and COSI images must be built from the same base image. Trident +validates that the COSI's `VARIANT_ID` in `/etc/os-release` matches the host's. +::: + +## Running the Servicing Scenario + +The servicing scenario requires root access for VM creation via `virt-install`: + +```bash +sudo bin/storm-trident run servicing -- \ + --artifacts-dir ./artifacts \ + --output-path /tmp/servicing-output \ + --platform qemu \ + --ssh-private-key-path ./artifacts/id_rsa \ + --verbose +``` + +### Test Cases + +The servicing scenario runs these test cases in order: + +1. **publish-sig-image** — Publishes the image to Azure SIG (skipped for QEMU) +2. **deploy-vm** — Finds a `trident-vm-*-testimage.qcow2` in artifacts, copies + it to `booted.qcow2`, and creates a QEMU VM +3. **check-deployment** — Verifies the VM booted and is accessible via SSH +4. **update-loop** — Runs repeated A/B updates: starts `netlisten` servers on + ports 8000/8001 to serve COSI images, SSHes into the VM, edits the Host + Configuration, and triggers `trident grpc-client update` with stage then + finalize operations +5. **rollback** — Tests rollback after update (only when `--rollback` is enabled) +6. **collect-logs** — Fetches Trident logs from the VM via SSH +7. **cleanup-vm** — Destroys the QEMU VM + +### Flags + +| Flag | Description | Default | +|------|-------------|---------| +| `--artifacts-dir` | Directory containing VM images and COSI files | `/tmp` | +| `--output-path` | Output directory for logs | `./output` | +| `--platform` | `qemu` or `azure` | `qemu` | +| `--ssh-private-key-path` | Path to SSH private key | `~/.ssh/id_rsa` | +| `--user` | SSH user on the VM | `testuser` | +| `--rollback` | Enable rollback testing | `false` | +| `--retry-count` | Number of update retry attempts | `3` | +| `--update-port-a` | Port for first update server | `8000` | +| `--update-port-b` | Port for second update server | `8001` | +| `--verbose` | Enable verbose logging | `false` | +| `--force-cleanup` | Force VM cleanup on exit | `false` | +| `--test-case-to-run` | Run a specific test case only | `all` | +| `--expected-volume` | Expected active volume after update | `volume-a` | +| `--rollback-retry-count` | Number of rollback retry attempts | `3` | diff --git a/docs/Development/Testing/Testing.md b/docs/Development/Testing/Testing.md new file mode 100644 index 000000000..9812dd4b7 --- /dev/null +++ b/docs/Development/Testing/Testing.md @@ -0,0 +1,61 @@ +--- +sidebar_position: 3 +--- + +# Testing Trident + +## Code Checks + +To ensure code quality and consistency, run `make check`. This verifies +formatting (`cargo fmt --check`), runs `cargo check` with all features, and +then runs `clippy` with `-D warnings`: + +```bash +make check +``` + +## Unit Testing + +To run Trident's unit tests, you can use the following command: + +```bash +cargo test --all +``` + +or + +```bash +make test +``` + +## Functional Testing + +Functional tests validate operations that cannot run in isolation (disk +manipulation, RAID arrays, mounts, filesystems, etc.) inside a libvirt/QEMU +virtual machine. + +See [Functional Tests](Functional-Tests.md) for architecture details, +prerequisites, building the test VM image, and running the tests. + +## E2E Testing + +E2E tests validate complete Trident install-and-update workflows using +`netlaunch` to boot a VM from an installer ISO, followed by pytest validation. + +See [E2E Tests](E2E-Tests.md) for full setup and run instructions. + +## Servicing and Rollback Testing + +Servicing and rollback tests use pre-built VM images (defined in +`tests/images/trident-vm-testimage/`) to test multi-update workflows and +manual rollback chains without using `netlaunch` or an installer ISO. + +- [Servicing Tests](Servicing-Tests.md) — multi-update loop with optional + rollback via `storm-trident run servicing` +- [Rollback Tests](Rollback-Tests.md) — full rollback chain (A/B + runtime + updates) via `storm-trident run rollback` + +## Code Coverage + +See [Coverage](Coverage.md) for instructions on generating and viewing code +coverage reports. diff --git a/docs/Development/Testing/_category_.json b/docs/Development/Testing/_category_.json new file mode 100644 index 000000000..5e70d90ab --- /dev/null +++ b/docs/Development/Testing/_category_.json @@ -0,0 +1,8 @@ +{ + "label": "Testing", + "position": 3, + "link": { + "type": "doc", + "id": "Development/Testing/Testing" + } +} diff --git a/tools/storm/e2e/README.md b/tools/storm/e2e/README.md deleted file mode 100644 index 6177b120c..000000000 --- a/tools/storm/e2e/README.md +++ /dev/null @@ -1,214 +0,0 @@ -# Trident E2E Testing with Storm - -- [Trident E2E Testing with Storm](#trident-e2e-testing-with-storm) - - [Running Locally](#running-locally) - - [Build](#build) - - [Query E2E Scenarios](#query-e2e-scenarios) - - [E2E Scenario Naming](#e2e-scenario-naming) - - [Run a Specific Scenario](#run-a-specific-scenario) - - [Help for E2E Scenario](#help-for-e2e-scenario) - - [How This Works](#how-this-works) - - [Test Rings](#test-rings) - - [Discovery](#discovery) - - [Matrix Generation in Pipelines](#matrix-generation-in-pipelines) - - [Pipeline Execution](#pipeline-execution) - - [E2E Test Code](#e2e-test-code) - -## Running Locally - -### Build - -```bash -make bin/storm-trident -``` - -### Query E2E Scenarios - -```bash -./bin/storm-trident list scenarios -t e2e -``` - -You can also filter by other tags for runtime and hardware: - -- `host` / `container` -- `vm` / `bm` - -### E2E Scenario Naming - -All E2E scenarios follow the naming convention: - -``` --- -``` - -Where: - -- `` is the name of the host config used (e.g., `base`, `simple`, `usrverity`). -- `` is either `vm` (virtual machine) or `bm` (bare metal). -- `` is either `host` (runs directly on the host) or `container` (runs inside a container). - -### Run a Specific Scenario - -```bash -./bin/storm-trident run -- -``` - -### Help for E2E Scenario - -```bash -./bin/storm-trident run -- --help -``` - - -All E2E scenarios have the same underlying code, so parameters are consistent across -scenarios. Common parameters include: - -- `-h, --help` Show context-sensitive help. -- `--iso=STRING` Path to the ISO to use for OS installation. -- `--pipeline-run` Indicates whether the scenario is being run in a pipeline - context. This will, among other things, install dependencies. -- `-i, --test-image-dir="./artifacts/test-image"` Directory containing the test - images to use for OS installation. -- `--logstream-file="logstream-full.log"` File to write logstream to. -- `--tracestream-file=STRING` File to write tracestream to. -- `--signing-cert=STRING` Path to certificate file to inject into VM EFI - variables. -- `--dump-ssh-key=STRING` If set, the SSH private key used for VM access will be - dumped to the specified file. - -## How This Works - -### Test Rings - -This E2E testing framework uses the concept of "test rings" to group scenarios by -their intended frequency of execution. - -Inner/lower/fast/earlier rings run more frequently, and outer/higher/slow/later -rings run less frequently. Rings are cumulative, so that all scenarios in inner -rings are also run when an outer ring is run. - -The inner-most ring is `pr-e2e`, which is run on every pull request. The -outer-most ring is `full-validation`, which is run weekly. - -Test rings are formally defined in -[`testrings/testrings.go`](testrings/testrings.go). - -### Discovery - -The first step is discovery of all configured E2E scenarios. In short, this -means looking at all existing Host Configurations and when they are supposed to -run. To ultimate goal is to produce instances of the struct `TridentE2EScenario` -(from [`scenario/trident.go`](scenario/trident.go)) representing each -combination of parameters. - -All test discovery happens in [`discover.go`](discover.go). The key function is -`DiscoverTridentE2EScenarios`, which returns a list of all discovered -`TridentE2EScenario` instances. - -[NOTE: IN DEVELOPMENT: Update once this changes.] While in development, all -configurations are defined in `tests/e2e_tests/trident_configurations/`, and the -configuration for when each is supposed to run is defined in -`tests/e2e_tests/target-configurations.yaml`. To port these over into go, we use -a combination of Go's generate directive, Go's embed package, and some custom -code in [`invert.py`](invert.py): - -```go -//go:generate cp -r ../../../tests/e2e_tests/trident_configurations configurations -//go:generate python3 invert.py -//go:embed configurations/* -var content embed.FS -``` - -The python script includes more thorough documentation on itself, but in short -it will produce the yaml file `configurations/configurations.yaml`, which has -this structure: - -```yaml -: - : - : -``` - -For example: - -```yaml -base: - vm: - host: pr-e2e -``` - -The function `DiscoverTridentE2EScenarios` will go over this data structure and -all the Host Configuration files to produce instances of `TridentE2EScenario`. -Each is configured to contain the Host Configuration, the target hardware type -(`vm`/`bm`), the target runtime (`host`/`container`), and the lowest test ring -it should be run in. If the specific combination is not configured to run in any -ring, the instance is not created. - -The type type `configs` in [`discover.go`](discover.go) contains the expected -structure of the YAML configuration data. - -Some configuration have special behaviors, such as expected failures. For those -special cases, the config can be further customized with the YAML keys defined -in the struct `TridentE2EHostConfigParams` in -[`scenario/trident.go`](scenario/trident.go). These are keys directly under the -configuration name, for example: - -```yaml -base: - maxExpectedFailures: 1 # Expect this config to fail at most once. - vm: - host: pr-e2e -``` - -### Matrix Generation in Pipelines - -The E2E testing framework includes functionality to generate ADO job matrixes -based on the defined E2E scenarios and their test rings. This allows pipelines to -dynamically adjust which scenarios to run based on the desired test ring. - -This is implemented by the `e2e-matrix` script added to `storm-trident` binary, -which is defined in [`matrix_script.go`](matrix_script.go). - -This script takes as input the desired test ring (e.g., `pr-e2e`, -`full-validation`) and will output one ADO job matrix per configuration -combination containing all scenarios that should be run at that ring or lower. -The matrices are directly saved as ADO variables that can be consumed by later -jobs in the pipeline, the contents are also printed to standard output for -debugging purposes. - -The variable names follow the pattern: - -``` -TEST_MATRIX_E2E__ -``` - -Where `` is either `VM` or `BM`, and `` is either `HOST` or -`CONTAINER`. - -Example: - -```text -$ ./bin/storm-trident script e2e-matrix pr-e2e -##vso[task.setvariable variable=TEST_MATRIX_E2E_VM_HOST;isOutput=true]{"base_vm-host":{"scenario":"base_vm-host","hardware":"vm","runtime":"host"}} -INFO[0000] Generated matrix for hardware 'vm' and runtime 'host' with 1 scenarios: - - base_vm-host -``` - -### Pipeline Execution - -To run this in pipelines, we depend on two YAML templates: - -- [`.pipelines/templates/stages/testing_e2e/storm_e2e.yml`](../../../.pipelines/templates/stages/testing_e2e/storm_e2e.yml) - This is the entry point for E2E testing in pipelines. It is responsible for - invoking the matrix generation script and then running the actual test - execution template for each configuration combination. -- [`.pipelines/templates/stages/testing_e2e/test_execution_template.yml`](../../../.pipelines/templates/stages/testing_e2e/test_execution_template.yml) - This template is responsible for running the actual E2E scenarios for a given - configuration combination. It consumes the matrix variable produced by the - previous template and runs each scenario in it. - -### E2E Test Code - -All actual E2E test code lives under [`scenario/`](scenario/). The main entry -point is the file [`trident.go`](scenario/trident.go), which contains the -`TridentE2EScenario` struct which implements the storm Scenario interface. \ No newline at end of file diff --git a/tools/storm/rollback/README.md b/tools/storm/rollback/README.md deleted file mode 100644 index 2b3281817..000000000 --- a/tools/storm/rollback/README.md +++ /dev/null @@ -1,66 +0,0 @@ -# Testing Manual Rollback and Runtime Updates - -This storm scenario can be run locally by following the instructions below. - -The test will do the following: - -* Update the standard QCOW2 to include sysext extension v1 -* Start a VM with the updated QCOW2 -* Verify extension is correct, active volume is A, and rollback info is as expected -* Run an A/B update that includes sysext extension v2 and new netplan -* Verify extension is v2, netplan is correct, active volume is B, and rollback info is as expected -* Run a runtime update that includes sysext extension v3 and new netplan -* Verify extension is v3, netplan is correct, active volume is B, and rollback info is as expected -* Run a runtime update that excludes sysext extension and netplan -* Verify extension and netplan do not exist, active volume is B, and rollback info is as expected -* Run manual rollback (of 2nd runtime update) -* Verify extension is v3, netplan is correct, active volume is B, and rollback info is as expected -* Run manual rollback (of 1st runtime update) -* Verify extension is v2, netplan is correct, active volume is B, and rollback info is as expected -* Run manual rollback (of A/B update) -* Verify extension is v1, active volume is A, and rollback info is as expected - -Test can be configured to skip some testing: - -* `--skip-runtime-updates` - skips testing associated with runtime update -* `--skip-manual-rollbacks` - skips testing associated with manual rollbacks -* `--skip-extension-testing` - skips testing associated with extensions -* `--skip-netplan-runtime-testing` - skips testing associated with netplans - -> Note: there are 2 test images (trident-vm-usr-verity-testimage and trident-vm-grub-verity-testimage) that can be used to run the tests, pick the desired image by setting `TEST_IMAGE_NAME` in the script below. If using trident-vm-grub-verity-testimage, extension testing will be skipped as Image Customizer cannot add an extension to the original QCOW2. - -``` sh -# TEST_IMAGE_NAME="trident-vm-grub-verity-testimage" -TEST_IMAGE_NAME="trident-vm-usr-verity-testimage" - -# Build storm-trident binary -make bin/storm-trident -# Build the test extensions -pushd ./artifacts -../bin/storm-trident script build-extension-images --build-sysexts --num-clones 3 -popd -# Build the Trident rpms -sudo rm bin/trident-rpms.tar.gz -sudo rm -rf bin/RPMS -make bin/trident-rpms.tar.gz -# Ensure that there are no previous test images that the test might pick up -sudo rm artifacts/trident-vm-*-testimage.qcow2 artifacts/trident-vm-*-testimage.cosi -# Build the required test images -make artifacts/$TEST_IMAGE_NAME.cosi -make artifacts/$TEST_IMAGE_NAME.qcow2 - -SKIP_FLAGS="" -if [ "$TEST_IMAGE_NAME" == "trident-vm-grub-verity-testimage" ]; then - # skip extension testing - SKIP_FLAGS="$SKIP_FLAGS --skip-extension-testing" -fi - -sudo ./bin/storm-trident run rollback -w --verbose \ - --artifacts-dir ./artifacts/ \ - --output-path /tmp/output \ - --platform qemu \ - --ssh-private-key-path ./artifacts/id_rsa \ - --ssh-public-key-path ./artifacts/id_rsa.pub \ - $SKIP_FLAGS - - ```