Testing

Testing

Tests are organized by what they need. Most need nothing but Go and Git. Some need a container engine. A few need a specially built binary.

Unit

go test -race ./...

Fast, hermetic, no engine. Covers the Git wrapper, fixtures and pinned ids, snapshots and materialization, the module proxy, profile refusals, validation parsing, gate rules, readiness, policy (as agent-runtime testkit conformance tables and never-assertions over the real tool specs), tools containment and schema fuzzing, the fake GitHub server, publication reconciliation, locks, and the CLI's argument handling.

Integration

go test -tags integration -p 1 ./...

Runs against a real engine: sandbox guarantees, cold-cache inspect, baseline maintain, scripted scenarios, model mode from recordings, publication end to end against the fake API and a bare repository, approvals across processes. -p 1 because the inspect path reaps every labelled container. Temporary directories come from testtmp, rooted under the home directory so a VM-backed engine can see them. The pinned image must be pulled first. The internal/vulnscan and internal/steward integration tests also need the network: they build govulncheck once per test binary through proxy.golang.org and sum.golang.org, and one of them fetches https://vuln.go.dev. Without the network they fail with a message saying so.

Fault injection

go test -tags faultinject ./internal/manifest/ ./internal/proposal/

The build tag compiles fault points into the binary; REPO_STEWARD_FAULT=<point> makes the process exit there. Tests build the binary, run it to a fault point, run recovery, and assert the journal and the workspace agree. Publication interruption tests in the integration job use the same binary.

Replay

Model mode in CI uses recordings with the server pointed at a closed port. A miss fails; a live call cannot happen.

Smoke steps

The CI workflow runs the quick start on fixtures with a built binary and checks exit codes. It must be a built binary: go run reports every non-zero exit as 1.

Smoke against public modules

go test -tags smoke -count=1 -p 1 -v ./internal/smoke/

Clones pinned public repositories and runs the deterministic pipeline through the real module proxy. Six scenarios: five run the plain pipeline, and TestSmokeVulnerable runs vulnerable selection on labstack/echo v4.15.4 against the live database, requiring GO-2026-5970 in golang.org/x/text v0.38.0 (indirect) found at base and cleared by v0.39.0 with nothing introduced; it checks that advisory, not totals, because the live database changes. Needs the network and the pinned images for Go 1.25 and 1.26. Runs in its own workflow on demand and weekly.

Repair on real repositories

REPO_STEWARD_SMOKE_MODEL=ollama:qwen3:30b-a3b REPO_STEWARD_SMOKE_OUT=$HOME/tmp/repair \
  go test -tags smoke -count=1 -p 1 -v -timeout 2h -run TestRepair ./internal/smoke/

Three scenarios, each a real repository where one upgrade breaks the build. TestRepair first runs the baseline pipeline, which must end regressed, then lets a local model try the repair. A run that ends without a proposal passes and is logged, since a failed repair is a result; a proposal must pass every check a correct one would, including hidden oracles. It is skipped unless REPO_STEWARD_SMOKE_MODEL names a model, only ollama: models are accepted, and neither TestSmoke nor the smoke workflow runs it. Only one scenario may run against an engine at a time. Results are on Evidence and Benchmark Results.

Vulnerability tests

The parser, the target computation, readiness rules, and the report's shape are unit tests. The mounts, the tool profile, the report end to end, and vulnerable selection on fixtures are integration tests. CI also runs fixture vulndb, vulns, and maintain -select vulnerable through the built binary on fixtures. Test names are in Vulnerability Scan Pipeline.

Live model runs

Not in CI. Locally, with Ollama running:

go run ./cmd/repo-steward bench run -mode model -model ollama:qwen3:30b-a3b -scenarios S1,S2

Re-recording

After any change to the system prompt, a tool description or schema, or a tool result shape:

go run ./cmd/repo-steward maintain <fixture-dir> -mode model -model ollama:qwen3:30b-a3b \
  -record internal/steward/testdata/recordings/qwen3-30b-a3b/<fixture> -fixture-proxy <proxy>

Then re-run the benchmarks so the committed results match the committed prompt.

What a change should come with

A control gets a named test, and the status table in docs/status.md points at it. A recovery path gets a fault point and an interruption test. A new fixture gets a pinned id. A new scenario gets an expectation and, where the agent could cheat, an oracle.