Skip to content

Sandbox

Joey King edited this page Oct 5, 2026 · 2 revisions

Sandbox

Every command that runs the Go toolchain against a repository runs in a container created through the Docker Engine API. The engine can be Docker Desktop, OrbStack, Colima, or Rancher Desktop; only the socket matters.

Why containers, and why through the API

A subprocess with restrictions on the host would still share the host filesystem, network, and environment. A container gives a separate root filesystem, a separate network namespace that can be disabled, an empty environment, and resource limits, all enforced by the engine rather than by this program. Talking to the engine over its API instead of shelling out to the docker binary keeps the argument shape explicit and avoids inheriting whatever the CLI reads from the user's configuration.

The client in sandbox/dockerapi implements only what is needed: create, start, wait, logs, remove, list by label, and inspect. It has no dependencies.

Profiles

Profile Source snapshot Module cache Staging Network Runs repository code
acquire read-only writable none as configured, proxy only no
mutate read-only writable read-write copy of the manifests as configured no
execute read-only read-only none none, enforced by the engine yes
tool none its own, fresh for each build none proxy.golang.org and sum.golang.org only, by configuration no; builds a pinned tool such as the scanner

When a scan is configured, execute also mounts the scanner directory at /tools and the vulnerability database at /vulndb, both read-only; no other profile mounts them, and the execute container cannot write either: the host, not a container, copies what the tool profile built into the data directory.

The tool profile exists to build a trusted, pinned tool. It mounts no source, not the repository's module cache, no configured tools or database directory, and fresh module, build, and output directories of its own; /tools is writable there as the Go install directory. It always uses proxy.golang.org and sum.golang.org, even when the run uses a fixture proxy. It has the same read-only root, dropped capabilities, unprivileged user, limits, and from-scratch environment as every profile, and its network egress is limited by configuration, not by the operating system, as for acquire.

The separation is the security argument in one table. The profile that runs repository code has no network and cannot write anything that persists. The acquire and mutate profiles run only the toolchain's own module commands, never a build or a test. The tool profile builds one pinned tool, shares nothing with the repository, and never runs repository code.

Common settings

Every container gets: an empty environment with only the variables the profile sets, an unprivileged user, all capabilities dropped and no new privileges, a read-only root filesystem, a tmpfs at /tmp mounted with exec because the toolchain builds test binaries there, a build cache directory of its own, cpu, memory, and pids limits, an in-container timeout that kills and removes the container, and a label that identifies it as this program's so orphans can be reaped.

Environment in the containers: GOFLAGS=-mod=mod, GOTOOLCHAIN=local, GOPROXY set to the configured proxy with no direct fallback, GOSUMDB on unless a fixture proxy is in use, GOPRIVATE empty, GOMODCACHE pointing at the mounted cache, GOCACHE at the build cache, HOME and TMPDIR under /tmp.

The mount probe

VM-backed engines share only some host paths with the virtual machine. A bind mount of an unshared path does not fail; it silently appears empty. Validating an empty directory would pass with zero packages and produce a false claim. So before any profile runs, a probe writes a uniquely named marker file into each host directory the run will mount, starts a container that lists them, and fails hard if any marker is missing. The marker name is unique per probe because some engines cache path lookups and a reused name could report a stale result.

Rules that came from real failures

  • Never delete and recreate a mounted path within a run. Path caching in the VM can leave a container looking at the old inode. Snapshot directories, staging directories, and build caches all get unique names.

  • Build output goes to /tmp. go build ./... writes into the package directory, which is read-only; the validator passes -o to a tmpfs path.

  • Test setup failures are on stderr only. go test -json reports a package that fails to build as text, not as a JSON event; the parser treats that as a conclusive package finding rather than silence.

  • Output is capped. Stdout and stderr are bounded and the truncation is flagged. A truncated validation is inconclusive, never passing.

  • A fresh build cache for every validation. Baseline, post, each run_validation, inspect, and every readiness or tool command in the execute profile mounts a new, empty, randomly named build cache, and the execute profile refuses to run without one. Acquire and mutate use a separate cache that execute never mounts, and the end of the run removes them all. Within one validation, build, vet, and test share a cache, so code that validation runs can still affect that validation's own result. The cost is time: a later validation no longer reuses what an earlier one compiled.

  • Each post-run engine call gets its own budget. Kill, log collection, and removal used to share one 30-second budget that began when the container was created, so any command that ran longer than that, such as acquisition over the network, failed with its output unread. Each now has its own, covered by TestRun_LongCommandStillCollectsLogsAndRemoves. Measuring repair on real repositories exposed it.

  • Build caches are removed through a container when the host cannot. On native Linux Docker the container user owns what the toolchain wrote; cleanup runs through a container after the cache directories are reopened. Verified on colima only; CI's integration job on native Linux had not yet run it.

  • The data directory is owner-only, because the module cache, build caches, and staging directories are mode 0777 for the container user. See Configuration.

Orphans and locks

Containers carry a label with the run id. inspect and maintain reap any labelled container left by a crashed process before starting. Because reaping is global, integration tests run serially.

Toolchain images

The image is golang:<minor>-bookworm pinned by digest, selected from the repository's go directive. A directive outside the table is a refusal. GOTOOLCHAIN=local means the toolchain never downloads a newer Go inside the container; a dependency that needs one fails at go get and the run reports requires_newer_toolchain. Runs never pull images; -pull or docker pull does that once.

Tests

Integration tests under -tags integration exercise each guarantee against a real engine, including the new mounts and the tool profile (TestExecute_ToolsAndVulnDBReadOnly, TestTool_IsolatedProfile, TestExecute_BuildCacheFreshPerValidation): no network in execute, source and cache read-only, environment built from scratch, timeout kills and removes, output cap flags truncation, orphans reaped, probe detects an unshared directory. They run with -p 1 because reaping is global, and their temporary directories live under the home directory so a VM-backed engine can see them.

Clone this wiki locally