Benchmarks

Benchmarks and Performance

Four benchmark families in internal/oci/benchmark_test.go, each run across eight payload sizes (1 KiB, 4 KiB, 64 KiB, 256 KiB, 1 MiB, 4 MiB, 16 MiB, 32 MiB - bounded there rather than continuing to 64/128 MiB, to keep the full -count=20 × 3-family sweep affordable in CI) unless noted otherwise:

Family	What it measures
BenchmarkBuild	The full, real path: input/ELF validation, reading the binary, building the deterministic tar layer, gzip-compressing it, marshaling config/manifest/index JSON, writing every blob to a fresh temp directory, and an atomic rename. No fsync anywhere in this path.
BenchmarkMakeLayer	Just the in-memory cost - tar framing, SHA-256, gzip - with no disk I/O at all. The gap between this and BenchmarkBuild at the same size is approximately what disk I/O costs on the runner's temp filesystem.
BenchmarkNaiveTarGzip	A reference point, not a competitor to beat: the simplest possible archive/tar + compress/gzip at default settings, none of Build()'s determinism, validation, or OCI structure. Shows what correctness and determinism cost over the bare minimum.
BenchmarkBuildParallel	BenchmarkBuild at fixed concurrency levels (1/2/4/8/16) via RunParallel, at a constant 1 MiB payload, run at the runner's real GOMAXPROCS (not pinned to 1, unlike the other three - see below).

Payload bytes are a fixed-seed math/rand fill: deterministic across runs, but not trivially compressible, so results don't overstate throughput the way an all-zero fixture would.

Two real bugs this benchmark suite found and got fixed

Profiling BenchmarkBuild at 16 MiB (-memprofile, go tool pprof -alloc_space) showed 74% of all allocated bytes came from bytes.Buffer growth-and-copy in makeLayer: neither the tar-stream buffer nor the gzip-compressed-output buffer were pre-sized, so both grew via repeated doubling as archive/tar and compress/flate wrote into them. Pre-sizing both with Buffer.Grow roughly halved allocation at 16 MiB (101.5 MiB/op → 51.2 MiB/op).

That fix then revealed a second bug at 32 MiB+: Grow(raw.Len()) reserved zero margin for the compressed buffer, and deflate can expand already-incompressible content slightly (measured: +0.031%, i.e. +10,263 bytes, compressing 32 MiB of random data) - that tiny overshoot forced a full reallocate-and-copy of the entire buffer, again. A 1% margin fixed it; allocation is now a flat, scale-invariant ~2.02–2.05x payload size from 1 KiB to 128 MiB (previously ballooning to ~4.1x at 32 MiB+).

A third finding, not a performance bug but real duplicate work: the layer's SHA-256 was computed twice - once for its descriptor, once again in writeLayout just to pick the blob's content-addressed filename. Fixed by threading the already-computed digest through instead of re-hashing.

BenchmarkNaiveTarGzip (unmodified stdlib, no pre-sizing) shows the same ~4x blowup at scale that was just fixed in makeLayer - so this project's own layer-building code is now measurably more memory-efficient at large payload sizes than a naive out-of-the-box tar+gzip, purely from that pre-sizing.

Statistics, not just a median

scripts/ci/benchmark-report.py computes mean, median, min, max, sample standard deviation, and p95 for time/op, throughput, heap/op, and allocations/op - across -count=20 independent repetitions per size, at a fixed -benchtime=5x (bounded, predictable CI time regardless of payload size, rather than the time-based default's open-ended iteration count for small sizes). benchmark.json carries the full statistics; benchmark.md shows median and p95 per row to stay readable.

BenchmarkBuild, BenchmarkMakeLayer, and BenchmarkNaiveTarGzip all run with -cpu=1 - none of them use goroutines internally, so pinning GOMAXPROCS=1 removes scheduler noise instead of leaving an unexplained runner-core-count suffix in the numbers. BenchmarkBuildParallel is the deliberate exception: it exists specifically to exercise concurrent builds, so it runs at the runner's actual GOMAXPROCS.

The reporter also self-checks: it re-parses the benchmark.md and benchmark.json it just wrote and confirms every family/parameter/run-count in the JSON is actually reflected in the Markdown, so the two outputs can't silently drift apart in a future change.

Reproducibility metadata recorded every run

Go version (pinned, currently 1.25.12), kernel/OS/architecture, CPU model, core count, the exact benchmark command, git commit, workflow run ID and attempt, and runner OS/architecture - all in benchmark.json.environment and benchmark.md's Environment section. A CPU profile and a memory profile (-cpuprofile/-memprofile, memprofilerate=1) are captured every run and rendered to a readable top-25 text summary (go tool pprof -top), so profiling evidence doesn't require downloading and opening a binary profile just to see if anything changed.

Leak and concurrency safety checks (not performance numbers)

TestBuildLeavesNoGoroutinesOrFilesBehind asserts runtime.NumGoroutine() and (on Linux) the /proc/self/fd count don't increase across a Build() call - Build() is fully synchronous today, so this pins that property and catches it immediately if a future change (streaming, a worker pool) introduces a leak. BenchmarkBuildParallel also runs once under -race, separately from the performance numbers, specifically to catch a data race in concurrent Build() calls without race instrumentation's overhead polluting the timing results.

Charts

scripts/ci/render-benchmark-charts.py renders three payload-size charts (throughput, memory/op, time/op - BenchmarkBuild/MakeLayer/NaiveTarGzip, one line per family) straight from benchmark.json as self-contained SVG - no matplotlib/numpy: this project doesn't carry a plotting dependency anywhere else, and adding one just for three lines didn't seem worth it. Memory and time use a log-scale y-axis (the size range spans seven orders of magnitude); throughput is linear. Line color follows this project's validated three-slot categorical palette (blue/orange/aqua - passes CVD and normal-vision separation checks across every pair, in both light and dark mode), each series also carries a direct end-of-line label so identity never depends on color alone. Uploaded as CI artifacts alongside the other evidence; not embedded inline in benchmark.md today (GitHub's artifact viewer doesn't render SVG previews without a download).

Reproduce locally

go test ./internal/oci -run '^$' \
  -bench '^(BenchmarkBuild|BenchmarkMakeLayer|BenchmarkNaiveTarGzip)$' \
  -benchmem -benchtime=5x -count=20 -cpu=1 \
  -cpuprofile=cpu.prof -memprofile=mem.prof | tee benchmark-sequential.txt
go test ./internal/oci -run '^$' -bench '^BenchmarkBuildParallel$' \
  -benchmem -benchtime=5x -count=20 | tee benchmark-parallel.txt
cat benchmark-sequential.txt benchmark-parallel.txt > benchmark.txt
python3 scripts/ci/benchmark-report.py benchmark.txt environment.txt benchmark.json benchmark.md

CI publication

ci-benchmark.yml has four jobs:

• benchmark - the full run above, plus the leak check and the -race pass, uploaded as benchmark-results (90-day retention): raw text, JSON, Markdown, environment, both profiles (binary and rendered top-25 text), and traceability.json.
• compare-with-main - runs on pull requests (and non-main pushes), skipped on main itself. Checks out the PR's base branch into a git worktree (what's measured) but always runs this branch's own scripts/ci/benchmark-report.py/compare-benchmarks.py (how it's measured and reported) - the same "trusted verifier, not trusted subject" pattern this project already uses for OCI layout verification, so an older reporting-script schema on the base branch can never produce something this comparison can't parse. Scoped to BenchmarkBuild only at a smaller -count=5, to keep this second full benchmark pass affordable. Produces two complementary comparisons, posted as a PR comment (edited in place on subsequent pushes, not reposted) and uploaded as benchmark-comparison:
  • benchstat (pinned by commit): a proper statistical A/B comparison with p-values (Mann-Whitney U), not just a percentage difference.
  • compare-benchmarks.py: a simple threshold check - flags a median throughput drop past 5%, a median memory increase past 5%, or any allocs/op change (allocation counts are otherwise deterministic per code path, so any change is worth a look regardless of size). Informational only - never fails the job. A single GitHub-hosted run isn't a reliable enough signal to gate merges on yet; see below.
• benchmark-history - on push to main: appends this run's results to an artifact-based JSONL history (benchmark-history-<sha>, 90-day retention) rather than committing to the tracked tree, to avoid a bot automatically pushing to a protected branch.
• sign-benchmark-evidence - on push to main only (never on pull_request, even from a fork that still gets a real OIDC token - the same reasoning ci-release.yml and ci-microvm.yml apply to their own signing jobs): signs benchmark.json with keyless Cosign and verifies it, the same evidence-signing pattern as the OCI image and the microVM kernel evidence.

Interpretation and regression policy

Throughput should increase with payload size as fixed overhead is amortized; time and heap use should scale approximately linearly with payload size (see above for the two places that didn't, until fixed). compare-with-main's thresholds (5% throughput/memory, any allocation change) are deliberately not a merge gate today: GitHub-hosted runners share hardware and produce real noise, and a hard gate on a single run would create false failures and invite threshold-inflation games. Once benchmark-history has accumulated enough runs to characterize normal noise, a future change can turn this into an actual blocking check - not before.

Scope limitations - what this benchmark suite does not claim

This is written out explicitly because it's easy to read "20 runs, 10 sizes, CPU/memory profiling, benchstat, signed evidence" and assume more rigor than is actually being claimed:

• Single machine, single campaign. Every run above happens on one GitHub-hosted ubuntu-24.04 runner, shared hardware, one campaign per CI run. This is not "multiple independent campaigns on dedicated hardware," and the numbers should be read as noisy-but-useful-for-catching-large- regressions, not as a characterized, low-variance baseline.
• No comparison against umoci, go-containerregistry/crane, BuildKit, or a bare tar+gzip CLI. BenchmarkNaiveTarGzip is a bare-minimum reference point using the same Go stdlib this project already uses - not a different tool doing the same job. A fair comparison against a different tool would require an equivalent scope (does it validate ELF closure? produce a full OCI layout? run daemonless?) that this project doesn't attempt to construct here. An apples-to-oranges throughput number against a tool with different guarantees would be a marketing chart, not evidence.
• No disk-type comparison. tmpfs vs. local SSD vs. GitHub Actions' actual backing storage vs. cloud block storage are not compared; whichever filesystem $RUNNER_TEMP resolves to on the hosted runner is what's measured, and it isn't recorded whether its cache is warm or cold.
• No fsync, no full-disk, no write-failure testing. Build()'s write path (os.MkdirTemp + os.WriteFile per blob + os.Rename) has no fsync call at all today - that's a fact about the code, not something benchmarked. Behavior under a full disk or a mid-write failure is exercised by existing error-path unit tests (a permission-denied or missing-parent case), not by an actual disk-exhaustion scenario.
• compare-with-main is not a blocking gate. See above - it's evidence for a human to read, not yet a regression gate with a track record behind it.

None of this is a defect in what's implemented - it's the difference between "a benchmark suite that will catch a 2x memory regression or a broken compression path" (what this is) and "a certified, apples-to-apples performance SLA across tools and hardware" (what it explicitly is not).