Fleet release: publish the 5-agent combo fleet + base-image CVE fixes

[elronbandel]

What this does

Adds one workflow that releases the whole eval-container fleet to GHCR (multi-arch), plus the security fixes that make it pass the CVE gate.

Fleet release (release-images.yml) — folds the old per-task publisher in and adds the combo + per-task publish. A vX.Y.Z tag (or a manual run) builds, in one pipeline:

• the shared base images once per arch (frozen), then every benchmark / agent / model leaf in parallel;
• eval combos for the 5 released agents — claude-code, codex, gemini-cli, openclaw, zerostack — × every benchmark, including the per-task ones (all 500 swe-bench tasks + all terminal-bench tasks), plus a single-container -standalone bundle for each combo (lean base + in-process gateway/otelcol/process-compose — the laptop / --mode container artifact);
• the per-task base images (swe-bench via bake, terminal-bench via its build script);
• combo + per-task matrices are sharded to fit under GitHub's 256-job cap;
• builds retry with jittered backoff so Hugging Face rate-limiting on the runner IP can't sink a leaf;
• combos + compose survive a partial leaf build — one failed leaf can't skip every combo/bundle;
• CVE gate on the base images, then :tag → :latest promotion + Helm chart;
• an end-of-run durable size report (fleet-size-report artifact: per-agent/-benchmark size + build time as JSON + CSV) for the audits + README tables.

Multi-arch (amd64 + arm64) — the fleet publishes multi-arch manifest lists via native-per-arch (NOT QEMU — 41 of 103 benchmarks still use pyarrow, which segfaults under emulation): bases / leaves / per-task each carry an arch: [amd64, arm64] matrix and build on their own runner (arm64 → ubuntu-24.04-arm, override with the FLEET_RUNNER_ARM repo Variable), each pushing :TAG-<arch>; a new merge job stitches them into the :TAG manifest list (imagetools create), skipping any image with no per-arch tag so a single-arch failure stays isolated. Combos build multi-arch directly with buildx --platform amd64,arm64 (thin overlays are QEMU-safe). swe-bench FROMs Epoch's per-arch base via EVAL_BASE_ARCH. The Dockerfiles + the per-arch tag flow are validated locally; the arm64 CI path is verified on the release run.

Base-image CVE fixes — the fleet passes the HIGH/CRITICAL trivy gate:

• litellm: install the four genuinely-needed patched deps (starlette / python-multipart / PyJWT / cryptography) into the base's pip-less uv venv, guarded by a functional RS256 sign+verify smoke (cryptography is forced past litellm's <47 cap);
• runtime-bundle: build gosu + process-compose from source with a current Go (clears a CRITICAL + the Go-stdlib CVEs the upstream release binaries still carry);
• otel: build the collector with a current Go toolchain (collector version unchanged → traces stay byte-identical); the grpc CRITICAL (CVE-2026-33186) is fixed via a builder-config grpc→1.79.3 bump, not deferred;
• gateways/litellm: bumped 1.83.3 → 1.89.1 with the same dep patches;
• the rest: OS-package upgrades. Remaining findings are no-fix upstream and documented in .trivyignore.

HF dataset auth — build-time Hugging Face downloads now authenticate (Bearer token), so cold parallel builds aren't throttled to anonymous rate limits.

Cleanup — removes frontiermath (a private dataset that can't be built from a clean checkout) and refreshes the counts/docs/tests.

Pre-release review

An independent multi-agent review (CVEs, workflow logic, multi-arch/build, regression) ran over the branch; every confirmed finding is folded in and validated locally: the grpc CRITICAL is now genuinely fixed (it had been a wrong-justification deferral); otel/runtime-bundle/duckdb no longer break on the classic build path (BuildKit-only $TARGETARCH was empty there → native build); combos + compose survive a partial leaf build; the gateway litellm was bumped; npm@latest was pinned; the .secrets.baseline was corrected (it had been regenerated against the pre-#171 layout).

Scope note

swe-bench-pro + swe-lancer per-task publishing is intentionally dropped vs the old publish-per-task.yml (which published skills-bench, terminal-bench, swe-bench-pro, swe-lancer). The unified workflow publishes terminal-bench + skills-bench + swe-bench per-task — matching the release scope (terminal-bench + swe-bench).

Validation

• CVE gate green across all base images (incl. the duckdb/slim bases); core/litellm rebuilt + trivy-scanned clean with the trimmed pin set (0 non-ignored HIGH/CRITICAL).
• otel / runtime-bundle / duckdb build + run as native arm64 on the classic DOCKER_BUILDKIT=0 path, and amd64; the per-arch TAG flows into both the image tag and the frozen-base context (bake --print).
• The standalone bake graph (eval + eval-standalone) resolves; the size-report JSON/CSV generation is validated.
• Full-fleet config resolves at scale (sharded matrices stay under the 256-job cap, ×2 arch accounted for).
• The full ~hours-long multi-arch publish + the arm64 runner path are verified only on the release tag (no local arm64 runner; Actions is billing-locked).

Releasing

After merge, push a v0.1.0 tag (matches the current Cargo/Chart version, so the version guard passes) — it fires this workflow + the CLI release.