Give the study a pinned runtime that refuses rather than degrades - #520
Merged
Conversation
CDEB-03. Every downstream ticket names this one, because a benchmark whose isolation silently degrades measures a different thing than it reports. The new bench/cdeb/runtime/ owns the pinned OCI runtime the study measures in: the pin manifest with every §8 identity field, the exact container shape a measured run gets (fresh isolated HOME, harness-written settings, empty MCP config, no session persistence, frozen tool policy), a provider-only network policy (internal network with no external route, plus an allowlist egress proxy run from the pinned image itself), and the raw provider stream captured byte-for-byte for CDEB-05 to reconcile. The gate is the point. isolation.ts checks eleven capabilities — §7.5's eight plus the container, HOME and image guarantees they rest on — and has exactly two outcomes: a token authorizing runs for one specific pin, or a throw that names every capability that failed or was never probed. Absence of evidence is refusal: a report that omits the network policy has not shown the network policy. The token is checked against the pin's digest before any run, so a preflight for another pin authorizes nothing. Model and CLI identity are checked the same way on the stream itself: an observed model that is not the pinned model, an empty model id, a subagent turn, or a CLI version drift is a hard stop, not a flag on the row. The pin manifest ships UNFROZEN: digests and the exact observed model id are null, because only the freeze ceremony that builds the image and observes the provider can produce them. The gate refuses measured runs until it does — which is also why this change could be verified end to end without pretending to have run a container it could not reach. The §4.6 runtime probe now takes its runtime explicitly. The host runtime stays exported so the pilot's sealed numbers remain reproducible; the pinned runtime is a second implementation of the same seam. Say it plainly: every wall-time number frozen into the PRD was measured on the host CLI, so the 0.48/1.00 split behind the 0.6 screen must be re-measured on the pinned runtime before any freeze relies on it, and that re-validation has not happened yet. The legacy fail-open driver in bench/drivers/ is untouched on purpose: §24.1 forbids reusing its degrade-to-warning fallback in the study runtime, and the study runtime simply does not contain a warning path to reuse. Limit: enforced by code — the capability gate refuses a missing or never-probed capability, the run spec cannot express host HOME or settings inheritance, provider credentials cross the container boundary only through a two-key allowlist, the captured stream is identity-checked before anything reads it as measurement, and the pin loader rejects hand-edited enforcement shapes. Still resting on the operator — building and pulling the image the freeze records, running preflight on the machine that runs the study, keeping the container runtime daemon itself trusted, and passing provider credentials into the preflight and run calls; the gate can verify what a container does, not that the host it runs on is honest Ruled-out: degrading with a warning when the CLI lacks an isolation flag, as the legacy driver does | that is exactly the silent degradation this ticket exists to remove — a run that inherited the operator's settings measures one laptop, not the study Ruled-out: container networking set to none with a unix-socket egress proxy | the agent's HTTP stack cannot be assumed to speak a socket proxy, so provider reachability would fail closed on every machine; an internal network plus allowlist proxy is enforceable and probe-verified instead Ruled-out: filling the pin manifest's digest fields with values read from this machine | the digests must come from the freeze ceremony that builds the image; a hand-filled manifest is an unfrozen pin pretending, and the gate is built to refuse it Warn: the two socket-level egress proxy tests skip where the sandbox denies listen — run them where binding is allowed before the freeze treats the proxy as tested; and the probe's frozen wall-time anchors were measured on the host CLI, so re-measure on the pinned runtime before any freeze relies on the 0.6 screen Blast: local Undo: easy Certainty: firm Verified: fifty-five cases pass, two skipping — a missing or never-probed capability refuses naming it; inherited HOME content, an unexpected mount and persisted session state each fail closed; model drift, empty model ids, subagent turns, CLI drift and tool-set divergence each stop the study; scripted-runtime orchestration yields one passing probe per capability and the generated network probe script passes a syntax check; the allowlist decision refuses foreign hosts, foreign ports and non-CONNECT; both typechecks, both verifiers and a double build with stable dist all pass Unverified: no container runtime was reachable from this machine, so the image was never built, no daemon ever answered a probe, and no run ever executed inside the pinned runtime — the Dockerfile, the probe orchestration against a real daemon, and the socket path of the egress proxy are written but unexercised; the probe's wall-time anchors need re-measurement on the pinned runtime Provenance: authored Record-Id: r-cdeb03rt
CommitLore — record lintTrailers: clean — 1 commit in Active constraints for the paths this PR touchesLimits (5)
Ruled out (11)
Warnings (5)
Trailer violations fail this check. Active constraints are informational — they are what the repository already decided, not a verdict on this PR. |
This was referenced Aug 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CDEB-03. Unblocks CDEB-04 and CDEB-05.
A benchmark whose isolation silently degrades measures a different thing than it reports. All three acceptance properties are failures, and each has a test that fails when the property is removed:
The second line is the one that matters most. A capability nobody probed is not a capability that passed — that is the path by which isolation degrades without anyone noticing, and it is closed by refusal rather than by a default.
What is enforced by code, and what is not
The commit's
Limit:splits them: the capability gate refuses a missing or never-probed capability, the run spec cannot express hostHOMEor settings inheritance, and executable drift stops the study. What remains is what the operator has to run correctly.And its
Warn:names a gap in this verification rather than hiding it:Those two are unverified here. They should be run on a machine that allows binding before anything relies on the proxy.
Note on §4.6
runtime-probe.tspromised the pinned runtime while spawning the host'sclaude. That gap is what this closes — but the probe's calibration was measured on the old path, and re-pointing it does not carry that calibration forward. The 0.6 threshold's derivation is already marked unverified by #518.57 cases pass; package and bench typechecks clean; both verifiers pass;
distunchanged.