Skip to content

Give the study a pinned runtime that refuses rather than degrades - #520

Merged
MongLong0214 merged 1 commit into
devfrom
cdeb-03-runtime
Aug 11, 2026
Merged

Give the study a pinned runtime that refuses rather than degrades#520
MongLong0214 merged 1 commit into
devfrom
cdeb-03-runtime

Conversation

@MongLong0214

Copy link
Copy Markdown
Owner

CDEB-03. Unblocks CDEB-04 and CDEB-05.

A benchmark whose isolation silently degrades measures a different thing than it reports. All three acceptance properties are failures, and each has a test that fails when the property is removed:

refuses a failed probe and names the capability and what is missing
refuses a capability that was never probed — absence of evidence is not a pass
refuses a malformed report that probes one capability twice
help without isolation flags fails the capabilities those flags protect
an image whose digest does not match the pin is refused, never retagged
executable drift against the pin is refused naming each drifted field
refuses provider env keys outside the allowlist — a host environment leak path
a HOME that starts non-empty is inheritance, and the gate names it
a mount outside the expected pair is refused naming the mount

The second line is the one that matters most. A capability nobody probed is not a capability that passed — that is the path by which isolation degrades without anyone noticing, and it is closed by refusal rather than by a default.

What is enforced by code, and what is not

The commit's Limit: splits them: the capability gate refuses a missing or never-probed capability, the run spec cannot express host HOME or settings inheritance, and executable drift stops the study. What remains is what the operator has to run correctly.

And its Warn: names a gap in this verification rather than hiding it:

the two socket-level egress proxy tests skip where the sandbox denies listen — run them where binding is allowed before the freeze treats the proxy as tested

Those two are unverified here. They should be run on a machine that allows binding before anything relies on the proxy.

Note on §4.6

runtime-probe.ts promised the pinned runtime while spawning the host's claude. That gap is what this closes — but the probe's calibration was measured on the old path, and re-pointing it does not carry that calibration forward. The 0.6 threshold's derivation is already marked unverified by #518.

57 cases pass; package and bench typechecks clean; both verifiers pass; dist unchanged.

CDEB-03. Every downstream ticket names this one, because a benchmark whose
isolation silently degrades measures a different thing than it reports. The
new bench/cdeb/runtime/ owns the pinned OCI runtime the study measures in:
the pin manifest with every §8 identity field, the exact container shape a
measured run gets (fresh isolated HOME, harness-written settings, empty MCP
config, no session persistence, frozen tool policy), a provider-only network
policy (internal network with no external route, plus an allowlist egress
proxy run from the pinned image itself), and the raw provider stream captured
byte-for-byte for CDEB-05 to reconcile.

The gate is the point. isolation.ts checks eleven capabilities — §7.5's eight
plus the container, HOME and image guarantees they rest on — and has exactly
two outcomes: a token authorizing runs for one specific pin, or a throw that
names every capability that failed or was never probed. Absence of evidence
is refusal: a report that omits the network policy has not shown the network
policy. The token is checked against the pin's digest before any run, so a
preflight for another pin authorizes nothing. Model and CLI identity are
checked the same way on the stream itself: an observed model that is not the
pinned model, an empty model id, a subagent turn, or a CLI version drift is a
hard stop, not a flag on the row.

The pin manifest ships UNFROZEN: digests and the exact observed model id are
null, because only the freeze ceremony that builds the image and observes the
provider can produce them. The gate refuses measured runs until it does —
which is also why this change could be verified end to end without pretending
to have run a container it could not reach.

The §4.6 runtime probe now takes its runtime explicitly. The host runtime
stays exported so the pilot's sealed numbers remain reproducible; the pinned
runtime is a second implementation of the same seam. Say it plainly: every
wall-time number frozen into the PRD was measured on the host CLI, so the
0.48/1.00 split behind the 0.6 screen must be re-measured on the pinned
runtime before any freeze relies on it, and that re-validation has not
happened yet.

The legacy fail-open driver in bench/drivers/ is untouched on purpose: §24.1
forbids reusing its degrade-to-warning fallback in the study runtime, and the
study runtime simply does not contain a warning path to reuse.

Limit: enforced by code — the capability gate refuses a missing or never-probed capability, the run spec cannot express host HOME or settings inheritance, provider credentials cross the container boundary only through a two-key allowlist, the captured stream is identity-checked before anything reads it as measurement, and the pin loader rejects hand-edited enforcement shapes. Still resting on the operator — building and pulling the image the freeze records, running preflight on the machine that runs the study, keeping the container runtime daemon itself trusted, and passing provider credentials into the preflight and run calls; the gate can verify what a container does, not that the host it runs on is honest
Ruled-out: degrading with a warning when the CLI lacks an isolation flag, as the legacy driver does | that is exactly the silent degradation this ticket exists to remove — a run that inherited the operator's settings measures one laptop, not the study
Ruled-out: container networking set to none with a unix-socket egress proxy | the agent's HTTP stack cannot be assumed to speak a socket proxy, so provider reachability would fail closed on every machine; an internal network plus allowlist proxy is enforceable and probe-verified instead
Ruled-out: filling the pin manifest's digest fields with values read from this machine | the digests must come from the freeze ceremony that builds the image; a hand-filled manifest is an unfrozen pin pretending, and the gate is built to refuse it
Warn: the two socket-level egress proxy tests skip where the sandbox denies listen — run them where binding is allowed before the freeze treats the proxy as tested; and the probe's frozen wall-time anchors were measured on the host CLI, so re-measure on the pinned runtime before any freeze relies on the 0.6 screen
Blast: local
Undo: easy
Certainty: firm
Verified: fifty-five cases pass, two skipping — a missing or never-probed capability refuses naming it; inherited HOME content, an unexpected mount and persisted session state each fail closed; model drift, empty model ids, subagent turns, CLI drift and tool-set divergence each stop the study; scripted-runtime orchestration yields one passing probe per capability and the generated network probe script passes a syntax check; the allowlist decision refuses foreign hosts, foreign ports and non-CONNECT; both typechecks, both verifiers and a double build with stable dist all pass
Unverified: no container runtime was reachable from this machine, so the image was never built, no daemon ever answered a probe, and no run ever executed inside the pinned runtime — the Dockerfile, the probe orchestration against a real daemon, and the socket path of the egress proxy are written but unexercised; the probe's wall-time anchors need re-measurement on the pinned runtime
Provenance: authored
Record-Id: r-cdeb03rt
@github-actions

Copy link
Copy Markdown

CommitLore — record lint

Trailers: clean — 1 commit in origin/dev..f0ad0913ae9a4f7a526d81986a77500946fdf717
Active constraints: 5 limits · 11 ruled-out · 5 warnings — from 5 records over 7 changed paths

Active constraints for the paths this PR touches

Limits (5)

  • r-cdeb03rt f0ad091 — enforced by code — the capability gate refuses a missing or never-probed capability, the run spec cannot express host HOME or settings inheritance, provider credentials cross the container boundary only through a two-key allowlist, the captured stream is identity-checked before anything reads it as measurement, and the pin loader rejects hand-edited enforcement shapes. Still resting on the operator — building and pulling the image the freeze records, running preflight on the machine that runs the study, keeping the container runtime daemon itself trusted, and passing provider credentials into the preflight and run calls; the gate can verify what a container does, not that the host it runs on is honest
  • r-unattshadow b7b532a — together the two features measure how often an unattended pipeline would have written, and remove the asking from the writing -- neither half can say whether what gets written is worth a reader's attention, so shadow's number for an unattended repository is a volume, not a value
  • r-cdebsurface a1e8833 — the pilot anchors now marked unverified were measured under Edit|Write|MultiEdit|NotebookEdit, not the shipping Read|Edit|Write -- the 1.45 ON/OFF overhead, the timeout wall times behind the 0.48/1.00 split and the 0.6 screen derived from it, and the zero-delivery count
  • r-probemodel 6966cb3 — pinning to sonnet means CDEB will say nothing about whether a stronger model needs delivered decisions less, which is a question worth its own study rather than a variable inside this one
  • r-runprobe 6a368ec — the gate screens runtime and says so -- completed means the process returned, which a no-op satisfies, and the pilot's 4.9x within-cell spread means two probes cannot bound the tail

Ruled out (11)

  • r-cdeb03rt f0ad091 — degrading with a warning when the CLI lacks an isolation flag, as the legacy driver does | that is exactly the silent degradation this ticket exists to remove — a run that inherited the operator's settings measures one laptop, not the study
  • r-cdeb03rt f0ad091 — container networking set to none with a unix-socket egress proxy | the agent's HTTP stack cannot be assumed to speak a socket proxy, so provider reachability would fail closed on every machine; an internal network plus allowlist proxy is enforceable and probe-verified instead
  • r-cdeb03rt f0ad091 — filling the pin manifest's digest fields with values read from this machine | the digests must come from the freeze ceremony that builds the image; a hand-filled manifest is an unfrozen pin pretending, and the gate is built to refuse it
  • r-unattshadow b7b532a — keeping the unattended branch's inline prepare body | the shadow refactor exists so both entry points share one side-effect-free half, and two copies of the same hashing and policy logic would drift the first time either changed
  • r-unattshadow b7b532a — checking unattended consent in the live path only | the refusal for mode "off" already lives in the shared half of prepare, and a consent check that guards one door but not the other is no guard for the next entry point added
  • r-cdebsurface a1e8833 — copying the shipping matcher string into each arm writer | a copied string drifts the first time the product changes it, and that drift is exactly the defect this fixes -- every arm must read what the product ships
  • r-cdebsurface a1e8833 — re-deriving or deleting the pilot anchors once their surface was known to differ | the numbers stay citable as what the pilot measured on the surface it ran; re-measurement is a separate decision this change does not make
  • r-probemodel 6966cb3 — qualifying on opus while measuring on sonnet | the gate would screen runtimes the study never produces, and its threshold is derived from sonnet data
  • r-probemodel 6966cb3 — leaving the probe model unchecked and documenting the requirement | the mismatch produces timeouts that read as task difficulty, so a note cannot catch what a comparison can
  • r-runprobe 6a368ec — averaging the two arms | the study runs six sessions per task, so the slower arm is the one that decides whether they fit, and a mean lets the faster arm carry a task in
  • r-runprobe 6a368ec — keeping the probe artifact out of storage | the freeze manifest is required to carry per-task probe results, and a qualification nobody can recheck is not a gate

Warnings (5)

  • r-cdeb03rt f0ad091 (claim) — the two socket-level egress proxy tests skip where the sandbox denies listen — run them where binding is allowed before the freeze treats the proxy as tested; and the probe's frozen wall-time anchors were measured on the host CLI, so re-measure on the pinned runtime before any freeze relies on the 0.6 screen
  • r-unattshadow b7b532a (claim) — the consent check now lives in the shared half of prepare, so a third entry point inherits it -- but it fires only on a declaration, and shadow writes nothing because it routes around createPending and stage, not because it is exempted from consent
  • r-cdebsurface a1e8833 (claim) — 0.6 stays frozen as the runtime screen with its derivation marked unverified; if the shipping matcher ever changes the arms pick it up automatically, but every pilot anchor in the PRD needs re-measuring before a freeze relies on it
  • r-probemodel 6966cb3 (claim) — a later model change invalidates the 0.6 derivation as well as the pinning, so both must be re-established rather than only the constant edited
  • r-runprobe 6a368ec (claim)runProbe costs two agent sessions per task, so qualifying a 30-task corpus is 60 sessions before a single measured run

git log --follow accepts exactly one pathspec, so renames are not followed for 7 paths; query one path at a time to follow its rename chain

Trailer violations fail this check. Active constraints are informational — they are what the repository already decided, not a verdict on this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant