Skip to content

Make CDEB token accounting fail closed - #524

Merged
MongLong0214 merged 1 commit into
devfrom
cdeb-05-ledger
Aug 11, 2026
Merged

Make CDEB token accounting fail closed#524
MongLong0214 merged 1 commit into
devfrom
cdeb-05-ledger

Conversation

@MongLong0214

Copy link
Copy Markdown
Owner

CDEB-05. Depends on #520 (merged).

A cost number that looks measured and is not is worse than no number, and the substitution is quiet: a gap filled with a mean, a zero, or a per-token extrapolation reads as diligence. docs/SELF-AUDIT.md exists because of exactly that shape.

So usage has two states and no third:

meaning
measured every NDJSON segment parsed, every turn carried final usage, terminal usage existed, and the two totals reconciled exactly
unavailable anything else — and the row schema has no place to put a number, so a reader cannot mistake a diagnostic for data

The schema split is a oneOf on availability. The unavailable branch carries reasons and the raw stream digest, and not one token field.

Dropping the row is the same defect one layer up

Aggregation propagates the gap rather than summing what is left. §14.6 already says an unrecoverable run stays in the denominator; a smaller total assembled from the runs that happened to parse describes a different population than the one that ran.

Observed, not requested

The models recorded are the ones observed answering. Drift between requested and observed is what the pinned runtime (#520) exists to catch, and the ledger is where it becomes evidence rather than an assumption. The verifier now binds a row to the retained raw bytes — decompress, digest, compare against both the sidecar and the row — so a summary whose stream was not kept is a finding, not a claim.

A timeout is not an excuse to synthesize a total. When SIGTERM lets the CLI emit terminal usage the parser uses it; when it does not, the outcome says so.

Two things worth arguing with

One unavailable run makes the whole aggregate unavailable. That is the strict reading of §14.6 and it will feel severe on a long matrix. The reasons are per run, so a report can always say which ones and why — but the headline number refuses rather than shrinks.

node:zlib zstd needs Node 22.15, above the package floor of 22. This is bench-only; no shipped code imports it. CI runs 22 and 24.

83 cases pass across the ledger, verifier and isolation suites — an unparsed segment, a delegated turn, a truncated stream, an absent terminal usage, and a byte-exact round trip of recorded NDJSON through the persisted artifact. Both typechecks clean, two builds leave dist unchanged, both verifiers pass.

Every stream tested here is a recorded fixture; no containerized agent run was performed.

A cost number that looks measured and is not is worse than no number, and the
substitution is quiet: a gap filled with a mean, a zero, or a per-token
extrapolation reads as diligence. CDEB-05 therefore gives usage exactly two
states. Measured means every NDJSON segment parsed, every turn carried final
usage, the terminal usage object existed, and the two totals reconciled
exactly. Anything else is unavailable, and an unavailable row carries no
numeric token field at all -- the row schema has no place to put one, so a
reader cannot mistake a diagnostic for data.

Dropping the row would be the same defect one layer up, so aggregation
propagates the gap instead of summing what is left. §14.6 already says an
unrecoverable run stays in the denominator; a smaller total assembled from the
runs that happened to parse is a different population than the one that ran.

The models recorded are the ones observed answering, not the ones requested.
Drift between the two is what the pinned runtime exists to catch, and the
ledger is where it becomes evidence rather than an assumption; the verifier now
binds a row to the retained raw bytes, so a summary whose stream was not kept
is a finding rather than a claim.

A timeout is not an excuse to synthesize a total. When SIGTERM lets the CLI
emit terminal usage the parser uses it, and when it does not the outcome says
so.

Limit: the ledger observes only bytes emitted before the agent process closes -- terminal usage never written to the stream, external billing, and work outside that stream are all unavailable to it rather than estimated
Ruled-out: inferring terminal usage from the turns that did arrive | it would be a plausible number carrying no evidence, and plausible is exactly what makes it survive review
Ruled-out: excluding an unavailable run from the denominator | that silently redefines the population the token claim describes
Warn: one unavailable run makes the whole aggregate unavailable, which is the strict reading of §14.6 and will feel severe on a long matrix; the reasons are per run, so a report can say which ones and why
Warn: the artifact path uses `node:zlib` zstd, which exists from Node 22.15 -- above the package floor of 22, though this is bench-only and no shipped code imports it
Blast: module
Undo: easy
Certainty: firm
Verified: eighty-three cases pass across the ledger, verifier and isolation suites, covering an unparsed segment, a delegated turn, a truncated stream, an absent terminal usage, and a byte-exact round trip of recorded NDJSON through the persisted artifact; package and bench typechecks clean; two builds leave dist unchanged; both bench verifiers pass
Unverified: no containerized agent run was performed, so every stream tested here is a recorded fixture rather than a live provider
Provenance: authored
Record-Id: r-cdeb05lg
@github-actions

Copy link
Copy Markdown

CommitLore — record lint

Trailers: clean — 1 commit in origin/dev..5d347b981baf3e357abcaaf9994005f1371719f8
Active constraints: 10 limits · 23 ruled-out · 11 warnings — from 10 records over 8 changed paths

Active constraints for the paths this PR touches

Limits (10)

  • r-cdeb05lg 5d347b9 — the ledger observes only bytes emitted before the agent process closes -- terminal usage never written to the stream, external billing, and work outside that stream are all unavailable to it rather than estimated
  • r-cdeb03rt f0ad091 — enforced by code — the capability gate refuses a missing or never-probed capability, the run spec cannot express host HOME or settings inheritance, provider credentials cross the container boundary only through a two-key allowlist, the captured stream is identity-checked before anything reads it as measurement, and the pin loader rejects hand-edited enforcement shapes. Still resting on the operator — building and pulling the image the freeze records, running preflight on the machine that runs the study, keeping the container runtime daemon itself trusted, and passing provider credentials into the preflight and run calls; the gate can verify what a container does, not that the host it runs on is honest
  • r-unattshadow b7b532a — together the two features measure how often an unattended pipeline would have written, and remove the asking from the writing -- neither half can say whether what gets written is worth a reader's attention, so shadow's number for an unattended repository is a volume, not a value
  • r-cdebsurface a1e8833 — the pilot anchors now marked unverified were measured under Edit|Write|MultiEdit|NotebookEdit, not the shipping Read|Edit|Write -- the 1.45 ON/OFF overhead, the timeout wall times behind the 0.48/1.00 split and the 0.6 screen derived from it, and the zero-delivery count
  • r-delivchk 59ba928 — the runtime probe still has no implementation, so a freeze cannot yet be written even though the schema now demands its evidence
  • r-v13fix f4b6859 — schemas and the verifier now require the qualification evidence, but nothing yet produces it -- CDEB-10 must build the probe and the deliverability check before a freeze can be written
  • r-cdebv13 e0bd61a — the calibration is measured on disposable tasks, so it estimates overhead on work that is not the corpus and a corpus with different injection density will sit off it
  • r-cdebpres e01b482 — one repository, four tasks, six usable runs per arm and a local evaluator -- no claim gate is evaluable and preregistration §8 forbids any number here reaching a public surface
  • r-cdebver01 ce7b278 — the schemas freeze protocol 1.2.0 constants -- thresholds, matrix size, category names -- so a protocol change is a schema change and CI notices
  • r-cdebprd12 c3fefd2 — CDEB measures the shipping trust configuration as it exists, which grades every record claim; a directive-path result requires a different study under a different product digest

Ruled out (23)

  • r-cdeb05lg 5d347b9 — inferring terminal usage from the turns that did arrive | it would be a plausible number carrying no evidence, and plausible is exactly what makes it survive review
  • r-cdeb05lg 5d347b9 — excluding an unavailable run from the denominator | that silently redefines the population the token claim describes
  • r-cdeb03rt f0ad091 — degrading with a warning when the CLI lacks an isolation flag, as the legacy driver does | that is exactly the silent degradation this ticket exists to remove — a run that inherited the operator's settings measures one laptop, not the study
  • r-cdeb03rt f0ad091 — container networking set to none with a unix-socket egress proxy | the agent's HTTP stack cannot be assumed to speak a socket proxy, so provider reachability would fail closed on every machine; an internal network plus allowlist proxy is enforceable and probe-verified instead
  • r-cdeb03rt f0ad091 — filling the pin manifest's digest fields with values read from this machine | the digests must come from the freeze ceremony that builds the image; a hand-filled manifest is an unfrozen pin pretending, and the gate is built to refuse it
  • r-unattshadow b7b532a — keeping the unattended branch's inline prepare body | the shadow refactor exists so both entry points share one side-effect-free half, and two copies of the same hashing and policy logic would drift the first time either changed
  • r-unattshadow b7b532a — checking unattended consent in the live path only | the refusal for mode "off" already lives in the shared half of prepare, and a consent check that guards one door but not the other is no guard for the next entry point added
  • r-cdebsurface a1e8833 — copying the shipping matcher string into each arm writer | a copied string drifts the first time the product changes it, and that drift is exactly the defect this fixes -- every arm must read what the product ships
  • r-cdebsurface a1e8833 — re-deriving or deleting the pilot anchors once their surface was known to differ | the numbers stay citable as what the pilot measured on the surface it ran; re-measurement is a separate decision this change does not make
  • r-delivchk 59ba928 — qualifying delivery through commitlore context | budget, trust grading, the guard, lifecycle projection, the matcher and output parsing all sit between it and the agent, and the defect being fixed was zero delivery through exactly those
  • r-delivchk 59ba928 — requiring every good-control path to carry every record | a record scoped to one file of a multi-file change still reaches an agent that opens that file, and demanding all of them rejects sound tasks
  • r-delivchk 59ba928 — throwing on a non-zero injector exit | the hook is fail-open by design, so the exit code belongs in the freeze manifest where a reader can weigh it
  • r-v13fix f4b6859 — keeping the derived threshold with the exponent corrected | any threshold that moves with measured overhead is a threshold the study can influence, and the fixed bar is the only one that means the same thing before and after the run
  • r-v13fix f4b6859 — lowering 15% to something the pilot's overhead clears | that measures a different claim and calls it the same one; if 45% overhead makes the bar hard, the bar is reporting a true fact about the product
  • r-cdebv13 e0bd61a — raising the 15% to a number that looked achievable | picking a threshold to be clearable is the same act as picking one after seeing results, and only the derivation makes it preregistered
  • r-cdebv13 e0bd61a — keeping token efficiency in the conjunction with the new threshold | the objection was never the number, it was that one gate could erase a measured behaviour result while the report called it a three-metric study
  • r-cdebv13 e0bd61a — widening the hook matcher so the two silent tasks would deliver | that redefines a delivery failure as a delivery success, and §9.5 exists to record exactly that failure
  • r-cdebpres e01b482 — reporting the completed-only table as the pilot's result without the timed-out task | the exclusion is the registered stop_reason rule, and a reader who cannot see that a quarter of the study never finished cannot judge the rest
  • r-cdebpres e01b482 — dropping the two zero-delivery tasks as broken | they are the intention-to-treat case the protocol is built around, and they are also how the qualification gap was found
  • r-cdebver01 ce7b278 — default-in discovery over the CDEB tree like the legacy gate | that gate's enemy was the unregistered file, CDEB's is the unregistered input, and bench:m5 reads every jsonl in bench/results, so the registered analysis would run over M1, M4 and files marked non-citable #441 showed a glob passing a stopping rule on contamination
  • r-cdebver01 ce7b278 — validating with the draft-07 Ajv entry and downgrading the schemas | the 2020-12 entry ships in the same package, and weakening six schemas to fit a meta-schema is backwards
  • r-cdebprd12 c3fefd2 — reopening any of the twenty locked decisions in §29 | v1.1 locked them deliberately and every v1.2 change is additive definition, disclosure, or leak-closure
  • r-cdebprd12 c3fefd2 — enabling capture in the ON arm for realism | an agent commit mid-run would stage records unattended and change repository state between repeats, destroying the same-history invariant the whole design rests on

Warnings (11)

  • r-cdeb05lg 5d347b9 (claim) — one unavailable run makes the whole aggregate unavailable, which is the strict reading of §14.6 and will feel severe on a long matrix; the reasons are per run, so a report can say which ones and why
  • r-cdeb05lg 5d347b9 (claim) — the artifact path uses node:zlib zstd, which exists from Node 22.15 -- above the package floor of 22, though this is bench-only and no shipped code imports it
  • r-cdeb03rt f0ad091 (claim) — the two socket-level egress proxy tests skip where the sandbox denies listen — run them where binding is allowed before the freeze treats the proxy as tested; and the probe's frozen wall-time anchors were measured on the host CLI, so re-measure on the pinned runtime before any freeze relies on the 0.6 screen
  • r-unattshadow b7b532a (claim) — the consent check now lives in the shared half of prepare, so a third entry point inherits it -- but it fires only on a declaration, and shadow writes nothing because it routes around createPending and stage, not because it is exempted from consent
  • r-cdebsurface a1e8833 (claim) — 0.6 stays frozen as the runtime screen with its derivation marked unverified; if the shipping matcher ever changes the arms pick it up automatically, but every pilot anchor in the PRD needs re-measuring before a freeze relies on it
  • r-delivchk 59ba928 (claim) — delivery qualification proves the record can arrive, not that it will -- an agent that edits another path first or uses a tool outside the matcher still gets nothing, and §9.5 is what records that
  • r-v13fix f4b6859 (claim) — the 0.6 probe budget is still a judgement call with no measurement behind it, and it now gates corpus membership through two agent sessions per task
  • r-cdebv13 e0bd61a (claim) — the 0.6 probe budget and the 1.15 calibration factor are judgement calls with no measurement behind them, unlike the three findings that forced this revision
  • r-cdebpres e01b482 (claim) — the exposure counter counts deliveries rather than opportunities, so a zero cannot distinguish a hook that never fired from one that fired on a path with no records -- CDEB v1 §9.5 requires that split and CDEB-05 must build it
  • r-cdebver01 ce7b278 (claim) — every failure fixture is the one valid row with a single mutation, so keep the valid-row builder in lockstep with result.schema.json -- a builder drift fails all eleven cases at once and reads as a broken verifier
  • r-cdebprd12 c3fefd2 (claim) — the corpus does not exist yet; nothing in this commit authorises a measured run, which stays behind the freeze gate and Definition of Done

git log --follow accepts exactly one pathspec, so renames are not followed for 8 paths; query one path at a time to follow its rename chain

Trailer violations fail this check. Active constraints are informational — they are what the repository already decided, not a verdict on this PR.

@MongLong0214
MongLong0214 merged commit 51f5102 into dev Aug 11, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant