Make CDEB token accounting fail closed - #524
Merged
Merged
Conversation
A cost number that looks measured and is not is worse than no number, and the substitution is quiet: a gap filled with a mean, a zero, or a per-token extrapolation reads as diligence. CDEB-05 therefore gives usage exactly two states. Measured means every NDJSON segment parsed, every turn carried final usage, the terminal usage object existed, and the two totals reconciled exactly. Anything else is unavailable, and an unavailable row carries no numeric token field at all -- the row schema has no place to put one, so a reader cannot mistake a diagnostic for data. Dropping the row would be the same defect one layer up, so aggregation propagates the gap instead of summing what is left. §14.6 already says an unrecoverable run stays in the denominator; a smaller total assembled from the runs that happened to parse is a different population than the one that ran. The models recorded are the ones observed answering, not the ones requested. Drift between the two is what the pinned runtime exists to catch, and the ledger is where it becomes evidence rather than an assumption; the verifier now binds a row to the retained raw bytes, so a summary whose stream was not kept is a finding rather than a claim. A timeout is not an excuse to synthesize a total. When SIGTERM lets the CLI emit terminal usage the parser uses it, and when it does not the outcome says so. Limit: the ledger observes only bytes emitted before the agent process closes -- terminal usage never written to the stream, external billing, and work outside that stream are all unavailable to it rather than estimated Ruled-out: inferring terminal usage from the turns that did arrive | it would be a plausible number carrying no evidence, and plausible is exactly what makes it survive review Ruled-out: excluding an unavailable run from the denominator | that silently redefines the population the token claim describes Warn: one unavailable run makes the whole aggregate unavailable, which is the strict reading of §14.6 and will feel severe on a long matrix; the reasons are per run, so a report can say which ones and why Warn: the artifact path uses `node:zlib` zstd, which exists from Node 22.15 -- above the package floor of 22, though this is bench-only and no shipped code imports it Blast: module Undo: easy Certainty: firm Verified: eighty-three cases pass across the ledger, verifier and isolation suites, covering an unparsed segment, a delegated turn, a truncated stream, an absent terminal usage, and a byte-exact round trip of recorded NDJSON through the persisted artifact; package and bench typechecks clean; two builds leave dist unchanged; both bench verifiers pass Unverified: no containerized agent run was performed, so every stream tested here is a recorded fixture rather than a live provider Provenance: authored Record-Id: r-cdeb05lg
CommitLore — record lintTrailers: clean — 1 commit in Active constraints for the paths this PR touchesLimits (10)
Ruled out (23)
Warnings (11)
Trailer violations fail this check. Active constraints are informational — they are what the repository already decided, not a verdict on this PR. |
This was referenced Aug 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CDEB-05. Depends on #520 (merged).
A cost number that looks measured and is not is worse than no number, and the substitution is quiet: a gap filled with a mean, a zero, or a per-token extrapolation reads as diligence.
docs/SELF-AUDIT.mdexists because of exactly that shape.So usage has two states and no third:
measuredunavailableThe schema split is a
oneOfonavailability. The unavailable branch carriesreasonsand the raw stream digest, and not one token field.Dropping the row is the same defect one layer up
Aggregation propagates the gap rather than summing what is left. §14.6 already says an unrecoverable run stays in the denominator; a smaller total assembled from the runs that happened to parse describes a different population than the one that ran.
Observed, not requested
The models recorded are the ones observed answering. Drift between requested and observed is what the pinned runtime (#520) exists to catch, and the ledger is where it becomes evidence rather than an assumption. The verifier now binds a row to the retained raw bytes — decompress, digest, compare against both the sidecar and the row — so a summary whose stream was not kept is a finding, not a claim.
A timeout is not an excuse to synthesize a total. When SIGTERM lets the CLI emit terminal usage the parser uses it; when it does not, the outcome says so.
Two things worth arguing with
One unavailable run makes the whole aggregate unavailable. That is the strict reading of §14.6 and it will feel severe on a long matrix. The reasons are per run, so a report can always say which ones and why — but the headline number refuses rather than shrinks.
node:zlibzstd needs Node 22.15, above the package floor of 22. This is bench-only; no shipped code imports it. CI runs 22 and 24.83 cases pass across the ledger, verifier and isolation suites — an unparsed segment, a delegated turn, a truncated stream, an absent terminal usage, and a byte-exact round trip of recorded NDJSON through the persisted artifact. Both typechecks clean, two builds leave
distunchanged, both verifiers pass.Every stream tested here is a recorded fixture; no containerized agent run was performed.