feat(pipeline): verification cannot lie — a blind empty diff never reads as "no changes" - #352
Merged
Merged
Conversation
…ound The harness enumerates the provider x isolation x class x model x witness x budget x bypass cross-product (288 scenarios) and asserts two invariants on EVERY one: (1) named termination — a run ends only by Ok(outcome) or a typed Err, an Ok(Aborted) carries a non-empty reason, and a terminal Complete/Error reaches the stream; (2) never choose nothing — a run that executed zero worker turns and did not complete may only be a RECOGNIZED cause (budget, scope-user-abort, unresolvable provider, or a deliberate witness artifact-integrity fail-closed), never a silent setup death. Deterministic, milliseconds, and a break names the exact scenario. Building it earned its keep immediately. My first cut was vacuous — it inferred "worker executed" from transcript length, but run() appends the user message before execution, so the check never fired. Fixed to key on the worker's StepUsage event, the harness went red on a 7th zero-work death that #345 had missed: witness AUTHORING failures (no TEST_COMMAND, an unusable command, a test that proves nothing, the author engine getting stuck) aborted the whole task. The task needs no witness to proceed — so these now degrade to a bare worker run. Only witness artifact-INTEGRITY violations (the author modified tracked files, produced a non-single-file/symlink artifact, or a runner/identity mismatch) stay fail-closed, surfacing the problem instead of silently completing unverified. witness_stage now returns a typed WitnessAbort { reason, degradable } to draw that line. Tests reconciled: the still-passes and metering cases degrade (author+repair still individually metered); the production-edit security case is scripted to reach the tracked-mutation check and stays fail-closed. Signed-off-by: macanderson <mac@oxagen.sh>
…ads as "no changes" Principle #4 of the loop-hardening thread, and the scariest failure this codebase hit: a wrong verification signal does not merely fail, it MISDIRECTS. On Terminal-Bench the judge was handed an empty diff and concluded "no changes were made to the repository" while the agent's files sat on disk — the agent's rational response was to reinitialize git to make the signal non-empty. An empty diff is ambiguous: the agent genuinely changed nothing, OR the diff machinery is blind (work committed, a baseline miss, an uncaptured file). The pipeline already carries the signal that resolves it — file_changes, the count of FileChange events the turn emitted. When that is positive but the gathered diff is empty, verification_honest_diff replaces the bare empty string with an explicit "the tree changed but the diff could not be captured — this is NOT evidence that nothing changed" marker. Applied at both diff-gather points (initial verify and post-revise), so every downstream consumer — the model judge, the distress-guidance call, the recorded evidence — sees a "couldn't verify", never a false "verified nothing". The judge's evidence summary now also states the file-change count explicitly. This is the confidence notion the principle calls for: distinguish "verified false" from "couldn't verify" so a blind signal can never be read as a negative one. The deterministic paths were already honest here (the zero-diff guard keyed on file_changes, not the diff). Tests pin the pure guard: a blind empty diff with file changes reports "uncaptured" and forecloses the "no changes" reading; a truly empty diff with zero file changes stays empty (no invented changes); a real diff passes through untouched. Signed-off-by: macanderson <mac@oxagen.sh>
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Sorry @macanderson, you have reached your weekly rate limit of 500000 diff characters.
Please try again later or upgrade to continue using Sourcery
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
macanderson
marked this pull request as ready for review
July 23, 2026 06:58
macanderson
added a commit
that referenced
this pull request
Jul 23, 2026
* fix(cgp): declare §B3-honest frame token_cost (budget_tokens) The CGP pin rebase (#354) claimed to adopt "canonical token accounting" but kept the old estimate: the store declared `estimate_tokens(content) + estimate_tokens(title)` and the graph declared `estimate_tokens(content).max(1)`. Under the pinned protocol, §B3 requires `token_cost == budget_tokens(content)` — the canonical inline count (ceil(bytes/4)), exact, no tolerance. Neither builder was honest: the store added the title's tokens (always non-zero) and used chars not bytes; the graph's `.max(1)` diverges on empty content. The existing conformance gate didn't catch it: its probe query returns zero frames from the mismatched seed, so §B3 never fired on a real frame. Fix: both production builders now declare `budget_tokens(content)`. `pack_to_budget` already packs against each frame's own token_cost, so the summed-budget invariant is preserved. New `recalled_frames_declare_honest_token_cost` drives the real `recall` builder with a query that provably surfaces a frame and asserts `declares_honest_token_cost()` — it fails on the pre-fix code (token_cost 18 vs canonical 15) and passes after. A matching assertion guards the graph builder. Dead `estimate_tokens` helpers removed. * fix(ci): clear two pre-existing clippy -D-warnings blocking CI `main` (b0117f3) fails `cargo clippy --workspace --all-targets -- -D warnings` — the exact CI gate — on two warnings that landed unprotected: - `run_shared_candidates` (pipeline.rs, from #352) lacks the `#[allow(clippy::too_many_arguments)]` its five sibling functions in the same file already carry. - `use contextgraph_types::Representation` (contextgraph.rs, from #354) is used only in a `#[cfg(test)]` helper, so it is unused in the bin build. Import removed; the one test use is fully-qualified. Both unrelated to the token_cost fix but required for this PR's CI to be green.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Principle #4 of the loop-hardening thread — the scariest one, because a wrong verification signal doesn't just fail, it misdirects.
The lie
On Terminal-Bench the judge was handed an empty diff and concluded "no changes were made to the repository" — while the agent's files sat on disk. The agent's rational response to a false signal was to reinitialize git to make the signal non-empty. Verification didn't just miss; it actively steered the agent wrong.
Why an empty diff is a lie waiting to happen
Empty is ambiguous: the agent genuinely changed nothing, or the diff machinery is blind (work committed, a baseline miss, an uncaptured file). Handed a bare empty string, a judge reads the second case as the first.
The pipeline already carries the signal that resolves the ambiguity:
file_changes, the count ofFileChangeevents the turn emitted.verification_honest_diffuses it — whenfile_changes > 0but the gathered diff is empty, it substitutes:Applied at both diff-gather points (initial verify + post-revise), so every downstream consumer — the model judge, the distress-guidance call, the recorded evidence — sees a "couldn't verify", never a false "verified nothing". The judge's evidence summary now states the file-change count explicitly too.
That's the confidence notion the principle calls for: distinguish "verified false" from "couldn't verify" so a blind signal can never be read as a negative one. (The deterministic paths were already honest — the zero-diff guard keyed on
file_changes, not the diff.)Tests
Pin the pure guard: a blind empty diff with file changes reports "uncaptured" and forecloses the "no changes" reading; a truly empty diff with zero file changes stays empty (no invented changes); a real diff passes through untouched.
cargo test -p stella-pipeline(156) +clippy -D warningsgreen; the chaos harness still passes.Note
The root-cause diff-blindness (bare
git diffblind to committed work) was fixed earlier by diffing against the session baseline commit. This is the structural guard on top: even if some future diff path goes blind again, it can no longer be read as "nothing changed".