Stop the citation gate passing on citations it could not verify - #260
Merged
Conversation
The protocol doc cites ~59 source line ranges and a required CI step checks they still point at the code they claim. It ended `return 1 if drifts else 0`, so the "unverified" category was structurally incapable of failing: the gate was most permissive exactly where its own confidence was lowest. Measured before this change: 57 ok, 0 drift, 2 unverified, exit 0. Both unverified citations are genuinely WRONG, hand-verified: - the doc's conversation_start section points at lines covering AllScansComplete; the model it names is 13 lines further down. - the doc says "the client-type registry has exactly one entry" and points at the SERVER registry, which has eight. Neither is merely unmechanisable. They reach the unverified bucket because every backticked hint in one context window is in the skip list and both cited lines sit at module level, so neither resolution pass fires. The interaction laundered real drift into a category that could not fail. The verdict is now three-valued. Drift stays exit 1 -- the gate did its job and found something. Anything the gate could not verify is exit 2 -- the gate could not do its job -- and the two are named differently in the report so they cannot be confused. A run that extracted or graded zero citations is also exit 2: an empty graded set reports clean having checked nothing, so every run prints checked=N of M extracted and asserts N > 0 and N == M. The two known-bad citations are registered in a ratcheted citation-baseline.json so the gate can mount before the doc repair it demands. Each entry carries the finding (what is wrong, with the corrected range) and the reason (why it is registered rather than fixed here), and its identity is the exact 4-tuple of doc line, path and cited range -- so widening a range, moving the paragraph or repairing the citation all stop matching and are reported stale rather than silently inheriting the blessing. The registry can never launder a drift; it covers only the unverifiable bucket. Adding an entry requires bumping frozen_count. The unverified report is now actionable instead of opaque. It names the nearest indexed symbol above and below the cited line with their ranges, and every doc hint it found and rejected -- including, by name, the ones rejected for being in the skip list. That opacity is why these two sat unnoticed. The gate outgrew the 500-line file law once the honest contract and the actionable report landed, so it is split by question rather than by size: what the doc says, what the source says, what a verdict is, and what the registry blesses. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CONTRACT
scripts/check_protocol_doc_citations.pyverifies that the ~59src/bonfire/...py:NNcitations indocs/scan-front-door-protocol.mdstill point at the code they claim. It is a required CI step (ci.yml, jobtest (3.12), required by the repository ruleset).It ended
return 1 if drifts else 0. Theunverifiedcategory was therefore structurally incapable of failing — the gate was most permissive exactly where its own confidence was lowest.WORLD — measured before the change
Both are genuinely wrong. Hand-verified:
conversation_startmodelclass AllScansComplete, its header and docstring openingConversationStartis at 120-123_SERVER_TYPES— the server registry, which has eight entries_CLIENT_TYPESis at 188-190Neither is merely unmechanisable. They land in
unverifiedbecause every backticked hint in one context window is in the skip list, and both cited lines sit at module level, so neither resolution pass fires. That interaction laundered real drift into a category that could not fail. There was also no non-vacuity assertion: a doc with zero extractable citations printed0 ok, 0 drift, 0 unverified, 0 totaland exited 0.The honest contract
checked > 0, andchecked == extractedunverified; absent/unparseable cited source; unreadable doc; malformed registry; count overfrozen_count; stale registry entry; zero citations extracted or graded1outranks2, and every blocker still prints under an exit-1 verdict, so nothing is masked. Every run now printschecked=N of M extractedandNon-vacuity: PASS|FAIL.The registry, and why it cannot drift onto a different citation
citation-baseline.json—{frozen_count, entries:[{doc_line, path, cited_start, cited_end, finding, reason}]}, mirroring the shapeexemptions.jsonalready uses.(doc_line, path, cited_start, cited_end). Every field a scalar; glob characters inpathare rejected outright;cited_endis required, so widening a range (107-110→107-140) stops matching instead of inheriting the blessing.finding/reasonis rejected as a silent suppression; duplicate identities are rejected because they inflate the ratchet while blessing one citation.unverifiedbucket. It can never launder a drift — a citation that resolves and is wrong exits 1 regardless of what the registry says, and a test pins that.findingstates what is wrong with the corrected range;reasonstates why it is registered rather than repaired here. Neither claims the citations are merely unmechanisable.The report is now actionable
The old message was
not inside any indexed symbol and no symbol hint resolvable. Now:That opacity is why these two sat unnoticed.
FAILURE — the control rods, run by hand
Rod 1 — remove the registry. The two known-bad citations must block.
Rod 2 — point it at a doc with no citations. It must not report clean.
The remaining six rods (drift → 1, absent source → 2, registry-cannot-launder-drift → 1,
frozen_countnot bumped → 2, real doc + real registry → 0 withchecked > 0, registered unverified → 0 with its reason printed) plus 13 supporting rods: 19 passed.The gate at harvest
Full kit battery from the repo root, at the new kit pin this rung stacks on:
ruff-formatwas red on one cosmetic line join in a new file and was fixed with the kit's pinned formatter; the board above is the post-fix state. LawGiver pre-gate:files_checked=8 added_lines_checked=1443,gauges unrun=(none), exit 0.Why four new modules
The honest contract plus the actionable report took the single file to 764 measured lines against a hard 500 cap that compression cannot buy down (the gauge measures
max(physical, statements)). Split by question, not by size, with no import cycles: what the doc says · what the source says · what a verdict is and what it exits · what the registry blesses. Final sizes 382 / 156 / 137 / 270 / 279 — every one under the cap. No number infile-budget.jsonwas touched, and neitherscripts/nortests/scripts/draws against a package total.The one existing assertion that changed, with its reason
test_current_doc_has_no_driftassertedrc == 0, and that 0 was satisfied by 57 ok plus 2 laundered unverified. Replaced bytest_real_doc_and_real_registry_exit_zero_on_a_nonempty_graded_set, which assertsrc == 0and parseschecked=N of M extractedassertingN > 0andN == MandNon-vacuity: PASS. Strictly stronger; the reason is written inline. No other assertion was weakened.🔴 Findings for a maintainer — deliberately not fixed here
docs/scan-front-door-protocol.md:179→107-110becomes 120-123, and:266→175-177becomes 188-190. Both are recorded in the registry'sfindingfields. Landing the gate is this change; repairing the doc is the doc's own work — and that follow-up must delete the two registry entries in the same PR and lowerfrozen_countto 0, or CI goes red at 2 on a stale entry. That coupling is deliberate.doc_lineis part of the entry identity, so inserting a paragraph above doc line 179 invalidates both entries until they are re-confirmed. That is the point — the doc moving is the failure mode — but it is real friction on doc edits.scripts/is lint-graded only by the kit workflow. The repo's ownci.ymlrunsruff check src/ tests/, so the gate tooling that now decides whethermainis mergeable is lint/format-graded exclusively byquality.yml's tree-wideruff check ., and never bymypyorcomplexipy(both source-root-scoped).file-budget.json'spackagesblock covers neitherscripts/nortests/scripts/, so sibling-file accretion there draws against no ceiling — the kit's own documented relocation-only open issue, now exercised by adding four files toscripts/.Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com