Skip to content

Stop the citation gate passing on citations it could not verify - #260

Merged
Antawari merged 1 commit into
mainfrom
catrina/gates-r4-citations-cannot-fail-open
Jul 29, 2026
Merged

Stop the citation gate passing on citations it could not verify#260
Antawari merged 1 commit into
mainfrom
catrina/gates-r4-citations-cannot-fail-open

Conversation

@Antawari

Copy link
Copy Markdown
Contributor

Stacked on #258. Base is catrina/gates-r3-bump-the-kit-pin, so this diff is this rung only. Merge #258 first; GitHub will retarget this to main.

CONTRACT

scripts/check_protocol_doc_citations.py verifies that the ~59 src/bonfire/...py:NN citations in docs/scan-front-door-protocol.md still point at the code they claim. It is a required CI step (ci.yml, job test (3.12), required by the repository ruleset).

It ended return 1 if drifts else 0. The unverified category was therefore structurally incapable of failing — the gate was most permissive exactly where its own confidence was lowest.

WORLD — measured before the change

2 citation(s) could not be mechanically verified:
  doc line 179: src/bonfire/onboard/protocol.py:107 (...no symbol hint resolvable (hint=None))
  doc line 266: src/bonfire/onboard/protocol.py:175 (...no symbol hint resolvable (hint=None))

Summary: 57 ok, 0 drift, 2 unverified, 59 total citations checked.
CITATION_EXIT=0

Both are genuinely wrong. Hand-verified:

doc says cited range actually covers truth
:179 the conversation_start model the blank line before class AllScansComplete, its header and docstring opening ConversationStart is at 120-123
:266 "the client-type registry has exactly one entry" the Parsing banner, a blank line, and the first line of _SERVER_TYPES — the server registry, which has eight entries _CLIENT_TYPES is at 188-190

Neither is merely unmechanisable. They land in unverified because every backticked hint in one context window is in the skip list, and both cited lines sit at module level, so neither resolution pass fires. That interaction laundered real drift into a category that could not fail. There was also no non-vacuity assertion: a doc with zero extractable citations printed 0 ok, 0 drift, 0 unverified, 0 total and exited 0.

The honest contract

exit meaning fires on
0 clean every citation resolved or registered, and checked > 0, and checked == extracted
1 DRIFT — the gate did its job a citation resolved to a symbol and the cited line is wrong
2 COULD NOT VERIFY — the gate could not do its job unregistered unverified; absent/unparseable cited source; unreadable doc; malformed registry; count over frozen_count; stale registry entry; zero citations extracted or graded

1 outranks 2, and every blocker still prints under an exit-1 verdict, so nothing is masked. Every run now prints checked=N of M extracted and Non-vacuity: PASS|FAIL.

The registry, and why it cannot drift onto a different citation

citation-baseline.json{frozen_count, entries:[{doc_line, path, cited_start, cited_end, finding, reason}]}, mirroring the shape exemptions.json already uses.

  • Identity is the exact 4-tuple (doc_line, path, cited_start, cited_end). Every field a scalar; glob characters in path are rejected outright; cited_end is required, so widening a range (107-110107-140) stops matching instead of inheriting the blessing.
  • Move the paragraph, move the citation, or repair the range and the entry matches nothing → reported stale → exit 2. A blessing cannot silently follow the doc somewhere nobody re-read it.
  • Unknown keys are a hard error ("a key this gate does not read cannot carry a claim"); blank finding/reason is rejected as a silent suppression; duplicate identities are rejected because they inflate the ratchet while blessing one citation.
  • The registry covers only the unverified bucket. It can never launder a drift — a citation that resolves and is wrong exits 1 regardless of what the registry says, and a test pins that.
  • finding states what is wrong with the corrected range; reason states why it is registered rather than repaired here. Neither claims the citations are merely unmechanisable.

The report is now actionable

The old message was not inside any indexed symbol and no symbol hint resolvable. Now:

gate saw: cited line 107 is not inside any indexed symbol;
  nearest symbol above: ScanComplete (lines 92-105);
  nearest symbol below: AllScansComplete (lines 108-117);
  doc hints found and rejected: `falcor_message` (listed in citation_doc.SKIP_HINTS),
    `question` (listed in citation_doc.SKIP_HINTS),
    `all_scans_complete` (listed in citation_doc.SKIP_HINTS);
  FIX: re-point the citation at the symbol the doc names, or name that symbol in
  backticks within the preceding doc lines so the hint pass can resolve it

That opacity is why these two sat unnoticed.

FAILURE — the control rods, run by hand

Rod 1 — remove the registry. The two known-bad citations must block.

Summary: 57 ok, 0 drift, 0 registered-unverifiable, 2 unregistered-unverifiable, 0 blocked, checked=59 of 59 extracted.
Non-vacuity: PASS — checked=59 (> 0 required).
Verdict: exit 2 — COULD NOT VERIFY. This is NOT 'the gate did its job and found drift' (exit 1); it is 'the gate could not do its job':
  - unregistered unverifiable citation: doc line 179, ... — resolve it, or register it with a reason and bump frozen_count
  - unregistered unverifiable citation: doc line 266, ... — resolve it, or register it with a reason and bump frozen_count
EXIT=2

Rod 2 — point it at a doc with no citations. It must not report clean.

Summary: 0 ok, 0 drift, 0 registered-unverifiable, 0 unregistered-unverifiable, 0 blocked, checked=0 of 0 extracted.
Non-vacuity: FAIL — checked=0 (> 0 required).
Verdict: exit 2 — COULD NOT VERIFY ...
  - stale citation-baseline.json entry: doc line 179 ... matches no unverifiable citation
  - stale citation-baseline.json entry: doc line 266 ... matches no unverifiable citation
  - the gate GRADED NOTHING: 0 citations extracted from LICENSE. A gate that grades an empty
    set reports clean having checked nothing, so an empty set is a failure, not a pass
EXIT=2

The remaining six rods (drift → 1, absent source → 2, registry-cannot-launder-drift → 1, frozen_count not bumped → 2, real doc + real registry → 0 with checked > 0, registered unverified → 0 with its reason printed) plus 13 supporting rods: 19 passed.

The gate at harvest

Full kit battery from the repo root, at the new kit pin this rung stacks on:

PASS  ruff-check          PASS  cf-exemptions        PASS  mypy
PASS  ruff-format         PASS  cf-no-bon-ref        PASS  complexipy
PASS  cf-sticky-check     PASS  cf-import-contract   PASS  pytest
PASS  cf-file-budget      PASS  cf-mirror-check      PASS  cf-recursion-check

cf-gate: PASS — every gate is clean   (exit 0)

ruff-format was red on one cosmetic line join in a new file and was fixed with the kit's pinned formatter; the board above is the post-fix state. LawGiver pre-gate: files_checked=8 added_lines_checked=1443, gauges unrun=(none), exit 0.

Why four new modules

The honest contract plus the actionable report took the single file to 764 measured lines against a hard 500 cap that compression cannot buy down (the gauge measures max(physical, statements)). Split by question, not by size, with no import cycles: what the doc says · what the source says · what a verdict is and what it exits · what the registry blesses. Final sizes 382 / 156 / 137 / 270 / 279 — every one under the cap. No number in file-budget.json was touched, and neither scripts/ nor tests/scripts/ draws against a package total.

The one existing assertion that changed, with its reason

test_current_doc_has_no_drift asserted rc == 0, and that 0 was satisfied by 57 ok plus 2 laundered unverified. Replaced by test_real_doc_and_real_registry_exit_zero_on_a_nonempty_graded_set, which asserts rc == 0 and parses checked=N of M extracted asserting N > 0 and N == M and Non-vacuity: PASS. Strictly stronger; the reason is written inline. No other assertion was weakened.

🔴 Findings for a maintainer — deliberately not fixed here

  1. The two doc citations are still wrong. The repair is docs/scan-front-door-protocol.md:179107-110 becomes 120-123, and :266175-177 becomes 188-190. Both are recorded in the registry's finding fields. Landing the gate is this change; repairing the doc is the doc's own work — and that follow-up must delete the two registry entries in the same PR and lower frozen_count to 0, or CI goes red at 2 on a stale entry. That coupling is deliberate.
  2. doc_line is part of the entry identity, so inserting a paragraph above doc line 179 invalidates both entries until they are re-confirmed. That is the point — the doc moving is the failure mode — but it is real friction on doc edits.
  3. A stale registry entry is exit 2, not a notice. Judged deliberately strict: an entry is a claim about one citation, and if it matches nothing then either the doc moved without the blessing being re-read, or the citation was repaired and the ratchet owes a shrink.
  4. scripts/ is lint-graded only by the kit workflow. The repo's own ci.yml runs ruff check src/ tests/, so the gate tooling that now decides whether main is mergeable is lint/format-graded exclusively by quality.yml's tree-wide ruff check ., and never by mypy or complexipy (both source-root-scoped).
  5. file-budget.json's packages block covers neither scripts/ nor tests/scripts/, so sibling-file accretion there draws against no ceiling — the kit's own documented relocation-only open issue, now exercised by adding four files to scripts/.

Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com

The protocol doc cites ~59 source line ranges and a required CI step checks they
still point at the code they claim. It ended `return 1 if drifts else 0`, so the
"unverified" category was structurally incapable of failing: the gate was most
permissive exactly where its own confidence was lowest.

Measured before this change: 57 ok, 0 drift, 2 unverified, exit 0.

Both unverified citations are genuinely WRONG, hand-verified:

- the doc's conversation_start section points at lines covering AllScansComplete;
  the model it names is 13 lines further down.
- the doc says "the client-type registry has exactly one entry" and points at the
  SERVER registry, which has eight.

Neither is merely unmechanisable. They reach the unverified bucket because every
backticked hint in one context window is in the skip list and both cited lines
sit at module level, so neither resolution pass fires. The interaction laundered
real drift into a category that could not fail.

The verdict is now three-valued. Drift stays exit 1 -- the gate did its job and
found something. Anything the gate could not verify is exit 2 -- the gate could
not do its job -- and the two are named differently in the report so they cannot
be confused. A run that extracted or graded zero citations is also exit 2: an
empty graded set reports clean having checked nothing, so every run prints
checked=N of M extracted and asserts N > 0 and N == M.

The two known-bad citations are registered in a ratcheted citation-baseline.json
so the gate can mount before the doc repair it demands. Each entry carries the
finding (what is wrong, with the corrected range) and the reason (why it is
registered rather than fixed here), and its identity is the exact 4-tuple of doc
line, path and cited range -- so widening a range, moving the paragraph or
repairing the citation all stop matching and are reported stale rather than
silently inheriting the blessing. The registry can never launder a drift; it
covers only the unverifiable bucket. Adding an entry requires bumping
frozen_count.

The unverified report is now actionable instead of opaque. It names the nearest
indexed symbol above and below the cited line with their ranges, and every doc
hint it found and rejected -- including, by name, the ones rejected for being in
the skip list. That opacity is why these two sat unnoticed.

The gate outgrew the 500-line file law once the honest contract and the
actionable report landed, so it is split by question rather than by size: what
the doc says, what the source says, what a verdict is, and what the registry
blesses.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Base automatically changed from catrina/gates-r3-bump-the-kit-pin to main July 29, 2026 16:55
@Antawari Antawari closed this Jul 29, 2026
@Antawari Antawari reopened this Jul 29, 2026
@Antawari
Antawari merged commit b3b611b into main Jul 29, 2026
5 checks passed
@Antawari
Antawari deleted the catrina/gates-r4-citations-cannot-fail-open branch July 29, 2026 17:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant