Skip to content

v0.5.0 — cross-family Pass I, and the convergence gaps a finished-looking corpus hides

Latest

Choose a tag to compare

@chrisgagne chrisgagne released this 21 Aug 19:57

Everything here comes from auditing a 55-source corpus end to end. Each change is the shortest fix for something that actually went wrong, and the numbers are measured rather than estimated.

A fresh Pass I subagent is necessary and not sufficient

The protocol has always required Pass I to run in a fresh subagent — never the session that produced the reference. That removes context leakage: a session cannot see the training-priors it just wrote in as source-grounded, so it certifies its own leakage.

It does not remove prior leakage. An auditor from the same model family holds the same background knowledge, finds the generator's leaked domain reasoning unremarkable, and passes it through as though the source had supplied it.

Forty-four references were audited by a fresh same-family subagent, then re-audited blind by a different model family before being shown the first audit's findings:

Confirmed by both 503
Missed by the same-family audit 221
Wrongly flagged by the same-family audit 5

Roughly 30% of real defects survive a cold same-family audit, at a false-positive rate under 1% — near-perfect precision, with a hole in recall shaped exactly like the generator's own priors. The rate held between 27.8% and 32.8% across nine independent batches.

Pass I now requires a cross-family leg, blind-first: the second auditor traces the reference against the source and writes its own findings before reading the first audit, because an anchored second opinion is not a second opinion.

The class that gets missed, now named

Source-external analytical framing — domain-prior explanation wearing the source-grounded voice. It survives audit because it is usually reasonable and often true. Pass I's fail conditions now enumerate the forms that actually recurred: evidence-quality rankings the source never makes, plausibility judgements, methodological verdicts, evaluative labels, causal explanation the source reports without giving, invented oppositions, and market or legal framing in a deep reference.

One example carries the whole class. A paper performs a processing step without citing anyone; the audit inferred the step was therefore established practice. Absence of a citation establishes only that the paper does not cite.

The test is not "is this true?" but "which passage says it?"

Two template slots were generating that framing

Positions the author explicitly frames against and Connections the author makes in the text were unconditional headings with bulleted placeholders, no None. branch, and nothing in Pass E saying an empty section is normal. A model handed a slot fills the slot.

42 of 55 references in the audited corpus carried invented oppositions there. The shipped demo corpus shows the same shape. Both sections now have an explicit None. branch, and Pass E states that most research papers argue against nobody.

[V] discipline: two rules that were implicit

  • A translation can never carry [V]. It is the ingester's wording, however faithful. One Chinese-language reference went from 51 verbatim markers to zero on this rule alone.
  • On OCR'd sources, check reading order before trusting a character match. Two-column conversions scramble line order, so a passage can be present word-for-word and exist nowhere as a contiguous sequence. Matching words is not verifying a quotation.

Pass H was missing its image-index assembly step

Image classification runs per source into staging, and nothing folded that into IMAGE-INDEX.yaml. A corpus with 46 careful staging files had no index at all — and by the skill's own rule, an image absent from that index is not part of the library. New script scripts/build_indexes/build_image_index.py normalises the four staging schemas found in the wild, indexes only substantive entries, verifies every indexed file exists, and refuses to replace a populated index with an empty one.

The convergence steps are the ones that get skipped

Per-source agents write staging faithfully and report success. The single corpus-level run at the end of a parallel batch belongs to nobody's task list. One corpus had missed four at once: no image index, 13 router rows still sitting in _ingest_*.md stubs so five distillations were unreachable from the runtime router, concept-index --assemble never run, 20 sidecars unwritten. Nothing looked broken from any per-source view.

Pass H now carries a convergence checklist, plus the two failure shapes no script reports: leftover _ingest_*.md stubs inside distillations/{task}/, and a stray --- splitting a phase table so the builder silently stops seeing rows after it.

Cross-reference rows are the least-verified content in a corpus

Pass H writes claims about other sources into distillations, and nothing checks them. Pass I verifies a reference against its own source; no pass verifies a distillation against a different reference's source. The same applies to corpus-scope superlatives. Now stated in the protocol rather than left to be mistaken for verified.

Operational lore

A stalled agent is usually not a failed agent — 5 of 7 reported failures had completed their writes. But an agent that truly dies mid-pass leaves no broken grammar, only confidently wrong classifications under an unearned stamp, and the retry inherits them blind. Agents now write their artefact before reporting and stamp only after it exists, so a stamp implies a completed pass.

docs/images/ is now gitignored: the image staging path is repo-root rather than corpus-root, so a private corpus's per-figure descriptions were landing in a tree that pushes to a public remote.

Note: package.json still reads 0.4.0; it carries unrelated in-flight work and was deliberately left untouched.