Check that evidence snippets are verbatim, instead of assuming it (#247) - #267
Conversation
EvidenceItem.snippet is specified as a verbatim quote, and CURATION_PLAYBOOK sharpens that to contiguous, no ellipsis, no paraphrase, diversified across edges. Nothing in this repo checked any of it — enforcement was a human PR reviewer — so the corpus already carried violations nobody had logged. What the corpus actually holds, across 4089 evidence items on 3402 edges: MISSING_SNIPPET 2586 a reference with no quote at all — 63% UNSUPPORTIVE_SNIPPET 71 'host', 'toxins', 'growth' ELLIPTICAL_SNIPPET 60 stitched, not contiguous [ERROR] REUSED_SNIPPET 13 one phrase across 3+ edges of a graph ECHOES_RESEARCH_REPORT 7 text taken from the trait's own report That REUSED_SNIPPET set includes 'virulence factors' on three edges of animal_pathogen — the literal example the playbook names. The decision #247 asked for: verbatim stands. A report's evidence text goes in notes:, which is where reasoning about a source belongs and has no verbatim requirement; the DOI transfers, the quote does not. Documented in the playbook, because the shortcut is otherwise the obvious way to work #183's backfill. Landed as a ratchet against conf/evidence_snippet_baseline.tsv, the same shape as audit-graphs: today's 2737 findings never fail, anything new exits 1. So it joins qc green while the backfill cannot make it worse. Canaried: injecting one elliptical snippet takes it to exit 1, removing it returns to 0. ECHOES_RESEARCH_REPORT was the hard one, and verification mattered — direction is not observable from a match alone. Two confounds, both found by hand-checking findings rather than by reading code: 257 findings naive whole-file match. Every curator-written snippet matched its own reflection, because the pipeline feeds existing evidence into the prompt and the provider echoes the prompt twice. 41 findings after trimming to the answer via research_answer() (#233). arsenic_tolerant still flagged: its quote was handed to the provider as evidence_summary, so the answer restating it says nothing about where the YAML got it. 7 findings after also subtracting what the provider was shown. Both are pinned by tests, since either regression silently re-inflates the count. Also added _safe_rel so the audit runs against a fixture tree rather than raising outside the repo root. 310 tests pass, `just qc` exits 0. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Claude finished @realmarcin's task in 6m 4s —— View job Review
The audit itself is sound. One thing needs changing. 🟡 The new tracked report is outside the staleness gate that exists for exactly this case
So a stale committed copy of this file is unobservable. Two ways out, either fine:
🔵 Smaller things
Not defects, noted
|
![culturebot-reviewer[bot]](https://avatars.githubusercontent.com/u/233210009?s=60&v=4){kind=small}
There was a problem hiding this comment.
The audit is sound — schema coverage is complete (both EvidenceItem use sites), and the direction handling that could have been quietly wrong isn't: evidence_summary is the only template_variable carrying curated snippet text, and edge snippets are never in the prompt. Confirmed the ECHOES detector is not vacuous: cellulolysis_cellulase_systems:cellobiohydrolase->cellobiose[0] is a verbatim lift of the falcon report's evidence column, ellipsis and all.
One 🟡: reports/evidence_snippet_audit.tsv is tracked and rewritten by audit-snippets earlier in the same qc run, but is not in audit-derived-reports — which is exactly the situation justfile:382-390 says forces a 'git show HEAD:' comparison, because a working-tree comparison lets a stale committed copy sail through. The baseline test does not cover it (findings ⊆ baseline is the growing direction; burn-down staleness shrinks). Either wire it into audit-derived-reports next to the cga block, or stop tracking it — it is byte-identical to conf/evidence_snippet_baseline.tsv.
Smaller: _fold strips punctuation without re-collapsing whitespace, so a standalone ellipsis leaves a double space and weakens matching on exactly the snippets most worth checking; --fail-on error is undocumented in the help; REUSED_SNIPPET's detail says 'edges' but counts evidence items; F541 on line 300.
Full detail in the review comment on the PR.
…268, #269) reports/evidence_snippet_audit.tsv was tracked and rewritten by audit-snippets during the same `just qc` run that would have to check it — the exact situation the justfile already documents as unobservable, since an earlier step refreshes the working tree and a stale committed copy sails through. It was also byte-identical to conf/evidence_snippet_baseline.tsv (md5 4891832e), so it added 2,738 duplicated lines whose only distinct value was being a snapshot, which is what #214 was filed about. Untracked and gitignored, with a test asserting it stays that way. The baseline remains tracked; it is the artifact with meaning. _fold stripped punctuation without re-collapsing whitespace, so a standalone "…" or " — " left a double space where the report's prose folded to one, and the substring test failed on exactly the snippets most worth checking against a source. `cellulolysis` matched only because the report carried the identical ellipsis. Now an elliptical snippet matches prose without one, pinned by a test. Also: --fail-on error was undocumented in the help though it is a valid choice; REUSED_SNIPPET said "N edges" while counting evidence items, which differ when one edge carries two; and a placeholder-free f-string would trip ruff F541. The detail rewording changed 13 baseline keys, so the baseline is regenerated — 13 lines, same 2737 findings. Canaried again after the changes: injecting one elliptical snippet exits 1, removing it returns to 0. 312 tests pass, `just qc` exits 0. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Claude finished @realmarcin's task in 5m 6s —— View job Review
The blocking 🟡 is resolved, and by the cleaner of the two routes I offered: Approving. Two things below are worth doing; neither should hold the merge. 🔵 The baseline key embeds an array index, which the sibling ratchet deliberately avoids
Concretely,
It fails closed in every case I could construct, which is why this is 🔵 and not blocking, and the mainline #183 move (adding a snippet to an existing item) only removes findings. But the fix is a few lines and the postmortem is already written next door. Fix this → 🔵 The "edges" → "evidence items" correction landed in the code but not where a curator reads itThe finding text is fixed, but the module docstring still says "the same snippet on several edges of one graph" ( Minor, in the same test: Noted, not defects
· branch |
There was a problem hiding this comment.
The blocking issue is resolved, by the cleaner route: reports/evidence_snippet_audit.tsv is untracked, gitignored with the reason recorded, and pinned by a test. No dangling references, so audit-derived-reports needed no change. All four smaller items landed. Baseline diff is exactly the 13 REUSED_SNIPPET rewordings; still 2,737 findings. CI green on this head.
Two 🔵s, neither blocking. (1) _key embeds the evidence-array index and the full detail string, so deleting or prepending an evidence item renumbers survivors and fails qc on a strict improvement — 114 baseline rows sit at index >= 1, and audit_causal_graphs.py:220-247 carries a written postmortem of this exact pattern. It fails closed in every case I could construct, hence not blocking. (2) The 'edges' -> 'evidence items' correction landed in the finding text but not in the module docstring (line 23) or the playbook's defect table.
Also worth knowing: the fold fix changes nothing on the corpus — 7 ECHOES and 60 ELLIPTICAL before and after — so it is pinned by its unit test, not by a detection delta.
Full detail in the review comment.
#271) The REUSED_SNIPPET finding text was corrected to count evidence items rather than edges, since one edge can carry several. The correction missed the two places someone goes to learn what the defect means: the module docstring and the playbook's defect table. Deferring #270 rather than fixing it here, deliberately. The baseline key embeds an evidence-array index, so renumbering can fail qc on a change that improved the corpus. The obvious fix — strip the index — collapses evidence[0] and evidence[1] to one key, so a THIRD missing snippet would match a baselined key and pass silently. Trading a false positive for a false negative in the integrity mechanism itself needs per-key occurrence counts and its own tests, not a tail-end edit. It fails closed today, and #183's mainline move only removes findings. 312 tests pass, `just qc` exits 0. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Claude finished @realmarcin's task in 1m 31s —— View job Review
The #270 deferral is right, and for the reason given. I checked the argument rather than taking it: stripping CI is green on this head — pytest, qc, pr-sanity, vendored-sync. Approving. 🔵 Still open, non-blockingThe one minor from last round didn't land: Noted, not defects
|
The previous commit closed sections 7 and 8 and opened section 5 — the same header-without-body failure, now in the section the header calls the largest remaining item. Three numbers were wrong against conf/causal_graph_audit_baseline.tsv, and the section still described the pre-#220 audit: files with >=1 UNREACHABLE_FROM_TRAIT said 219 is 220 UNREACHABLE_FROM_TRAIT findings said 1314 is 1321 morphology FRAGMENTED_GRAPH rows said 46 is 47 baseline total said 1314 is 1541 The 219-vs-220 discrepancy the section explained at length as a real one-file gap is gone: #220 closed in #227, and FRAGMENTED_GRAPH now catches a graph splitting into components that each carry their own TRAIT node — the dumbbell_shaped case. So the two measures now agree at 220, and the section says why rather than still explaining the old gap. Also records what the header could not: backfill progress is 1 of 220, and that one — cellulolysis — is simultaneously #183's worked example and a member of #267's ECHOES_RESEARCH_REPORT set, so it demonstrates both the target and the shortcut to avoid. Every figure re-derived from the tracked baseline rather than carried forward. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Reconcile NEXT_TASKS.md (2026-08-05) The file was written before this session and claimed the Edison sweep was "being re-run"; it completed in #241. Reconciled against what actually merged. Eight of the fifteen issues it listed are closed (#192, #193, #203, #205, #208, #214, #218, #220). Sixteen are open, and nearly all the new ones come from the reviews of the five PRs that landed since — which is the review loop working rather than a backlog blowing out. Recommends #252 next: third recurrence of one bug class (#184, #200, #250), the gate machinery is fresh from #272, and docs/WORKFLOW_CONVENTIONS.md currently has to say "nothing checks this invariant". Names #270 as runner-up and says plainly what is NOT actionable as a next item — #183's backfill is a campaign, not a PR, and the vendored-file items want the hub repo. Adds section 10 for the evidence-snippet audit landed in #267, with the frozen backlog's shape, since the 2,586 missing snippets are the same edges #183's backfill has to touch. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Reconcile the section bodies, not just the headings they sit under (#283) The first pass changed section headings and the header block and stopped there, so the file asserted both halves of three contradictions: section 8's heading said the sweep was complete while its body said "is under way" and displayed `manifest ok rows with a missing artifact: 331` as a live figure, and section 7's pending table listed six issues the header's own closed list named on the same screen. Section 8 is rewritten to what happened: 353/353 tracked, the manifest's 714 rows as a spend record with all 13 failures later succeeding, the clean --verify output including the malformed-CURIE scan #242 added, and the five issues the sweep left behind (#244, #245, #246, #248, #249) which the header table routes here but which appeared nowhere in the section. It now carries #244's caveat that --verify is narrower than its clean output suggests. Section 7's table is the current residuals — #191, #197, #198, #209, #217, #252, #275 — with the closed ones listed as closed rather than pending. Checked mechanically rather than by eye, since the file is 600 lines: zero table rows name a closed issue, and every open issue appears somewhere. That check found #283 itself missing and it is now listed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Reconcile section 5, where the header sends readers for #183 (#283) The previous commit closed sections 7 and 8 and opened section 5 — the same header-without-body failure, now in the section the header calls the largest remaining item. Three numbers were wrong against conf/causal_graph_audit_baseline.tsv, and the section still described the pre-#220 audit: files with >=1 UNREACHABLE_FROM_TRAIT said 219 is 220 UNREACHABLE_FROM_TRAIT findings said 1314 is 1321 morphology FRAGMENTED_GRAPH rows said 46 is 47 baseline total said 1314 is 1541 The 219-vs-220 discrepancy the section explained at length as a real one-file gap is gone: #220 closed in #227, and FRAGMENTED_GRAPH now catches a graph splitting into components that each carry their own TRAIT node — the dumbbell_shaped case. So the two measures now agree at 220, and the section says why rather than still explaining the old gap. Also records what the header could not: backfill progress is 1 of 220, and that one — cellulolysis — is simultaneously #183's worked example and a member of #267's ECHOES_RESEARCH_REPORT set, so it demonstrates both the target and the shortcut to avoid. Every figure re-derived from the tracked baseline rather than carried forward. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Reconcile the paragraphs the tables sit under, and drop a false example (#283, #284) Fourth round on the same defect class, so this pass swept the file mechanically instead of reacting to the sections named. Section 5's opening still said "the audit-graphs gate does not catch graph fragmentation" and "nothing currently reports it", three lines under a heading saying DETECTION DONE. audit_causal_graphs.py reports it 220 times. Rewritten to past tense, with FRAGMENTED_GRAPH added to the list of what the audit now emits. The dumbbell_shaped rationale I added last round is FALSE, and the reviewer checked it against the tree rather than taking it. That record declares exactly one TRAIT node and carries 7 UNREACHABLE_FROM_TRAIT rows, so reachability catches it fine. I had propagated it from audit_causal_graphs.py's own comment without verifying — now filed as #284. The blind spot FRAGMENTED_GRAPH closes is real, but no record exhibits it: a sweep for fragmented graphs with >1 TRAIT node and zero unreachable rows returns 0, and the two measures cover the identical 220 files. Stated abstractly, with that fact said out loud. Section 8 said "8 fail:1 rows" 22 lines above its own correct 13. Section 7's prose still had #217 as a page with "nowhere to live" and #218 as unimplemented, under a table listing both as landed. #217 now says what actually remains — where the page lives, not whether it exists. Swept for the rest: the only other stale claim was #216's "still unproven at the time of writing", which is proven now — claude-review has run to completion on every PR through #282 without cancelling itself. Mechanically checked after: zero table rows name a closed issue, every open issue appears somewhere, and #284 is listed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Fix the backfill arithmetic and a bold span I broke (#283) "1 of 220" was wrong in the direction that matters. cellulolysis.yaml has ZERO rows in conf/causal_graph_audit_baseline.tsv — it was repaired before the baseline was frozen — so the 220 there are all still to do, and "1 of 220" read as 219 remaining, contradicting the table twenty lines below. That also explains a discrepancy the file could not previously account for: #183's 2026-07-30 measurement says 220 fragmented graphs and today's baseline says 220, with one file fixed in between. Both are right because the 2026-07-30 figure was already post-cellulolysis. The corpus was 221. Now stated where the measurement is quoted, so the next reader does not have to re-derive it. The previous commit also dropped an opening ** while rewriting the paragraph above it, leaving an unmatched closer that GFM renders literally. Checked the whole file for the same shape rather than just that line — every other bold span is balanced. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Revert my own retraction: the dumbbell_shaped example was right (#283) Round 4 "corrected" section 5 by retracting the dumbbell_shaped worked example as fabricated, and filed #284 against audit_causal_graphs.py's comment on the same reasoning. Both were wrong, and the repo had written proof I did not read. v_shaped_daughters WAS typed TRAIT. Both components of dumbbell_shaped_snapping_division were therefore reachable-from-a-trait, the graph reported clean, and #227 retyped that node to QUALITY in the same change that added FRAGMENTED_GRAPH. Recorded in history/records/dumbbell_shaped/2026-08-03T230903Z-claude-code-90a277.yaml (linking issues: #220, prs: #227) and in the record's own curation_history. So the 7 UNREACHABLE_FROM_TRAIT rows I cited as disproof are the fix working. The example was accurate for the data as it stood; it is historical now, not invented. Section 5 says that, and says to read history/ alongside the record, because the current node types alone tell the opposite story. #284 is closed with the correction rather than left as a wrong record. The failure was checking the current corpus and stopping. history/ exists to explain why present data looks as it does, and I did not open it — which is also why the "no live instance" sweep is now stated as #227 having removed the one instance, rather than as the check guarding against nothing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
… (#290) * Key the snippet baseline by occurrence count, not by array index (#270) The baseline keyed on (file, locator, defect, detail), and locator embeds the evidence-array index. 114 baselined findings sat at index >= 1, so deleting a duplicate evidence item renumbered the survivors and turned unchanged findings into new ones — `just qc` failing on a change that strictly improved the corpus. The remedy a curator reaches for is --write-baseline, which re-freezes anything genuinely new in the same PR. That is how a ratchet rots. detail went too, for the same reason at a different granularity: it carries the full snippet for ELLIPTICAL/UNSUPPORTIVE and the DOI for MISSING, so retyping a still-elliptical snippet or correcting the DOI on a still-snippet-less reference also flipped the key. audit_causal_graphs.py learned this first and keys on only the leading fragment of its detail; the docstring claimed "same shape" and this was the one place it wasn't. Dropping both collapses findings onto shared keys, which is why #267 deferred the obvious fix: set membership would then let a THIRD missing snippet match a baselined pair and pass silently, trading a false positive for a false negative inside the integrity mechanism. So the baseline is read as a COUNT per key — "two of these were accepted" — and only occurrences in excess are new. Fewer than baselined is an improvement and passes. Canaried on the record the issue names. Deleting biofilm_formation's evidence[0] so evidence[1] renumbers: 2736 findings, 0 new, exit 0 — the case that used to fail. Adding a third snippet-less item: 2738 findings, 1 new, exit 1. The baseline file format is unchanged, so no regeneration: load_baseline aggregates the existing rows by key. 348 tests pass (6 new), `just qc` exits 0. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Ratchet REUSED_SNIPPET's magnitude as a value, not as identity (#291) Dropping detail from the key cost nothing on the index problem for per-item defects, and disarmed the ratchet on the one aggregate this audit emits. REUSED_SNIPPET's locator is `{graph_id}:*` with no index, and its magnitude lives entirely in detail — so a graph going from 3 shared snippets to 9 is one finding either way, same key, occurrence count unchanged, and it passed. All 13 baselined rows were exposed; gc_content is already at 5. The naive repair fails the other way: putting the count back in the key makes 3 -> 2, an improvement, an unbaselined finding that exits 1 — the exact rot #270 was about. So the magnitude is ratcheted as a VALUE. load_baseline records the worst accepted per key; compare() flags a finding whose magnitude exceeds it even when the count does not. Scoped to REUSED_SNIPPET deliberately. UNSUPPORTIVE_SNIPPET's detail also leads with an integer, but that one is a character count where larger is better, so ratcheting it would flag a snippet growing from 6 chars to 10 as a regression. MAGNITUDE_DEFECTS says which, and why. Canaried against the real baseline: 3 -> 9 flagged, 3 -> 2 and 3 -> 3 pass. 352 tests pass (4 new), `just qc` exits 0. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Make the baseline a value, and per-snippet rather than per-graph (#291) Four review items, one of them a hole the previous commit opened. The baseline-currency test — the only one comparing the committed baseline to the live corpus — still read `_key(r) not in baseline`. Against the new dict that tests key PRESENCE, so it silently stopped checking both the occurrence count and the magnitude: a third snippet-less reference, or a graph growing from 5 shared snippets to 50, would fail `just qc` while the test passed. Now uses compare(), so it asserts what qc enforces. Magnitudes were keyed per _key(), and REUSED_SNIPPET's locator is `{graph_id}:*` — so every reused snippet in one graph shared a key and a per-key max let the smaller of an uneven pair grow up to the larger unnoticed. trophic_type_classification_axes already carries two such rows, equal today. The snippet itself is now the discriminator. compare() took the baseline as an argument while reading magnitudes from a module global, which the tests exposed by having to clear and poke it by hand. load_baseline returns a Baseline NamedTuple carrying both maps, so compare() is a pure function of its arguments and the tests construct a baseline instead. The module docstring still claimed "same shape as audit_causal_graphs.py". It is not, and the divergence is the point: that shape is set membership on a 4-tuple carrying the index and a detail fragment, which is exactly what rotted here. Now says what this does and why. compare() also documents that for a count excess it names the LAST row at a key rather than the new one — the occurrences are interchangeable by construction, so no better answer exists. 353 tests pass (1 new), `just qc` exits 0, and the real-baseline canary still flags 3 -> 9 while passing 3 -> 2. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Document the magnitude key's cost (#292) Editing a REUSED_SNIPPET's shared quote produces an unseen magnitude key, so its baselined value reads as 0 and any count beats it — a rewording reported as new when nothing got worse. The obvious fallback to the graph's per-key max would restore the sheltering #291 removed, so this fails closed and is tracked rather than patched. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
… (#294) * Backfill 3 evidence-backed edges, connecting 2 fragmented graphs (#183) The detection half of #183 shipped in #185/#227; this is the first increment of the content half, and establishes the method. metabolism/dissimilatory_iron_reduction: the Fe(III)-pool pair (dissolved_fe3_om_complex -> solid_fe3_mineral) sat with no path to the trait — a free-standing geochemical rate comparison beside a respiration graph. Both are pools of the acceptor the organism reduces, so dir_trait now uses each as an electron acceptor, grounded METPO:2000008 to match the 13 existing uses-electron-donor edges of the same TRAIT -> CHEMICAL shape. morphology/sulfur_globule: persulfide_dioxygenase -> sulfite was stranded as a correct enzymatic fact with nothing tying it to globule formation. PDO acts downstream of SQR in the same sulfide-oxidation pathway the rest of the graph models, so it now participates in oxidative_sulfur_metabolism. THE SNIPPETS ARE THE POINT. #247 established that a research report is not a snippet source — its evidence text paraphrases, so what it quotes is unknowable without opening the paper — and #267 made that enforceable, which means the backfill cannot take the shortcut. Each snippet here was pulled from the PubMed abstract via efetch and checked character-for-character against it in the same run: PMID:15518832 for both iron edges, PMID:35653932 for the sulfur edge. That constraint is load-bearing rather than ceremonial. An edge added with a reference and no snippet is schema-valid and is what 63% of existing evidence items do — but it creates a new locator, so audit-snippets sees a new MISSING_SNIPPET and qc exits 1. The ratchet makes the honest path the only path. FRAGMENTED_GRAPH 220 -> 218 UNREACHABLE_FROM_TRAIT 1321 -> 1317 audit-snippets 0 new findings backfill 1 -> 3 of 221 Baseline regenerated so the ratchet tightens rather than tolerating the old count. pages/ regenerated: the stamp derives from the corpus, so appending curation history rebuilds all 489. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Narrow two claims to what their quotes establish, and record both edits (#297, #296) Review found that both snippets support their edges less directly than the surrounding prose claimed. Both edges stand; the wording did not. The dissolved Fe(III) quote is about chelators the organism SECRETES, and its second half describes reduction of oxides not in contact with the cell — a different mechanism from "chelated Fe(III) is an acceptor pool", while the node is ambient Fe(III)-organic-matter complex. The description no longer says "chelator-solubilised", and the notes now say the quote supports the pool's existence rather than its origin. The PDO quote covers heterotrophic bacteria and human mitochondria, not the Thiomargarita/Beggiatoa exemplars of this trait. It supports the generic pathway claim its object node makes, and the notes now say so and no more. Both caveats are written into the record rather than into a commit message, because that is the failure mode #183's backfill will most easily repeat at scale: a true quote attached to a slightly wider claim. Verbatim-ness does not protect against it — it is exactly what verbatim-ness cannot see. Tracked in #297. Also added the history/ records the review flagged as missing. history/README's own worked example is this issue, and the one existing record is what let the #282 review catch a false retraction the current data alone could not explain. Written by hand because `just new-history` requires a claw checkout that is not always present — tracked in #296. 353 tests pass, `just qc` exits 0. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
1 participant
Closes #247. Settles the decision it asked for, and ships the check that makes it enforceable.
The decision: verbatim stands
#247 offered three ways out — re-fetch the source, add a separate verbatim field to reports, or relax the schema. Re-fetch. A report's evidence text goes in
notes:, which is where reasoning about a source belongs and carries no verbatim requirement; the DOI transfers, the quote does not. Relaxingsnippet:was never attractive: the anti-hallucination check is the reason the field is worded as it is.That is now written into
docs/CURATION_PLAYBOOK.md, because pasting the report's evidence column is otherwise the obvious way to work through #183's backfill, and nine reports state outright that their evidence text "closely paraphrases or quotes" the source — so which of the two you copied is unknowable without opening the paper.The enforcement gap was the real finding
The requirement was documented; nothing checked it. Enforcement was a human PR reviewer, so the corpus already carried violations nobody had logged. Across 4,089 evidence items on 3,402 edges:
MISSING_SNIPPETUNSUPPORTIVE_SNIPPEThost,toxins,growthELLIPTICAL_SNIPPETREUSED_SNIPPETECHOES_RESEARCH_REPORTThe
REUSED_SNIPPETset includesvirulence factorson three edges ofanimal_pathogen— the literal example the playbook names as the thing to avoid. TheECHOESset includescellulolysis, the worked example #183 holds up as the template for what "done" looks like.Worth stating plainly: 63% of evidence items assert a mechanism on a bare DOI.
snippet:is schema-optional so those are valid records, but burning that down is the same job as #183's backfill, on the same edges.A ratchet, so it lands green
conf/evidence_snippet_baseline.tsvfreezes today's 2,737 findings and only new ones fail — the same shape asaudit-graphs(#185). It joinsqcpassing, and the backfill cannot make it worse. Canaried: injecting one elliptical snippet takes it to exit 1; removing it returns to 0.Direction was the hard part, and hand-checking found what reading didn't
A curated snippet appearing in a report proves nothing on its own. Two confounds, each found by inspecting actual findings:
evidence_summary, and the provider echoes the whole prompt twice.research_answer()(reused from The per-trait research block never renders: the renderer and the research pipeline disagree on the filename #233).arsenic_tolerantstill flagged, and I checked it: its quote was handed to the provider in the prompt, so the answer restating it says nothing about where the YAML got it.Both confounds are pinned by tests, since either regression silently re-inflates the count. The finding is worded as a prompt to verify, not a verdict — a report may legitimately quote the same sentence you did.
Verification
Also added
_safe_relso the audit runs against a fixture tree instead of raising outside the repo root.🤖 Generated with Claude Code