Gate snippets truncated at a genus abbreviation; fix the four found (#299) - #303
Merged
Conversation
…299) **The obvious fix would have been wrong.** #299 proposed a minimum-length rule for evidence snippets. Measuring first killed that idea: of 5106 snippets, 627 are under 40 characters and the shortest ones are correct — `pH 6.8`, `150 g/L`, `glucose 10 g/L`, `Firmicutes`. Those are exactly the right evidence for a pH, a concentration or a taxon, so a length floor would have buried a real signal under hundreds of false positives. The signature that does work is specific and self-verifying: the snippet ends in a single capital letter and a period, *and the cached reference continues with a lowercase word*. The second half is what gives it precision — it separates a truncated genus from a sentence that legitimately ends in an abbreviation or a unit. Across the whole KB it flags 4 snippets, all genuine, with no false positives; `27 °C.` and `1.0 ppm F.` are correctly ignored because nothing lowercase follows them. All four are fixed. Each had stopped at precisely the word it existed to identify: The main products from C. -> + ljungdahlii fermentation at a pH of 6.0… …by P. aeruginosa and G. -> + sulfurreducens respectively. (x2) benzoate-fermenting bacterium S. -> + gentianae is approximately -45 kJ… The second signature from #299 — stopping immediately before a non-ASCII character in the cache, the U+2010 case — is deliberately *not* gated. It fires once repo-wide, on a snippet that ends a complete clause before an em-dash, so it is currently all false positive; the reasoning is recorded in the test's docstring rather than lost. Snippets absent from their cache (291) are skipped rather than reported: that is a stale-or-mis-rendered cache, which the snippet audit already buckets, and conflating it with truncation would make this gate untrustworthy. Mutation-checked: re-truncating one snippet fails exactly its file's case. 899 tests pass; lint clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Network integrity findingsReporting only — this check does not fail the build (see issue #273). The full report is attached to the workflow run as an artifact. |
There was a problem hiding this comment.
Pull request overview
Adds a targeted regression test to detect evidence snippets that were mechanically truncated at an abbreviated genus (e.g., ending in C. where the cached reference continues with the species epithet), and fixes the four affected community records by extending their snippets to full, meaningful clause boundaries.
Changes:
- Add
tests/test_snippet_truncation.pyto gate the “genus abbreviation truncation” signature described in #299. - Update three community YAMLs to replace truncated snippets (
… C.,… G.,… S.) with complete, cache-verbatim clauses (4 occurrences total). - Ensure the new gate skips uncached references to avoid conflating truncation with cache-rendering/caching defects.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| tests/test_snippet_truncation.py | New parametrized test that flags snippets ending in a single-letter genus abbreviation when the cache continues with a lowercase epithet. |
| kb/communities/Syntrophus_Methanospirillum_Gentianae_Benzoate_Coculture.yaml | Extend truncated S. snippet to include S. gentianae … clause. |
| kb/communities/Geobacter_Pseudomonas_Formate_Fumarate_Electroactive_Coculture.yaml | Extend two duplicated truncated G. snippets to include G. sulfurreducens respectively. |
| kb/communities/Clostridium_Ljungdahlii_Kluyveri_Syngas_Alcohol_Coculture.yaml | Extend truncated C. snippet to include C. ljungdahlii … clause. |
Suppressed comments (1)
tests/test_snippet_truncation.py:53
- Consider memoizing _cache_text() (e.g., with functools.lru_cache) so each reference’s cache content is read/normalized at most once per test run.
def _cache_text(reference: str) -> str | None:
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+55
to
+58
| key = reference.replace("PMID:", "PMID_").replace("doi:", "DOI_").replace("/", "_") | ||
| for candidate in CACHE.glob(key + ".*"): | ||
| return " ".join(candidate.read_text(errors="replace").split()) | ||
| return None |
Comment on lines
+37
to
+38
| import re | ||
| from pathlib import Path |
Review of this PR found the gate's own cache resolution was filesystem-order dependent. `references_cache/` holds 379 .md, 214 .txt and 78 .json files, and 63 references carry more than one — usually a .md alongside a .txt for the same PMID. Those are not interchangeable: per #265 the .md typically holds open-access full text while the .txt may be only the abstract. Taking the first glob match therefore read an arbitrary one. Measured by running the scan twice with opposite preferences, 4471 snippets are locatable via .md against 4400 via .txt — so **71 snippets were checked or silently skipped depending on directory iteration order**, which is not stable across machines. It never surfaced as a failure because the four truncations this PR fixes are flagged under either preference. It was latent: a future truncation among those 71 would be caught or missed by accident. All non-.json variants are now searched. They are deliberately not concatenated — this check inspects what *follows* the snippet, and joining two files would manufacture a continuation across the boundary. .json files are CrossRef metadata rather than prose and are excluded outright. Incidentally about twice as fast (6.7s -> 3.7s), because the cheap regex on the snippet now short-circuits before any cache file is read. Filed as #306, which also notes that evidence_snippet_audit.py and the vendored reference validator are worth checking for the same ordering dependence — if they resolve caches this way, their RENDERING/MISMATCH/NOCONTENT buckets inherit it. Mutation-checked again: re-truncating a snippet whose reference has only a .md cache still fails exactly its file's case. 899 tests pass; lint clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #299.
Measuring first killed the obvious fix
#299 proposed a minimum-length rule for evidence snippets. The data says that would have been wrong. Of 5,106 snippets, 627 are under 40 characters — and the shortest ones are correct:
pH 6.8150 g/LFirmicutesglucose 10 g/LA length floor would have buried a real signal under hundreds of false positives. Short snippets are the normal shape of evidence for quantitative and nominal fields.
The signature that works
The snippet ends in a single capital letter and a period, and the cached reference continues with a lowercase word. That second condition is what makes it precise — it separates a truncated genus from a sentence that legitimately ends in an abbreviation or a unit.
Across the whole KB it flags 4 snippets, all genuine, zero false positives.
27 °C.and1.0 ppm F.are correctly ignored, because nothing lowercase follows them.The four, each truncated at the word it existed to identify
All extended to a clean clause boundary and re-verified verbatim against their caches.
Two deliberate non-decisions, recorded rather than lost
The U+2010 signature is not gated. #299's other mechanical cause — a snippet stopping immediately before a non-ASCII character present in the cache — fires exactly once repo-wide, on a snippet that ends a complete clause before an em-dash. It is currently all false positive, so gating it would cost trust for no benefit. The reasoning lives in the test's docstring so the next person does not re-derive it.
Snippets absent from their cache are skipped, not reported. There are 291 of them. That is a stale or mis-rendered cache — a different defect, already bucketed by the snippet audit (RENDERING / MISMATCH / NOCONTENT per #257/#265). Folding it into a truncation gate would make this test untrustworthy for its actual purpose.
Verification
black,ruff,mypyclean; all three edited records pass schema validation.🤖 Generated with Claude Code
Review round
Reviewing the gate found a defect in the gate itself: its cache resolution was filesystem-order dependent.
references_cache/holds 379.md, 214.txtand 78.jsonfiles, and 63 references carry more than one — usually a.mdbeside a.txtfor the same PMID. Those are not interchangeable: per #265 the.mdtypically holds open-access full text while the.txtmay be only the abstract. Taking the firstglobmatch read an arbitrary one.Measured by running the scan twice with opposite preferences:
.md.txt71 snippets were checked or silently skipped depending on directory iteration order, which is not stable across machines. It never surfaced as a failure because the four truncations fixed here are flagged under either preference — it was latent, and a future truncation among those 71 would have been caught or missed by accident.
All non-
.jsonvariants are now searched. They are deliberately not concatenated: this check inspects what follows the snippet, so joining two files would manufacture a continuation across the boundary..jsonfiles are CrossRef metadata rather than prose and are excluded outright.Filed as #306, which also flags that
evidence_snippet_audit.pyand the vendored reference validator are worth checking for the same ordering dependence — if they resolve caches this way, their RENDERING/MISMATCH/NOCONTENT buckets inherit it.Incidentally about twice as fast (6.7s → 3.7s): the cheap regex on the snippet now short-circuits before any cache file is read.
Mutation-checked again — re-truncating a snippet whose reference has only a
.mdcache still fails exactly its file's case. 899 tests pass; lint clean.