No DOI-based full-text cache path: blocks non-PubMed OA sources (e.g. exact-pair formate data for 000068)

[realmarcin]

Found while curating #254.

scripts/cache_fulltext.py resolves full text only via PMID → Europe PMC → PMC id, and refuses to start without an existing references_cache/PMID_<id>.{txt,md} abstract. Sources that are open access but not indexed in PubMed therefore cannot be snippet-verified at all, so they cannot be curated.

Concrete blocked case

Li et al. 2024, Water 16:3551, doi:10.3390/w16243551 — fully open access, and the only retrieved study that perturbs the exact CommunityMech:000068 pair (S. fumaroxidans + M. formicicum, which matters because most of that record's mechanistic literature is actually S. fumaroxidans + M. hungatei; see the discussion added in #254).

It reports graded-formate dosing on the exact coculture:

• 5–10 mM formate promotes syntrophic propionate oxidation
• ≥30 mM inhibits it (strongest at 50 mM), with propionate accumulation at days 5–7
• FDH1–FDH4 and hydrogenase transcripts downregulated under formate stress
• caveat worth curating: added formate raised total CH4/H2 yield while impeding propionate-derived carbon flow

That is several well-perturbed causal edges plus environmental factors that 000068 currently lacks. Europe PMC returns no pmid, no pmcid, isOpenAccess: null for this DOI, so the existing tooling cannot touch it.

Proposed fix

Add a DOI-based path to cache_fulltext.py:

1. Try Europe PMC by DOI (query=DOI:"<doi>") — covers OA DOIs that do have a PMC record.
2. Fall back to Unpaywall (literature.py already has an Unpaywall integration) → OA PDF/HTML → text.
3. Write to the DOI cache filename the reference validator already reads (DOI_<doi with / → _>.md), appending under the same ===== OPEN-ACCESS FULL TEXT marker so the operation stays idempotent.

Non-OA sources must keep failing loudly rather than fabricating — the current "report and skip, never fabricate" behaviour is right and should be preserved.

[realmarcin]

Partially addressed in #260: cache_fulltext.py now accepts DOIs and resolves them against Europe PMC by DOI, writing to the DOI_<slug>.md convention the validator reads. Verified on 10.1128/spectrum.00941-23 → PMC10580878 and 10.1111/1462-2920.14119 → PMC5947623.

Leaving this open because the motivating case is still blocked: doi:10.3390/w16243551 has no Europe PMC record, and MDPI returns HTTP 403 to programmatic PDF download. I did not add a PDF dependency, since the block is the publisher rather than the parser — pypdf would not have retrieved it either.

Unpaywall lookup is wired in behind UNPAYWALL_EMAIL so the skip message at least names the OA location for manual retrieval. Options for full closure: a curator-supplied local file path, or a polite publisher-specific fetcher.

[realmarcin]

Retrieval half resolved in #261. cache_fulltext.py now takes --from-file <path>, so a curator-supplied PDF/HTML/text file can be appended to either a PMID or DOI cache entry under the existing OA marker. pypdf is imported lazily (message points at uv run --with pypdf) so it stays off the dependency list, and extraction under 500 chars is refused so a failed parse can't produce an empty cache.

The motivating source, doi:10.3390/w16243551, is now cached (66,496 chars) and its exact-pair formate results are curated into CommunityMech:000068.

Remaining open: automated retrieval for publishers that block programmatic download. The --from-file route is a manual escape hatch, not a fix for that. Closing this issue would be reasonable if manual retrieval is considered acceptable for such sources.