Ground book citations against Open Library / Internet Archive (read path, PRD-0009)#147
Draft
tieguy wants to merge 5 commits into
Draft
Ground book citations against Open Library / Internet Archive (read path, PRD-0009)#147tieguy wants to merge 5 commits into
tieguy wants to merge 5 commits into
Conversation
…ce 1)
Draft ADR-0018, the Open Library / Internet Archive read contract spawned
by PRD-0009 (Q6a): identifier-gated resolution, a strictly side-effect-free
Books API catalog lookup (never the import-on-miss /isbn/{isbn}.json),
Read API exact-vs-similar scan availability, and the search-inside
grounding-source rules including the scoped ShortBody bypass.
Implement the first read-only slice:
- sp42-platform: BookIdentifier (ISBN checksum-validated, OCLC, LCCN
LC-normalized, OLID canonicalized with author ids rejected) and
BookSource; BlockRef carries book_sources alongside cited sources.
- sp42-parsoid: cite-template data-mw extraction now also reads
isbn/oclc/lccn/ol params (and the cited page), independent of the
url= gate, so a URL-less {{cite book}} finally yields something.
- sp42-citation extract: a url-less ref with a validated identifier is
skipped as BookSource (with its identifiers) instead of blending into
NonUrlSource; the page report prints those identifiers.
- sp42-citation openlibrary: pure build/parse pairs for the Books API
catalog lookup and the Read API scan availability, partitioned so only
exact-match scans can ever feed grounding.
Wiring resolution into the verify path and report is the next slice.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012u5MVp3vAN44KhsVg3vroY
The page orchestrator now resolves every skipped book ref through the
side-effect-free Open Library lane (ADR-0018) and the report shows what
the catalog said, so a book citation prints a resolved record instead of
a bare identifier.
- sp42-citation openlibrary: resolve_book_source executor — tries a
ref's identifiers in template order against the Books API, then checks
Read API scan availability on the first hit. A clean miss is NotFound
(never a create), a transport failure is LookupFailed (existence
unknown), and an availability failure degrades to scan-unknown without
losing the catalog hit. A recording-client test asserts the whole lane
addresses only /api/books and /api/volumes/brief, never the
import-on-miss /isbn/{isbn}.json path.
- verify_page: a bounded (<=3, REST politeness) resolution pass over the
skipped refs' book sources; failures stay inside the books lane and
never fail the page. Report gains book_resolutions plus
books_resolved/books_not_found/book_lookups_failed stats, all
serde-default so older reports still deserialize.
- Renderer: a Books section (resolved record with title/authors/record
URL and an exact/similar_only/none/unknown scan label; distinct
not_found vs lookup_failed lines) and a books lead tally.
- SkippedRef now carries its BookSource structures (identifiers + cited
page) instead of a flattened identifier list, so resolution keeps
per-template identifier order and the cited page for Layer 2.
Search-inside grounding (Layer 2) is the next slice.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012u5MVp3vAN44KhsVg3vroY
Layer 2 of the book lane (ADR-0018 Decision 4): a resolved book citation with an exact-edition scan now gets a real verdict instead of a skip. - sp42-citation search_inside: the Internet Archive grounding source — item-metadata read for the designated search server (texts-only gate), BookReader full-text search build/parse (verbatim OCR snippets with highlight markers stripped, page numbers kept), conservative claim→query term selection, page-anchored deep links, and prepare_book_grounding mapping every degraded outcome onto the ADR-0007 split: no usable body (not a text item / no index) -> SourceUnavailable; an indexed scan searched with zero snippets -> not_supported, never SourceUnavailable. Cited page first, whole-book fallback, per PRD-0009 Q5. - verify: FetchedSource gains a book_snippet provenance flag whose only effect is bypassing the generic ShortBody floor at the usability gate — arbitrary short web pages still short-circuit, an empty snippet stays unusable (covered by a three-way test). CitationFinding gains optional BookScanProvenance (ocaid, scanned vs cited page, refined note) and two no-model book outcomes for the unavailable/searched-empty cases. - extract: a url-less ref with validated identifiers is now a BookUseSite (same sentence attach as URL refs, shared ordinal sequence) instead of an unconditional skip; only identifier-less refs skip at extract time. - verify_page: the book pass resolves each site against Open Library, grounds resolved books through the snippet body and the model panel (resolved record as context-only metadata sidecar), degrades similar- only/no-scan books to an honest SourceUnavailable pointing at the OL record, and re-skips catalog misses with the refined BookSource reason. The scanned page the passage actually located on is recorded so pagination mismatches surface instead of causing false negatives. - Renderer: finding lines show scan/scanned_page/cited_page/note; the Books section and stats are unchanged from slice 2. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012u5MVp3vAN44KhsVg3vroY
Manual, network-touching, never CI (ADR-0009 keeps tests on fixtures).
Walks the exact read chain verify-page uses — Books API catalog lookup,
Read API scan availability, archive.org item metadata, BookReader
search-inside — with read-only GETs and asserts precisely the response
fields the PRD-0009 parsers read. A pass means production still matches
the fixtures; a failure names the drifted contract and keeps the real
payload for fixture refresh. By construction it never touches the
import-on-miss /isbn/{isbn}.json path or any write endpoint.
Header notes that Open Library 403s datacenter IPs (observed from this
environment), so the smoke runs from a developer connection — the same
caveat applies to any future cloud-hosted SP42 deployment.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012u5MVp3vAN44KhsVg3vroY
Rebased the PRD-0009 read/grounding slices onto current origin/main (~90 commits of drift). Renumber the read-contract ADR off contested 0018 (claimed by PR #106) to 0024. Reconcile struct/signature drift where main's re-verify path (PRD-0014) and the book lane both extended shared types: keep both parsoid maps (ref_bare + RefSources), thread book_resolutions through tally_stats, and stamp book_sources/is_bare_url_ref/book_scan on the merged struct literals. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Book citations — a
{{cite book}}with an ISBN but nourl=— currently getno verdict and no affordance: the verifier records them as
skipped { NonUrlSource }because there is nothing to fetch. This adds a read-only paththat turns a book identifier into a grounded citation verdict:
template's
data-mw) against Open Library's catalog — side-effect-free, noimport-on-miss, no write endpoints touched.
snippet when an exact-edition scan is available, feeding that snippet into the
existing verification pipeline and anchoring the finding to a page-level deep
link.
not_supportedwhen the text is searched andthe claim isn't found, distinct from
SourceUnavailablewhen no scan exists.Everything here is read-only. The enrichment write lane (proposing edits
back to Open Library) is deliberately out of scope and lands separately as a
gated follow-up — this PR is independently mergeable without it.
Design: ADR-0024 (Open Library / Internet Archive read contract), implementing
PRD-0009 Layers 1–2.
scripts/openlibrary-contract-smoke.shis a manuallive-network smoke for the read endpoints.
Provenance / review notes
claude/openlibrary-resolvebranch (~90 commitsbehind); rebased onto current
main.in-flight by PR Docs: MCP citation write surface — agent proposes, human confirms (add path blocked on PRD-0012/ADR-0003 insertion) #106 (the citation write-surface / token seam), which is a
different ADR. Main-side references to that 0018 are left untouched.
main's re-verify path (PRD-0014) and thisbook lane both extended shared structs/signatures in
sp42-parsoid,sp42-citation, andsp42-mcp(both the bare-URL-repair and book-source mapssurvive;
tally_statsthreadsbook_resolutions).cargo build/test/clippy --workspacegreen (the Tauribundle step in
ci-allcan't run in this environment).Draft — for review, not merge-ready.