Skip to content

feat: Unpaywall integration for open-access PDF resolution #45

Description

@zoharbabin

Why

academic_search finds papers with DOIs, but a researcher often can't reach the full text without an institutional subscription. Unpaywall (by OurResearch, the team behind OpenAlex) maps DOIs to legal open-access copies for tens of millions of articles. Adding it completes the discover → metadata → access → extract chain: academic_search → OA resolution → scrape_page on the PDF. No other single MCP server offers the full chain.

academic_search now spans multiple providers — openalex, crossref, and exa (research-paper category) all return AcademicResults. OA enrichment sits below the provider layer, so it applies uniformly across all of them: any DOI-bearing result missing a PDF link can be enriched, regardless of which provider produced it.

What

OA resolution layered onto the existing academic pipeline (not a standalone tool):

  1. Enrichment: after academic_search returns DOI-bearing results, optionally resolve each via Unpaywall and populate pdfUrl + openAccess when the provider didn't already supply them. Provider-agnostic — works for openalex, crossref, exa, and any future academic provider.
  2. Direct resolution: accept a doi input on academic_search that triggers a direct DOI → metadata + best-OA-location lookup.

How

  1. Endpoint: GET https://api.unpaywall.org/v2/{DOI}?email={email}is_oa, oa_status (gold/green/hybrid/bronze), best_oa_location.url_for_pdf. Email-only auth, reusing the existing OpenAlexEmail / CrossRefEmail config pattern in AcademicProviderConfig (internal/search/domain.go). Generous 100k/day limit.
  2. Enrichment runs in internal/tools/academic.go after results return; populate the existing AcademicResult.PDFUrl / OpenAccess fields (defined in internal/search/domain.go) only when empty (never overwrite a value the provider already supplied — e.g. an exa or openalex PDF link).
  3. SSRF-safe client (scraper.NewSSRFSafeClient()), per-provider circuit breaker, 24h cache TTL keyed by DOI (OA status changes slowly).
  4. Best-effort: an Unpaywall failure must never fail the underlying academic_search — results return unenriched.

Acceptance criteria

  • DOI-bearing academic_search results from any provider gain pdfUrl/openAccess when Unpaywall has an OA copy and the provider didn't supply one
  • doi input performs a direct resolution
  • Existing provider-supplied pdfUrl is never overwritten
  • Unpaywall failure degrades gracefully (unenriched results, no error)
  • Email config + 24h DOI cache; unit tests with httptest mock
  • docs/TOOLS.md notes the enrichment behavior

Dependencies

Metadata

Metadata

Assignees

No one assigned

    Labels

    P0Highest priorityacademicAcademic/scholarly search featuresenhancementNew feature or requestproviderNew search/data provider integration

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions