Literature Provider Adapters

Literature Provider Adapters

Author: Ciprian Ștefan Pleșca

OpenLongevity ingests bibliographic and trial-registry data through a set of read-only adapters, each implementing the LiteratureProvider protocol defined in src/openlongevity/providers/base.py. This page documents the shared contract, the fully implemented PubMed adapter, and the design constraints that apply to every adapter in the set (Europe PMC, OpenAlex, Crossref, ClinicalTrials.gov).

The shared contract

classDiagram
    class LiteratureProvider {
        <<protocol>>
        +str name
        +search(query: SearchQuery) list~Publication~
        +get_by_id(external_id: str) Publication~
    }

    class SearchQuery {
        +str query
        +int limit
        +int page
        +Mapping filters
        +__post_init__() normalizes whitespace, enforces 1..200 chars, 1..100 limit, page >= 1
    }

    class Provenance {
        +str source_provider
        +str source_identifier
        +str source_url
        +str retrieved_at
        +str~ checksum
        +str normalization_version
        +str parser_version
    }

    class Publication {
        +str identifier
        +str title
        +str abstract
        +tuple authors
        +Provenance~ provenance
        +str retraction_status
        +tuple corrections
    }

    class ProviderError {
        <<exception>>
    }

    LiteratureProvider ..> SearchQuery : accepts
    LiteratureProvider ..> Publication : returns
    Publication --> Provenance
    LiteratureProvider ..> ProviderError : raises on recoverable failure

Loading

Every adapter is expected to:

1. Accept a validated SearchQuery — validation happens once, in the dataclass itself, so no adapter needs to re-check query length or limit bounds.
2. Return a list[Publication], never a provider-specific shape.
3. Attach a Provenance to every returned Publication, with a source_provider name and a resolvable source_url.
4. Raise ProviderError — never let raw transport exceptions escape — for any recoverable upstream failure (timeout, malformed payload, rate limiting exhausted).

PubMed adapter: ingestion pipeline

PubMedProvider is the only adapter implemented as a full, hardened I/O pipeline in this repository, and it establishes the pattern that other adapters are expected to follow.

flowchart TD
    Start([search-SearchQuery]) --> Lock["acquire asyncio.Lock: serialize this instance's requests"]
    Lock --> ESearch["esearch.fcgi\ndb=pubmed, term, retmax, retstart"]
    ESearch --> Rate["fixed 0.35s pre-request delay\n(<3 req/s per instance)"]
    Rate --> Stream[stream response, abort if > 5MB]
    Stream --> StatusCheck{"status 429 or >= 500?"}
    StatusCheck -- yes, attempts remain --> Backoff["respect Retry-After header,\nelse exponential backoff (0.5 x 2^n)"]
    Backoff --> ESearch
    StatusCheck -- no --> ParseIds{"idlist valid\n(list of numeric strings)?"}
    ParseIds -- no --> Err1[raise ProviderError: malformed search data]
    ParseIds -- empty --> Empty[return empty list]
    ParseIds -- yes --> EFetch["efetch.fcgi\ndb=pubmed, id=joined ids, retmode=xml"]
    EFetch --> Defused["defusedxml parse\n(blocks unsafe XML entities)"]
    Defused --> ParseFail{"parse error?"}
    ParseFail -- yes --> Err2[raise ProviderError: malformed or unsafe XML]
    ParseFail -- no --> PerArticle[for each PubmedArticle node]
    PerArticle --> Extract["extract pmid, title, abstract,\nauthors, journal, doi, types, corrections"]
    Extract --> Skip{"pmid, title\nboth present?"}
    Skip -- no --> Drop[skip this article]
    Skip -- yes --> Canon["canonicalize per-article XML,\nSHA-256 checksum"]
    Canon --> Build[build Publication + Provenance]
    Build --> Collect[append to results]
    Collect --> Done([return list of Publications])

Loading

Resilience characteristics

• Retry budget: up to 3 attempts per request stage, with exponential backoff (0.5 × 2^attempt) on network/timeout errors, and honoring an upstream Retry-After header (capped at 10 seconds) on 429/5xx responses.
• Bounded memory: the response body is streamed and aborted with a ProviderError the moment it exceeds max_bytes = 5_000_000, protecting the process from an unexpectedly large or malicious upstream payload.
• Safe XML parsing: defusedxml.ElementTree.fromstring is used instead of the standard library parser, mitigating XML entity-expansion and external-entity attacks — a meaningful hardening choice for any adapter parsing third-party XML.
• Deterministic checksums: the checksum is computed over ElementTree.canonicalize() output of the individual article node, not the whole batch response and not including the retrieval timestamp. This means the same article fetched on two different days produces the same checksum, which is exactly the property PublicationRepository depends on to distinguish "re-fetched, unchanged" from "genuinely revised."
• Malformed-identifier rejection: esearch results are validated as a list of purely numeric strings before use; anything else raises ProviderError rather than being passed downstream.

get_by_id semantics

async def get_by_id(self, external_id: str) -> Publication | None:
    pmid = external_id.removeprefix("PMID:")
    if not pmid.isdigit():
        raise ValueError("A numeric PMID is required")
    records = await self.search(SearchQuery(f"{pmid}[PMID]", limit=1))
    return next((record for record in records if record.identifier == f"PMID:{pmid}"), None)

Note the distinction in error handling: an invalid PMID format raises ValueError (a caller programming error — the input was never a valid identifier), while any upstream failure during the underlying search() call raises ProviderError (a recoverable runtime condition). This distinction matters for api.py, which catches ValueError as a 422 client error and ProviderError as a 502 upstream error.

The remaining adapters (Europe PMC, OpenAlex, Crossref, ClinicalTrials.gov)

These adapters share the same LiteratureProvider contract and the Publication/Provenance output shape. Architecturally, they are expected to:

• Target their respective public APIs (Europe PMC REST API, OpenAlex works endpoint, Crossref works endpoint, and the ClinicalTrials.gov v2 API for ClinicalTrial records).
• Reuse providers/http.py for shared transport concerns rather than each adapter reimplementing retry and timeout logic independently.
• Populate source_provider with their own name (europe_pmc, openalex, crossref, clinicaltrials) so that downstream consumers — including the concentration gap heuristic described in Research-Gap-Detection — can distinguish where a record actually originated.

flowchart LR
    subgraph Adapters["providers/ (all implement LiteratureProvider)"]
        direction TB
        PM[pubmed.py — fully hardened reference implementation]
        EP[europe_pmc.py]
        OA[openalex.py]
        CR[crossref.py]
        CT[clinicaltrials.py — returns ClinicalTrial records]
    end
    HTTP[http.py: shared transport helpers] --> PM
    HTTP --> EP
    HTTP --> OA
    HTTP --> CR
    HTTP --> CT
    PM --> OUT[list of Publication]
    EP --> OUT
    OA --> OUT
    CR --> OUT
    CT --> OUT2[list of ClinicalTrial]

Loading

Next

See Persistence-and-Data-Governance for what happens to a Publication once an adapter returns it, or API-Reference for the HTTP surface that triggers ingestion.