sourcecheck

Source-integrity checks for AI-generated claims. Does the cited source exist, is it retracted, is the quoted text actually there? sourcecheck resolves each citation to a real source via OpenAlex and Crossref, flags retracted sources, checks the claim against the source text, and abstains when it can't confirm, rather than guessing.

It checks whether a claim is sourced, not whether it is true. That scope is the point: the checks it makes, existence, retraction, and presence-in-source, are exactly the checks a language model cannot make for itself. It hallucinates DOIs and cannot know a retraction past its training cutoff; an authoritative registry can. So most tools measure "faithfulness" with one LLM judging another, and sourcecheck takes the opposite stance: resolve the citation to ground truth first, because for these checks a language model is the thing being checked, not the judge.

The 2026 paper Cited but Not Verified found frontier models keep citation link validity above 94% but reach only 39-77% factual accuracy, and accuracy drops ~42% as tool calls scale. Citations are everywhere and verified almost nowhere.

Install

Not on PyPI yet (the trusted-publishing release workflow is in place; the first GitHub release will publish it). Until then, install from source:

git clone https://github.com/aberaio/sourcecheck && cd sourcecheck
pip install -e .                      # core (httpx only)
pip install -e ".[entailment]"        # + NLI grounding (sentence-transformers)
pip install -e ".[demo]"              # + web demo (fastapi/uvicorn)

Use

from sourcecheck import Verifier, OpenAlexCrossrefBackend

v = Verifier(provenance=OpenAlexCrossrefBackend(mailto="you@example.com"))

report = v.verify([
    {"claim": "The Transformer is based solely on attention mechanisms.",
     "citation": {"doi": "10.48550/arXiv.1706.03762"}},
    {"claim": "LLMs achieve 100% factual accuracy on cited claims.",
     "citation": {"doi": "10.1234/fabricated"}},
])

for c in report.claims:
    print(c.verdict.value, "-", c.reason)
print(report.precision_note)

Or paste a whole AI summary (inline DOIs or a References section) and let it extract the claims:

report = v.verify_text(ai_summary_with_citations)

Or from the command line:

sourcecheck paper.md                       # composite grounding (no model deps)
sourcecheck paper.md --grounding entailment
sourcecheck paper.md --full-text           # check against OA full text, not just abstracts
cat summary.txt | sourcecheck

As a CI gate for AI-written docs

The CLI is built to run in CI like a linter: nonzero exit when the things you care about go wrong, markdown output for the PR.

sourcecheck docs/summary.md --fail-on contradicted,retracted,unresolved
sourcecheck docs/summary.md --format markdown >> "$GITHUB_STEP_SUMMARY"
sourcecheck docs/summary.md --format json | jq '.claims[].reason_code'

--fail-on conditions: contradicted, unresolved (citation resolves to no real work — the fabricated-citation case), retracted (cites a retracted paper), unsupported (strictest: anything not positively supported). Resolutions are cached on disk for 7 days (--no-cache to opt out), and the batch's unique citations resolve concurrently (--max-workers, default 4).

Three honest verdicts:

• supported — the citation resolved to a real source and the claim is grounded in it.
• contradicted — the source actually contradicts the claim.
• not_verifiable — the citation didn't resolve, the source had no checkable text, or the claim couldn't be confirmed. This is a feature, not a failure.

Every result also carries a machine-readable reason_code (supported, contradicted, unresolved, retracted, no_content, no_grounding, low_grounding).

Retracted sources are flagged

The single worst thing a citation can do is point at a paper that has been formally withdrawn. sourcecheck checks retraction status on every resolution (OpenAlex is_retracted, plus Crossref's Retraction Watch records) and a retracted work is never accepted as evidence for or against a claim — even a verbatim match returns not_verifiable with reason_code=retracted, and --fail-on retracted turns it into a hard CI failure.

For screening a whole corpus, a request per DOI is the wrong tool. Load the Retraction Watch dataset (one CSV, ~65K retractions) into an offline index and the backend joins against it locally, flagging even retractions the live registries haven't caught up to yet:

from sourcecheck import RetractionIndex, OpenAlexCrossrefBackend

index = RetractionIndex.from_csv("retraction_watch.csv")
backend = OpenAlexCrossrefBackend(mailto="you@example.com", retraction_index=index)
# index.lookup(doi) -> RetractionRecord(notice_doi, date) for batch reports

Only rows whose nature is a retraction gate; corrections and expressions of concern leave a paper live, because over-flagging a corrected-but-valid paper is its own precision failure.

Design: narrow focus, pluggable guts

Five interfaces make the domain swappable while the gate stays the same:

Interface	Job	Flagship implementation
ProvenanceBackend	resolve a citation to a real source	OpenAlexCrossrefBackend, FederalRegisterBackend
SourceStore	fetch the source's checkable text	OpenAlex abstract; EuropePMCFullTextStore (OA full text)
GroundingChecker	is the claim supported by the text?	number-match + lexical; entailment/MiniCheck/HHEM drop in
AbstentionPolicy	verdict, precision-first	PrecisionFirstPolicy
Verifier	orchestrate the pipeline	—

Point the ProvenanceBackend at a legal registry, municipal bylaws, or an internal corpus and the same abstention gate applies unchanged. The scientific literature is the hardest, most rigorous case, and the flagship — but this isn't a hypothetical: FederalRegisterBackend (below) is a working second domain built on the exact same interfaces.

A second domain, same gate: US regulations

To show the interfaces aren't scholarly-only, sourcecheck ships a FederalRegisterBackend that resolves against every rule and notice the US government publishes (each with a stable document number) via the free Federal Register API:

from sourcecheck import Verifier
from sourcecheck.backends.federal_register import FederalRegisterBackend

with Verifier(provenance=FederalRegisterBackend()) as v:
    r = v.verify_claim(
        "The rule announces acceptance of applications for the ReConnect program.",
        {"raw": "2024-03484"},   # a Federal Register document number
    )
    print(r.verdict.value, "-", r.reason)

Nothing else changes: the same precision-first gate, reason codes, extraction, and CLI work unaltered. A document number is authoritative (resolve by number only); a free-text query must clear the same title-overlap bar as the scholarly backend or it abstains.

Full-text grounding (open access)

By default grounding checks the abstract. Wrap the backend in EuropePMCFullTextStore to check open-access claims against the full article body (Europe PMC, free, no key), which is where Results/Methods claims live:

from sourcecheck import OpenAlexCrossrefBackend, Verifier
from sourcecheck.backends.europepmc import EuropePMCFullTextStore

backend = OpenAlexCrossrefBackend(mailto="you@example.com")
store = EuropePMCFullTextStore(fallback=backend, mailto="you@example.com")
v = Verifier(provenance=backend, source_store=store)   # or: sourcecheck paper.md --full-text

Precision is unchanged: the grounding gates operate per-sentence, so more text raises recall without loosening the support bar. References, tables, and figure captions are stripped so a bibliography match can't masquerade as support. When full text is unavailable (closed access, no PMCID, a fetch error) it falls back to the abstract, so it's always safe to wrap.

Async

AsyncVerifier runs the pipeline off your event loop so it fits into a FastAPI handler without blocking. It offloads the (synchronous-HTTP) pipeline to a worker thread via asyncio.to_thread and bounds concurrency with a semaphore — honestly not native async I/O, but exactly what you need to not stall the loop:

from sourcecheck import OpenAlexCrossrefBackend
from sourcecheck.aio import AsyncVerifier

async with AsyncVerifier(provenance=OpenAlexCrossrefBackend(mailto="you@example.com")) as v:
    report = await v.verify(claims)

Grounding backends

• composite (default, no model deps): a conservative lexical checker. It marks SUPPORTED only when the claim appears near-verbatim in a single source sentence, with every number present in that sentence and no polarity/direction cue mismatch. It abstains on anything less (paraphrase, scattered terms, antonyms). High precision, low recall, and no semantic understanding, by design.
• entailment (pip install "sourcecheck[entailment]"): sentence-level NLI. Adds real contradiction detection, with topical-relevance and number gates on both the support and contradiction paths so a source that is merely silent on a claim abstains instead of false-supporting or false-contradicting. Model label order is resolved from the model config at load time (no silent inversion). Its thresholds are calibrated on a held-out set (see below), and a directional/numeric guard suppresses the NLI model's habit of calling quantifier entailments ("18% more" → "more") contradictions.

Both are the same GroundingChecker interface; wrap MiniCheck/HHEM/Lynx the same way.

Entailment threshold calibration

The entailment checker's operating point (support / contradiction / relevance thresholds) is tuned on a hand-labelled held-out set rather than guessed. The calibration is reproducible and, crucially, split so the tuning runs without a GPU or the network:

python benchmark/calibrate.py --rescore   # run the NLI model once, record raw scores (committed)
python benchmark/calibrate.py             # sweep thresholds over recorded scores (offline), write JSON
python benchmark/calibrate.py --check     # CI: assert the committed point stays zero-fault

The sweep only considers operating points with zero false-support and zero false-contradiction on the set, then maximizes accuracy — precision-first is a hard constraint, not a term in a loss. The chosen thresholds are written to a packaged JSON that EntailmentChecker loads at construction, gated by model name so they are never silently applied to a different model. Calibrating this model surfaced a real failure mode — nli-deberta-v3-small confidently labels "18% more" → "more" a contradiction — which the directional/numeric guard now catches structurally. On the live smoke benchmark the calibrated backend scores 11/11 with all real contradictions still caught; test_entailment_calibration.py enforces the zero-fault guarantee offline in CI.

Composes with your stack

sourcecheck is a verification gate, not another eval framework. It wraps existing detectors as GroundingCheckers (MiniCheck, HHEM, Lynx), and ships adapters in sourcecheck.integrations so its verdicts flow into the tools you already run:

from sourcecheck.integrations import to_ragas_records, make_guardrails_validator

# RAGAS: one record per claim, with the resolved source as retrieved context and
# sourcecheck's verdict/score as extra columns. Pure dicts — no RAGAS import needed.
records = to_ragas_records(report)      # -> ragas.EvaluationDataset.from_list(records)

# Guardrails AI: an on-fail validator that rejects LLM output whose claims are
# contradicted or rest on a retracted source. pip install "sourcecheck[guardrails]"
validator = make_guardrails_validator(verifier, fail_on=["contradicted", "retracted"])

Neither heavy library is imported until you use it; import sourcecheck.integrations works with neither installed.

Benchmark (smoke-grade, not a leaderboard)

benchmark/ is a tiny hand-labelled regression check (~10 items), not a statistical evaluation. It exists to catch the errors a precision-first tool must never make, and the run fails (exits nonzero) if any occurs:

• false support — a not-actually-supported claim marked SUPPORTED
• false contradiction — a fine claim marked CONTRADICTED
• missed retraction — a retracted source not flagged retracted

python benchmark/run.py --offline                 # hermetic (fixtured abstracts); runs in CI on every push
python benchmark/run.py --grounding entailment    # against the live APIs

Do not read its accuracy number as evidence of general performance; the set is far too small. A larger, versioned benchmark with fixtured sources and adversarial negatives is on the roadmap. See Limitations below.

Limitations (read before trusting a verdict)

sourcecheck is v0.3, still pre-1.0. A SUPPORTED verdict is a useful signal, not a proof. Known limits:

• Grounding defaults to the abstract. Without --full-text it checks the claim against the abstract (and, via Crossref, its JATS text), so a claim supported only in methods/results correctly comes back not_verifiable. --full-text / EuropePMCFullTextStore closes most of this gap for open-access work, but closed-access papers still fall back to the abstract.
• The default composite checker is a lexical heuristic, not a semantic model. It only marks SUPPORTED when the claim appears near-verbatim in a single source sentence (with numbers and polarity consistent). It abstains on paraphrase and cannot detect argument-structure swaps that reuse the same words ("A causes B" vs "B causes A"). For semantic matching and real contradiction detection, use the entailment backend.
• The entailment model is small. Its thresholds are now calibrated on a held-out set (zero false-support / false-contradiction there) and a directional/numeric guard blocks the model's known quantifier-contradiction errors, but the held-out set is tiny and the model can still miss or misjudge on inputs unlike it. Numeric-magnitude contradictions (e.g. "at least 40" vs "at least 15") are deliberately abstained on rather than asserted.
• Citation extraction is structured-text-oriented: inline DOIs/arXiv, [n] markers (including [1-3] ranges) against a References section, and author-year citations — parenthetical (Smith et al., 2020) groups and narrative Smith et al. (2020) — matched to the References by first-author surname + year. It still does not parse every citation style; supply {claim, citation} pairs directly for full control. An author-year citation with no matching reference entry is kept and honestly reported unresolved (surname + year alone is too vague to search safely).
• Retraction data is only as good as the registries. OpenAlex and Crossref/Retraction Watch flag most retractions, but coverage is not total, and the on-disk cache means a brand-new retraction can take up to the cache TTL (default 7 days) to surface.
• English-oriented, and limited to what OpenAlex/Crossref index.
• Not for high-stakes automated decisions yet. Treat verdicts as review aids.

Precision-first means it prefers not_verifiable (a false abstention) over a wrong supported. When it cannot confirm, it says so.

Web demo

pip install "sourcecheck[demo]"
uvicorn demo.app:app          # then open http://127.0.0.1:8000

Paste an AI summary and watch each claim get marked supported, contradicted, or not verifiable, with the abstentions front and center. The demo escapes all rendered text and caps input size / citation count; still, treat it as a local demo, not a hardened public service.

Privacy

Citations and claim text you submit are sent to the OpenAlex and Crossref public APIs to resolve sources. Nothing is stored by sourcecheck, but those third-party services receive the queries.

Performance & caching

• Verifier.verify() resolves the batch's unique citations concurrently (max_workers=4 by default, out of politeness to the public APIs); grounding runs sequentially so GroundingChecker implementations never need to be thread-safe.

• Wrap any backend in CachedBackend to cache successful resolutions on disk (default 7 days, under the platform cache dir or $SOURCECHECK_CACHE_DIR):

  from sourcecheck import CachedBackend, OpenAlexCrossrefBackend, Verifier
  
  backend = CachedBackend(OpenAlexCrossrefBackend(mailto="you@example.com"))
  v = Verifier(provenance=backend)

  Negative results are never cached — a transient outage must not freeze into "citation not found" — and entries expire so later retractions surface.

Status

v0.3. Working: OpenAlex/Crossref resolution (DOI-direct, plus validated title search), retraction flagging, open-access full-text grounding (Europe PMC), composite + entailment grounding, the precision-first abstention gate with machine-readable reason codes, claim/citation extraction (inline DOI/arXiv, [n], author-year), concurrent batch resolution, an async API, an on-disk cache, a second-domain provenance backend (US Federal Register), RAGAS and Guardrails adapters, a CI-gate CLI (--fail-on, --format json|markdown, --full-text), a web demo, and an offline benchmark enforced in CI.

The v0.2→v0.3 roadmap — full-text retrieval, async API, a second provenance backend, RAGAS/Guardrails adapters, and entailment threshold calibration — is done. Next: native-async backends (httpx AsyncClient end-to-end), a larger versioned benchmark with adversarial negatives, and full-text sources beyond Europe PMC (arXiv, PubMed Central OA mirrors).

Contributing & security

See CONTRIBUTING.md and SECURITY.md.

License

MIT (see LICENSE) · built by Mikias Abera