A pipeline for checking whether references produced by large language models actually exist, by resolving each one against Crossref and OpenAlex and scoring how well the generated metadata agrees with the real record.
Given a CSV of AI-generated citations, it labels every reference as Verified, Auto-correctable, Needs attention, or Unknown, and writes back the matched record so you can see exactly what the model got wrong: a fabricated title, a real paper with the wrong year, a plausible author list attached to someone else's work.
The run is resumable, deduplicates identical references before hitting the network, and caches every API response, so re-running over a growing dataset only pays for what is new.
Attribution. This project is a derivative work of hallubib by Endre Márk Borza, used under the MIT License. The verification concept, the four-way status taxonomy, and the core matching approach originate there. See Relationship to hallubib for what this fork changes, and LICENSE for the retained copyright notices.
| File | Purpose |
|---|---|
verify_references.py |
Main pipeline. Verifies references against Crossref + OpenAlex. |
title_matching.py |
Optional large-scale stage: semantic title matching against a local corpus dump using FAISS + sentence-transformers. |
git clone https://github.com/<your-username>/ai-reference-verification.git
cd ai-reference-verification
pip install -r requirements.txtThe main pipeline needs only requests. Python 3.9+.
Make a small CSV to try it on. The first reference below is real; the second is fabricated:
cat > refs.csv <<'CSV'
authors,title,journal,year,DOI,link,id,LLMs_model
Timothy W. Lyons,The rise of oxygen in Earth's early ocean and atmosphere,Nature,2014,10.1038/nature13068,,r1,example-model
Jane Q. Researcher,Quantum entanglement in medieval manuscript bindings,Nature,2019,,,r2,example-model
CSV
python verify_references.py --input-csv refs.csv --mailto you@example.comThat writes three files next to the input:
refs_checked.csv-- the resultsrefs_checked.sqlite-- per-reference checkpoint (lets you resume)refs_checked.cache.sqlite-- cached API responses
The first row should come back Verified via DOI/Crossref; the second
Unknown, with no matched record.
Re-run the exact same command after an interrupt and it picks up where it stopped instead of starting over.
Crossref and OpenAlex ask automated clients to identify themselves with a
contact address, and both route identified traffic through a faster "polite
pool". Pass your own address via --mailto, or set it once:
export CONTACT_EMAIL=you@example.comThe script runs without it but will warn, and you will get slower, less reliable service from both APIs.
These six column headers must be present. Individual values may be empty -- that is the normal case, since models often omit a DOI or a venue -- but a missing header is a hard error:
| Column | Notes |
|---|---|
title |
The main matching key. Rows with a short or empty title are not searched. |
authors |
Any common separator; used for the author score. |
journal |
Venue / container title. |
year |
Publication year. |
DOI |
When present it is tried first, and it is the strongest signal. |
link |
Only checked with --check-url. |
Any other columns you include are carried through to the output untouched. Two that are worth adding:
| Column | Notes |
|---|---|
id |
Your own stable identifier, so you can join results back to the source. |
LLMs_model |
Which model produced the reference; enables per-model breakdowns. |
Example:
authors,title,journal,year,DOI,link,id,LLMs_model
Timothy W. Lyons,The rise of oxygen in Earth's early ocean and atmosphere,Nature,2014,10.1038/nature13068,https://www.nature.com/articles/nature13068,9_19_0_1,llama3.1_405bThe input columns, plus:
Verdict
| Column | Meaning |
|---|---|
status |
Verified / Auto-correctable / Needs attention / Unknown |
source |
Which lookup produced the match: DOI/Crossref, Crossref title, OpenAlex DOI, OpenAlex title, or None |
confidence |
0-1 composite score |
reason |
Human-readable explanation of the verdict |
The record that was matched
matched_title, matched_authors, matched_journal, matched_year,
matched_doi, matched_url
Field-level agreement
title_score, author_score, author_match, year_match,
year_off_by_one, journal_score, doi_match, reachable_url
| Status | Condition | Reading |
|---|---|---|
Verified |
title_score >= 0.95 and (author_score >= 0.50 or DOI matches) and the year agrees (or is absent, or is off by one) |
The reference is real and the metadata is right. |
Auto-correctable |
title_score >= 0.85 with corroborating evidence, but year, journal, or DOI disagrees |
Real paper, wrong details -- fixable from matched_*. |
Needs attention |
A partial or weak match (title_score >= 0.72) |
Ambiguous. Worth a human look. |
Unknown |
No plausible match found | No such record was located -- the strongest hallucination signal. |
Unknown is not proof of fabrication on its own. Very recent work, non-indexed
venues, books, preprints, and non-English titles are all under-covered by
Crossref and OpenAlex. Treat the four labels as a triage ordering, not a
ground truth, and spot-check before reporting rates.
The composite confidence is built from the title score, plus weighted
contributions from author overlap (0.20), year agreement (0.10, or 0.05 when
off by one), journal similarity (up to 0.08), and an exact DOI match (0.20),
capped at 1.0.
--input-csv PATH Input CSV (required)
--output-csv PATH Default: <input>_checked.csv
--db-path PATH Checkpoint DB. Default: <input>_checked.sqlite
--cache-db-path PATH API cache DB. Default: <input>_checked.cache.sqlite
--mailto EMAIL Contact address for the polite pool ($CONTACT_EMAIL)
--check-url Also test whether the `link` column resolves (slow)
--no-openalex Crossref only
--max-workers N Worker threads (default 8)
--crossref-interval SEC Min seconds between Crossref calls (default 0.20)
--openalex-interval SEC Min seconds between OpenAlex calls (default 0.08)
--cache-hit-ttl-days N How long to reuse a successful lookup (default 90)
--cache-miss-ttl-days N How long to reuse a miss (default 7)
--log-level LEVEL DEBUG / INFO / WARNING / ERROR
Run python verify_references.py --help for the full list, including cache
flush and connection-pool tuning.
The default pacing is deliberately conservative and stays inside both APIs'
published rate guidance. Raising --max-workers while lowering the intervals
will get you throttled or blocked; do not do it without checking each service's
current terms.
title_matching.py handles the case where per-reference API calls are not
practical -- millions of titles to resolve against a full corpus snapshot. It
embeds both sides with sentence-transformers, indexes the corpus with a FAISS
IVFPQ index, then reranks candidates with a combined semantic + fuzzy-string
score.
pip install -r requirements-matching.txt
python title_matching.py \
--corpus-csv /path/to/openalex_id_doi_title.csv.gz \
--queries-csv /path/to/ai_generated_titles.csv \
--output-csv matches.csv \
--work-dir ./matching_workdir--corpus-csv needs id and title columns; --queries-csv needs id,
title, and LLMs_model. The FAISS index, lookup table, checkpoint, and log
all land in --work-dir, and the run resumes from the checkpoint if
interrupted. A GPU is used automatically when available and is strongly
recommended at corpus scale.
Match thresholds are tunable via --min-combined-score (default 0.9) and
--alpha (default 0.85, the weight given to the semantic score over the fuzzy
string score).
- Both scripts write real-time output; killing them mid-run loses at most one batch.
- The checkpoint and cache databases are separate on purpose: delete the checkpoint to force a full re-verification while keeping the (expensive) API cache.
- Deduplication happens on the normalized reference, so the same fabricated citation appearing across many model outputs costs exactly one lookup.
hallubib is a packaged, tested, PyPI-distributed CLI tool that checks the bibliography of a single manuscript. This project reuses its verification logic but re-targets it at a different problem: verifying millions of AI-generated references as a batch research pipeline. That difference drives everything below.
Inherited from hallubib
- The four-way status taxonomy (
Verified/Auto-correctable/Needs attention/Unknown) and its underlying idea - Resolving references against OpenAlex and Crossref, with DOI as a fast path
- Scoring a candidate on normalized-title similarity (
difflib.SequenceMatcher), author last-name agreement, year, and journal - Accent-stripping / punctuation-stripping text normalization before comparison
- Tolerating a one-year discrepancy as online-first vs. print
- TTL-based caching of API responses
Changed or added here
| Area | hallubib | This project |
|---|---|---|
| Input | .bib and .tex parsing |
CSV (no bibliography parsing at all) |
| Output | Terminal summary, Markdown, HTML report | CSV with 26 columns for downstream analysis |
| Scale model | One manuscript, in-memory | Millions of rows, SQLite-backed |
| Resume | None | pending/running/done state machine; interrupt and re-run at will |
| Deduplication | None | Bibliographic key hash, so a repeated fabrication costs one lookup |
| Cache | Filesystem JSON, single TTL | SQLite, async batched writer thread, separate hit/miss TTLs |
| Rate limiting | None | Per-source configurable intervals, Retry-After handling |
| Concurrency | ThreadPoolExecutor |
Thread-local sessions and DB connections, bounded in-flight futures, tunable pools |
| Sources | OpenAlex, Crossref, arXiv, Semantic Scholar | OpenAlex, Crossref, DOI only (arXiv and Semantic Scholar dropped) |
| Journal names | Abbreviation expansion table | Raw similarity only |
| Author matching | Last-name set overlap | Last-name + initials signature matching |
| Scoring | Rank by 0.5*title + 0.3*author + 0.2*year; verify at title >= 0.90 with first-author match |
Additive confidence with an explicit DOI term (+0.20); verify at title >= 0.95 with author >= 0.50 or DOI match |
| Semantic matching | None | title_matching.py: FAISS IVFPQ + sentence-transformers over a full corpus dump |
| Packaging | PyPI package, test suite, CI | Research scripts |
Two honest caveats about the comparison: hallubib covers more sources and handles journal abbreviations, both of which this project gave up; and it ships a real test suite, which this does not. If your goal is checking one paper's bibliography, use hallubib -- it is the better tool for that job. Use this if you need to verify references at dataset scale with interruptible runs.
If you use this code, please cite both this repository and the original:
@software{hallubib,
author = {Borza, Endre M{\'a}rk},
title = {hallubib: Check bibliography for hallucinations},
url = {https://github.com/endremborza/hallubib},
year = {2026}
}MIT -- see LICENSE.
This is a derivative work of hallubib, Copyright 2026 Endre Márk Borza, used under the MIT License. The original copyright notice is retained in LICENSE as that license requires.