Repository navigation
PR 58 Measurements
← #58 References and citations
Measured 2026-10-09 on the ns-pond corpus on beast, read-only. The scripts, full
outputs, README and hand-read gold are on beast in /data/alejandro/jk-references/
(src/, out/, README.md, gold/).
- Sample: 200 pubget, 100 PMC, 200 Elsevier, 200 ACE-only and 150 PDF-only records, plus 60 papers each that have both ACE and pubget, or both ACE and Elsevier.
-
Exact gold: every JATS
xref ref-type="bibr"and Elsevier bibliography cross-ref, placed intext.txt, for 620 papers (out/xml_cites.jsonl). -
Hand-read gold: 100 citations across six routes, with verdicts
(
gold/handcheck.jsonl).
| JATS (360) | Elsevier (260) | ACE-only (200) | |
|---|---|---|---|
| reference list found | 100% | 100% | 70.5% of pages |
| entries with DOI · PMID | 69.8% · 86.3% | 16.1% · 0% | 46.6% · 39.2% |
| link ids that resolve | 100% | 100% | — |
citations placed in text.txt
|
98.1% | 99.2% | 95.9% |
| marker missing from text (before #66) | 17.5% | 0% | — |
| hand check | 20/20 | 15/15 | 23/25 |
ACE pages that hold a list, by publisher: OUP 39/40, Wiley 39/40, J Neurosci 19/19, Springer 8/23, ScienceDirect 8/19, MIT Press 0/11.
Scored against the exact links:
| list | papers | precision | recall (visible markers) |
|---|---|---|---|
| the XML's own | JATS, 358 | 0.98 | 0.91 |
| the XML's own | Elsevier, 260 | 0.99 | 0.96 |
| Crossref's + OpenAlex names | author-year, 110 | 0.97 | 0.87 |
| Crossref's + OpenAlex names | numbered, 110 | 0.87 | 0.78 |
Precision by pattern, with the XML list: [n] 0.997, (n) 0.96, loose numbers
0.97, author-year 0.97–0.99. In PDFs, read by hand: brackets 3/3, author-year
14/15, loose numbers 13/15 (once Docling's | table rows, decimals and figure
labels were excluded).
| papers with a list | ids on entries | notes | |
|---|---|---|---|
| Crossref | 182/198 | DOI 91.9% | ordered; APA, AMA and some of J Neurosci deposit nothing |
| OpenAlex | 198/200 | DOI 94.5%, PMID 87.1% | unordered; keyless use metered at $0.10/day |
| Europe PMC | 191/200 | PMID 94.7%, DOI 1.6% | unmatched entries become "AUTHOR UNKNOWN" placeholders |
| Semantic Scholar | 453/620 gold papers | DOI 90.5% | withheld for 152 of 260 Elsevier papers |
On licensing: Crossref references are open facts (since June 2022), and OpenAlex is
CC0. Semantic Scholar's API licence forbids redistribution, but its bulk
citations dataset is ODC-BY. Europe PMC's terms for references could not be
confirmed.
- 98.6% of contexts are found in
text.txt, 76% verbatim. - A context found verbatim touches a sentence citing its reference 87% of the time.
- If the reference's own marker is found inside the context, it is right 98.8% of the time; otherwise 34%.
- S2 recovers 38% of all gold (sentence, reference) pairs.
The sample averages 63 references and 79 citations per paper. That is 46 KB per paper as JSON (2.9 GB for the 61k corpus) and 9.7 KB as zstd Parquet (0.6 GB).