preprint-fulltext v0.2.1 — 4-Tier Full-Text Engine, Indexed S3 PKZIP Range, & pf-text/2 Fidelity Overhaul

preprint-fulltext v0.2.1 — 4-Tier Full-Text Engine, Indexed S3 PKZIP Range Extraction, & pf-text/2 Fidelity Overhaul

preprint-fulltext v0.2.1 ships a new 4-tier full-text resolution engine, indexed 2-step PKZIP Range extraction for the requester-pays bioRxiv/medRxiv S3 buckets, and the complete pf-text/2 text-extraction fidelity overhaul (D1–D15).

1. 4-Tier Full-Text Engine & Indexed S3 PKZIP Range Extraction (v0.2.1)

• Tier 1 — Europe PMC REST API (inEPMC == "Y"): Free, $O(1)$ JATS XML for open-access preprints, gated on inEPMC == "Y" to avoid HTTP 500 responses on metadata-only records.
• Tier 2 — Direct openRxiv JATS XML (biorxiv_jats_xml): Preserves the jatsxml (https://www.biorxiv.org/content/early/YYYY/MM/DD/<id>.source.xml) URL from api.biorxiv.org/details/{server}/{doi} and parses it directly with parse_jats() when --html (allow_html=True) is enabled. Resolves cc_no, cc_by_nc, and cc_by_nc_nd preprints (including legacy 6-digit DOIs) in 0.3s–2.2s at $0.00 S3 cost, guarded by a process-wide Cloudflare circuit breaker.
• Tier 3 — Indexed 2-Step PKZIP Range S3 (BiorxivS3 + S3Index):
  • Indexes S3 month folders (Current_Content/<Month_Year>/) into a persistent local SQLite database (~/.cache/preprint-fulltext/s3_index.sqlite) by reading only the last 4 KB (Range: bytes={size-4096}-{size-1}) of each .meca archive in parallel (ThreadPoolExecutor(max_workers=32)) to parse the PKZIP End-of-Central-Directory and Central Directory.
  • Downloads and inflates only the ~20–30 KB compressed content/<id>.xml slice (Range: bytes={lfh_offset}-{end}, zlib.decompress(raw, -15)) instead of full 5–120+ MB .meca archives (e.g., extracting all 19 sections of 10.1101/486415 from a 122.9 MB .meca archive by downloading only 30.4 KB in 0.90s — a 4,038× bandwidth reduction).
  • Resolves legacy numeric DOIs (10.1101/NNNNNN) on S3 using version posting dates (candidate_dates) from BiorxivAPI.get_versions() and verifies parsed.doi == doi to disambiguate post-2020 6-digit manuscript counter wrap-arounds.
• Tier 4 — HighWire .full HTML Fallback (BiorxivHTML): Used for pre-JATS legacy preprints when --html (allow_html=True) is passed.

2. pf-text/2 Text-Extraction Fidelity Overhaul (D1–D15)

• Single-Pass Non-Mutating Tree Renderer (preprint_fulltext/core/render.py):
  • D1 (External Links): Preserves hyperlinks inline ("visible (URL)") and populates FullText.links[] with character offsets.
  • D2 (Citations): Strips bracketed and superscript reference groups cleanly without leaving empty (), [], or (,,,); preserves narrative "Author (Year)" citations; populates FullText.citations[].
  • D3 (Cross-References): Preserves "Figure 1e", "Table 1", "Section 1.4", and "Equation (6)" in prose and populates FullText.xrefs[].
  • D4 & D5 (Figures & Tables): Extracts FullText.figures[] and FullText.tables[] (Markdown pipe tables, plus opt-in --fetch-tables for HTML and --legends-as-section).
  • D6 (Superscripts & Subscripts): Renders exponents with caret notation (10^6, 10^−8, Ca^2+) or Unicode (--scripts unicode) instead of flattening 10^6 to 106.
  • D7 (JATS Sentence Reordering Fixed): Eliminates lxml .remove() tail-text corruption that previously scrambled ~40% of JATS paragraphs containing <italic> + <xref>.
  • D8–D15: MathML/TeX formula rendering (FullText.formulas[]), HTML footnotes/funding & JATS <back> matter, anchored section classification (Section.role), and S3 .meca Windows-backslash manifest normalisation.