v0.2.0 — Table fidelity
A sustained pass over how the pipeline handles textbook tables, driven by measuring the real
corpus rather than by intuition. Every change defaults to preserving prior output (a kill-switch
restores byte-identical behavior) unless it corrects output that was already wrong.
Added
- Cross-page table merge (
T2N_TABLE_MERGE=1, default OFF) — stitch a table that ends near
a page bottom to a geometrically-matching table at the top of the next page; dedupes the
repeated header, leaves a<!-- table continues from page N -->trace. - Docling table rung (
T2N_DOCLING=1, default OFF) — a layout-model table extractor that
gets multi-column shape right wherepdfplumbercollapses borderless grids; falls back to
pdfplumberper-page when it finds nothing, so no page is left worse off. - Table QC gate — flags structural damage (ragged rows, empty first cell, run-together
text, single-column collapse, content-retention) with a<!-- ⚠️ … -->trace, never
auto-"fixing" it. - Out-of-band review queue (
T2N_REVIEW_QUEUE=1, default OFF; recommended ON for clinical
corpora) — the QC gate sees structure but not misbinding (a value merged into the wrong
row of an otherwise clean grid). There is no safe automatic fix, so the high-risk subset
(continuation-page tables + dosage/threshold tables) is flagged for a bring-your-own-model
second opinion. In testing ~1 in 6 continuation×dose tables carried a high-severity misbinding
vs ~0 in a random sample. Seedocs/table-review.md. - Spanned category-header collapse (
T2N_TABLE_HEADER_COLLAPSE=1, default ON) — a section
header broadcast across every column of a wide grid becomes a phantom full-width data row. A
real data row never repeats one ≥15-char string across ≥3 columns, so the row is re-cast as a
single header cell — structural only, never moves a value between rows. Hit 130/232 (56%) of
one dense pharmacology reference's tables. - Book-level table-reliability banner (detection only) — when ≥40% of a book's tables (given
≥10) trip a QC flag, a> [!caution]banner is hung at the top of the markdown telling the
downstream model to verify every table against the source PDF;reliability_flagged/
flag_rateland in the per-book stats. One reference ran 66%. - Whole-book table-failure warning (
T2N_BOOK_TABLE_CHECK, default ON) — table loss is
bimodal (a book extracts fine or loses every table); a loud warning is emitted when a book
yields 0 tables despite ≥10 captions, orpdfplumberparses 0 pages whilefitzopens fine.
Fixed
- Page-frame pseudo-table rejection (
T2N_TABLE_FRAME_REJECT, default ON) — page-decoration
rectangles madepdfplumber"find" a whole-page 1-column table that dumps every word into one
cell (real multi-column tables arrive column-interleaved but caption-and-values intact, reading
as clean data while the binding is destroyed). Rejected and replaced by a trace comment.
Measured at 9.9% of extracted tables across 128 books. - Two false-positive table detections — running prose and a navigation strip that the frame
heuristic misread as tables, narrowed without regressing real rejections. - Docling ligature corruption — repair ligature glyphs against the page's own text layer
before emitting, so the markdown and the QC retention check both see corrected text. - Furniture false-rejections — a geometry-only running-header/footer band rule, measured for
its false-kill rate and narrowed.