Skip to content

v0.2.0 — Table fidelity

Choose a tag to compare

@drpwchen drpwchen released this 30 Jul 15:39

A sustained pass over how the pipeline handles textbook tables, driven by measuring the real
corpus rather than by intuition. Every change defaults to preserving prior output (a kill-switch
restores byte-identical behavior) unless it corrects output that was already wrong.

Added

  • Cross-page table merge (T2N_TABLE_MERGE=1, default OFF) — stitch a table that ends near
    a page bottom to a geometrically-matching table at the top of the next page; dedupes the
    repeated header, leaves a <!-- table continues from page N --> trace.
  • Docling table rung (T2N_DOCLING=1, default OFF) — a layout-model table extractor that
    gets multi-column shape right where pdfplumber collapses borderless grids; falls back to
    pdfplumber per-page when it finds nothing, so no page is left worse off.
  • Table QC gate — flags structural damage (ragged rows, empty first cell, run-together
    text, single-column collapse, content-retention) with a <!-- ⚠️ … --> trace, never
    auto-"fixing" it.
  • Out-of-band review queue (T2N_REVIEW_QUEUE=1, default OFF; recommended ON for clinical
    corpora) — the QC gate sees structure but not misbinding (a value merged into the wrong
    row
    of an otherwise clean grid). There is no safe automatic fix, so the high-risk subset
    (continuation-page tables + dosage/threshold tables) is flagged for a bring-your-own-model
    second opinion. In testing ~1 in 6 continuation×dose tables carried a high-severity misbinding
    vs ~0 in a random sample. See docs/table-review.md.
  • Spanned category-header collapse (T2N_TABLE_HEADER_COLLAPSE=1, default ON) — a section
    header broadcast across every column of a wide grid becomes a phantom full-width data row. A
    real data row never repeats one ≥15-char string across ≥3 columns, so the row is re-cast as a
    single header cell — structural only, never moves a value between rows. Hit 130/232 (56%) of
    one dense pharmacology reference's tables.
  • Book-level table-reliability banner (detection only) — when ≥40% of a book's tables (given
    ≥10) trip a QC flag, a > [!caution] banner is hung at the top of the markdown telling the
    downstream model to verify every table against the source PDF; reliability_flagged /
    flag_rate land in the per-book stats. One reference ran 66%.
  • Whole-book table-failure warning (T2N_BOOK_TABLE_CHECK, default ON) — table loss is
    bimodal (a book extracts fine or loses every table); a loud warning is emitted when a book
    yields 0 tables despite ≥10 captions, or pdfplumber parses 0 pages while fitz opens fine.

Fixed

  • Page-frame pseudo-table rejection (T2N_TABLE_FRAME_REJECT, default ON) — page-decoration
    rectangles made pdfplumber "find" a whole-page 1-column table that dumps every word into one
    cell (real multi-column tables arrive column-interleaved but caption-and-values intact, reading
    as clean data while the binding is destroyed). Rejected and replaced by a trace comment.
    Measured at 9.9% of extracted tables across 128 books.
  • Two false-positive table detections — running prose and a navigation strip that the frame
    heuristic misread as tables, narrowed without regressing real rejections.
  • Docling ligature corruption — repair ligature glyphs against the page's own text layer
    before emitting, so the markdown and the QC retention check both see corrected text.
  • Furniture false-rejections — a geometry-only running-header/footer band rule, measured for
    its false-kill rate and narrowed.