Structural tables: documents are no longer flattened before the model sees them.
Everything here was tuned against a 314-file corpus of real invoices and then
measured for accuracy against 76 Factur-X documents with embedded EN16931 ground
truth, rather than against intuition. Two of those measurements changed the design
and one stopped a change from shipping — details inline.
Fixed
- Trailing-minus accounting notation in
LenientDecoding.parseDecimal. SAP /
German invoice amounts print the sign after the digits ("1,12 -","12-"), and
a real corpus document does exactly that. Those forms now decode as negatives
(−1.12, −12) instead of being rejected. Optional whitespace before the sign is
allowed; a trailing+is accepted as an explicit positive. Two-sign forms
(-12-,12--,+12-, parenthesised-plus-trailing) stay rejected, as do the
existing multi-dot / multi-sign / junk-exponent hardenings, pinned as a table so
the boundary is visible in one place. Parenthesised negatives and the scientific
path are unchanged.
Added
- Structural tables surface to the model and callers (stage 2). Detected tables
from geometric reconstruction are returned onExtractionResult.tablesand may be
appended to the extraction prompt as a labelled Markdown section (linear document
text is left unchanged — additive, not a substitute). When no tables are found or
tableDetectionis.off, the prompt is byte-identical to the pre-feature shape.
Chunked runs assign whole tables by page (never a half table); if page filtering
would drop every table, the full set is attached to the first chunk. Docs:
API, Examples, README limitations.
Changed
-
Schema-gated table prompt injection under
.automatic. Always appending
detected tables to the prompt hurt header-field accuracy on types without
collections (measured −11.4 pp header accuracy on 76 invoices) while helping
line-item structure when the schema has arrays..automaticstill runs geometric
detection and always exposes grids onresult.tables; the prompt section is
included only when the targetextractionSchemacontains a collection (array)
anywhere, including nested. Header-only types get a prompt byte-identical to
.off. No new enum case; callers who need grids without line items still use
result.tables.Measured on 76 Factur-X invoices with embedded EN16931 ground truth, same local
MLX model per run,temperature = 0, tables.automaticvs.off:Qwen2.5 1.5B 4-bit Qwen2.5 7B 4-bit header accuracy, tables off 62.7% 95.2% header accuracy, tables on 51.3% 95.8% Δ on the files where tables were injected −11.4 pp +0.8 pp Δ line description −1.0 pp +7.1 pp So the distraction is a small-model artefact, not a property of the approach:
at 7B the table section no longer costs header accuracy and clearly helps line
descriptions. Schema gating is kept because it is exactly right for weak models
and for header-only schemas, and costs a strong model nothing. On header-only
types the two arms produced byte-identical extractions on all 76 documents —
only the reportedtablescount differs, which is the intended behaviour. -
Table detection precision against real invoices.
TableDetectornow splits
multi-column regions on large vertical gaps (line items vs totals), bridges short
single-column description lines under items, keeps only “spine” rows (numeric /
header-like) when clustering columns, drops sparse and all-empty columns, and
requires fill density ≥ 0.55 before emission. On a 314-file invoice corpus this
raised median density from ~0.71 to 1.0, cutpdf/average tables/file from 3.4
to 1.8, and eliminated all-empty columns, while preserving line-item recall on
Coolblue, Sammy Maystone,fixtures/invoice.pdf, andfixtures/receipt.png.
Unit tests cover a Coolblue-style merged-region layout and an empty-column case. -
Recursive XY-cut (horizontal column bands). Before row grouping, the detector
splits a page on a vertical whitespace corridor — the largest mid-X gap between
blocks — when the gap is ≥ 0.06, ≥ 1.25× each band’s internal mid-X structure,
both bands have ≥ 2 multi-column rows, and their Y-ranges overlap (≥ 25% of the
shorter band). Each band then runs the existing vertical region split. This
recovers side-by-side documents that previously merged into one sparse mega-grid
(or density-rejected to zero), e.g.hard/invoice_table_detect_img1.jpg, without
bisecting single-document line-item tables (Coolblue, Sammy). Unit test covers two
independent 3-column grids separated by a wide corridor. -
XY-cut corridor vs table-gutter discrimination. Width alone cannot tell a
vertical region boundary from a table’s inter-column gutter, so pure XY-cut
shattered genuine line-item grids (e.g. Kostenrechnung4×6→ six half-width
fragments; Hetzner8×4→8×2/4×2/11×2). A candidate corridor is now kept
only when multi-column row baselines across the gap largely fail to align: the cut
is refused when the shorter multi-col side’s mid-Y match is ≥ 0.80 (strong
co-tabular signal, even if one side has extra chrome rows) or when both directed
matches are ≥ 0.50 (balanced shared grid). Side-by-side documents with unrelated
line positions still split (hard/invoice_table_detect_img1.jpg→4×4+6×5).
On the 90-filepdf/set, line-item-shaped grids (rows ≥ 3, cols 3–6, density
≥ 0.80) recover to 36/90 (40%) from 32/90 under pure XY-cut, matching the
pre-horizontal-cut tightening rate, while median density stays 1.0, p90 columns
5, andpdf/avg tables/file 1.79. Unit test pins a dense 6-column grid staying
one table against two misaligned 3-column grids staying two.