Repository navigation
PR 29 Keep Every Table
Merged 2026-09-29. Frozen. The system as it stands is described in Design.
Before this change the extract stage was also a filter. Four separate places threw away part of what the document said, and each one was found by measuring what actually reached the serialiser rather than by reading the code.
The full measurements and the reasoning behind each fix are in the pull request. This page records what it did to the corpus, and the one thing that went wrong afterwards.
| where | what was lost | how it was found |
|---|---|---|
| ace | every table ACE did not itself recognise as an activation table | 100% of 500 sampled ace tables carried coordinates — a filtered set has no negative class |
| the header flags and spans Docling detected |
# header marker present on 3.6% of pdf tables against 92.3% after |
|
| ace, elsevier, pdf | the caption and the footnote | elsevier's footnote is in ce:legend, populated 59.9%, and the extractor read only ce:table-foot, populated 0% |
| the footnote Docling did not link | reading the linked field alone recovered 3.9%; position recovers the rest |
15.7% of real tables state their coordinate space only in that surrounding text, so a dropped caption is a dropped space.
Re-extraction of all 29,525 ace articles:
| before | after | |
|---|---|---|
| tables | 35,404 | 108,381 |
| rescued from the HTML | 0 | 72,977 |
| carrying coordinates | — | 33,563 (31%) |
Across ace, elsevier and pdf, 131,667 articles were re-extracted. 93.7% of the serialised text of the curated training tables changed, and 1,591 of them gained a footnote they had never had.
Keeping every table is only half of it. extract no longer decides which
tables are worth anything — it extracts, and a separate stage judges. That
judgement is #30 Triage, which exists because this change
left 1.33M tables where there had been 385K, and something has to say which
of them earn a model call.
An extract artifact's summary still carries tables_with_coordinates,
because the API reads it, but nothing in the pipeline routes on it any more.
The re-extraction ran with the default catalog_root, which is a relative
./.catalog. cache_root was absolute and shared, so the payloads landed in
the shared cache and the results landed in a private catalog nobody reads.
Every stage reported success. Nothing said otherwise.
It was found three days later, when curated training data rebuilt from the
catalog still carried the old captions. The recovery merged the cache payloads
into the catalog under the fingerprints the stage would have written — the
same eight keys, blob = sha256 of compact sorted JSON, fingerprint = blake2b
over ("extract", source, EXTRACTOR_VERSIONS[source]) with the download
artifact's fingerprint as upstream — and was verified against 1,638 artifacts
the pipeline itself had produced: fingerprint, blob and summary matched on all
of them. 129,751 rows merged; no other stage was touched.
Settings.ensure_directories now logs both roots and warns when an absolute
cache_root sits beside a relative catalog_root. See
Data safety.