academic-drain: mech-first shortcut — skip VLM OCR on pages with a clean text layer - #150
Merged
Conversation
…oven text layer (#147) paper-e2e.js resolvePage() now runs mech first and resolves source:mech without rendering or any GPU node when both engines clear a word floor and agree with each other; raster moves inside the VLM lane. compare.sh grows an optional [min-words] arg for the floor. drain.sh passes mechSelfAgreementPermille=930 mechMinWords=200 and bumps max-nodes to 7*pages+30. Calibration on 2,374 already-drained pages: the 930/200 gate shortcuts 78% of pages with 0.3% residual mech-vs-VLM disagreement; the floor, not the Dice threshold, is the safety-critical half (the bad tail — figure/cover/ UI-chrome pages — self-agrees at 1000 permille but runs sparse). All residuals were vlm32b-disputed under the old flow anyway. drain.sh's supersede handler now also matches script-changed-mid-run: shipping a new flow script invalidates every persisted run-id pin, which previously burned the 3-failure fuse instead of rotating. Live validation (2026-08-04): two 11-page papers drained end-to-end on the new flow — 51s and 101s wall-clock vs ~13min at the old pace; 21/22 pages resolved mech, the one sub-floor page (159 words) correctly took the VLM lane; supersede rotation observed on a stale-pinned paper; absorb epilogue committed both. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
mecattaf
added a commit
that referenced
this pull request
Aug 6, 2026
Tom's ruling, 2026-08-06: tables route to the VLM lane. This commit is that ruling's only record — the issue describing it was deliberately deleted. The defect. On a source:mech page the text layer is extracted column-major: every value in a table survives, and the row/column association does not. The mech-first shortcut (#150, b5cfc43) cannot see this. Its gate asks whether two mechanical engines agree, and poppler and mupdf linearize a table identically — they agree perfectly, at 1000 permille, on a reading in which the numbers no longer belong to the rows they are printed in. Nothing downstream recovers it, and the result reads as authoritative in the canonical corpus. Side by side: paper 28666de6 page 7 (mech: column headers, then all row labels, then a bare column of 24 values) against e3969f95 pages 5-9 (vlm8b: proper GFM tables). The fix. After the self-agreement gate passes, tables.sh looks for table evidence in the mechanical text; if it finds any, the page falls through to the VLM lane, which sees the page as a picture and emits a real GFM table. Mech-side table extractors (camelot, pdfplumber) were considered and not chosen. Two signals, calibrated 2026-08-06 against 2,181 already-drained pages that cleared the word floor and went to the VLM anyway, labelled by whether the VLM output actually contains a GFM table (240 do): caption alone 76.2% recall 2.0% false positive caption OR run>=4 85.0% recall 3.6% false positive <- shipped caption OR run>=6 82.1% recall 3.1% false positive The second signal — consecutive bare-numeric lines — is the direct fingerprint of column-major linearization, which emits one cell per line. It buys 9 points of recall for 1.6 of false positives, and the trade is not symmetric: a false positive costs one VLM page, a false negative costs a silently scrambled table in the corpus. Per-line numeric *density* was tried and dropped — it fired on author-affiliation numbering and stats-in-prose, and every sampled hit it added was wrong. Cost, honestly. Measured over every page that has resolved source:mech to date: 463 of 2,662, so 17.4% of would-be-shortcut pages now take the VLM lane. That is above the ~12% quoted when the ruling was made. The caption signal alone is 15.6% on this corpus, not 12%, so most of the gap predates the numeric-run half, which adds the last 1.8 points. Flagging the number, not re-deciding the ruling. Verified on the live flow (paper 28666de6, scratch state root, new package): the gate's per-page verdicts reproduce the standalone prediction exactly — pages 6, 7, 8, 9, 11 carry evidence and are the only five rastered, the other nine shortcut on mech as before. Also here: backfill-tables.sh, the one-time repair driver for pages already receipted as source:mech on a linearized table (462 pages across 142 papers at ship time). It re-runs whole papers through paper-e2e.js on a fresh flow-run id rather than carrying a second copy of the per-page ladder, and re-absorbs what absorb.py cannot (a repair overwrites an already-placed note). Not wired to any timer; running it is an operator step after this ships. Per-page worst case is now 8 nodes, and the assembler stamps tally-flow-e2e-2026-08-06 so a repaired paper.md is distinguishable. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #147.
resolvePage()now runsmechfirst; when poppler and mupdf both clear a word floor and agree with each other (Dice, reusingcompare.sh), the page resolves assource:mechwith no render and no GPU node. Raster moves inside the VLM lane, so shortcut pages produce no PNG at all. Fallback to the existing vlm8b → compare → vlm32b ladder is unchanged for scanned pages, sparse text layers, and engine disagreement.Threshold calibration (2,374 already-drained pages)
The floor, not the Dice gate, is the safety-critical half: the bad tail (figure pages, covers, LinkedIn-style print-to-PDFs) self-agrees at a perfect 1000‰ on a text layer that doesn't cover the visual content, but runs sparse. All 6 residual pages at 930/200 resolved
vlm32b-disputedunder the old flow anyway. The floor also keepscompare.sh's #127 truncation guard voucher-eligible on every shortcut page.Also fixed
drain.sh's supersede handler only matchedargs-changed-mid-run; shipping a new flow script pins out every persisted run-id withscript-changed-mid-run, which burned the 3-consecutive-failure fuse instead of rotating. Now both variants rotate.Live validation on the coordinator
Two 11-page papers drained end-to-end on the built package: 51s and 101s wall-clock vs ~13min at the old pace. 21/22 pages resolved
mech; the one page under the word floor (159 words) correctly took the VLM lane (exactly one render produced, resolvedvlm8b); supersede rotation observed once on the stale-pinned paper; absorb epilogue committed both to notes.At ~78% shortcut coverage the remaining 3,003-paper / ~193k-page queue drops from an estimated ~145–170 GPU-days to roughly 3–4 weeks of GPU-bound work, dominated by the genuinely-scanned minority.
🤖 Generated with Claude Code