v0.3.1 — Contributor fixes and production hardening

Contributor fixes and production hardening

The first release to carry outside contributions. Three fixes to corpus text fidelity — typographic ligatures, en/em dashes in figure references, and presentational HTML in EPUBs — arrived as pull requests from @pig18888; a follow-up collapses the project's scattered dash definitions into one. Alongside them: two fixes proven on a full 154-book production run before landing here, and a docs-only split into three usage profiles so an agent installs only the stack the user actually asked for.

Fixed

• Typographic ligatures are expanded so the corpus stays searchable (#6, thanks @pig18888) — PDF text layers emit fi fl ff ffi ffl ſt st as single codepoints, so specific finds zero hits in a book that holds 116 occurrences of specific. Mapped to ASCII pairs explicitly — not via NFKC, which would also rewrite the scientific notation (μ, U+2212 minus, superscripts) the corpus needs to keep.
• Figure and table references match the full range of dashes (#8, thanks @pig18888) — Katzung 16e runs on en dashes throughout (717 en dashes to 8 ASCII hyphens); the old [-.] pattern matched 14 references and left figure extraction blind for the whole book. Detection now spans U+2010–U+2015 and U+2212.
• Presentational HTML is stripped from converted EPUBs (#9, thanks @pig18888) — calibre's <a> / <div> / <span> wrappers were 41% of every character in one 52-chapter book, diluting each chunk the indexer builds. <sup> / <sub> and table tags are deliberately kept.
• One dash definition, shared by both sides of the figure pipeline (#10) — converged into shared/config.py (DASH_CHARS / SEP_CLASS), imported at all four call sites. Tests 242 / 0 / 5, unchanged from baseline.
• Docling degraded-book warning no longer flags every later book after one failure — the sticky last_error check is replaced with a cumulative failure_count delta (proven on the 154-book run).
• A PDF whose page tree pdfminer refuses is repaired through fitz instead of silently losing every table — rewritten to a temp copy and retried when pdfplumber parses 0 pages but fitz reads the book fine (proven on the 154-book run).

Changed

• Three usage profiles replace the implicit all-or-nothing setup (docs) — A converter-only, B + note workflow and figures, C + semantic search via vault-search. AGENTS.md asks which profile first, before any install.
• The Surya-adapter reference now reports measured GPU numbers (closes #4) — ~0.15 pages/s end to end and a capped inference server fitting in ~2.3 GB VRAM on an 8 GB card. Flagged as an order-of-magnitude anchor (n=5 pages, measured once, on a shared card) — not a benchmark.

Full changelog: CHANGELOG.md

---

外部貢獻修正與正式批次強化

第一個帶外部貢獻的版本。三項語料庫文字保真修正——ligature、圖表引用的 en/em dash、EPUB 裝飾性 HTML——以 pull request 形式來自 @pig18888；再加一個把散落的 dash 定義收斂成單一來源的 follow-up。另有兩項在 154 本正式批次跑過才落地的修正，以及把安裝拆成三條使用路徑的 docs 改動。

修正 (Fixed)

• Ligature 展開，語料庫維持可搜尋（#6，感謝 @pig18888）— PDF 文字層把 fi fl ff ffi ffl ſt st 存成單一碼位，specific 在含 116 個 specific 的書裡搜不到。顯式對映 ASCII，不走 NFKC，以保住 μ／U+2212／上標。
• 圖表引用跨全 dash 範圍（#8，感謝 @pig18888）— Katzung 16e 全書用 en dash（717 vs 8 個 ASCII），舊 [-.] 只中 14 個、整本抽圖失明。改成涵蓋 U+2010–U+2015 與 U+2212。
• EPUB 裝飾性 HTML 清除（#9，感謝 @pig18888）— calibre 的 <a>／<div>／<span> 在一本 52 章書佔 41% 字元、稀釋每個 chunk。保留 <sup>／<sub> 與 table 標籤。
• 單一 dash 定義，圖片管線兩側共用（#10）— 收斂到 shared/config.py，四個使用點 import。測試 242／0／5 不變。
• Docling 降級警告不再一錯報全部— sticky last_error 改用累計 failure_count 差值（154 本正式批次驗證）。
• pdfminer 拒讀的 page tree 改用 fitz 重寫，救回原本會整本消失的表格— pdfplumber 讀到 0 頁但 fitz 正常時，重寫暫存檔重試（154 本正式批次驗證）。

變更 (Changed)

• 三條使用路徑取代隱含的全有全無安裝（docs）— A 只轉檔／B ＋筆記工作流與圖片／C ＋語意搜尋（vault-search）。AGENTS.md 先問走哪條路徑，再安裝。
• Surya adapter 參考補上實測 GPU 數字（closes #4）— ~0.15 pg/s、capped server 在 8GB 卡上約 2.3 GB VRAM。標明 n=5／單次／共卡，是量級錨點而非 benchmark。

完整變更紀錄：CHANGELOG.md

v0.3.0 — The OCR rung, and telling the truth in the docs

Where 0.2.0 was about table fidelity, this one is about the two places the project was
quietly asking users to trust something that wasn't there: an OCR rung whose central
component was never shipped, and documentation describing behaviour the code did not have.
Both were found the same way — by an outside user's agent following the docs literally and
burning hours on it (#3).

Added

• A reference OCR adapter ships — converter/surya_adapter.py, targeting Surya 0.22.x, plus
  docs/surya-adapter.md (#4).
  SURYA_ADAPTER had been a first-class config value, required for surya_available(), executed
  as a subprocess — and the file it pointed at existed only on the author's machine, with no
  published interface. An agent following the setup guide reasonably assumed it was part of the
  repo, could not find it, and reverse-engineered one against the removed surya.ocr API. The doc
  now publishes the contract (JSON Lines on stdout, fixture + blocks[].text + required
  blocks[].bbox, one line per image even when blank, logging on stderr, non-zero exit fails the
  batch) so any engine can take that rung, and states plainly why an adapter written for Surya
  0.17 dies on 0.22: surya.ocr is gone, FoundationPredictor became SuryaInferenceManager,
  and blocks now carry html rather than flat text.
• Documented, tested memory caps for the OCR inference server. Surya 0.22 serves its VLM
  behind llama.cpp or vllm, and llama.cpp sizes its KV cache as
  parallel × ctx_per_slot — defaulting to 8 × 12288 = 98304 tokens for a model whose weights
  are ~1.4 GB. That cache, not the model, is what became a reported 48 GB run for one user. The
  doc ships a launch command with the caps in it; measured peak RSS with them was 3121 MB.
• OCR output QC gate — the OCR rung now fails loud like every other rung. surya_ocr_pdf()
  counts unparseable adapter lines, images with no answer, and empty pages, and raises instead
  of writing markdown when the empty-page ratio exceeds T2N_OCR_EMPTY_PAGE_MAX (0.35) or mean
  characters per page falls below T2N_OCR_MIN_CHARS_PER_PAGE (200). Previously a JSON parse
  failure was silently continued and a near-empty book reported success in KB — the shape of a
  real incident where a scanned book produced 20 KB where ~1.6 MB was expected (~25 chars/page).
  Thresholds are calibrated against two real scanned references (689 and 788 pages: empty ratios
  0.054 / 0.003, mean 1791 / 2005 chars per page), so they clear legitimate blank and plate pages
  by roughly an order of magnitude. Per-book counters are surfaced in the batch report.

Fixed

• Two-column reading order on OCR'd pages — the OCR path sorted blocks by (y0, x0), which
  walks across the gutter and back on every band of a two-column page, interleaving the columns.
  Every character is present and individually correct, so no downstream check could see it — the
  same defect the fitz path already carried a dedicated column sort for. Both paths now share one
  column_order_boxes() (T2N_COLUMN_SORT=0 restores the old sort on both). Byte-identity of the
  fitz path across two full books was verified as part of the same re-conversion run used for #5.
• DOCLING_DEVICE defaults to auto instead of a hardcoded "cuda", resolving CUDA → Apple
  Silicon MPS → CPU inside the worker. A CPU-only machine no longer asks Docling for CUDA, and
  mps is now reachable at all (previously anything that wasn't the literal "cuda" mapped to
  CPU). Resolution logic is unit-tested without torch or a GPU; the real MPS path is unverified
  — there is no Apple Silicon in the development environment.
• pdfplumber page cache is released after each page (#5).
  convert_pdf() opened one pdfplumber.PDF for a whole book and reached into plumber.pages[i]
  per page. PDF.pages holds every materialized Page for the object's lifetime, and each Page
  caches its chars / edges / rects on first use, so peak memory grew roughly linearly with the
  number of pages the table gate let through — nothing released them until close() at the end of
  the book. Every page is read exactly once, so flush_cache() + close() after its single use
  cannot change output, and byte-identity was verified on two books (a 1297-page table-dense
  reference, 627 tables, and a 638-page ordinary one): 23/23 output files identical each, before
  and after. Measured peak RSS: 6899 MB → 803 MB on the dense book (8.6x), 6209 MB → 133 MB
  on the ordinary one (46.6x), and the curve flattens instead of climbing to the last page.
  The ordinary book is the more interesting number: the issue predicted table-sparse books would
  barely leak because the gate skips them, but the gate admits far more pages than actually yield
  tables (638 pages, 12 tables extracted, still 6.2 GB), so the leak was never confined to
  table-dense books.

Changed

• Documentation audited against the code, and corrected where it disagreed. Each of these
  was somewhere a reader could act on the docs and get a different result than the repo delivers:
  semantic search was described as if an indexer shipped (it does not — post_convert.py --index
  prints [skip] and returns success without INDEXER_SCRIPT, which is now stated, with the
  companion repo vault-search named as the thing to
  point it at); requirements.txt was said to cover the semantic-search stack (lines 7-16 are all
  comments); the figure-remap contract was documented with a match_method key that the
  validator actively rejects (the real key is match_quality, and qc_degraded / qc_skipped
  were undocumented); docs used a textbook-md/ output directory that is really OUTPUT_DIR
  (default ./output); architecture.md pointed at a section of itself that does not exist; and
  scan_fix_negatives.py read OLLAMA_VISION_MODEL while the rest of the repo uses the T2N_
  namespace (now T2N_OLLAMA_VISION_MODEL, with the old name kept as a fallback so existing
  environments don't silently switch models). Also corrected: the skill claimed a strict figure
  hard_fail exits 2 (it exits 1), and that chapter splitting is not attempted on OCR'd books
  (it is, best-effort).
• README no longer claims per-page OCR engine selection (both variants). The detection
  signals are per-page; the routing decision is per-book — one trip of the check sends the whole
  PDF to OCR. Per-page routing is future work and is now labelled as such rather than described
  as shipped.
• Setup guide no longer provisions OCR up front (AGENTS.md, #3).
  Step 1 asked whether the user had a GPU and pointed at the OCR ladder before a single page had
  been converted, which reads to a coding agent as "install the OCR stack now." It is an exception
  path: OCR routing exists only in the --batch-dir code path, and the single-file path never
  invokes it at all. A first-time user's agent installed Surya, hit the missing adapter (#4),
  wrote its own against the removed surya.ocr API, chased that into a local VLM inference server,
  and exhausted 48 GB of RAM — on a born-digital PDF that converted correctly with no OCR at all.
  The GPU question is gone from Step 1 and OCR now lives in a new Step 4.5, entered only when
  Step 4's output is actually garbled or empty, with a diagnose-before-installing checklist.
• Related-projects section now links note-supplement
  (both README variants). It covers the direction this pipeline deliberately does not: merging new
  source material into notes that already exist, where the risk is not missing content but silently
  overwriting content the existing note already got right.

v0.2.0 — Table fidelity

A sustained pass over how the pipeline handles textbook tables, driven by measuring the real
corpus rather than by intuition. Every change defaults to preserving prior output (a kill-switch
restores byte-identical behavior) unless it corrects output that was already wrong.

Added

• Cross-page table merge (T2N_TABLE_MERGE=1, default OFF) — stitch a table that ends near
  a page bottom to a geometrically-matching table at the top of the next page; dedupes the
  repeated header, leaves a <!-- table continues from page N --> trace.
• Docling table rung (T2N_DOCLING=1, default OFF) — a layout-model table extractor that
  gets multi-column shape right where pdfplumber collapses borderless grids; falls back to
  pdfplumber per-page when it finds nothing, so no page is left worse off.
• Table QC gate — flags structural damage (ragged rows, empty first cell, run-together
  text, single-column collapse, content-retention) with a <!-- ⚠️ … --> trace, never
  auto-"fixing" it.
• Out-of-band review queue (T2N_REVIEW_QUEUE=1, default OFF; recommended ON for clinical
  corpora) — the QC gate sees structure but not misbinding (a value merged into the wrong
  row of an otherwise clean grid). There is no safe automatic fix, so the high-risk subset
  (continuation-page tables + dosage/threshold tables) is flagged for a bring-your-own-model
  second opinion. In testing ~1 in 6 continuation×dose tables carried a high-severity misbinding
  vs ~0 in a random sample. See docs/table-review.md.
• Spanned category-header collapse (T2N_TABLE_HEADER_COLLAPSE=1, default ON) — a section
  header broadcast across every column of a wide grid becomes a phantom full-width data row. A
  real data row never repeats one ≥15-char string across ≥3 columns, so the row is re-cast as a
  single header cell — structural only, never moves a value between rows. Hit 130/232 (56%) of
  one dense pharmacology reference's tables.
• Book-level table-reliability banner (detection only) — when ≥40% of a book's tables (given
  ≥10) trip a QC flag, a > [!caution] banner is hung at the top of the markdown telling the
  downstream model to verify every table against the source PDF; reliability_flagged /
  flag_rate land in the per-book stats. One reference ran 66%.
• Whole-book table-failure warning (T2N_BOOK_TABLE_CHECK, default ON) — table loss is
  bimodal (a book extracts fine or loses every table); a loud warning is emitted when a book
  yields 0 tables despite ≥10 captions, or pdfplumber parses 0 pages while fitz opens fine.

Fixed

• Page-frame pseudo-table rejection (T2N_TABLE_FRAME_REJECT, default ON) — page-decoration
  rectangles made pdfplumber "find" a whole-page 1-column table that dumps every word into one
  cell (real multi-column tables arrive column-interleaved but caption-and-values intact, reading
  as clean data while the binding is destroyed). Rejected and replaced by a trace comment.
  Measured at 9.9% of extracted tables across 128 books.
• Two false-positive table detections — running prose and a navigation strip that the frame
  heuristic misread as tables, narrowed without regressing real rejections.
• Docling ligature corruption — repair ligature glyphs against the page's own text layer
  before emitting, so the markdown and the QC retention check both see corrected text.
• Furniture false-rejections — a geometry-only running-header/footer band rule, measured for
  its false-kill rate and narrowed.

v0.1.0 — Initial public release

PDF/EPUB textbook → AI-searchable markdown → structured, fully-cited notes. Five stages:
convert (0-token, silent-failure detection, column-aware reading order), chunk (heading-aware),
retrieve (local LanceDB semantic search with source weighting), write (template-driven,
citation-enforced, non-destructive), extract figures (geometric match + deterministic QC gate).
Bilingual READMEs and note templates; ships as skills an AI agent installs from AGENTS.md.