Contributor fixes and production hardening
The first release to carry outside contributions. Three fixes to corpus text fidelity — typographic ligatures, en/em dashes in figure references, and presentational HTML in EPUBs — arrived as pull requests from @pig18888; a follow-up collapses the project's scattered dash definitions into one. Alongside them: two fixes proven on a full 154-book production run before landing here, and a docs-only split into three usage profiles so an agent installs only the stack the user actually asked for.
Fixed
- Typographic ligatures are expanded so the corpus stays searchable (#6, thanks @pig18888) — PDF text layers emit
fi fl ff ffi ffl ſt stas single codepoints, sospecificfinds zero hits in a book that holds 116 occurrences ofspecific. Mapped to ASCII pairs explicitly — not via NFKC, which would also rewrite the scientific notation (μ, U+2212 minus, superscripts) the corpus needs to keep. - Figure and table references match the full range of dashes (#8, thanks @pig18888) — Katzung 16e runs on en dashes throughout (717 en dashes to 8 ASCII hyphens); the old
[-.]pattern matched 14 references and left figure extraction blind for the whole book. Detection now spans U+2010–U+2015 and U+2212. - Presentational HTML is stripped from converted EPUBs (#9, thanks @pig18888) — calibre's
<a>/<div>/<span>wrappers were 41% of every character in one 52-chapter book, diluting each chunk the indexer builds.<sup>/<sub>and table tags are deliberately kept. - One dash definition, shared by both sides of the figure pipeline (#10) — converged into
shared/config.py(DASH_CHARS/SEP_CLASS), imported at all four call sites. Tests 242 / 0 / 5, unchanged from baseline. - Docling degraded-book warning no longer flags every later book after one failure — the sticky
last_errorcheck is replaced with a cumulativefailure_countdelta (proven on the 154-book run). - A PDF whose page tree pdfminer refuses is repaired through fitz instead of silently losing every table — rewritten to a temp copy and retried when pdfplumber parses 0 pages but fitz reads the book fine (proven on the 154-book run).
Changed
- Three usage profiles replace the implicit all-or-nothing setup (docs) — A converter-only, B + note workflow and figures, C + semantic search via vault-search.
AGENTS.mdasks which profile first, before any install. - The Surya-adapter reference now reports measured GPU numbers (closes #4) — ~0.15 pages/s end to end and a capped inference server fitting in ~2.3 GB VRAM on an 8 GB card. Flagged as an order-of-magnitude anchor (n=5 pages, measured once, on a shared card) — not a benchmark.
Full changelog: CHANGELOG.md
外部貢獻修正與正式批次強化
第一個帶外部貢獻的版本。三項語料庫文字保真修正——ligature、圖表引用的 en/em dash、EPUB 裝飾性 HTML——以 pull request 形式來自 @pig18888;再加一個把散落的 dash 定義收斂成單一來源的 follow-up。另有兩項在 154 本正式批次跑過才落地的修正,以及把安裝拆成三條使用路徑的 docs 改動。
修正 (Fixed)
- Ligature 展開,語料庫維持可搜尋(#6,感謝 @pig18888)— PDF 文字層把
fi fl ff ffi ffl ſt st存成單一碼位,specific在含 116 個specific的書裡搜不到。顯式對映 ASCII,不走 NFKC,以保住μ/U+2212/上標。 - 圖表引用跨全 dash 範圍(#8,感謝 @pig18888)— Katzung 16e 全書用 en dash(717 vs 8 個 ASCII),舊
[-.]只中 14 個、整本抽圖失明。改成涵蓋 U+2010–U+2015 與 U+2212。 - EPUB 裝飾性 HTML 清除(#9,感謝 @pig18888)— calibre 的
<a>/<div>/<span>在一本 52 章書佔 41% 字元、稀釋每個 chunk。保留<sup>/<sub>與 table 標籤。 - 單一 dash 定義,圖片管線兩側共用(#10)— 收斂到
shared/config.py,四個使用點 import。測試 242/0/5 不變。 - Docling 降級警告不再一錯報全部— sticky
last_error改用累計failure_count差值(154 本正式批次驗證)。 - pdfminer 拒讀的 page tree 改用 fitz 重寫,救回原本會整本消失的表格— pdfplumber 讀到 0 頁但 fitz 正常時,重寫暫存檔重試(154 本正式批次驗證)。
變更 (Changed)
- 三條使用路徑取代隱含的全有全無安裝(docs)— A 只轉檔/B +筆記工作流與圖片/C +語意搜尋(vault-search)。
AGENTS.md先問走哪條路徑,再安裝。 - Surya adapter 參考補上實測 GPU 數字(closes #4)— ~0.15 pg/s、capped server 在 8GB 卡上約 2.3 GB VRAM。標明 n=5/單次/共卡,是量級錨點而非 benchmark。
完整變更紀錄:CHANGELOG.md