Skip to content

v1.4.0

Choose a tag to compare

@LarFii LarFii released this 02 Sep 08:01
· 127 commits to main since this release
45e6ba6

What's new

Features

  • Native MinerU content list v2 (#347, closes #224): MinerU 3.x content_list_v2.json is normalized into the existing content-list contract — titles, paragraphs, lists, equations, images, tables, charts, code and algorithms, with page_idx/bbox/heading levels/captions preserved. Layout furniture (page_header, page_footer, page_number, …) is filtered by default; pass include_layout_blocks=True to keep it. Legacy content lists work unchanged, with an automatic fallback chain.
  • Direct .txt/.md parsing (#333, closes #331): plain text and markdown no longer take a ReportLab→PDF→OCR round trip (~3× faster measured; the whole family of PDF-rendering fidelity bugs #24/#226/#316 becomes structurally impossible on this path). Markdown ATX headings carry text_level; encoding falls back UTF-8 → gbk → latin-1 → cp1252.
  • force_multimodal_reprocess flag (#318, closes #154): opt-in parameter on process_document_complete() / insert_content_list() to re-run image/table/equation processing for an already-processed document — e.g. after switching graph/vector storage backends. Deterministic chunk ids are deduplicated on re-runs.
  • Vision prompts handle diagrams (#319, closes #313): the default prompts now transcribe text labels verbatim and enumerate nodes/edges/topology for schematics instead of describing "colors and lighting".
  • --no-recursive for the batch CLI (#327).

Fixes

  • Documents inserted through process_document_complete_lightrag_api() were silently skipped by LightRAG's dedup and indexed zero chunks (#334, fixes #328).
  • Multimodal chunks_list bookkeeping: merges are idempotent and a missing doc_status record is surfaced (#335, refs #332).
  • Parse cache now fingerprints file content (SHA-256), so in-place replacements with preserved mtime can no longer serve stale parses (#325).
  • Multimodal query cache keys include full media identity and all query options, eliminating cross-file collisions (#324).
  • Batch inputs with the same filename no longer share an output directory (#346) — batch outputs now live in <stem>_<hash>/.
  • LibreOffice conversion timeout is configurable via LIBREOFFICE_CONVERT_TIMEOUT (default 600 s) and timeouts are reported as timeouts, not as a missing installation (#329).
  • LA (grayscale+alpha) images are composited onto white correctly before MinerU (#317).
  • Markdown with bare angle brackets (score <threshold) no longer crashes convert_text_to_pdf (#316).
  • Retry delays are capped at max_delay after jitter (#323).
  • insert_content_list([]) now raises ValueError instead of creating an empty document record (#326).
  • MinerU/pdftext PageChars crashes are self-diagnosing (#321) and prevented for fresh installs by requiring mineru[core]>=3.4.1 (#337).
  • LM Studio example tolerates servers without a working /models endpoint (#349).

Migration notes (one-time effects on upgrade)

  1. .txt/.md re-ingest as new documents. The content list for text files changes shape, so content-derived doc ids change. Previously ingested text documents will be treated as new on re-processing; remove the old entries if duplicates matter to you.
  2. Parse caches invalidate once. #325 (content fingerprint) and #347 (new cache-relevant options) change the parse-cache key; every document re-parses once after upgrading.
  3. insert_content_list([]) now raises ValueError instead of silently succeeding.
  4. Batch output directories move from <output_dir>/<stem>/ to <output_dir>/<stem>_<pathhash>/. Incremental manifests keep skipping already-processed files; delete the manifest if you want a directory-layout rebuild.
  5. MinerU lower bound: mineru[core]>=3.4.1 (pairs with pdftext ≥0.7; older MinerU crashes on Office documents).

Full history: v1.3.1...v1.4.0