Skip to content

v0.3.0 — Structure-Aware Ingestion Foundation

Choose a tag to compare

View all tags
@Rcloudso Rcloudso released this 27 Jul 06:15
v0.3.0
04a42e3
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

v0.3.0 — Structure-Aware Ingestion Foundation

v0.3.0 replaces format-specific “extract text and chunk immediately” behavior
with one versioned, inspectable ingestion boundary:

validate → parse → normalize → clean → quality_gate → chunk → embed → publish

Product outcome

Administrators continue uploading TXT, Markdown, text-layer PDF, and DOCX from
the existing Documents page. The system now preserves the structure it can
reliably extract, records why content was accepted or withheld, and exposes
safe processing evidence without returning file paths or embeddings.

The document detail shows:

  • quality and index status;
  • parser, representation, cleaner, and chunker versions;
  • structure counts and parser/cleaner warnings;
  • a paginated Block inspector with type, heading path, page, content preview,
    inclusion state, and source relationship;
  • the fixed eight-stage processing timeline;
  • published chunks with heading/page provenance.

Engineering boundary

  • DocumentIR v1 is Zod-validated, limited to 2 MiB and 2,000 ordered Blocks.
  • TXT retains paragraphs; Markdown retains heading hierarchy, lists, and fenced
    code; text PDF retains page numbers and deterministic page order; DOCX
    retains reliably extracted headings, paragraphs, lists, tables, and image
    references.
  • Scan-only PDFs and image-only DOCX content return review-required quality
    results. v0.3.0 does not fabricate OCR or visual descriptions.
  • Cleaning is deterministic. PDF headers or footers are excluded only when the
    same short Block appears in the same first/last page position on at least
    three pages and at least 60% of pages.
  • Lists remain item-boundary aware. Tables repeat the extracted header when
    rows require multiple chunks. Paragraphs continue using semantic chunking.
  • New uploads do not expose chunks until every pre-publication stage succeeds.
    Explicit legacy reprocessing uses a shadow build; failed publication restores
    the previous database state and reconciles the in-memory index.
  • SQLite remains the default store and the current VectorStore, hybrid
    retrieval, FAQ, SSE, Grounding, and no-key contracts remain compatible.

API additions

  • GET /api/admin/documents/:id adds optional representation, quality, index,
    and processing summaries.
  • GET /api/admin/documents/:id/blocks?page=1&pageSize=20 returns safe,
    paginated Block DTOs.
  • POST /api/admin/documents/:id/reprocess explicitly upgrades a published
    legacy document; an already-current successful result is reused.
  • Existing upload, list, chunk, retry, enable/disable, delete, chat, and SSE
    interfaces remain available.

Non-goals and limits

OCR/VLM execution, Qdrant, Agentic Retrieval, business tools, domain-specific
field extraction, manual force-publish, background workers, and scheduled
reprocessing are not included. See
v0.3.0 implementation evidence for the current
verification state and ROADMAP.md for the next releases.

中文说明

v0.3.0 将原来的“按格式提取文本后直接切片”升级为统一、版本化且可检查的入库
管线。管理员仍在原有“文档知识”页面上传 TXT、Markdown、含文本层 PDF 和
DOCX；详情中可以检查质量/索引状态、各处理版本、结构统计、警告、分页 Block、
八阶段时间线和已发布切片。

DocumentIR v1 保留可靠提取到的标题、段落、列表、表格、页码、顺序与图片
引用。扫描 PDF 和仅图片 DOCX 会进入复核且不建立索引，系统不会伪造 OCR
结果。新上传只有在全部阶段成功后才可检索；历史 ready 文档使用影子重处理，
发布失败时恢复旧数据库状态并同步内存索引。

本版本继续使用 SQLite、现有 VectorStore、混合检索和无 Key 路径；不包含
OCR/VLM、Qdrant、Agentic Retrieval、业务工具、人工强制放行、后台 Worker 或
定时重处理。

Assets 2
Loading