Skip to content

v4.2 — Structured ingestion and reviewable knowledge bases

Choose a tag to compare

View all tags
@ChenSiyun1234 ChenSiyun1234 released this 14 Jul 14:55
v4.2
d983557
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

v4.2 — Structured ingestion and reviewable knowledge bases

This release rebuilds course-material ingestion around a lightweight, source-grounded knowledge layer. Every imported source, page, question, answer, asset, warning, and AI-assisted correction now has an explicit place in a pipeline that can be inspected, replayed, and validated before teaching begins.

Highlights

  • One-command course ingestion: ingest_course.py is the normal entry point for PDF, DOCX, PPTX, text, and Markdown materials. It distinguishes ready, usable_with_gaps, and blocked instead of treating a successful process exit as proof that the course is complete.
  • Reviewable AI takeover: parser warnings, skipped content, missing answers, uncertain pages, and unbound evidence enter typed ReviewIssue records. Validated ReviewPatch operations are append-only, replayable after a rebuild, and never disguised as teacher-provided truth.
  • Stable provenance and crash recovery: strict schemas, stable IDs, source hashes, page anchors, asset hashes, and multi-file ingestion transactions make knowledge bases rebuildable and detect source drift without silently leaving partial state.
  • Lightweight retrieval with measurable quality: structure-aware chunks, concept postings, stale-index integrity checks, and deterministic Recall@1/5 plus MRR evaluation improve retrieval without adding a heavy vector-database runtime.
  • Safer document and visual extraction: DOCX/PPTX handling covers tables, content controls, formulas/list review signals, notes, hidden objects, images, and unsupported structures. Uncertain or visually dependent content fails into review instead of disappearing.
  • A smaller, clearer project: the main skill, sub-skills, bilingual language packs, validator, and file-format contracts now share the same readiness and provenance rules. Duplicate indexes, the caption-only gallery, historical root clutter, and the retired LlamaIndex spike were removed from the maintained runtime.

中文摘要

  • ingest_course.py 成为 PDF、DOCX、PPTX、文本与 Markdown 的统一建库入口，并区分“可学习 / 有缺口但可用 / 被内容问题阻断”。
  • 所有解析警告、跳过内容、缺失答案和低置信页面进入类型化 AI 复核队列；修复补丁经过校验后可在重建时重放，不冒充老师或教材答案。
  • 稳定 ID、源文件哈希、页锚点、资产哈希与多文件事务让知识库可追溯、可重建，并能在中断后回滚。
  • 结构感知切块、概念索引、索引完整性检查和 Recall@1/5、MRR 评估提升命中率，同时保持纯标准库、轻量运行时。
  • DOCX/PPTX 提取补强表格、内容控件、公式/列表信号、备注、隐藏对象与图片；无法可靠恢复的内容不会静默丢失，而是进入复核流程。
  • 主 SKILL、子 SKILL、双语语言包和脚本契约完成对齐，并清理重复索引、旧实验与历史根目录文件。

Compatibility and migration

  • Existing build_raw_input_from_workspace.py and ingest.py workflows remain available; new course workspaces should use ingest_course.py.
  • Exit code 10 means engineering compilation completed but content readiness is still blocked. Resolve the review queue before teaching or quizzing.
  • Legacy workspaces retain their compatibility path. New workspaces use .ingest/ and study_state.json as structured sources of truth.
  • PDF parsing and printable output remain optional, content-aware, and consent-gated; no external skill or dependency is installed silently.

Validation

  • 1,509 tests total; 27 optional tests skipped and all remaining tests passed
  • GitHub CI passed on Ubuntu and Windows with Python 3.8 and Python 3.12, both before and after merge
  • the v4.2 release tag / root metadata / CHANGELOG integrity gate passed locally before publication
  • the pre-release runtime build contains 79 files, reports root metadata version: "4.2", and excludes tests, benchmarks, CI files, and historical plans
  • independent cold-reader checks passed urgent teaching, missing-bank quiz, warning-bearing ingestion, one-shot PDF, and missing-visual English routes

Full implementation and review details: PR #23 · implementation record

Assets 3
Loading