v0.23.1
v0.23.1 is about trusting what you import. Five parsing paths got hardened this round, and every fix came from feeding real documents, real books, real subtitles and real slide decks through the pipeline and measuring what fell out.
E-book chapters land where they belong. The e-book parser assumed one spine file equals one chapter. Eight real Gutenberg books said otherwise, five of them packed many chapters into single files, and chapter numbers were often plain text lines rather than headings. Titles came from a stray table-of-contents label, content started at the first real chapter, and the AI invented numbering like "Chapter 3" for books that never had any. Chapters now split on real marks with sequence checks, a Pride and Prejudice import of 61 chapters across 15 files comes out as lessons that match their headings one to one, and the structuring AI is told plainly that a book without numbered chapters must not get numbers invented for it.
Chinese PDFs stop speaking in radicals. The PDF engine emitted Kangxi radical block codepoints for Chinese text, so 手 turned into ⼿ and search, highlighting and read-aloud could not match a single word. One real machine-learning textbook carried 23,559 of these characters. Extracted text now normalizes character by character at both engine exits, and scanned PDFs without a text layer return an honest empty result instead of garbage.
Web articles lose the furniture and keep the author. Scraped articles used to drag site templates into the course, a return-home line here, sidebar headings there, bare image paths at the end. The rules now delete only cross-article machine fingerprints. Whether a trailing promotional paragraph belongs to the author is a semantic question, and it goes to the AI that designs the course. That boundary was learned the hard way, an early rule ate a CSDN author's own closing paragraph and a reverse-guard test now keeps that from coming back.
Subtitles stop stuttering. YouTube auto-captions repeat each line across a rolling window of cues, the old adjacent-only dedup missed the pattern, and every sentence came out twice. Chinese transcripts were joined with a space between every line, 78 and 262 hits in two real transcripts. And when yt-dlp saved several languages, alphabetical order meant English always won over Chinese. All three are fixed, along with entity decoding and machine markers like [Music].
PowerPoint tables come back. The parser walked paragraphs, images and notes, then silently dropped table nodes whole, so a real three-slide deck full of tables yielded three bare headings and nothing else. Tables now render as proper markdown tables with pipes escaped, and placeholder tables that were never filled in are skipped instead of littering the course with empty rows.
The tests grew teeth. Real arXiv papers and web articles now flow through the entire pipeline into the database inside CI, behind flags that degrade honestly when no API key or network is present. Four new corpus suites replay six to twelve real samples per format, from Chinese textbooks to subtitled courses to pathological PowerPoint files, and every fix in this release carries a recorded break-test, break the code on purpose, watch the test go red, restore, watch it go green again.
The seed course also got a content pass, six lessons whose wording had drifted behind the product were rewritten in both languages.
v0.23.1 讲的是让你敢信导入进来的东西。五条解析路径这轮全部加固,每个修法都来自把真实的文档、真书、真字幕、真课件灌进管线,量出哪里掉了东西。
电子书的章节回到自己的位置上。 解析器过去假设一个 spine 文件等于一章。八本 Gutenberg 真书说事情没有这么整齐,其中五本把许多章塞在单个文件里,章号还常常是裸文本行而非标题。于是标题取自歪掉的目录标签,内容从真正的第一章开始,而对没有编号体系的书,AI 会编造「第 3 章」这样的序号。现在章节按真实标记切分并做序列校验,傲慢与偏见 15 个文件里的 61 章导入后课课与标题对齐,负责结构化的 AI 也被明确告知,原书没有编号就不许发明编号。
中文 PDF 不再满纸部首。 PDF 引擎对中文文本输出康熙部首区的码点,「手」变成「⼿」,检索、画线、朗读一个字都配不上。一本真实的机器学习教材里数出 23,559 个这样的字符。现在提取文本在两个引擎出口逐字符归一,没有文本层的扫描件则诚实返回空,不再产出垃圾。
网页文章去掉家具,留住作者。 抓取的文章过去会把站点模板拖进课程,这边一条返回首页,那边几行侧栏标题,结尾挂着裸图路径。规则现在只删跨文章稳定的机器指纹。尾部那段推广是不是作者自己写的,这是语义问题,交给设计课程的 AI 判断。这条边界是付了学费才学会的,早期一条规则吃掉过 CSDN 作者亲手写的收尾段,现在有一个反向守卫测试锁住,不许它回来。
字幕不再口吃。 YouTube 自动字幕会在滚动窗口里逐条复述,旧的相邻去重挡不住这个结构,成文里每句话出现两遍。中文转写逐句拼接时每行之间塞一个空格,两份真实转写稿里数出 78 处和 262 处。yt-dlp 落下多种语言时,字母序让英文永远压过中文。三处都已修好,顺带把实体解码和 [Music] 这类机器标记处理干净。
PPT 表格起死回生。 解析器会走段落、图片和备注,然后把表格节点整个丢掉,一份装满表格的三页真课件只产出三个光秃秃的标题。表格现在渲染成规范的 markdown 表格,竖线做转义,从未填过内容的占位空表则整张跳过,不给课程留一地空行。
测试长出了牙。 真实的 arXiv 论文和网页文章现在会在 CI 里走完整管线一路落库,无 key 或无网络时按档位诚实降级。四个新的语料套件按格式各回放六到十二个真样本,从中文教材到带字幕的课程再到病理课件。本版每个修复都带着录在提交记录里的破坏验证,故意改坏代码,看测试变红,还原,再看它变绿。
种子课程也做了一遍内容核对,六课落在产品后面的过时表述用两种语言重写了。