fix(knowledge-base): 修复 Corpus chunks 计数虚高与摄取写入路径非幂等导致的孤儿数据累积 - #488
Merged
Conversation
根因： 1. ingest_text/file/url 不幂等——同一 (corpus_id, source_uri) 多次摄取纯 INSERT，旧 chunks 残留 2. replace_source/sync_source 等的 DELETE+INSERT 分属独立事务，异常时旧删新未插 3. list_corpora 的 chunks 计数含 hierarchical child，与 document chunks 页口径不一致 修复（三层防线）： - Repository 新增 replace_knowledge_by_source（同事务 DELETE→INSERT，async with db.begin() 原子回滚） - Service _ingest_text_with_tracker 注入 persist_mode（replace/append），按 source_uri 自动决定语义 - 7 个 replace/sync pipeline 统一切到原子方法，消除两段式事务 - list_corpora/get_corpus/dashboard 切换为 top-level 口径（排除 child），新增 chunk_count_total 辅字段 - 前端 CorpusList 展示 "14 chunks · 70 vectors" 双数字（仅 hierarchical 时显示 vectors） 🤖 Generated with [Claude Code](https://github.com/claude), [CodeX](https://openai.com), [Gemini](https://github.com/apps/gemini-code-assist) Co-Authored-By: Aurelius Huang<threefish.ai@gmail.com>
- cleanup_orphan_chunks CLI: 按 created_at 时间聚类识别重复摄取批次，保留最新批次，物理删除孤儿 chunks（默认 dry_run）
- POST /base/{corpus_id}/admin/cleanup-orphans 管理端点: 支持 dry_run 查询参数
- 14 个单元测试覆盖：时间聚类算法、chunk_index 完整性校验、persist_mode 语义、PipelineTracker stage output
- 修复 _cluster_by_time 中 datetime.total_seconds() 误判导致的聚类失效（hasattr 检查对象错误）
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
page.tsx 中的 corpus 卡片也需展示 chunk_count_total， 与 CorpusList.tsx 保持一致，修复遗漏的渲染点。 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
回退到 clusters[-2] 后重新检查 chunk_index 连续性， 与 CLI 脚本行为对齐，防止 dry_run=False 时保留不完整批次。 🤖 Generated with [Claude Code](https://github.com/claude), [CodeX](https://openai.com), [Gemini](https://github.com/apps/gemini-code-assist) Co-Authored-By: Aurelius Huang<threefish.ai@gmail.com>
ThreeFish-AI
changed the title
- FakeRepository 补充 get_corpus_by_id 与 replace_knowledge_by_source， 使 rebuild pipeline 测试 RuntimeError 从正确位置抛出而非 AttributeError - test_pipeline_tracker 断言从 delete stage 更新为 persist stage（原子化后 delete 已合并） - KnowledgeBasePage UI 测试断言从 "chunks: 3" 更新为 "3 chunks" 🤖 Generated with [Claude Code](https://github.com/claude), [CodeX](https://openai.com), [Gemini](https://github.com/apps/gemini-code-assist) Co-Authored-By: Aurelius Huang<threefish.ai@gmail.com>
集成测试的 FakeRepository 缺少 replace_knowledge_by_source 方法， 导致 ingest_text（source_uri 非空 → effective_mode=replace）触发 AttributeError。 🤖 Generated with [Claude Code](https://github.com/claude), [CodeX](https://openai.com), [Gemini](https://github.com/apps/gemini-code-assist) Co-Authored-By: Aurelius Huang<threefish.ai@gmail.com>
原 _cleanup 函数仅收集扁平 ID 列表传给 _write_backup_csv， 导致 CSV 除 id 外所有列（source_uri、chunk_index、role、created_at）均为空， 无法支撑脚本文档声称的"可手工恢复"。 修复： - all_deleted_ids → all_deleted_chunks，收集完整 chunk 字典 - _get_chunks_for_source 返回值补充 source_uri 字段 - _write_backup_csv 优先使用 per-chunk source_uri 🤖 Generated with [Claude Code](https://github.com/claude), [CodeX](https://openai.com), [Gemini](https://github.com/apps/gemini-code-assist) Co-Authored-By: Aurelius Huang<threefish.ai@gmail.com>
ThreeFish-AI
added a commit
that referenced
this pull request
Jul 5, 2026
…-2026-4372 处置） (#1057) * fix(deps): ts-deepmerge 补丁地板至 8.0.0 修复 GHSA-87mf-gv2c-c62c (#481); 🤖 Generated with [Claude Code](https://github.com/claude), [CodeX](https://openai.com), [Gemini](https://github.com/apps/gemini-code-assist) Co-Authored-By: Aurelius Huang<threefish.ai@gmail.com> * fix(negentropy): pydantic-settings 升 2.14.2 + joserfc>=1.6.7 补丁地板修复 #482/#488; 🤖 Generated with [Claude Code](https://github.com/claude), [CodeX](https://openai.com), [Gemini](https://github.com/apps/gemini-code-assist) Co-Authored-By: Aurelius Huang<threefish.ai@gmail.com> * fix(cognizes): joserfc/msgpack/python-socketio/python-engineio 补丁地板修复 #483/#486/#487/#490/#491; 🤖 Generated with [Claude Code](https://github.com/claude), [CodeX](https://openai.com), [Gemini](https://github.com/apps/gemini-code-assist) Co-Authored-By: Aurelius Huang<threefish.ai@gmail.com> * docs(perceives): ISSUE-092 增补 CVE-2026-4372 威胁模型；transformers 钉 4.x + dismiss #492（marker 5.x 不兼容实测）; 🤖 Generated with [Claude Code](https://github.com/claude), [CodeX](https://openai.com), [Gemini](https://github.com/apps/gemini-code-assist) Co-Authored-By: Aurelius Huang<threefish.ai@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
背景
ingest_text/file/url不幂等——同一(corpus_id, source_uri)多次摄取纯 INSERT 不清理旧 chunks，导致 chunks 数量虚高（如 849 vs 预期 84）knowledge_count统一为全量行数，hierarchical 切分下无法区分用户感知的 chunks 与实际参与检索的 vectors核心变更
replace_knowledge_by_source方法，同事务内完成删除旧数据+插入新数据，消除孤儿 chunks 累积；_ingest_text_with_tracker新增persist_mode参数（replace/append/auto），所有摄取入口统一走 replace 路径_top_level_role_expr单一事实源过滤 child 子块，list_corpora_with_counts/get_corpus_with_counts返回 top-level + total 双计数，前端展示 "X chunks · Y vectors"（仅 hierarchical 场景下两数不同时展示）cleanup_orphan_chunks.py（干跑默认 + backup CSV）+ 管理 API 端点POST /base/{corpus_id}/admin/cleanup-orphans，按 created_at 时间聚类识别多批次摄取，保留最新完整批次get_stage_output读取已完成 stage 输出，合成 delete stage 保持前端时间轴语义一致风险与回滚
replace_knowledge_by_source在同事务内 DELETE+INSERT，并发同 source_uri 摄取可能导致数据翻倍（但窗口比旧方案显著缩小）验证证据
test_cleanup_orphans.py（聚类算法）、test_corpus_count_semantics.py（persist_mode 行为）、test_repository_replace.py（repository 辅助方法）影响范围
CorpusList.tsx、page.tsx、knowledge-api.ts（双数字展示）repository.py、service.py、api.py、schemas.py（原子写入 + 双口径计数 + 清理端点）cleanup_orphan_chunks.pyNext Best Action
cleanup_orphan_chunks.py --apply清理历史孤儿数据