Skip to content

refactor(knowledge-graph): 变更 KG 构建单位与 Corpus chunk 计数规则为 parent chunk - #491

Merged
ThreeFish-AI merged 2 commits into
feature/1.x.xfrom
ThreeFish-AI/kg-parent-chunk-unit
May 10, 2026
Merged

refactor(knowledge-graph): 变更 KG 构建单位与 Corpus chunk 计数规则为 parent chunk#491
ThreeFish-AI merged 2 commits into
feature/1.x.xfrom
ThreeFish-AI/kg-parent-chunk-unit

Conversation

@ThreeFish-AI

Copy link
Copy Markdown
Owner

背景

  • 本次变更要解决的问题:当前知识图谱(KG)构建以所有 chunk(parent + child + leaf)为单位,导致 child chunk 参与实体/关系抽取造成冗余;同时 Corpus chunk 总数统计基于 chunk_role != 'child'(parent + leaf),对 hierarchical 语料库显示虚高(如 Harness Engineering 显示 156 而实际只有 20 个 parent chunk)
  • 关联上下文:知识图谱构建流水线优化、Corpus 统计口径对齐

核心变更

  • Repository 层:新增 _parent_role_expr() 辅助函数,list_corpora_with_counts / get_corpus_with_counts 计数改为 chunk_role = 'parent',非 hierarchical 语料库(无 parent chunk)fallback 到全量计数,保持兼容
  • Dashboard:计数查询从 _top_level_role_expr() != 'child' 切换为 _parent_role_expr() == 'parent',同样含非 hierarchical fallback
  • KG 构建入口:API 端点 build_knowledge_graph 在传递 chunks 给 build_graph 前过滤为 parent chunk only(有 parent 时仅用 parent,否则保持全量)
  • Paper KG Pipelineenqueue_kg_build 在调用 _records_to_chunks 前过滤 records 为 parent chunk

风险与回滚

  • 主要风险:非 hierarchical 语料库无 chunk_role = 'parent' 的 chunk,所有 fallback 路径已覆盖(parent_count = 0 时回退到全量计数 / 全量 chunks)
  • 回滚方式:git revert 该提交即可,无数据库 schema 变更

验证证据

  • 单元测试:knowledge 目录 657 passed,paper_kg_pipeline 8 passed(含 fail-open 兜底、空 records 短路、后台 task 自清理等用例)
  • 集成测试:无需额外集成测试(改动仅涉及查询过滤条件变更,无新外部依赖)
  • E2E/Workflow:待浏览器验证 Corpus 列表 chunk 数量、Dashboard 计数、KG Build 行为
  • 覆盖率/关键截图:单元测试覆盖率 19%(项目整体,非本次变更专属)

影响范围

  • 前端:无代码变更,knowledge_count 字段语义调整后 UI 显示自然适配
  • 后端:3 个文件(repository.py、api.py、paper_kg_pipeline.py),_top_level_role_expr 保留不删供 document chunks 页等继续使用
  • GitHub Actions / 文档:无影响

Next Best Action

  • 浏览器端到端验证:确认 Harness Engineering Corpus 显示 20 chunks、Dashboard 计数正确、KG Build 仅处理 parent chunk

- Repository 层新增 _parent_role_expr() 辅助函数,list_corpora_with_counts / get_corpus_with_counts 计数改为 chunk_role = 'parent'(非 hierarchical 语料库 fallback 到全量计数)
- Dashboard 计数同步切换为 parent chunk 口径,含非 hierarchical fallback
- API 层 build_knowledge_graph 端点在传递 chunks 给 build_graph 前过滤为 parent chunk only
- Paper KG Pipeline enqueue_kg_build 在构建 chunks 前过滤 records 为 parent chunk

🤖 Generated with [Claude Code](https://github.com/claude), [CodeX](https://openai.com), [Gemini](https://github.com/apps/gemini-code-assist)
Co-Authored-By: Aurelius Huang<threefish.ai@gmail.com>
- Dashboard 计数改为复用 KnowledgeRepository.list_corpora_with_counts 按 corpus 粒度应用 parent/fallback 逻辑后再汇总,修复混合 hierarchical 与 non-hierarchical 语料库时 non-hierarchical chunks 被排除的计数错误
- 移除已废弃的 _top_level_role_expr 函数(生产代码无引用),同步更新测试覆盖为 _parent_role_expr

🤖 Generated with [Claude Code](https://github.com), [CodeX](https://openai.com), [Gemini](https://github.com/apps/gemini-code-assist)
Co-Authored-By: Aurelius Huang<threefish.ai@gmail.com>
@ThreeFish-AI
ThreeFish-AI merged commit 149242a into feature/1.x.x May 10, 2026
7 checks passed
@ThreeFish-AI
ThreeFish-AI deleted the ThreeFish-AI/kg-parent-chunk-unit branch May 10, 2026 13:51
ThreeFish-AI added a commit that referenced this pull request Jul 5, 2026
…-2026-4372 处置) (#1057)

* fix(deps): ts-deepmerge 补丁地板至 8.0.0 修复 GHSA-87mf-gv2c-c62c (#481);

🤖 Generated with [Claude Code](https://github.com/claude), [CodeX](https://openai.com), [Gemini](https://github.com/apps/gemini-code-assist)
Co-Authored-By: Aurelius Huang<threefish.ai@gmail.com>

* fix(negentropy): pydantic-settings 升 2.14.2 + joserfc>=1.6.7 补丁地板修复 #482/#488;

🤖 Generated with [Claude Code](https://github.com/claude), [CodeX](https://openai.com), [Gemini](https://github.com/apps/gemini-code-assist)
Co-Authored-By: Aurelius Huang<threefish.ai@gmail.com>

* fix(cognizes): joserfc/msgpack/python-socketio/python-engineio 补丁地板修复 #483/#486/#487/#490/#491;

🤖 Generated with [Claude Code](https://github.com/claude), [CodeX](https://openai.com), [Gemini](https://github.com/apps/gemini-code-assist)
Co-Authored-By: Aurelius Huang<threefish.ai@gmail.com>

* docs(perceives): ISSUE-092 增补 CVE-2026-4372 威胁模型;transformers 钉 4.x + dismiss #492(marker 5.x 不兼容实测);

🤖 Generated with [Claude Code](https://github.com/claude), [CodeX](https://openai.com), [Gemini](https://github.com/apps/gemini-code-assist)
Co-Authored-By: Aurelius Huang<threefish.ai@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant