Document 删除路径联动 Knowledge chunks 生命周期（Phase 2 · stacked on #483） - #484
Closed
ThreeFish-AI wants to merge 2 commits into
Closed
Document 删除路径联动 Knowledge chunks 生命周期（Phase 2 · stacked on #483）#484ThreeFish-AI wants to merge 2 commits into
ThreeFish-AI wants to merge 2 commits into
Conversation
ISSUE-078 Phase 2 应用层级联：堵未来产生孤儿 / 检索污染的入口。Phase 1 修计数 口径，Phase 2 修数据闭环，两者正交、可独立回滚。 变更: - 硬删: delete_document 在 db.delete(doc) 之前先按 (corpus_id, app_name, source_uri) 删除全部 Knowledge 行（hierarchical 父+子共享 source_uri，单次 清理一并覆盖），同事务原子提交，杜绝 FK 意义孤儿 - 软删: 批量更新对应 chunks 的 metadata.archived=true 与 is_enabled=false， 让既有 _active_filter_expr / _enabled_filter_expr 过滤生效，防 RAG 检索仍 命中已软删 doc；语义可逆（reactivation 兜底） - Reactivation: 复活 soft-deleted doc 时直接 hard delete 旧 chunks，让重新 ingest 写一份干净的，避免新旧 chunks 叠加 - delete_source: 加注释说明 _repository.delete_knowledge_by_source 仍保留 为 fallback 兜底（覆盖 source_uri 非 gs:// 与 doc 已被先前路径硬删两类场景） - source_uri 解析下沉: api.py 的 _resolve_document_source_uri 下沉到 storage.service 模块级 resolve_document_source_uri，api.py 同名函数变为 thin wrapper，消除跨层重复 新增 helper: - DocumentStorageService._hard_delete_chunks_in_session: 硬删 chunks ORM 删除 - DocumentStorageService._archive_chunks_in_session: 软删 archive，复用 archive_knowledge_by_source 的 jsonb_set 范式 测试: - test_storage_service_delete_cascade.py 5 场景串行覆盖：硬删级联 + 邻居 doc 不误删、软删 archive + Phase 1 口径联动验证、reactivation 清理旧 chunks、 跨 corpus 同名 source_uri 隔离、URL 类文档 origin_url 解析；GCS 通过 monkeypatch stub 避免真实网络调用 - 既有 19 项 API 单测无回归 检索路径过滤覆盖复核: - semantic_search / keyword_search: 三态（archived/searchable/is_enabled）齐全 - hybrid_search / rrf_search 走 PG 函数 kb_hybrid_search / kb_rrf_search， 函数内未过滤三态，仅靠 Python 后过滤检查 archived+searchable，缺 is_enabled ——是该 PG 函数既有缺陷，与本 phase 解耦。Phase 2 软删同时打 archived=true + is_enabled=false，前者覆盖全部 4 路径，软删主目标达成 详见 docs/issue.md ISSUE-078 Phase 2 补充章节。 🤖 Generated with [Claude Code](https://github.com/claude), [CodeX](https://openai.com), [Gemini](https://github.com/apps/gemini-code-assist) Co-Authored-By: Aurelius Huang<threefish.ai@gmail.com>
11 tasks
* feat(knowledge-base): Knowledge.document_id FK + 孤儿清算 CLI (Phase 3);
ISSUE-078 Phase 3 — DB Schema 防御层 + 历史脏数据一次性清算。Phase 1 修计数口径、
Phase 2 修应用层级联、Phase 3 在 DB 层加 FK ON DELETE CASCADE 兜底，三层防御
正交可独立回滚。
变更:
- alembic 0030: ADD COLUMN knowledge.document_id UUID NULL +
FK fk_knowledge_document_id REFERENCES knowledge_documents(id) ON DELETE
CASCADE + 部分索引 ix_knowledge_document_id WHERE document_id IS NOT NULL；
仅加列与约束、不做数据变更（避免大表 UPDATE 长锁）；downgrade 仅 DROP COLUMN
- ORM 同步: models/perception.py Knowledge.document_id Mapped[UUID|None]
- KnowledgeChunk dataclass: 加 document_id 字段；types.py
- 写入路径 stamp: _ingest_text_with_tracker 加 document_id 参数，单点
dataclasses.replace stamp 到每个 chunk；repository.add_knowledge values 加
document_id 字段。仅 2 个 ingest 入口（execute_ingest_url_document_pipeline /
execute_ingest_file_pipeline）传入 document_id，其余 10 个入口（纯文本、URL、
replace、rebuild、KG）保留 None
- 独立 CLI scripts/cleanup_orphan_knowledge.py:
uv run python -m negentropy.scripts.cleanup_orphan_knowledge \
--dry-run | --commit [--corpus-id ...] [--app-name ...] [--json]
三步流水线: 回填 document_id → 按 corpus 分组报告四档计数 → --commit 才
真正 DELETE（白名单 source_uri 形态防误删）；不放进 alembic 迁移因为
DELETE 不可逆需 dry-run + 审批
- 观测函数 count_orphan_knowledge: 与 CLI 共享过滤口径，cron 接入将
total_orphans 上报为 metric negentropy.knowledge.orphan_count{corpus_id}
dev DB 实地验证:
- 在 user 真实数据上跑 --dry-run 报告 Harness Engineering corpus
total=849, would_delete=0；--commit 后 849 条 Knowledge.document_id 全部
回填，与对应 KnowledgeDocument 建立 FK 关联
- Phase 1 → Phase 3 三层防御端到端在用户实际数据上闭环
测试: test_phase3_document_fk.py 8 场景覆盖 ORM 字段读写、DB 层 FK CASCADE
（直接 DELETE knowledge_documents 行触发 Knowledge 级联清理）、CLI dry-run/
commit、KG NULL source_uri 保留、非白名单 URI 形态保留、观测函数、migration
round-trip；既有 67 项单测无回归；alembic upgrade/downgrade 已在 dev DB 验证
详见 docs/issue.md ISSUE-078 Phase 3 补充章节。
🤖 Generated with [Claude Code](https://github.com/claude), [CodeX](https://openai.com), [Gemini](https://github.com/apps/gemini-code-assist)
Co-Authored-By: Aurelius Huang<threefish.ai@gmail.com>
* fix(knowledge-base): cleanup CLI backfill 改用标量子查询优先选取 active doc (review #1);
🤖 Generated with [Claude Code](https://github.com/claude), [CodeX](https://openai.com), [Gemini](https://github.com/apps/gemini-code-assist)
Co-Authored-By: Aurelius Huang<threefish.ai@gmail.com>
* fix(knowledge-base): 修复 CI 集成测试三处失败 (review #2);
- conftest: 补丁 storage.service / cleanup_orphan_knowledge 的模块级
AsyncSessionLocal 引用（from-import 绑定问题），修复 asyncpg 事件循环边界
RuntimeError
- _sync_document_chunk_stats: 包裹 try/except 降级为 warning，避免纯单元
测试（FakeRepository 无真实 DB）触发 DB 连接崩溃
- _backfill_sql: 改回 UPDATE ... FROM + DISTINCT ON 子查询，仅更新有文档
匹配的行（孤儿行自然排除），修复 backfilled 计数断言失败
🤖 Generated with [Claude Code](https://github.com/claude), [CodeX](https://openai.com), [Gemini](https://github.com/apps/gemini-code-assist)
Co-Authored-By: Aurelius Huang<threefish.ai@gmail.com>
* fix(knowledge-base): 修复 CI 集成测试两处失败 (review #3);
- _backfill_sql: 子查询内 scope 条件引用外部别名 k → 改为 k_sub，
修复 PG UndefinedTableError
- conftest: 补丁 knowledge.api 的模块级 AsyncSessionLocal 引用，
修复 _scenario_soft_delete_archives 调用 list_corpora 时的事件循环边界错误
🤖 Generated with [Claude Code](https://github.com/claude), [CodeX](https://openai.com), [Gemini](https://github.com/apps/gemini-code-assist)
Co-Authored-By: Aurelius Huang<threefish.ai@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
背景
ISSUE-078 Phase 2 应用层级联：堵未来产生孤儿 / 检索污染的入口。Phase 1 修计数口径，Phase 2 修数据闭环，两者正交、可独立回滚。
Phase 1 PR #483 解决了「软删 doc 的 chunks 仍计入 corpus 计数 / 父子混算」的展示口径问题——即时让 849 → 14 在 UI 上恢复正常，但未触及数据闭环：
is_enabled=true+searchable=true+archived未标）hard_delete=True）路径未级联清理 Knowledge 行 → 任何走过此路径的历史调用都留下 FK 意义孤儿本 PR 通过应用层级联补齐这三类场景。
改动
1.
_resolve_document_source_uri下沉到 storage 层apps/negentropy/src/negentropy/knowledge/api.py:1957原有 helper 下沉为negentropy.storage.service.resolve_document_source_uri模块级函数（URL 类取metadata.origin_url/ 否则取gcs_uri），api.py 同名函数变为 thin wrapper 复用之，消除跨层重复。2. 硬删级联清理 chunks（
DocumentStorageService.delete_document硬删分支）在
db.delete(doc)之前先按(corpus_id, app_name, source_uri)删除全部 Knowledge 行：_hard_delete_chunks_in_sessionstaticmethod（ORMdelete()表达式），logger 输出chunks_deleted计数供审计3. 软删 archive chunks（
DocumentStorageService.delete_document软删分支）在
doc.status='deleted'之前批量更新对应 chunks 的metadata.archived=true与is_enabled=false：metadata.archived=true→ 既有_active_filter_expr过滤生效，覆盖semantic_search/keyword_search/hybrid_search/rrf_search全部 4 路径is_enabled=false→ 冗余防御层，覆盖_enabled_filter_expr路径_archive_chunks_in_sessionstaticmethod，复用archive_knowledge_by_source的jsonb_set范式4. Reactivation 清理旧 chunks（
_reactivate_document）复活 soft-deleted doc 时（同 hash 重新上传），在重置 doc 状态前直接 hard delete 旧 chunks：
resolve_document_source_uri(doc)解析旧 URIreactivate_purge_chunks+old_source_uri字段5.
KnowledgeService.delete_source双重删除冗余说明storage_service.delete_document已自带 chunks 清理后，service.py:1742中的_repository.delete_knowledge_by_source不再删除（returns 0），但保留为 fallback 兜底，覆盖两类场景：source_uri不是gs://...（如 URL 类的origin_url）走不到 storage 分支加注释明确语义。
检索路径过滤覆盖复核结论
semantic_searchkeyword_searchhybrid_search(PGkb_hybrid_search)rrf_search(PGkb_rrf_search)hybrid/rrfPG 函数缺is_enabled过滤是该函数本身既有缺陷（与本 PR 解耦）；Phase 2 软删同时打archived=true与is_enabled=false，前者覆盖全部 4 路径，软删主目标达成。后者作为冗余防御层在semantic/keyword路径生效，hybrid/rrf不生效列入 follow-up。集成测试
新增
test_storage_service_delete_cascade.py，5 场景串行覆盖：hard_delete_cascadessoft_delete_archivesarchived=true+is_enabled=false；Phase 1 口径联动后 corpus count = 0reactivation_purges_old_chunksactive、original_filename已更新neighbor_corpus_isolationurl_doc_resolves_origin_urlmetadata.origin_url）的 chunks 也被正确级联清理GCS 通过
monkeypatch.setattr(DocumentStorageService, "_get_gcs_client", ...)替换为 stub，避免真实网络调用。受现有
db_engine函数级 fixture 与 asyncpg 跨事件循环边界限制，采用「单 test 函数 + 多场景子断言」模式（与 Phase 1 同范式）。影响面 & 兼容性
DocumentStorageService.delete_document签名/返回值不变try/except StorageError警告语义，chunks 清理在 GCS 操作之后执行（顺序：删 GCS → 删 chunks → 删 doc）Phase 化交付（本 PR 是 Phase 2）
Knowledge.document_idFK + 独立 CLIcleanup_orphan_knowledgedry-run/commit + 观测 metric — DB 层 belt-and-suspenders + 历史脏数据清算Test plan
pytest tests/integration_tests/knowledge/test_storage_service_delete_cascade.py5 场景全 passpytest tests/integration_tests/knowledge/test_corpus_chunk_count_filter.pyPhase 1 测试无回归pytest tests/unit_tests/knowledge/test_api_corpus.py tests/unit_tests/knowledge/test_api_documents.py19 项既有用例无回归docs/issue.mdISSUE-078 追加 Phase 2 补充章节Knowledge行数为 0🤖 Generated with Claude Code