fix(collection): identity dedup + chunk IN() + kill tree + help TTL + safe CDP cleanup [修复组⑥] - #30
Conversation
… + help cache TTL + safe CDP cleanup [C7,C15,C16,C17,C20]
|
Warning Review limit reached
Next review available in: 29 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (17)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
✅ Health: 6.8 📋 At a glance Files & modules (2)
🚨 Change risk: 9.5/10 (high)
🔎 More signals (2)🔥 Hotspots touched (5)
2 more
🔗 Hidden coupling (1 file)
👀 Suggested reviewers @xujinghua 📊 Full report · ⭐ Star Repowise · 📥 Install bot · Last updated 2026-07-18 18:45 UTC |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces several robustness improvements to the data collection pipeline, including TTL-based caching for OpenCLI commands to support binary upgrades, process-tree termination on CLI timeouts, and safe CDP tab cleanup that avoids closing user tabs on snapshot failures. It also implements stable identity-based deduplication and in-place updates for collected records, with chunked database queries to prevent exceeding SQLite's variable limits. The review feedback highlights a critical logic bug in the batch deduplication process where duplicate updates can occur for pre-existing identities, and advises against committing local .omc session cache files.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| if identity is not None: | ||
| existing = existing_by_identity.get(identity) | ||
| if existing is not None: | ||
| if existing.content_hash == content_hash: | ||
| # Same source-native item, content unchanged: a genuine | ||
| # duplicate, same as the content_hash-only path always did. | ||
| skipped += 1 | ||
| else: | ||
| # Same source-native item, content changed (e.g. a feed | ||
| # fixed a title typo) — update in place instead of | ||
| # inserting a duplicate row (C7's fix). | ||
| existing.raw_data = raw | ||
| existing.normalized_data = normalized | ||
| existing.content_hash = content_hash | ||
| existing.status = "normalized" | ||
| existing.ai_enrichment = None | ||
| updated_records.append(existing) | ||
| continue | ||
| if identity in seen_identities_in_batch: | ||
| # Two triples in this same batch share an identity (e.g. a | ||
| # feed listed the same entry twice) — keep the first, skip | ||
| # the rest rather than fight over which one "wins". | ||
| skipped += 1 | ||
| continue | ||
| seen_identities_in_batch.add(identity) |
There was a problem hiding this comment.
问题分析：
当前逻辑中，seen_identities_in_batch 的去重检查被放在了 existing is not None 的判断之后。
如果当前批次（batch）中包含多个具有相同 identity 的项，且该 identity 在数据库中已存在：
- 第一项会触发
existing is not None分支，更新existing并将其加入updated_records，然后通过continue跳过后续代码（因此该identity不会被加入seen_identities_in_batch）。 - 第二项（以及后续相同
identity的项）也会触发existing is not None分支，再次更新同一个existing实例，并再次将其追加到updated_records中。
这会导致：
updated_records中存在对同一个CollectedRecord实例的重复引用。- 返回的
new_records + updated_records包含重复元素，导致下游（如 AI 处理、通知发送）对同一条记录进行多次重复处理，且accepted计数不准确。
解决方案：
应该将 seen_identities_in_batch 的去重检查移到 existing is not None 判断之前。这样，批次中后续重复的 identity 会被直接作为 duplicate 跳过。
if identity is not None:
if identity in seen_identities_in_batch:
# Two triples in this same batch share an identity (e.g. a
# feed listed the same entry twice) — keep the first, skip
# the rest rather than fight over which one "wins".
skipped += 1
continue
seen_identities_in_batch.add(identity)
existing = existing_by_identity.get(identity)
if existing is not None:
if existing.content_hash == content_hash:
# Same source-native item, content unchanged: a genuine
# duplicate, same as the content_hash-only path always did.
skipped += 1
else:
# Same source-native item, content changed (e.g. a feed
# fixed a title typo) — update in place instead of
# inserting a duplicate row (C7's fix).
existing.raw_data = raw
existing.normalized_data = normalized
existing.content_hash = content_hash
existing.status = "normalized"
existing.ai_enrichment = None
updated_records.append(existing)
continue| @@ -0,0 +1 @@ | |||
| {"session_id":"b4f7d02c-1da1-4b4d-9835-33e44401355e","context_samples":[{"ts_ms":1783878055117,"tokens":109500},{"ts_ms":1783878057926,"tokens":109939},{"ts_ms":1783878059350,"tokens":109939},{"ts_ms":1783878062169,"tokens":109939},{"ts_ms":1783878065137,"tokens":112312},{"ts_ms":1783878066527,"tokens":112312},{"ts_ms":1783878069643,"tokens":112437},{"ts_ms":1783878070036,"tokens":112437},{"ts_ms":1783878070541,"tokens":112437},{"ts_ms":1783878071081,"tokens":112437},{"ts_ms":1783878077649,"tokens":113096},{"ts_ms":1783878078111,"tokens":113096},{"ts_ms":1783878083683,"tokens":113513},{"ts_ms":1783878084794,"tokens":113513},{"ts_ms":1783878091897,"tokens":114044},{"ts_ms":1783878093626,"tokens":114044},{"ts_ms":1783878095116,"tokens":114044},{"ts_ms":1783878100551,"tokens":114749},{"ts_ms":1783878117149,"tokens":114749},{"ts_ms":1783878475480,"tokens":116909},{"ts_ms":1783878477603,"tokens":116909},{"ts_ms":1783878481720,"tokens":118189},{"ts_ms":1783878483660,"tokens":118189},{"ts_ms":1783878488025,"tokens":118636},{"ts_ms":1783878488449,"tokens":118636},{"ts_ms":1783878495026,"tokens":119031},{"ts_ms":1783878514349,"tokens":119031},{"ts_ms":1783878521236,"tokens":121158},{"ts_ms":1783878586833,"tokens":121464},{"ts_ms":1783878587475,"tokens":121464},{"ts_ms":1783878597857,"tokens":121836},{"ts_ms":1783878604356,"tokens":122080},{"ts_ms":1783878604705,"tokens":122080},{"ts_ms":1783878614717,"tokens":122341},{"ts_ms":1783878621949,"tokens":122341},{"ts_ms":1783878630484,"tokens":123658}],"last_updated_ms":1783878630484,"session_start_tokens":72839} No newline at end of file | |||
There was a problem hiding this comment.
这些 .omc/state/ 目录下的 session 缓存文件是本地运行状态，不应该提交到版本控制中。建议将 .omc/ 目录添加到 .gitignore 中，并从 Git 中移除 these 文件。
1 participant
修复组⑥ — 杂项 (账本 C7/C15/C16/C17/C20)
Sonnet 实施, Fable 审计 + rebase 修 migration 冲突。已 rebase 到含 ④⑤ 的最新 main。
CollectedRecord.identity_key列 +(source_id, identity_key)复合索引; storer 按 identity supplementary 匹配: 命中同 content_hash 跳过, 不同则原地 update (刷新 content/hash, 清 ai_enrichment 重跑) 并入 new_records; identity 为 None 的通道走原 content_hash 路径不变。migrationFable 收口 (关键)
t9y0z1a2b3c4(各自基于旧 head 生成"下一个"), ④ 先合占用 → 会造成 alembic 双 revision/multiple heads。已重链: 新 idu0a1b2c3d4e5, down_revision=t9y0z1a2b3c4(④ 之后)。alembic heads 现单一u0a1b2c3d4e5(chain: s8x→t9 cursor→u0 identity)Test
合并顺序
已 rebase 到最新 main (含 #28/#29)。migration 挂 ④ 之后, 直接可合。