Skip to content

fix(dispatch): 孤儿任务恢复重投与 Edge hubTaskId 去重 - #2074

Merged
DeliciousBuding merged 4 commits into
masterfrom
fix/dispatch-orphan-recovery
Aug 30, 2026
Merged

fix(dispatch): 孤儿任务恢复重投与 Edge hubTaskId 去重#2074
DeliciousBuding merged 4 commits into
masterfrom
fix/dispatch-orphan-recovery

Conversation

@DeliciousBuding

Copy link
Copy Markdown
Collaborator

Summary

Closes #2066

Hub 在「任务入队」与「outbox 记录」之间存在崩溃窗口与信号量回退两条静默丢失通道。本 PR 实现孤儿任务恢复重投机制,确保每个入队任务都必然收敛——被投递或被显式处理,不再静默滞留。

Changes

Task 1: Edge hubTaskId 去重

  • Run struct 新增 HubTaskID 字段,store 层支持 GetRunByHubTaskID/SetRunHubTaskID
  • runcontrol.Create 在 runCreationMu 锁内检查已有同 hubTaskID 的 run 并幂等返回
  • SQLite migration v5 添加 edge_runs.hub_task_id 列与部分索引
  • 验收: go test ./internal/runcontrol/ -run TestCreateHubTaskIDDedup -v → PASS

Task 2: pending_agent_tasks 持久化 model_params

  • Migration 0069 添加 model_params JSONB 列(up→down→up 已在临时 PG 验证)
  • TriggerAgentTask 创建任务时写入 modelParams
  • 验收: go test -short ./... → ALL PASS

Task 3: 孤儿恢复 sweeper

  • Repository: ClaimOrphanedTasks(CAS UPDATE + NOT EXISTS delivery_outbox)
  • dispatchsvc: StartOrphanRecoveryLoop 定期扫描,redeliverOrphanedTask 重建上下文
  • 上下文重建:prompt←trigger_message, AI←agent_instance, customAgent preload, targetType re-derive
  • 接入 app/wiring.go background group
  • 验收: go test ./internal/service/dispatchsvc/ -run TestClaimOrphanedTasks -v → 5/5 PASS

Task 4: Metrics + Logging

  • 新文件 metrics/dispatch_orphan.go(DispatchOrphanDiscovered/Redelivered)
  • sweeper 结构化 slog 日志
  • 验收: go vet clean

Task 5: L1 集成测试(真 PG)

  • TestOrphanRecovery_ClaimAndRedeliver: queued+无 outbox+超 grace → claimed ✅
  • TestOrphanRecovery_GraceWindowNotClaimed: grace 内不被触碰 ✅
  • TestOrphanRecovery_TaskWithOutboxNotClaimed: 有 outbox 行不捞 ✅
  • 验收: AGENTHUB_DB_PASSWORD=dev_password go test -tags integration ./tests/integration/ -run TestOrphanRecovery -v → 3/3 PASS

Design Decisions

  • Grace 窗口默认 120s(远大于 dispatch goroutine 启动延迟)
  • 恢复重投复用现有 DispatchService.DispatchTask 入口
  • model_params 落库类型 jsonb,参照 custom_agents.model_params
  • targetType 不落库,由 target_id 重新推导
  • Edge 去重在 store 层实现(持久化), survives Edge restart

@coderabbitai

coderabbitai Bot commented Aug 29, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 48db4d00-6371-4beb-8e02-f5a20139018e

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

DeliciousBuding and others added 4 commits August 31, 2026 01:02
Run struct 新增 HubTaskID 字段,runcontrol.Create 在 runCreationMu
锁内检查已有同 hubTaskID 的 run 并幂等返回,防止孤儿恢复重投时
Edge 创建重复 run。SQLite migration v5 添加 edge_runs.hub_task_id
列与部分索引。

Closes #2066

Co-authored-by: Cursor <cursor@vectorcontrol.tech>
Migration 0069 添加 model_params JSONB 列;TriggerAgentTask 创建任务
时写入 modelParams,使孤儿恢复重投能重建完整 dispatch 上下文。

Closes #2066

Co-authored-by: Cursor <cursor@vectorcontrol.tech>
新增 orphan recovery sweeper:定期扫描 status=queued 且无
delivery_outbox 行的过期任务,CAS claim 后通过现有 DispatchTask
路径重投。上下文从 trigger_message/agent_instance/model_params 重建。
新增 dispatch_orphan metrics(discovered/redelivered)。
L1 集成测试覆盖正向 claim、grace 窗口负向、有 outbox 行负向。

Closes #2066

Co-authored-by: Cursor <cursor@vectorcontrol.tech>
@DeliciousBuding
DeliciousBuding force-pushed the fix/dispatch-orphan-recovery branch from eea9f23 to 4addc90 Compare August 30, 2026 17:13
@DeliciousBuding
DeliciousBuding merged commit 864eb41 into master Aug 30, 2026
38 checks passed
@DeliciousBuding
DeliciousBuding deleted the fix/dispatch-orphan-recovery branch August 30, 2026 17:27
DeliciousBuding added a commit that referenced this pull request Aug 31, 2026
Co-authored-by: Cursor <cursor@vectorcontrol.tech>
DeliciousBuding added a commit that referenced this pull request Aug 31, 2026
Co-authored-by: Cursor <cursor@vectorcontrol.tech>
DeliciousBuding added a commit that referenced this pull request Aug 31, 2026
* fix(db): team route-decision JSONB COUNT 执行计划优化(#2102 F14)

Co-authored-by: Cursor <cursor@vectorcontrol.tech>

* chore(db): 迁移改号 0069→0070 避让 #2074 同号迁移

Co-authored-by: Cursor <cursor@vectorcontrol.tech>

---------

Co-authored-by: DeliciousBuding <DeliciousBuding@users.noreply.github.com>
Co-authored-by: Cursor <cursor@vectorcontrol.tech>
DeliciousBuding added a commit that referenced this pull request Aug 31, 2026
重启 dev 栈后现场抓到:master hub-server 在 "health check passed" 后
卡死,goroutine dump 显示主 goroutine 停在
startServer → StartOrphanRecoveryLoop 的 select 循环上——
该 sweeper 自 #2074 起阻塞调用方(startServer 主路径),
HTTP server 永远走不到 "server starting",:8080 无监听。

修复:sweeper 循环移入内部 goroutine(与 delivery retry/cleanup、
run-event retention 等兄弟循环一致),方法立即返回。
补回归测试 TestStartOrphanRecoveryLoopDoesNotBlockCaller(3s 超时)。
本地验证:dispatchsvc+app 包测试、go build、sleep-ratchet 全绿。

Co-authored-by: Cursor <cursor@vectorcontrol.tech>
DeliciousBuding added a commit that referenced this pull request Aug 31, 2026
重启 dev 栈后现场抓到:master hub-server 在 "health check passed" 后
卡死,goroutine dump 显示主 goroutine 停在
startServer → StartOrphanRecoveryLoop 的 select 循环上——
该 sweeper 自 #2074 起阻塞调用方(startServer 主路径),
HTTP server 永远走不到 "server starting",:8080 无监听。

修复:sweeper 循环移入内部 goroutine(与 delivery retry/cleanup、
run-event retention 等兄弟循环一致),方法立即返回。
补回归测试 TestStartOrphanRecoveryLoopDoesNotBlockCaller(3s 超时)。
本地验证:dispatchsvc+app 包测试、go build、sleep-ratchet 全绿。

Co-authored-by: DeliciousBuding <DeliciousBuding@users.noreply.github.com>
Co-authored-by: Cursor <cursor@vectorcontrol.tech>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[P0 事件一致性] 切片 A:dispatch 业务写与 outbox 记录非同事务——崩溃/回退窗口任务静默丢失(审计 + 收口)

1 participant