From e778088771a87acff72f19b28f59be582b736a1a Mon Sep 17 00:00:00 2001 From: Pierre Date: Tue, 14 Jul 2026 12:13:38 +0200 Subject: [PATCH 1/3] fix: preserve orchestration sessions across restarts --- docs-web/architecture/virtual-workers.md | 6 + .../docs/architecture-virtual-workers.mdx | 6 + .../docs/settings-restart-behavior.mdx | 23 + .../content/docs/user-automation-and-ci.mdx | 4 + .../docs/user-sprint-orchestration.mdx | 4 + docs-web/settings/restart-behavior.md | 23 + docs-web/user/automation-and-ci.md | 4 + docs-web/user/sprint-orchestration.md | 4 + docs/architecture/quality-assurance-agent.md | 15 +- docs/architecture/virtual-workers.md | 16 +- docs/development/mockup-sprint-pentest.md | 2 +- docs/settings/restart-behavior.md | 23 + docs/sprint-loop/atomic-loop.md | 5 +- .../e2e/mockup-sprint-pentest-scenarios.mjs | 1 + scripts/e2e/run-mockup-sprint-pentest.mjs | 95 +- src/contracts/project-attention-types.ts | 24 + src/domain/qa-review/qa-review-budget.ts | 22 + .../sprint/orchestrator/cycle-runner.ts | 10 +- .../workers/project-attention-service.ts | 8 + .../cli/workspace-artifact-service.ts | 101 +- .../project-attention-repository.ts | 24 + .../session-tracking-repository.ts | 4 - src/server/code-ux-server.ts | 5 +- .../code-ux-default-assets-service.ts | 12 +- src/services/quality-assurance-service.ts | 1229 ++++++++++++++--- .../runtime-recovery/qa-review-recovery.ts | 39 +- .../runtime-startup-recovery-service.ts | 41 +- src/services/virtual-worker-service.ts | 936 ++++++++++--- .../domain/qa-review/qa-review-budget.test.ts | 25 + .../sprint/orchestrator/cycle-runner.test.ts | 79 ++ .../cli/workspace-artifact-service.test.ts | 47 +- .../session-tracking-repository.test.ts | 25 + .../mockup-sprint-pentest-runner.test.ts | 79 ++ .../quality-assurance-service.test.ts | 805 ++++++++++- .../runtime-startup-recovery-service.test.ts | 76 + .../services/virtual-worker-service.test.ts | 739 +++++++++- 36 files changed, 4132 insertions(+), 429 deletions(-) diff --git a/docs-web/architecture/virtual-workers.md b/docs-web/architecture/virtual-workers.md index f6176dbbda..7ec202ad85 100644 --- a/docs-web/architecture/virtual-workers.md +++ b/docs-web/architecture/virtual-workers.md @@ -120,6 +120,10 @@ Docker-backed planning uses a read-only snapshot workspace instead of a mutable Provider CLI workspace preparation is centralized through `InvocationWorkspacePreparer`. Its shared provider-invocation option builder constructs snapshot checkout, git policy, and fresh/continue lifecycle values for Docker provider calls, while its continuation resolver locates preserved workspaces and their current branches. Fresh Docker invocations in `REMOTE` git mode use explicit remote refs only: planning, project setup, dashboard/chat replies, worker inbox replies, node-flow provider prompts, QA review snapshots, task coding, QA follow-up, CI autofix, and merge-conflict repair all materialize from `origin/` refs rather than local branches or the host repo's current checkout. Dashboard/chat replies resolve dashboard settings with the project scope before building this policy, so local Git projects keep `LOCAL` snapshot behavior and do not require `origin/`. Continuation/restart flows may reuse a preserved workspace for provider-session continuity; if a preserved workspace is missing and a new workspace must be materialized, the same remote-only branch policy applies. +QA reviewers, standalone CI-fix workers, and merge-conflict workers checkpoint their own logical session and workspace before provider execution. Under restart invocation policy `continue`, the replacement invocation reuses that workspace and resumes the provider-native conversation when available. Merge-conflict continuation also recognizes an in-progress Git merge and continues resolving it instead of replaying the merge operation. + +CI-fix and merge-conflict repair checkpoints retain the original workspace Git baseline, the finalized repair head, and the host-publication phase. A restart after provider or merge completion exports from that original baseline and resumes publication instead of treating the repair commit as a new baseline, rerunning the provider, or replaying the merge. Merge recovery also recognizes a completed merge commit by its target-branch ancestry when the restart landed immediately before the finalization checkpoint. Host publication commits carry the repair head as a trailer; recovery from `host_publishing` finds that marker and idempotently pushes the existing remote branch before settlement, preventing a second patch application when the process exited after materialization but before the `host_published` checkpoint. For legacy unmarked publications, Code UX derives the effective workspace tree including uncommitted and newly created files and searches matching reachable repair commits even if the branch later advanced. Settlement requires the exact repair tree and subject, baseline ancestry, and the target parent for merge repairs. + Docker-volume artifact export performs Git discovery, staging, binary diffing, and temporary-index cleanup in one helper-container invocation instead of paying four or five Docker control-plane round trips per completed task. Host-side patch transaction files live under Git's administrative directory so materialization stays on the warm project Git helper rather than creating one-shot helpers for external temporary binds. When a LOCAL branch advances while an isolated worker is running, patch materialization applies the diff against its true workspace base and three-way merges the resulting tree onto the current descendant tip. Concurrent work is retained, identical already-landed file additions are de-duplicated, and genuine overlapping edits remain conflicts. ## Session lifecycle @@ -174,6 +178,8 @@ The virtual worker can claim and act on these attention item categories: Repair attention is scheduled before ordinary coding dispatches. Code UX does not lease a coding task while CI-fix or merge-conflict attention is waiting, and capacity is checked against the provider selected by the invocation-specific route rather than the generic virtual-worker provider. The final provider-slot wait is bounded to 30 seconds so sprint finalization cannot wait forever on a saturated or stale route. +Startup recovery releases `ci_fix_required` and `merge_conflict` items claimed by stopped virtual-worker endpoints and returns them to the queue with their repair-session checkpoint intact. Continuing that same interrupted attempt does not spend another guardrail attempt. Retryable interruption preserves the repair workspace; terminal success or exhaustion follows normal cleanup. + Task-scoped CI repair continues the originating coding session, native provider session, effective model, coding-agent instructions, and preserved workspace by default. Settings → AI Models → CI fix can disable this behavior and force the standalone CI Fix route; sprint-level final-merge repair always uses that route. Failed invocations return attention to an unclaimed retryable state while the guardrail budget remains. When the default five-attempt limit is reached, Code UX creates a human handoff containing the last error and attempt count. Immediately before every Docker provider launch attempt, Code UX reasserts runtime-volume ownership for the container's effective non-root UID/GID. This repairs newly created, stale, or concurrently recreated root-owned provider HOME/cache volumes at the atomic `docker run` boundary, including standalone final-merge CI repair. Workspace seed helpers explicitly trust mounted `/workspace` while initializing Git and then restore the provider UID/GID, so restart recovery does not trip Git's dubious-ownership protection on a correctly non-root-owned volume. diff --git a/docs-web/content/docs/architecture-virtual-workers.mdx b/docs-web/content/docs/architecture-virtual-workers.mdx index 4e8e3830f5..fca81310dd 100644 --- a/docs-web/content/docs/architecture-virtual-workers.mdx +++ b/docs-web/content/docs/architecture-virtual-workers.mdx @@ -120,6 +120,10 @@ Docker-backed planning uses a read-only snapshot workspace instead of a mutable Provider CLI workspace preparation is centralized through `InvocationWorkspacePreparer`. Its shared provider-invocation option builder constructs snapshot checkout, git policy, and fresh/continue lifecycle values for Docker provider calls, while its continuation resolver locates preserved workspaces and their current branches. Fresh Docker invocations in `REMOTE` git mode use explicit remote refs only: planning, project setup, dashboard/chat replies, worker inbox replies, node-flow provider prompts, QA review snapshots, task coding, QA follow-up, CI autofix, and merge-conflict repair all materialize from `origin/` refs rather than local branches or the host repo's current checkout. Dashboard/chat replies resolve dashboard settings with the project scope before building this policy, so local Git projects keep `LOCAL` snapshot behavior and do not require `origin/`. Continuation/restart flows may reuse a preserved workspace for provider-session continuity; if a preserved workspace is missing and a new workspace must be materialized, the same remote-only branch policy applies. +QA reviewers, standalone CI-fix workers, and merge-conflict workers checkpoint their own logical session and workspace before provider execution. Under restart invocation policy `continue`, the replacement invocation reuses that workspace and resumes the provider-native conversation when available. Merge-conflict continuation also recognizes an in-progress Git merge and continues resolving it instead of replaying the merge operation. + +CI-fix and merge-conflict repair checkpoints retain the original workspace Git baseline, the finalized repair head, and the host-publication phase. A restart after provider or merge completion exports from that original baseline and resumes publication instead of treating the repair commit as a new baseline, rerunning the provider, or replaying the merge. Merge recovery also recognizes a completed merge commit by its target-branch ancestry when the restart landed immediately before the finalization checkpoint. Host publication commits carry the repair head as a trailer; recovery from `host_publishing` finds that marker and idempotently pushes the existing remote branch before settlement, preventing a second patch application when the process exited after materialization but before the `host_published` checkpoint. For legacy unmarked publications, Code UX derives the effective workspace tree including uncommitted and newly created files and searches matching reachable repair commits even if the branch later advanced. Settlement requires the exact repair tree and subject, baseline ancestry, and the target parent for merge repairs. + Docker-volume artifact export performs Git discovery, staging, binary diffing, and temporary-index cleanup in one helper-container invocation instead of paying four or five Docker control-plane round trips per completed task. Host-side patch transaction files live under Git's administrative directory so materialization stays on the warm project Git helper rather than creating one-shot helpers for external temporary binds. When a LOCAL branch advances while an isolated worker is running, patch materialization applies the diff against its true workspace base and three-way merges the resulting tree onto the current descendant tip. Concurrent work is retained, identical already-landed file additions are de-duplicated, and genuine overlapping edits remain conflicts. ## Session lifecycle @@ -174,6 +178,8 @@ The virtual worker can claim and act on these attention item categories: Repair attention is scheduled before ordinary coding dispatches. Code UX does not lease a coding task while CI-fix or merge-conflict attention is waiting, and capacity is checked against the provider selected by the invocation-specific route rather than the generic virtual-worker provider. The final provider-slot wait is bounded to 30 seconds so sprint finalization cannot wait forever on a saturated or stale route. +Startup recovery releases `ci_fix_required` and `merge_conflict` items claimed by stopped virtual-worker endpoints and returns them to the queue with their repair-session checkpoint intact. Continuing that same interrupted attempt does not spend another guardrail attempt. Retryable interruption preserves the repair workspace; terminal success or exhaustion follows normal cleanup. + Task-scoped CI repair continues the originating coding session, native provider session, effective model, coding-agent instructions, and preserved workspace by default. Settings → AI Models → CI fix can disable this behavior and force the standalone CI Fix route; sprint-level final-merge repair always uses that route. Failed invocations return attention to an unclaimed retryable state while the guardrail budget remains. When the default five-attempt limit is reached, Code UX creates a human handoff containing the last error and attempt count. Immediately before every Docker provider launch attempt, Code UX reasserts runtime-volume ownership for the container's effective non-root UID/GID. This repairs newly created, stale, or concurrently recreated root-owned provider HOME/cache volumes at the atomic `docker run` boundary, including standalone final-merge CI repair. Workspace seed helpers explicitly trust mounted `/workspace` while initializing Git and then restore the provider UID/GID, so restart recovery does not trip Git's dubious-ownership protection on a correctly non-root-owned volume. diff --git a/docs-web/content/docs/settings-restart-behavior.mdx b/docs-web/content/docs/settings-restart-behavior.mdx index 1f0a8dab34..bbdd5070f8 100644 --- a/docs-web/content/docs/settings-restart-behavior.mdx +++ b/docs-web/content/docs/settings-restart-behavior.mdx @@ -15,12 +15,31 @@ Use it when you are configuring a new project, auditing inherited settings, or d Sprint policy continues, pauses, or cancels active sprints; invocation policy continues, cancels, or restarts interrupted work. +The invocation policy applies to every provider-backed orchestration stage, not only task coding. Under `continue`, Code UX durably resumes task coding, QA review, QA-requested coding follow-up, CI-fix, and merge-conflict work from their recorded logical session and workspace. When the provider exposed a resumable native session, the replacement invocation continues that native conversation as well. + | Control Surface | Runtime Effect | Review Before Saving | | --- | --- | --- | | Settings card fields | Updates the active Settings scope after you save the page. | Confirm whether you are editing System or Project scope. | | Inherited values | Values can flow from system defaults into project and sprint behavior. | Check the source badge before assuming a value is project-specific. | | Related runtime paths | The affected service reads the saved settings during planning, dispatch, dashboard rendering, or maintenance work. | Re-run the affected workflow after changing operational settings. | +### Continue-policy recovery contract + +When `restartSprintPolicy = continue` and `restartInvocationPolicy = continue`, startup recovery: + +- resumes the existing sprint run and watch loop instead of creating a replacement sprint run +- correlates each interrupted QA reviewer with its exact execution invocation, reviewer preset, logical provider session, and isolated review workspace +- reuses the QA review workspace and provider conversation for the retry, so a restart does not discard reviewer investigation already completed before the interruption +- checkpoints every configured reviewer in a multi-reviewer cycle before invoking the first reviewer; recovery keeps completed verdicts, resumes only interrupted reviewers, and fills any reviewer row missing from a legacy partial cycle without spending another QA cycle +- preserves task-level and sprint-completion `changes_requested` verdicts before starting their coding handoffs; if restart occurs between the verdict and the follow-up invocation, the next cycle resumes that pending handoff instead of leaving QA indefinitely blocked +- returns an abruptly failed QA coding handoff to `CODING_COMPLETED`/`QA_PENDING` and retries it from the recorded coding session and workspace. A successful or reconciled handoff remains in that verification-ready state until the next QA review starts, preventing the restart window from launching unrelated coding work. Provider failures are bounded to three continuation attempts, while resuming a `running` checkpoint after a runtime restart does not consume another failure allowance; exhaustion then follows the configured QA exhaustion policy instead of redispatching the task as unrelated coding or heartbeating forever. +- records the original worker-branch baseline before invoking a QA coding follow-up and reuses it after restart, so provider commits made before host-branch publication are still exported and published instead of being mistaken for an empty follow-up +- reconciles the recovered coding task-run and dispatch after a successful handoff, preventing an earlier transient failure marker from incorrectly failing the sprint during terminal evaluation +- requeues interrupted worker-owned CI-fix and merge-conflict attention, clearing ownership left by the stopped virtual worker +- resumes those repair workers with the same logical session, native provider session when available, and preserved workspace, so uncommitted repair progress survives the process boundary + +Recovery closes the interrupted invocation row for auditability and creates a correlated continuation invocation. That terminal audit row does not mean the logical work was abandoned. + ## Recommended Configuration Continue sprints and continue invocations for local development; pause when you want manual review after downtime. @@ -35,6 +54,8 @@ A practical review flow is: Restarting interrupted work can duplicate provider effort if the previous CLI run was still externally active. +The continuation guarantee depends on the provider's resumable session support and on the workspace volume still being available. Code UX preserves managed workspace volumes during a normal shutdown and fails closed when it cannot safely recover required Git state; manually deleting Docker volumes or provider-side conversations removes information the runtime cannot reconstruct. + Before applying changes, check: - Whether the value affects provider credentials, Docker runtime behavior, Git automation, memory retention, or destructive cleanup. @@ -49,6 +70,8 @@ If the saved setting does not appear to take effect: - Check for a project or sprint override that takes precedence over the system value. - Refresh the affected dashboard page if the setting controls a rendered surface. - Restart the local runtime only when the setting explicitly controls startup, listener, or process-level behavior. +- For a task parked at `QA_PENDING`, inspect the latest QA row for a pending fix handoff and confirm that a correlated `cli_task_followup` invocation was resumed or already completed. +- For CI-fix or merge-conflict work, confirm the attention item returned to the queue after startup and that its continuation invocation retained the prior workspace/session identifiers. ## Related Documentation diff --git a/docs-web/content/docs/user-automation-and-ci.mdx b/docs-web/content/docs/user-automation-and-ci.mdx index f5dfc1f465..87dd6989e5 100644 --- a/docs-web/content/docs/user-automation-and-ci.mdx +++ b/docs-web/content/docs/user-automation-and-ci.mdx @@ -122,6 +122,8 @@ If a QA agent preset is wired to `qa_review` in routing, completed tasks pass th Provider or infrastructure errors do not immediately create this handoff. Recovered failed, cancelled, and errored attempts retry within a bounded infrastructure grace, and every terminal attempt counts toward the hard ceiling. A CLI QA fix that produces no new patch is escalated as no progress instead of starting another QA cycle; existing branch commits do not renew the cycle. A coding run that produces no changes must explicitly confirm completion or it is blocked for attention. The sprint watch loop stays alive while worker or human attention is active. +QA review has its own durable session, separate from the coding session that receives fixes. With the restart invocation policy set to `continue`, Code UX resumes an interrupted reviewer in the same isolated review workspace and continues the provider conversation when the provider supports native session resumption. A requested-fix verdict is saved before its coding handoff starts. If restart lands in that gap, the next watch cycle resumes the pending handoff; if the coding follow-up already completed, Code UX advances to verification without repeating it. An abruptly failed coding handoff returns to the saved QA checkpoint and retries the same coding session/workspace up to the bounded continuation limit before applying the configured QA exhaustion policy. + Task and sprint summary badges select one reviewer from the newest QA cycle. Within that cycle, running reviews appear first, followed by requested changes, provider failures (`failed`, `cancelled`, or `errored`), passes, and other states. This keeps a passing reviewer from hiding another reviewer that is still active, has blocked the work, or did not return a usable verdict. ## Attention items: who handles them @@ -137,6 +139,8 @@ The eligible attention items per provider: Humans can claim and resolve items at any time from the dashboard. +Under the restart `continue` policy, an interrupted Code UX-owned CI-fix or merge-conflict item is released from the stopped virtual worker and returned to the worker queue. Its replacement invocation reuses the prior logical session and repair workspace, and resumes the provider-native conversation when available. This preserves partial edits and provider reasoning across a normal runtime restart instead of spending a new guardrail attempt on the same interrupted work. + ## Recommended settings recipes ### Conservative (recommended starting point) diff --git a/docs-web/content/docs/user-sprint-orchestration.mdx b/docs-web/content/docs/user-sprint-orchestration.mdx index e4a6b47112..7ff15839e1 100644 --- a/docs-web/content/docs/user-sprint-orchestration.mdx +++ b/docs-web/content/docs/user-sprint-orchestration.mdx @@ -120,6 +120,10 @@ Docker-backed task workspaces prepare independently. Code UX locks only the work On restart, interrupted local CLI task runs may be cancelled and redispatched, but their workspace volumes are preserved. If the coding provider had already finished, the resumed run continues with Git finalization from that workspace instead of invoking the coding agent again. Session sync treats finished local CLI task runs as terminal even if a stale cached session snapshot still reports the old session as running. +With restart invocation policy `continue`, the same continuity applies to QA reviewers, CI-fix workers, and merge-conflict workers. Each resumed invocation keeps its logical session and preserved workspace, and continues the provider-native conversation when supported. Code UX releases repair attention left claimed by the stopped virtual worker and returns it to the queue without consuming another repair attempt. + +QA fix handoffs are durable as well. Code UX records a requested-fix handoff before invoking the task's coding session. If restart occurs in that gap, the watch loop resumes the pending handoff; if the coding follow-up finished before its final QA update was written, the loop recognizes that execution and proceeds to verification instead of repeating the fix or leaving the task at `QA_PENDING`. + When a worker resolves a merge conflict, Code UX clears the task's stale `MERGE_CONFLICT` marker while keeping the task unmerged. The next protocol cycle retries the normal merge path instead of reopening the same attention item. That clear history is separate from merge-required suppression: suppression applies only after Git confirms the source branch has no commits ahead of the target feature branch. If the branch still has merge work, the task remains merge-required. Task-level human conflict handoffs are dismissed automatically once the task marker is cleared, while main-merge and unrelated manual handoffs remain visible. CLI tasks that complete with a worker branch but no PR use a branch-only merge path in both LOCAL and REMOTE git modes; REMOTE mode then pushes the sprint feature branch. If the task snapshot lost the worker branch, Code UX recovers it from the completed task run before checking merge readiness. For CLI-backed runs, branch-only classification and protocol merge-required attention wait for the git-finalize event (`cli_git_pushed` or `cli_git_no_changes`) so provider/session completion cannot race ahead of branch materialization. Task QA reviews run from an isolated snapshot of that selected branch in both Docker and host execution, so a visible default-branch checkout cannot create a false missing-file rejection. That merge runs in a temporary worktree through the containerized Git helper so the visible checkout and `.code-ux/` runtime files do not interfere with task settlement. When several clean LOCAL worker branches are ready in one cycle, they share that worktree while each successful merge is committed and published to the feature branch independently. Feature-branch publication is atomic: if a task merge overlaps a CI-fix patch, the writer that loses the ref race rebuilds on the new tip and retries, so neither result is discarded and provider work remains parallel. Code UX normalizes temporary worktree gitdir metadata after creation so later helper-container Git calls resolve the same repository. Once the task is settled as merged, stale task-run worker branch evidence is suppressed from live status so old branches do not keep re-entering merge scans. diff --git a/docs-web/settings/restart-behavior.md b/docs-web/settings/restart-behavior.md index 1f0a8dab34..bbdd5070f8 100644 --- a/docs-web/settings/restart-behavior.md +++ b/docs-web/settings/restart-behavior.md @@ -15,12 +15,31 @@ Use it when you are configuring a new project, auditing inherited settings, or d Sprint policy continues, pauses, or cancels active sprints; invocation policy continues, cancels, or restarts interrupted work. +The invocation policy applies to every provider-backed orchestration stage, not only task coding. Under `continue`, Code UX durably resumes task coding, QA review, QA-requested coding follow-up, CI-fix, and merge-conflict work from their recorded logical session and workspace. When the provider exposed a resumable native session, the replacement invocation continues that native conversation as well. + | Control Surface | Runtime Effect | Review Before Saving | | --- | --- | --- | | Settings card fields | Updates the active Settings scope after you save the page. | Confirm whether you are editing System or Project scope. | | Inherited values | Values can flow from system defaults into project and sprint behavior. | Check the source badge before assuming a value is project-specific. | | Related runtime paths | The affected service reads the saved settings during planning, dispatch, dashboard rendering, or maintenance work. | Re-run the affected workflow after changing operational settings. | +### Continue-policy recovery contract + +When `restartSprintPolicy = continue` and `restartInvocationPolicy = continue`, startup recovery: + +- resumes the existing sprint run and watch loop instead of creating a replacement sprint run +- correlates each interrupted QA reviewer with its exact execution invocation, reviewer preset, logical provider session, and isolated review workspace +- reuses the QA review workspace and provider conversation for the retry, so a restart does not discard reviewer investigation already completed before the interruption +- checkpoints every configured reviewer in a multi-reviewer cycle before invoking the first reviewer; recovery keeps completed verdicts, resumes only interrupted reviewers, and fills any reviewer row missing from a legacy partial cycle without spending another QA cycle +- preserves task-level and sprint-completion `changes_requested` verdicts before starting their coding handoffs; if restart occurs between the verdict and the follow-up invocation, the next cycle resumes that pending handoff instead of leaving QA indefinitely blocked +- returns an abruptly failed QA coding handoff to `CODING_COMPLETED`/`QA_PENDING` and retries it from the recorded coding session and workspace. A successful or reconciled handoff remains in that verification-ready state until the next QA review starts, preventing the restart window from launching unrelated coding work. Provider failures are bounded to three continuation attempts, while resuming a `running` checkpoint after a runtime restart does not consume another failure allowance; exhaustion then follows the configured QA exhaustion policy instead of redispatching the task as unrelated coding or heartbeating forever. +- records the original worker-branch baseline before invoking a QA coding follow-up and reuses it after restart, so provider commits made before host-branch publication are still exported and published instead of being mistaken for an empty follow-up +- reconciles the recovered coding task-run and dispatch after a successful handoff, preventing an earlier transient failure marker from incorrectly failing the sprint during terminal evaluation +- requeues interrupted worker-owned CI-fix and merge-conflict attention, clearing ownership left by the stopped virtual worker +- resumes those repair workers with the same logical session, native provider session when available, and preserved workspace, so uncommitted repair progress survives the process boundary + +Recovery closes the interrupted invocation row for auditability and creates a correlated continuation invocation. That terminal audit row does not mean the logical work was abandoned. + ## Recommended Configuration Continue sprints and continue invocations for local development; pause when you want manual review after downtime. @@ -35,6 +54,8 @@ A practical review flow is: Restarting interrupted work can duplicate provider effort if the previous CLI run was still externally active. +The continuation guarantee depends on the provider's resumable session support and on the workspace volume still being available. Code UX preserves managed workspace volumes during a normal shutdown and fails closed when it cannot safely recover required Git state; manually deleting Docker volumes or provider-side conversations removes information the runtime cannot reconstruct. + Before applying changes, check: - Whether the value affects provider credentials, Docker runtime behavior, Git automation, memory retention, or destructive cleanup. @@ -49,6 +70,8 @@ If the saved setting does not appear to take effect: - Check for a project or sprint override that takes precedence over the system value. - Refresh the affected dashboard page if the setting controls a rendered surface. - Restart the local runtime only when the setting explicitly controls startup, listener, or process-level behavior. +- For a task parked at `QA_PENDING`, inspect the latest QA row for a pending fix handoff and confirm that a correlated `cli_task_followup` invocation was resumed or already completed. +- For CI-fix or merge-conflict work, confirm the attention item returned to the queue after startup and that its continuation invocation retained the prior workspace/session identifiers. ## Related Documentation diff --git a/docs-web/user/automation-and-ci.md b/docs-web/user/automation-and-ci.md index 86606d7089..5da6aad0c8 100644 --- a/docs-web/user/automation-and-ci.md +++ b/docs-web/user/automation-and-ci.md @@ -122,6 +122,8 @@ If a QA agent preset is wired to `qa_review` in routing, completed tasks pass th Provider or infrastructure errors do not immediately create this handoff. Recovered failed, cancelled, and errored attempts retry within a bounded infrastructure grace, and every terminal attempt counts toward the hard ceiling. A CLI QA fix that produces no new patch is escalated as no progress instead of starting another QA cycle; existing branch commits do not renew the cycle. A coding run that produces no changes must explicitly confirm completion or it is blocked for attention. The sprint watch loop stays alive while worker or human attention is active. +QA review has its own durable session, separate from the coding session that receives fixes. With the restart invocation policy set to `continue`, Code UX resumes an interrupted reviewer in the same isolated review workspace and continues the provider conversation when the provider supports native session resumption. A requested-fix verdict is saved before its coding handoff starts. If restart lands in that gap, the next watch cycle resumes the pending handoff; if the coding follow-up already completed, Code UX advances to verification without repeating it. An abruptly failed coding handoff returns to the saved QA checkpoint and retries the same coding session/workspace up to the bounded continuation limit before applying the configured QA exhaustion policy. + Task and sprint summary badges select one reviewer from the newest QA cycle. Within that cycle, running reviews appear first, followed by requested changes, provider failures (`failed`, `cancelled`, or `errored`), passes, and other states. This keeps a passing reviewer from hiding another reviewer that is still active, has blocked the work, or did not return a usable verdict. ## Attention items: who handles them @@ -137,6 +139,8 @@ The eligible attention items per provider: Humans can claim and resolve items at any time from the dashboard. +Under the restart `continue` policy, an interrupted Code UX-owned CI-fix or merge-conflict item is released from the stopped virtual worker and returned to the worker queue. Its replacement invocation reuses the prior logical session and repair workspace, and resumes the provider-native conversation when available. This preserves partial edits and provider reasoning across a normal runtime restart instead of spending a new guardrail attempt on the same interrupted work. + ## Recommended settings recipes ### Conservative (recommended starting point) diff --git a/docs-web/user/sprint-orchestration.md b/docs-web/user/sprint-orchestration.md index a8b24fcd81..0a5bb3de87 100644 --- a/docs-web/user/sprint-orchestration.md +++ b/docs-web/user/sprint-orchestration.md @@ -120,6 +120,10 @@ Docker-backed task workspaces prepare independently. Code UX locks only the work On restart, interrupted local CLI task runs may be cancelled and redispatched, but their workspace volumes are preserved. If the coding provider had already finished, the resumed run continues with Git finalization from that workspace instead of invoking the coding agent again. Session sync treats finished local CLI task runs as terminal even if a stale cached session snapshot still reports the old session as running. +With restart invocation policy `continue`, the same continuity applies to QA reviewers, CI-fix workers, and merge-conflict workers. Each resumed invocation keeps its logical session and preserved workspace, and continues the provider-native conversation when supported. Code UX releases repair attention left claimed by the stopped virtual worker and returns it to the queue without consuming another repair attempt. + +QA fix handoffs are durable as well. Code UX records a requested-fix handoff before invoking the task's coding session. If restart occurs in that gap, the watch loop resumes the pending handoff; if the coding follow-up finished before its final QA update was written, the loop recognizes that execution and proceeds to verification instead of repeating the fix or leaving the task at `QA_PENDING`. + When a worker resolves a merge conflict, Code UX clears the task's stale `MERGE_CONFLICT` marker while keeping the task unmerged. The next protocol cycle retries the normal merge path instead of reopening the same attention item. That clear history is separate from merge-required suppression: suppression applies only after Git confirms the source branch has no commits ahead of the target feature branch. If the branch still has merge work, the task remains merge-required. Task-level human conflict handoffs are dismissed automatically once the task marker is cleared, while main-merge and unrelated manual handoffs remain visible. CLI tasks that complete with a worker branch but no PR use a branch-only merge path in both LOCAL and REMOTE git modes; REMOTE mode then pushes the sprint feature branch. If the task snapshot lost the worker branch, Code UX recovers it from the completed task run before checking merge readiness. For CLI-backed runs, branch-only classification and protocol merge-required attention wait for the git-finalize event (`cli_git_pushed` or `cli_git_no_changes`) so provider/session completion cannot race ahead of branch materialization. Task QA reviews run from an isolated snapshot of that selected branch in both Docker and host execution, so a visible default-branch checkout cannot create a false missing-file rejection. That merge runs in a temporary worktree through the containerized Git helper so the visible checkout and `.code-ux/` runtime files do not interfere with task settlement. When several clean LOCAL worker branches are ready in one cycle, they share that worktree while each successful merge is committed and published to the feature branch independently. Feature-branch publication is atomic: if a task merge overlaps a CI-fix patch, the writer that loses the ref race rebuilds on the new tip and retries, so neither result is discarded and provider work remains parallel. Code UX normalizes temporary worktree gitdir metadata after creation so later helper-container Git calls resolve the same repository. Once the task is settled as merged, stale task-run worker branch evidence is suppressed from live status so old branches do not keep re-entering merge scans. diff --git a/docs/architecture/quality-assurance-agent.md b/docs/architecture/quality-assurance-agent.md index e0dd6afd6c..43c0473d15 100644 --- a/docs/architecture/quality-assurance-agent.md +++ b/docs/architecture/quality-assurance-agent.md @@ -165,6 +165,10 @@ Provider/infrastructure failures in sprint-completion QA are also retryable with Recovery guarantees: - task QA no longer depends only on catching a single in-cycle transition edge; if a task is already code-complete and still has no successful QA run, Code UX will enqueue the missing review on the next orchestration cycle instead of leaving the task parked in `QA_PENDING` +- every task- and sprint-completion reviewer row records the exact review execution invocation, logical reviewer session, isolated workspace session, reviewer preset, and continuation provenance. Under the restart `continue` policy, a retry reuses that reviewer workspace and continues the provider's native session when available instead of starting the review again without its prior investigation context. +- a `changes_requested` result persists its coding-handoff state before Code UX invokes the target coding session. If the runtime stops after saving the verdict but before finishing that handoff, the next cycle resumes the pending handoff. If the same-session coding follow-up already completed, recovery settles the handoff from that execution evidence and schedules verification rather than invoking the coding provider again. This closes the post-verdict crash window that could otherwise leave a task parked indefinitely at `QA_PENDING`. +- a transient provider exit during that coding handoff preserves the original target session/workspace, restores the task to `CODING_COMPLETED` with `QA_PENDING`, and leaves the handoff retryable. Successful and execution-reconciled handoffs also remain `CODING_COMPLETED`/`QA_PENDING` until verification is scheduled, so a restart cannot mistake the crash window for ordinary task work. Continuation failures are capped by `QA_INFRA_FAILURE_GRACE`; exhausting the cap records no-progress evidence so the normal QA exhaustion policy settles or escalates the task. +- when a later same-session handoff succeeds, Code UX also reconciles the original task-run and dispatch back to completed before sprint terminal evaluation. An earlier failed continuation therefore cannot leave stale runtime evidence that falsely fails an otherwise healthy sprint. - if a QA run row is left behind in `running` state after its backing execution invocation has already finished, Code UX now automatically converts that stale row into a retryable failed run so the gate can recover instead of blocking indefinitely - before task QA starts, Code UX polls feature PR status with any task-level PR URLs already recorded by Jules. This lets orchestration recover the PR head branch even when the Jules PR base branch has drifted from the currently configured sprint feature branch. - if a prior task QA run requested changes, Code UX sends fix instructions back to the same task session when possible and tracks that work as same-session follow-up instead of creating a new task branch. @@ -244,13 +248,20 @@ Behavior: ## Session Continuation -QA does not open an isolated side-channel for fixes. +QA review and QA-requested fixes use two related but distinct session tracks: -Instead: +- the reviewer runs in an isolated review workspace and owns its own durable logical/provider session +- any requested implementation fix returns to the target task's coding session and worktree + +For the reviewer track, Code UX persists the exact execution invocation and workspace binding on the reviewer-specific QA row before dispatch. A runtime restart under the `continue` invocation policy closes the interrupted audit invocation, preserves the snapshot workspace, and starts a correlated continuation with the same logical session. When the CLI provider supplied a native session id, Code UX passes it to the resumed invocation so the reviewer retains conversation context as well as filesystem state. Multi-reviewer cycles recover each preset independently; timestamp proximity is only a compatibility fallback for older rows that predate exact invocation correlation. + +For the fix track, QA does not open an isolated side-channel. Instead: - Jules tasks receive a follow-up message on the existing Jules session - CLI tasks resume the existing worker session/worktree when possible +The QA row records a pending continuation before that follow-up begins. Completion updates that same handoff record, which lets startup and later watch cycles distinguish work that still needs dispatch from a follow-up that already finished before the final QA payload write. + For CLI follow-up runs, Code UX: - preserves the successful worktree after completion when QA is enabled for task completion diff --git a/docs/architecture/virtual-workers.md b/docs/architecture/virtual-workers.md index c8a3095900..1053c373a9 100644 --- a/docs/architecture/virtual-workers.md +++ b/docs/architecture/virtual-workers.md @@ -152,9 +152,11 @@ For merge conflicts, Code UX: - counts task-scoped merge-conflict attempts in the guardrail ledger; sprint-level final merge conflicts have no task row, so their retry count is stored on the attention item payload as `mergeConflictResolutionAttempts` before each real provider run - stops opening new worker-owned repair attempts for a task once the existing `merge_conflict` item has escalated to an active human handoff, preventing repeated container startup failures from cycling after the guardrail limit is reached -Merge-conflict handling intentionally stays isolated from the original task workspace. It always runs in a dedicated ephemeral Docker workspace so conflict resolution cannot pollute the task's normal follow-up workspace. +Merge-conflict handling intentionally stays isolated from the original task workspace, so conflict resolution cannot pollute the task's normal follow-up workspace. Its dedicated Docker workspace is ephemeral after terminal settlement, but it is preserved while an interrupted attempt remains eligible for restart continuation. The attention payload checkpoints the repair session, workspace, provider/model, native provider session, attempt id, and phase before provider execution; a resumed worker can therefore continue the same merge-in-progress state without replaying the merge or charging the interrupted attempt twice. -For task-scoped CI autofix, Code UX defaults to continuing the original task coding session exactly like a QA follow-up: it reuses the logical/native provider session, provider family, effective model, coding-agent instructions, and preserved task workspace. Settings → AI Models → CI fix exposes **Continue from same session and model as coding task** as an opt-out; disabling it uses the standalone CI Fix route. Sprint-level final-merge repair has no originating task session and always uses that route. The CI-fix prompt also receives the active agent's memory context and writes new durable learnings back into memory from the reused workspace. +Repair checkpoints also persist the workspace's original Git baseline, its finalized repair head, and the host-publication phase. CI-fix and merge-conflict recovery therefore always export against the pre-repair baseline even when the provider committed inside the preserved workspace before a restart. Once workspace finalization is durable, recovery skips provider execution and resumes host publication; merge recovery can also recognize a completed merge commit by its changed head and target-branch ancestry if the process exited immediately before that phase checkpoint. Host publication commits carry the durable repair head as a commit trailer. Recovery from `host_publishing` searches the worker branch for that trailer and, in remote mode, idempotently pushes the existing branch before settling, so a restart after patch materialization but before the `host_published` checkpoint never applies the repair twice. For legacy unmarked checkpoints, recovery derives the effective workspace tree including uncommitted tracked and untracked edits, then searches reachable host repair commits rather than only the branch tip. It accepts a commit only when that tree and the exact repair subject match, the saved baseline is its ancestor, and merge repairs retain the target parent; a tree-identical merge is allowed only when its saved repair commit differs from the baseline. The published host head is then checkpointed before attention settlement. + +For task-scoped CI autofix, Code UX defaults to continuing the original task coding session exactly like a QA follow-up: it reuses the logical/native provider session, provider family, effective model, coding-agent instructions, and preserved task workspace. Settings → AI Models → CI fix exposes **Continue from same session and model as coding task** as an opt-out; disabling it uses the standalone CI Fix route. Sprint-level final-merge repair has no originating task session and always uses that route. Standalone CI repair checkpoints its own logical/native session and isolated workspace on the attention item, giving it the same restart-continuation semantics as merge-conflict repair. The CI-fix prompt also receives the active agent's memory context and writes new durable learnings back into memory from the reused workspace. Workspace artifact export captures both tracked edits and newly created untracked files from the worker workspace. This matters for CI autofix follow-ups that add missing modules or tests after the original task run; the exporter uses a temporary Git index for untracked files and still excludes the transient `.task-learnings.md` memory-capture file, legacy `.code-ux-home/` provider state, and root `.pnpm-store/` package-cache state from commits. It asks Git to discover untracked files internally before diffing, so preserved Docker workspaces with many untracked paths cannot exceed command argument limits. Docker-volume exports fuse discovery, staging, diffing, and temporary-index cleanup into one helper-container invocation, avoiding four or five Docker control-plane round trips per completed task. Host-side patch transaction files stay inside Git's administrative directory, keeping materialization commands on the warm project Git helper instead of forcing one-shot helpers for an external temporary-index bind. When a LOCAL branch advances while an isolated worker is running, patch materialization first applies the diff against the workspace's true base, then three-way merges that tree onto the current descendant worker tip. Concurrent branch work is retained, an identical file already materialized by the original task is de-duplicated, and genuine overlapping edits still fail as conflicts. Current Docker workers keep provider HOME and Code UX-managed npm/pnpm cache paths in a paired runtime volume mounted outside `/workspace`, so fresh workspaces contain only the coding checkout. @@ -174,7 +176,7 @@ When feature-PR CI retries exhaust their guardrail, the runtime task remains blo If Docker is unavailable when the CI autofix flow starts, Code UX degrades that specific repair run to a host-backed worktree instead of looping on an unrecoverable Docker failure. Merge-conflict resolution does not use this fallback: it remains Docker-only so conflict repair stays isolated from the reusable task workspace. -For QA review execution, Code UX now runs the review itself against a fresh snapshot workspace rather than the mutable task workspace. This keeps review inspection isolated while still allowing QA-requested coding follow-ups to continue in the original task workspace when appropriate. Both the review agent and QA-requested coding follow-ups now receive their current memory context, and QA follow-up edits capture fresh learnings back into memory from the actual workspace used for the fix. +For QA review execution, Code UX now runs the review itself against a fresh snapshot workspace rather than the mutable task workspace. This keeps review inspection isolated while still allowing QA-requested coding follow-ups to continue in the original task workspace when appropriate. The QA row checkpoints the reviewer logical session, exact execution invocation, and review workspace before dispatch; restart continuation reuses those records and the provider-native session when available. Both the review agent and QA-requested coding follow-ups now receive their current memory context, and QA follow-up edits capture fresh learnings back into memory from the actual workspace used for the fix. Unsupported worker-owned attention types are escalated back to human attention with a summary. @@ -192,7 +194,13 @@ It deduplicates this project set and explicitly ignores all other projects in th ## Recovery -Startup cleanup prunes orphaned `virtual_cli` endpoints from previous runs. +Startup cleanup prunes orphaned `virtual_cli` endpoints from previous runs. Under the restart invocation `continue` policy, it also requeues claimed `ci_fix_required` and `merge_conflict` attention owned by those stopped endpoints. The requeued payload retains its repair-runtime checkpoint, while the next virtual worker: + +- reuses the same logical session and workspace session +- resolves the latest provider invocation for the previous native session and OpenCode usage baseline +- resumes an in-progress Git merge instead of starting it again +- avoids recording a second guardrail attempt for the same interrupted attempt id +- preserves the workspace again if the continuation is cancelled or fails while still retryable, and clears it through normal cleanup only after terminal settlement Startup cleanup also removes stale Code UX Docker assets through a background, label-filtered prune so server boot does not wait on full Docker daemon scans (managed via `DockerAssetPruneService`): diff --git a/docs/development/mockup-sprint-pentest.md b/docs/development/mockup-sprint-pentest.md index 7bbf57a9f1..d5bbcbc69b 100644 --- a/docs/development/mockup-sprint-pentest.md +++ b/docs/development/mockup-sprint-pentest.md @@ -117,7 +117,7 @@ The current runner covers: - Merge conflict handling through `merge-conflict-dag`, which creates sibling edits and validates the resolved join task. - Parallel task execution through `parallel-independent`, which fans out independent tasks before an aggregate task. - Dirty local checkout finalization through `dirty-checkout-final-merge`, which leaves an uncommitted visible checkout before orchestration and verifies LOCAL final merge preserves that work. -- Sprint-completion merge conflict repair through `completion-merge-conflict`, which mutates the default branch during orchestration so the final LOCAL sprint merge must invoke the mockup merge-conflict worker. +- Sprint-completion merge conflict repair through `completion-merge-conflict`, which waits for a running `task_coding` provider invocation before mutating the default branch. This places the mutation after sprint preflight and workspace preparation but during the fixture task's built-in delay, so the final LOCAL sprint merge deterministically invokes the mockup merge-conflict worker instead of racing feature-branch synchronization or worker-branch publication. The fixture synchronizes a checked-out default worktree after its detached mutation and requires a completed `merge_conflict` invocation, preventing stale-index dirty-work false positives or silent clean-merge passes. - CI-sized QA DAG orchestration through `ci-qa-dag`, a deterministic graph that covers task QA pass, task QA decline, follow-up file creation on the recovered worker branch, a second task QA pass, sprint QA, and final repository assertions. This scenario is the default no-secret GitHub Actions orchestration lane. - Large-DAG orchestration through `large-dag-stress`, a heavy 129-task graph with 96 leaf tasks, 24 batch joins, 6 group joins, a final manifest, and a validation task. This scenario is excluded from default `all` runs and included by `--scenario pentest`. - Runtime restart recovery by restarting the isolated compiled-runtime server during active project runs with `--restart-every-ms` and `--restart-count`; polling tolerates transient API failures while the server is down and then validates terminal sprint/task state after recovery. diff --git a/docs/settings/restart-behavior.md b/docs/settings/restart-behavior.md index 14686a3282..52fadf312f 100644 --- a/docs/settings/restart-behavior.md +++ b/docs/settings/restart-behavior.md @@ -15,12 +15,31 @@ Use it when you are configuring a new project, auditing inherited settings, or d Sprint policy continues, pauses, or cancels active sprints; invocation policy continues, cancels, or restarts interrupted work. +The invocation policy applies to every provider-backed orchestration stage, not only task coding. Under `continue`, Code UX durably resumes task coding, QA review, QA-requested coding follow-up, CI-fix, and merge-conflict work from their recorded logical session and workspace. When the provider exposed a resumable native session, the replacement invocation continues that native conversation as well. + | Control Surface | Runtime Effect | Review Before Saving | | --- | --- | --- | | Settings card fields | Updates the active Settings scope after you save the page. | Confirm whether you are editing System or Project scope. | | Inherited values | Values can flow from system defaults into project and sprint behavior. | Check the source badge before assuming a value is project-specific. | | Related runtime paths | The affected service reads the saved settings during planning, dispatch, dashboard rendering, or maintenance work. | Re-run the affected workflow after changing operational settings. | +### Continue-policy recovery contract + +When `restartSprintPolicy = continue` and `restartInvocationPolicy = continue`, startup recovery: + +- resumes the existing sprint run and watch loop instead of creating a replacement sprint run +- correlates each interrupted QA reviewer with its exact execution invocation, reviewer preset, logical provider session, and isolated review workspace +- reuses the QA review workspace and provider conversation for the retry, so a restart does not discard reviewer investigation already completed before the interruption +- checkpoints every configured reviewer in a multi-reviewer cycle before invoking the first reviewer; recovery keeps completed verdicts, resumes only interrupted reviewers, and fills any reviewer row missing from a legacy partial cycle without spending another QA cycle +- preserves task-level and sprint-completion `changes_requested` verdicts before starting their coding handoffs; if restart occurs between the verdict and the follow-up invocation, the next cycle resumes that pending handoff instead of leaving QA indefinitely blocked +- returns an abruptly failed QA coding handoff to `CODING_COMPLETED`/`QA_PENDING` and retries it from the recorded coding session and workspace. A successful or reconciled handoff remains in that verification-ready state until the next QA review starts, preventing the restart window from launching unrelated coding work. Provider failures are bounded to three continuation attempts, while resuming a `running` checkpoint after a runtime restart does not consume another failure allowance; exhaustion then follows the configured QA exhaustion policy instead of redispatching the task as unrelated coding or heartbeating forever. +- records the original worker-branch baseline before invoking a QA coding follow-up and reuses it after restart, so provider commits made before host-branch publication are still exported and published instead of being mistaken for an empty follow-up +- reconciles the recovered coding task-run and dispatch after a successful handoff, preventing an earlier transient failure marker from incorrectly failing the sprint during terminal evaluation +- requeues interrupted worker-owned CI-fix and merge-conflict attention, clearing ownership left by the stopped virtual worker +- resumes those repair workers with the same logical session, native provider session when available, and preserved workspace, so uncommitted repair progress survives the process boundary + +Recovery closes the interrupted invocation row for auditability and creates a correlated continuation invocation. That terminal audit row does not mean the logical work was abandoned. + ## Recommended Configuration Continue sprints and continue invocations for local development; pause when you want manual review after downtime. @@ -35,6 +54,8 @@ A practical review flow is: Restarting interrupted work can duplicate provider effort if the previous CLI run was still externally active. +The continuation guarantee depends on the provider's resumable session support and on the workspace volume still being available. Code UX preserves managed workspace volumes during a normal shutdown and fails closed when it cannot safely recover required Git state; manually deleting Docker volumes or provider-side conversations removes information the runtime cannot reconstruct. + Before applying changes, check: - Whether the value affects provider credentials, Docker runtime behavior, Git automation, memory retention, or destructive cleanup. @@ -49,6 +70,8 @@ If the saved setting does not appear to take effect: - Check for a project or sprint override that takes precedence over the system value. - Refresh the affected dashboard page if the setting controls a rendered surface. - Restart the local runtime only when the setting explicitly controls startup, listener, or process-level behavior. +- For a task parked at `QA_PENDING`, inspect the latest QA row for a pending fix handoff and confirm that a correlated `cli_task_followup` invocation was resumed or already completed. +- For CI-fix or merge-conflict work, confirm the attention item returned to the queue after startup and that its continuation invocation retained the prior workspace/session identifiers. ## Related Documentation diff --git a/docs/sprint-loop/atomic-loop.md b/docs/sprint-loop/atomic-loop.md index af5a695cf1..cfeb25ed2d 100644 --- a/docs/sprint-loop/atomic-loop.md +++ b/docs/sprint-loop/atomic-loop.md @@ -149,6 +149,7 @@ When `action=orchestrate`, `wait` is true, and `watchLoop` is enabled: - Finalisation only runs on terminal conditions. - Startup recovery and dashboard **Resume** restart monitoring through the existing-run recovery path. A resumed paused run keeps its original sprint-run id, is moved back to `running`, and then starts the watch loop without creating a duplicate run. Resume is refused while another queued/running/cancel-pending run for the same sprint is active. - If shutdown lands after a coding provider has completed but before Git finalization records the task as code-complete, recovery preserves the workspace and marks that exact crash window. The replacement task run resumes at Git finalization and reuses the completed provider result instead of invoking the coding agent a second time. A missing preserved workspace falls back to a normal fresh invocation rather than trusting unavailable changes. +- Under the restart invocation `continue` policy, startup treats QA review, CI fix, and merge-conflict repair as durable work streams just like task coding. It closes each interrupted audit invocation, preserves its logical/native session and workspace binding, and requeues only the work needed to create the correlated continuation. Claimed repair attention is released from the stopped virtual-worker endpoint before the next worker poll. - Existing-run recovery first checks the in-memory active-orchestrator registry and returns without starting another watch loop when the same project/sprint is already being monitored by the current process. - Sprint-run lifecycle updates are mirrored to the parent sprint row for dashboard/operator consistency. Active run states (`queued`, `running`, `cancel_requested`) keep the sprint `running`; pause, completion, failure, and cancellation transitions update the sprint row to the matching summary state. Heartbeats also repair drift after restarts, so a live run cannot remain hidden behind an `idle` sprint summary. - Human-escalated merge conflicts stop counting as worker activity. If a task conflict has already been handed to a human and no runnable work remains, the watch loop pauses the sprint instead of keeping the run alive with only heartbeat traffic. @@ -209,7 +210,8 @@ For `action=status`: - Completed CLI tasks with a worker branch but no PR use the deterministic branch-only merge path in both LOCAL and REMOTE git modes; REMOTE mode then pushes the sprint feature branch. If the markdown/task snapshot lost `worker_branch`, the gate recovers it from the latest completed task run before deciding whether merge work exists. For CLI-backed task runs, the gate waits for a git-finalize event (`cli_git_pushed` or `cli_git_no_changes`) before classifying branch-only work, so provider/session completion cannot race ahead of worker-branch materialization. Task QA always receives an isolated snapshot of its selected worker branch: Docker uses a volume snapshot and HOST uses a detached Git worktree, so the visible default-branch checkout cannot create false missing-file review failures. The task merge runs in a temporary worktree through the containerized Git helper, so the visible checkout, user dirt, and `.code-ux/` runtime files cannot block or be modified by worker-branch settlement. A LOCAL gate reuses one detached temporary worktree for all clean worker branches found in the same cycle, but updates the feature-branch ref after every successful merge; a conflict is aborted and isolated without rolling back earlier published merges. Feature-ref publication uses an atomic expected-old-tip update. When a task merge and CI-fix patch materialization overlap, the losing writer rebuilds on the newly published tip and retries, preserving both histories without serializing independent provider work behind a process-wide lock. Temporary worktree gitdir metadata is normalized to relative paths after creation so every later helper-container Git call resolves the same host repository. - Once a task is settled as merged, runtime projection suppresses stale `task_runs.worker_branch` evidence so old completed runs do not re-enter branch-only merge scans or keep the dashboard showing dead per-task branches. - In `WHEN_GREEN` feature PR mode, a clean PR with no check-rollup entries and no tracked CI runs is treated as CI-skipped after a 10 minute grace window. This prevents feature PRs from heartbeating forever when the repository has PR workflows but no run is ever materialized for that branch. -- When QA requests fixes and Code UX applies them through a same-session CLI follow-up, the next sprint cycle treats the completed `cli_task_followup` invocation as fresh task work and reruns QA verification instead of waiting for a separate task-run completion timestamp. If a restart happens before that invocation marker is persisted, the cycle can also use a later completed task run for the task's current session as recovery evidence, preventing a completed fix attempt from staying parked at `CODING_COMPLETED`/`QA_PENDING` forever. +- When QA requests fixes and Code UX applies them through a same-session CLI follow-up, the next sprint cycle treats the completed `cli_task_followup` invocation as fresh task work and reruns QA verification instead of waiting for a separate task-run completion timestamp. The task remains `CODING_COMPLETED`/`QA_PENDING` between handoff settlement and verification dispatch, closing the restart window in which ordinary coding could otherwise be launched. If a restart happens before that invocation marker is persisted, the cycle can also use a later completed task run for the task's current session as recovery evidence, preventing a completed fix attempt from staying parked forever. +- QA reviewer execution is correlated by exact invocation id and reviewer preset, with a distinct logical session and isolated workspace for each reviewer row. Restart continuation reuses that workspace and resumes the provider-native conversation when available. A `changes_requested` row records `pending` before its coding continuation begins, so the next cycle can resume an interrupted handoff or settle an already-completed follow-up instead of heartbeating forever behind the saved verdict. - Task-completion and completed-without-PR QA use the trigger's ordered `agentPresetIds` list. `[]` means zero custom reviewer IDs plus one built-in/default QA fallback; one ID runs one reviewer; multiple IDs run multiple reviewers in order. Each resolved reviewer creates its own `qa_review_runs` row in the same review cycle with the same `run_index`, for example `agent-qa-security` and `agent-qa-regression` can both review `project-123` task `T02` in run `2`. The cycle passes only when every reviewer passes. Any reviewer that requests changes, fails, or is still running keeps the task blocked under the existing QA gate rules, and reviewer rows remain visible per agent. - QA self-reflection is available under `agents.selfReflection.qualityAssurance` and also defaults to disabled. When enabled, QA results follow the same rate-and-improve loop as planning, and any improved QA output must pass the normal normalized QA schema before it can replace the previous valid result. - When CLI QA follow-up work creates or reuses a task PR after an earlier PR for the same task was already merged, Code UX clears stale merged state and persists the task as code-complete again so the feature PR gate can evaluate and auto-merge the follow-up PR. @@ -248,6 +250,7 @@ For `action=status`: - If feature PR checks fail, the sprint loop keeps the task in work state and enters the CI-fix guardrail path. When `waitForJulesCiAutofix` (the legacy-named configuration for CI autofix) is enabled for a hosted-provider-managed task, Code UX first notifies the hosted provider session with failed-check context from only the newest branch-matched failed CI run: its id/URL and every failed job, step, and actionable error/assertion excerpt. Older matching failures are excluded. When that toggle is disabled, or the task is not hosted-provider-managed, Code UX skips the hosted provider notification and dispatches a worker-owned `ci_fix_required` item. - CI autofix retries are capped by `julesCiAutofixMaxRetries` (the legacy-named setting); once exhausted, the task is escalated as intervention-needed with exact task id, PR URL, failed check names, the newest failed-run summary, and every failed job name (focus: fix CI before merge). The cap applies to the generic CI-fix loop, including worker repairs. - Worker-owned CI autofix attempts are de-duplicated across watch-loop cycles. While a matching `ci_fix_required` attention item is still open or claimed, Code UX treats that attempt as in-flight, keeps the task in `RUNNING`, and does not consume another retry until the worker attempt resolves. This includes the final main-merge gate: after a worker pushes a CI fix and GitHub reports replacement checks as pending, the main-merge `ci_fix_required` item stays active until checks pass, the merge completes, or another blocker replaces it. +- If restart interrupts a claimed worker-owned CI-fix or merge-conflict attempt, startup clears the stale endpoint claim and requeues the same attention item with its repair-session continuation metadata. The next virtual worker reuses the preserved worktree and logical provider session, passes the previous native session id when supported, and does not count the process interruption as a new guardrail attempt. Successful settlement clears the continuation marker before normal CI/merge gating resumes. - Repair attention takes precedence over queued coding dispatches, and virtual-worker admission checks the provider/limit selected by the actual `ci_fix` route. The final provider-slot claim has a 30-second wait bound, so finalization retries or escalates instead of hanging forever behind a saturated or stale provider slot. - Failed-job log collection searches the complete log of every failed job in the newest branch-matched failed run for failed-step names, error markers, assertion failures, stack traces, and expected/received output. The prompt receives bounded evidence windows around those signals instead of generic runner bootstrap and cleanup output, plus the exact `gh run view --job --log-failed` fallback command for each failed job. Older matching runs are not included. Task and final-merge CI repairs use the same evidence payload. - A task blocked after exhausting CI-fix retries is persisted as `coding_completed` in the planning layer while retaining its runtime CI block and intervention state. Reloading the sprint therefore cannot reinterpret the completed implementation as a pending task and relaunch ordinary `task_coding` work. diff --git a/scripts/e2e/mockup-sprint-pentest-scenarios.mjs b/scripts/e2e/mockup-sprint-pentest-scenarios.mjs index d0ac7b0888..a7150c340b 100644 --- a/scripts/e2e/mockup-sprint-pentest-scenarios.mjs +++ b/scripts/e2e/mockup-sprint-pentest-scenarios.mjs @@ -827,6 +827,7 @@ export const SCENARIOS = [ duringOrchestration: { defaultBranch: "main", timeoutMs: 60_000, + waitForRunningTaskCoding: true, defaultBranchMutations: [ { commitMessage: "mockup default branch conflict before sprint finalization", diff --git a/scripts/e2e/run-mockup-sprint-pentest.mjs b/scripts/e2e/run-mockup-sprint-pentest.mjs index cae6ad63f6..868c4b892a 100644 --- a/scripts/e2e/run-mockup-sprint-pentest.mjs +++ b/scripts/e2e/run-mockup-sprint-pentest.mjs @@ -553,20 +553,43 @@ async function applyBeforeOrchestrationHook(repoDir, projectRun) { } } -async function waitForNonDefaultBranch(repoDir, defaultBranch, timeoutMs = 60_000) { +export function selectDuringOrchestrationBranch(branches, defaultBranch, requireTaskBranch = false) { + const candidates = branches.filter((entry) => entry !== defaultBranch); + return requireTaskBranch + ? candidates.find((entry) => entry.startsWith("task/")) || null + : candidates[0] || null; +} + +async function waitForNonDefaultBranch( + repoDir, + defaultBranch, + timeoutMs = 60_000, + requireTaskBranch = false, +) { const startedAt = Date.now(); while (Date.now() - startedAt < timeoutMs) { const result = await spawnLogged("git", ["for-each-ref", "--format=%(refname:short)", "refs/heads"], { cwd: repoDir }); const branches = result.stdout.split(/\r?\n/).map((entry) => entry.trim()).filter(Boolean); - const branch = branches.find((entry) => entry !== defaultBranch); + const branch = selectDuringOrchestrationBranch(branches, defaultBranch, requireTaskBranch); if (branch) return branch; await delay(500); } - throw new Error(`Timed out waiting for a non-${defaultBranch} branch before applying orchestration hook.`); + throw new Error( + requireTaskBranch + ? "Timed out waiting for a task branch before applying orchestration hook." + : `Timed out waiting for a non-${defaultBranch} branch before applying orchestration hook.`, + ); } -async function mutateDefaultBranch(repoDir, mutation) { +export async function mutateDefaultBranch(repoDir, mutation) { const defaultBranch = mutation.defaultBranch || "main"; + const visibleBranch = (await spawnLogged("git", ["branch", "--show-current"], { cwd: repoDir })).stdout.trim(); + if (visibleBranch === defaultBranch) { + const visibleStatus = (await spawnLogged("git", ["status", "--porcelain"], { cwd: repoDir })).stdout.trim(); + if (visibleStatus) { + throw new Error(`Refusing to mutate checked-out ${defaultBranch}: fixture worktree is unexpectedly dirty.`); + } + } const tempWorktree = await fs.mkdtemp(path.join(path.dirname(repoDir), ".default-branch-mutation-")); try { await spawnLogged("git", ["worktree", "add", "--detach", tempWorktree, defaultBranch], { cwd: repoDir }); @@ -581,12 +604,46 @@ async function mutateDefaultBranch(repoDir, mutation) { await spawnLogged("git", ["commit", "-m", mutation.commitMessage || "mockup default branch mutation"], { cwd: tempWorktree }); const head = await spawnLogged("git", ["rev-parse", "HEAD"], { cwd: tempWorktree }); await spawnLogged("git", ["update-ref", `refs/heads/${defaultBranch}`, head.stdout.trim()], { cwd: repoDir }); + if (visibleBranch === defaultBranch) { + // update-ref intentionally bypasses Git's checked-out-branch protection. Synchronize the + // fixture-owned visible checkout immediately so its old index/worktree is not misread as + // operator dirty work by final LOCAL merge preservation. + await spawnLogged("git", ["reset", "--hard", head.stdout.trim()], { cwd: repoDir }); + } } finally { await spawnLogged("git", ["worktree", "remove", "--force", tempWorktree], { cwd: repoDir }).catch(() => undefined); await fs.rm(tempWorktree, { recursive: true, force: true }).catch(() => undefined); } } +async function waitForRunningTaskCodingInvocation(homeDir, records, timeoutMs = 60_000) { + const databasePath = path.join(homeDir, ".code-ux", "app.db"); + const startedAt = Date.now(); + while (Date.now() - startedAt < timeoutMs) { + let database; + try { + database = new DatabaseSync(databasePath, { readOnly: true }); + const invocation = database.prepare(` + SELECT id + FROM provider_invocations + WHERE project_id = ? + AND sprint_id = ? + AND purpose = 'task_coding' + AND status = 'running' + ORDER BY started_at DESC + LIMIT 1 + `).get(records.project.id, records.sprint.id); + if (invocation?.id) return invocation.id; + } catch { + // The runtime may still be creating or migrating its database. + } finally { + try { database?.close(); } catch { /* best effort */ } + } + await delay(250); + } + throw new Error("Timed out waiting for a running task_coding invocation before applying orchestration hook."); +} + async function injectMainCiFixAttention(homeDir, records, repoDir, branchName, config = {}) { const databasePath = path.join(homeDir, ".code-ux", "app.db"); const startedAt = Date.now(); @@ -660,9 +717,17 @@ async function injectMainCiFixAttention(homeDir, records, repoDir, branchName, c async function applyDuringOrchestrationHook(repoDir, projectRun, context) { const hook = projectRun.duringOrchestration; if (!hook) return null; + if (hook.waitForRunningTaskCoding === true) { + await waitForRunningTaskCodingInvocation(context.homeDir, context.records, hook.timeoutMs || 60_000); + } let branchName = null; if (hook.waitForNonDefaultBranch !== false) { - branchName = await waitForNonDefaultBranch(repoDir, hook.defaultBranch || projectRun.project?.defaultBranch || "main", hook.timeoutMs || 60_000); + branchName = await waitForNonDefaultBranch( + repoDir, + hook.defaultBranch || projectRun.project?.defaultBranch || "main", + hook.timeoutMs || 60_000, + hook.waitForTaskBranch === true, + ); } if (hook.injectMainCiFix) { branchName ||= await waitForNonDefaultBranch( @@ -1495,6 +1560,23 @@ export function assertExpectedTaskCodingInvocations(invocations, expectedCounts) } } +export function assertExpectedMergeConflictInvocation(invocations, createsMergeConflict) { + if (!createsMergeConflict) return; + const completedCount = invocations.filter((invocation) => ( + invocation.purpose === "merge_conflict" && invocation.status === "completed" + )).length; + if (completedCount < 1) { + throw new Error(`expected at least one completed merge_conflict invocation, received ${completedCount}`); + } +} + +function assertMergeConflictHistory(homeDir, records, projectRun) { + if (!projectRun.expected?.createsMergeConflict) return null; + const history = readQaHistory(homeDir, records.project.id, records.sprint.id); + assertExpectedMergeConflictInvocation(history.invocations, true); + return { invocations: history.invocations }; +} + function assertInvocationHistory(homeDir, records, projectRun) { const expected = projectRun.expected?.invocations; if (!expected) return null; @@ -1601,7 +1683,8 @@ async function runProjectRunInner(server, scenario, projectRun, options) { await assertExpectedFiles(records.repoDir, projectRun); commandResults = await assertExpectedCommands(records.repoDir, projectRun); qaHistory = assertQaHistory(options.homeDir, records, projectRun); - invocationHistory = assertInvocationHistory(options.homeDir, records, projectRun); + invocationHistory = assertInvocationHistory(options.homeDir, records, projectRun) + || assertMergeConflictHistory(options.homeDir, records, projectRun); } catch (error) { assertionFailure = error.message; } diff --git a/src/contracts/project-attention-types.ts b/src/contracts/project-attention-types.ts index 78efa91415..1e20ab4321 100644 --- a/src/contracts/project-attention-types.ts +++ b/src/contracts/project-attention-types.ts @@ -16,6 +16,30 @@ export type ProjectAttentionOwnerType = "worker" | "human" | "system"; export type ProjectAttentionStatus = "open" | "claimed" | "resolved" | "dismissed" | "expired"; export type WorkerAttentionOutcome = "handled_locally" | "needs_dashboard_reply" | "needs_human_escalation"; +export type RepairPublicationPhase = + | "pending" + | "workspace_finalized" + | "host_publishing" + | "host_published"; + +export interface RepairAttentionRuntimeState { + purpose: "ci_fix" | "merge_conflict"; + sessionId: string; + workspaceSessionId: string; + provider: string; + providerConfigId: string; + model: string; + nativeSessionId: string | null; + activeAttemptId: string | null; + attemptRecorded: boolean; + phase: "claimed" | "workspace_ready" | "provider_running" | "interrupted"; + workspaceBaselineHead: string | null; + workspaceRepairHead: string | null; + publicationPhase: RepairPublicationPhase; + publishedHeadSha: string | null; + updatedAt: string; +} + export interface ProjectAttentionItemRecord { id: string; projectId: string; diff --git a/src/domain/qa-review/qa-review-budget.ts b/src/domain/qa-review/qa-review-budget.ts index 4e5c97ed56..2278e7fba3 100644 --- a/src/domain/qa-review/qa-review-budget.ts +++ b/src/domain/qa-review/qa-review-budget.ts @@ -22,6 +22,28 @@ export function shouldVerifyContinuedQaFix(run: QaReviewRunRecord | null): boole && run.payload?.postExhaustionVerificationEligible === true; } +export function isPendingQaContinuation(run: QaReviewRunRecord | null): boolean { + if (run?.status !== "completed" || run.outcome !== "changes_requested" || !run.fixInstructions?.trim()) { + return false; + } + const continuationStatus = run.payload?.continuationStatus; + if (continuationStatus === "pending" || continuationStatus === "running") { + return true; + } + if (continuationStatus === "failed") { + const attemptCount = run.payload?.continuationAttemptCount; + const exhaustedAttempts = typeof attemptCount === "number" + && Number.isFinite(attemptCount) + && attemptCount >= QA_INFRA_FAILURE_GRACE; + return run.payload?.followUpNoProgress !== true && !exhaustedAttempts; + } + if (typeof continuationStatus === "string") { + return false; + } + return typeof run.payload?.continued !== "boolean" + && typeof run.payload?.continuationMode !== "string"; +} + export interface QaReviewBudgetArgs { existingRuns: number; decisiveRuns: number; diff --git a/src/domain/sprint/orchestrator/cycle-runner.ts b/src/domain/sprint/orchestrator/cycle-runner.ts index 06b73f4deb..419051dc2b 100644 --- a/src/domain/sprint/orchestrator/cycle-runner.ts +++ b/src/domain/sprint/orchestrator/cycle-runner.ts @@ -33,7 +33,7 @@ import { matchPrForTask } from "../ci/feature-pr/pr-matcher.js"; import { resolveCiEscalationOwner } from "../ci/feature-pr/ci-autofix-policy.js"; import type { MemoryCategory, CreateMemoryInput } from "../../../contracts/memory-types.js"; import { isTaskCodeComplete } from "../task-merge-state.js"; -import { shouldVerifyContinuedQaFix } from "../../qa-review/qa-review-budget.js"; +import { isPendingQaContinuation, shouldVerifyContinuedQaFix } from "../../qa-review/qa-review-budget.js"; import pLimit from "p-limit"; import { workerBranchHasMergeWork } from "../../../infrastructure/git/local-merge.js"; import { PROVIDER_IDS } from "../../../repositories/settings-defaults.js"; @@ -1339,6 +1339,7 @@ export class CycleRunner { const taskIsCodeComplete = isTaskCodeComplete(task); const hasSameSessionFollowUpAfterLatestQaRequest = taskIsCodeComplete && this.hasCompletedTaskFollowUpAfterLatestQaRequest(task, qaGate, args.sprintRunId); + const hasPendingQaFollowUp = isPendingQaContinuation(qaGate.latestRun); // QA spent its budget without ever clearing this task (no pass — either // changes still outstanding at the cap or the reviewer kept failing for @@ -1346,7 +1347,7 @@ export class CycleRunner { // it quietly settle as completed or loop forever. const qaNeedsExhaustionPolicy = qaGate.reason === "retries_exhausted" || qaGate.reason === "follow_up_no_progress"; - if (taskIsCodeComplete && qaNeedsExhaustionPolicy && !hasSameSessionFollowUpAfterLatestQaRequest) { + if (taskIsCodeComplete && qaNeedsExhaustionPolicy && !hasSameSessionFollowUpAfterLatestQaRequest && !hasPendingQaFollowUp) { const policy = settings.agents.qualityAssurance.exhaustionPolicy; if (this.applyQaExhaustionPolicy(task, qaGate, args, policy)) { if (policy === "FINISH_TASK") { @@ -1357,7 +1358,8 @@ export class CycleRunner { } const newlyCodeComplete = taskIsCodeComplete && !isTaskCodeComplete({ status: prev }); - const shouldRunQaReview = taskIsCodeComplete + const shouldRunQaReview = hasPendingQaFollowUp + || (taskIsCodeComplete && ( qaGate.reason === "pending_review" || qaGate.reason === "review_failed" @@ -1366,7 +1368,7 @@ export class CycleRunner { newlyCodeComplete || hasSameSessionFollowUpAfterLatestQaRequest )) - ); + )); if (!shouldRunQaReview) { continue; diff --git a/src/domain/workers/project-attention-service.ts b/src/domain/workers/project-attention-service.ts index d7885f9889..8fd325a65e 100644 --- a/src/domain/workers/project-attention-service.ts +++ b/src/domain/workers/project-attention-service.ts @@ -188,6 +188,14 @@ export class ProjectAttentionService { return requeued; } + requeueInterruptedVirtualRepairItems(): ProjectAttentionItemRecord[] { + const requeued = this.projectAttentionRepository.requeueInterruptedVirtualRepairItems(); + for (const item of requeued) { + this.onWorkerAttentionOpenedCallback?.(item.projectId); + } + return requeued; + } + private requireItem(itemId: string): ProjectAttentionItemRecord { const item = this.projectAttentionRepository.getAttentionItem(itemId); if (!item) { diff --git a/src/infrastructure/providers/cli/workspace-artifact-service.ts b/src/infrastructure/providers/cli/workspace-artifact-service.ts index 5b9667af2d..30b6588882 100644 --- a/src/infrastructure/providers/cli/workspace-artifact-service.ts +++ b/src/infrastructure/providers/cli/workspace-artifact-service.ts @@ -11,6 +11,18 @@ import type { IWorkspaceManager } from "./workspace-manager.js"; const TEMP_EXPORT_PATHSPEC = ":(exclude).code-ux-export-*"; +const workspaceExportPathspecs = (): string[] => [ + ".", + `:(exclude)${LEARNINGS_FILENAME}`, + TEMP_EXPORT_PATHSPEC, + ":(exclude).code-ux-home", + ":(exclude).code-ux-home/**", + ":(exclude).pnpm-store", + ":(exclude).pnpm-store/**", + ":(exclude,glob)**/logs/openai/**", + ":(exclude,glob)logs/openai/**", +]; + export interface AppliedWorkspacePatchResult { hasChanges: boolean; commitSha?: string; @@ -106,17 +118,7 @@ export class WorkspaceArtifactService { // against the base. Git still owns discovery of new, modified, and deleted // files, including ignore handling, while Code UX avoids passing a large // changed-path list through Docker argv. - const excludePathspecs = [ - `:(exclude)${LEARNINGS_FILENAME}`, - TEMP_EXPORT_PATHSPEC, - ":(exclude).code-ux-home", - ":(exclude).code-ux-home/**", - ":(exclude).pnpm-store", - ":(exclude).pnpm-store/**", - ":(exclude,glob)**/logs/openai/**", - ":(exclude,glob)logs/openai/**", - ]; - const pathspecs = [".", ...excludePathspecs]; + const pathspecs = workspaceExportPathspecs(); const tempIndexFilename = `.code-ux-export-${process.pid}-${Date.now()}-${Math.random().toString(16).slice(2)}.index`; if (workspaceRef.startsWith("docker-volume://")) { return await this.exportDockerVolumeBinaryPatch( @@ -173,6 +175,83 @@ export class WorkspaceArtifactService { } } + async resolveWorkspaceTree(workspaceRef: string): Promise { + const pathspecs = workspaceExportPathspecs(); + const tempIndexFilename = `.code-ux-export-${process.pid}-${Date.now()}-${Math.random().toString(16).slice(2)}.index`; + if (workspaceRef.startsWith("docker-volume://")) { + const tempPathListFilename = tempIndexFilename.replace(/\.index$/, ".paths"); + const script = [ + "index_file=$1", + "path_file=$2", + "shift 2", + "trap 'rm -f \"$index_file\" \"$path_file\"' EXIT", + "export GIT_INDEX_FILE=$index_file", + "git read-tree HEAD", + "git ls-files --modified --deleted --others --exclude-standard -z -- \"$@\" > \"$path_file\"", + "if [ -s \"$path_file\" ]; then", + " git add -A --pathspec-from-file=- --pathspec-file-nul < \"$path_file\"", + "fi", + "git write-tree", + ].join("\n"); + return (await this.workspaceManager.runWorkspaceCommand( + workspaceRef, + "sh", + [ + "-ceu", + script, + "code-ux-tree", + tempIndexFilename, + tempPathListFilename, + ...pathspecs, + ], + )).stdout.trim(); + } + + const tempIndexPath = !path.isAbsolute(workspaceRef) + ? tempIndexFilename + : path.join(workspaceRef, tempIndexFilename); + const tempIndexEnv = { + ...process.env, + GIT_INDEX_FILE: tempIndexPath, + }; + const tempPathListPath = path.join(os.tmpdir(), `code-ux-export-paths-${process.pid}-${Date.now()}-${Math.random().toString(16).slice(2)}.paths`); + try { + await this.workspaceManager.runWorkspaceCommand( + workspaceRef, + "git", + ["read-tree", "HEAD"], + { env: tempIndexEnv }, + ); + const changedPaths = await this.workspaceManager.runWorkspaceCommand( + workspaceRef, + "git", + ["ls-files", "--modified", "--deleted", "--others", "--exclude-standard", "-z", "--", ...pathspecs], + { env: tempIndexEnv, trimOutput: false }, + ); + if (changedPaths.stdout.length > 0) { + await fs.writeFile(tempPathListPath, changedPaths.stdout, "utf8"); + await this.workspaceManager.runWorkspaceCommand( + workspaceRef, + "git", + ["add", "-A", "--pathspec-from-file=-", "--pathspec-file-nul"], + { env: tempIndexEnv, stdinFile: tempPathListPath }, + ); + } + return (await this.workspaceManager.runWorkspaceCommand( + workspaceRef, + "git", + ["write-tree"], + { env: tempIndexEnv }, + )).stdout.trim(); + } finally { + await fs.rm(tempPathListPath, { force: true }).catch(() => undefined); + const hostTempIndexPath = path.isAbsolute(tempIndexPath) + ? tempIndexPath + : path.join(workspaceRef, tempIndexPath); + await fs.rm(hostTempIndexPath, { force: true }).catch(() => undefined); + } + } + private async exportDockerVolumeBinaryPatch( workspaceRef: string, baseRef: string, diff --git a/src/repositories/project-attention-repository.ts b/src/repositories/project-attention-repository.ts index 2cf068f249..4f5ac35bab 100644 --- a/src/repositories/project-attention-repository.ts +++ b/src/repositories/project-attention-repository.ts @@ -666,6 +666,30 @@ export class ProjectAttentionRepository { return this.requireAndNotifyItem(itemId, current.projectId, true); } + requeueInterruptedVirtualRepairItems(): ProjectAttentionItemRecord[] { + const rows = this.db.prepare(` + SELECT attention.* + FROM project_attention_items attention + LEFT JOIN worker_endpoints endpoint + ON endpoint.id = attention.assigned_worker_endpoint_id + WHERE attention.owner_type = 'worker' + AND attention.status = 'claimed' + AND attention.attention_type IN ('ci_fix_required', 'merge_conflict') + AND ( + attention.assigned_worker_endpoint_id IS NULL + OR endpoint.endpoint_type = 'virtual_cli' + OR json_extract(attention.payload_json, '$.repairRuntime.sessionId') IS NOT NULL + ) + ORDER BY attention.opened_at ASC, attention.id ASC + `).all() as unknown as ProjectAttentionItemRow[]; + + return rows.map((row) => this.requeueAttentionItem(row.id, { + recoveredByStartup: true, + repairRecoveryReason: "startup_interrupted_virtual_repair", + repairRecoveredAt: new Date().toISOString(), + })); + } + private requireAndNotifyItem(itemId: string, projectId: string, includeOverview: boolean): ProjectAttentionItemRecord { const item = this.mapRow(requireRecord(this.db.prepare('SELECT * FROM project_attention_items WHERE id = ?').get(itemId) as any, "Project attention item", itemId)); this.notifyProjectRefresh(projectId, includeOverview); diff --git a/src/repositories/session-tracking-repository.ts b/src/repositories/session-tracking-repository.ts index 9890e60313..91a5d7941c 100644 --- a/src/repositories/session-tracking-repository.ts +++ b/src/repositories/session-tracking-repository.ts @@ -285,7 +285,6 @@ export class SessionTrackingRepository { update_time AS updateTime FROM provider_sessions WHERE provider IN (${TRACKED_CLI_PROVIDER_SQL}) - AND id LIKE 'cli-%' ORDER BY update_time DESC `).all() as unknown as TrackedCliSessionRow[]; } @@ -354,7 +353,6 @@ export class SessionTrackingRepository { AND feature_branch = ? AND (repo_path = ? OR repo_path = ? OR repo_path = '/workspace') AND state IN (${statePlaceholders}) - AND id LIKE 'cli-%' AND worker_branch IS NOT NULL ORDER BY create_time DESC, update_time DESC, id DESC LIMIT 1 @@ -392,7 +390,6 @@ export class SessionTrackingRepository { WHERE (repo_path = ? OR repo_path = ? OR repo_path = '/workspace') AND worker_branch = ? AND provider IN (${placeholders}) - AND id LIKE 'cli-%' ORDER BY create_time DESC, update_time DESC, id DESC LIMIT 1 `).get( @@ -419,7 +416,6 @@ export class SessionTrackingRepository { FROM provider_sessions WHERE state = 'RUNNING' AND provider IN (${TRACKED_CLI_PROVIDER_SQL}) - AND id LIKE 'cli-%' ORDER BY create_time ASC `).all() as unknown as SessionIdRow[]; diff --git a/src/server/code-ux-server.ts b/src/server/code-ux-server.ts index 188bd9cc00..1738ff43c5 100644 --- a/src/server/code-ux-server.ts +++ b/src/server/code-ux-server.ts @@ -1349,6 +1349,10 @@ export class CodeUxServer { } catch (error) { this.logger.error("Failed to recover runtime state on startup", { error }); } finally { + // Repair attention can retain a claimed virtual endpoint and a live Docker + // container across a process boundary. Reconcile and stop that stale owner + // before virtual workers are allowed to claim the preserved workspace. + this.virtualWorkerService.start(); this.startupRecoveryCompleted = true; } } @@ -1610,7 +1614,6 @@ export class CodeUxServer { this.startSprintPreviewLoop(); this.startLiveSnapshotLoop(); this.startWalCheckpointLoop(); - this.virtualWorkerService.start(); this.scheduleBackgroundStartupTasks(); } } diff --git a/src/services/code-ux-default-assets-service.ts b/src/services/code-ux-default-assets-service.ts index ca3b3c1a8d..5b5606116c 100644 --- a/src/services/code-ux-default-assets-service.ts +++ b/src/services/code-ux-default-assets-service.ts @@ -101,17 +101,19 @@ export async function ensureDefaultCodeUxAssetsInstalled( } async function buildDefaultAssetTargetDirectorySignature(): Promise { - const directories = [ + const targetPaths = [ getHomeCodeUxPath("agents"), getHomeCodeUxPath("container"), getHomeCodeUxPath("quicksprints", "templates"), + ...DEFAULT_AGENT_FILES.map((fileName) => getHomeCodeUxPath("agents", fileName)), + getHomeCodeUxPath("container", DEFAULT_CONTAINER_SETUP_FILE), ]; - const stats = await Promise.all(directories.map(async (directory) => { + const stats = await Promise.all(targetPaths.map(async (targetPath) => { try { - const stat = await fs.stat(directory, { bigint: true }); - return `${directory}:${stat.mtimeNs}:${stat.size}`; + const stat = await fs.stat(targetPath, { bigint: true }); + return `${targetPath}:${stat.mtimeNs}:${stat.size}`; } catch { - return `${directory}:missing`; + return `${targetPath}:missing`; } })); return stats.join("\0"); diff --git a/src/services/quality-assurance-service.ts b/src/services/quality-assurance-service.ts index 3515305e03..7d88253a45 100644 --- a/src/services/quality-assurance-service.ts +++ b/src/services/quality-assurance-service.ts @@ -1,4 +1,3 @@ -import { randomUUID } from "node:crypto"; import { buildProviderSettingsOverride } from "./provider-settings-override.js"; import { buildProviderPrompt, @@ -49,7 +48,12 @@ import type { SkillService } from "./skill-service.js"; import type { AgentPresetRepository } from "../repositories/agent-preset-repository.js"; import type { McpConnectionInfo } from "../contracts/mcp-connection-types.js"; import { syncRemoteBranchIfAvailable } from "./git-branch-sync-service.js"; -import { evaluateQaReviewBudget, isRecoveredStaleQaRun } from "../domain/qa-review/qa-review-budget.js"; +import { + evaluateQaReviewBudget, + isPendingQaContinuation, + isRecoveredStaleQaRun, + QA_INFRA_FAILURE_GRACE, +} from "../domain/qa-review/qa-review-budget.js"; import { isQaReviewCancellationError, parseQaError } from "../domain/qa-review/qa-review-types.js"; import { normalizeQaReviewResult } from "../domain/qa-review/qa-review-result-normalizer.js"; import type { NormalizedQaReviewResult } from "../domain/qa-review/qa-review-types.js"; @@ -137,6 +141,7 @@ export class QualityAssuranceService { private readonly providerExecutionService: ProviderExecutionService; private readonly structuredAgentRequestService: StructuredAgentRequestService; + private readonly activeQaContinuationRunIds = new Set(); constructor(private readonly deps: QualityAssuranceServiceDependencies) { this.providerExecutionService = new ProviderExecutionService({ @@ -217,18 +222,65 @@ export class QualityAssuranceService { const existingRuns = this.deps.qaReviewRepository.countTaskRuns(taskId); const decisiveRuns = this.deps.qaReviewRepository.countDecisiveTaskRuns(taskId); const latestRun = this.deps.qaReviewRepository.getLatestTaskRun(taskId); + const previousTaskCycleRuns = this.deps.qaReviewRepository.listLatestTaskCycleRuns(taskId); const taskRun = this.resolveTaskRunForSubtask(args.task, args.sprintRunId); + const project = this.deps.projectManagementRepository.getProject(args.projectId); + const sprint = this.deps.projectManagementRepository.getSprint(args.sprintId); + if (!project || !sprint) { + return { reviewed: false, reopenedTask: false, mergeBlocked: false, reportText: "" }; + } + const sprintFeatureBranch = sprint.featureBranch?.trim() + || `${settings.git.featureBranchPrefix || "feature/"}sprint-${sprint.number ?? 0}`; + + const pendingTaskContinuation = previousTaskCycleRuns.find((run) => this.isPendingTaskQaContinuation(run)) ?? null; + if (pendingTaskContinuation) { + return await this.continuePendingTaskQaRun({ + run: pendingTaskContinuation, + task: args.task, + taskRun, + repoPath: args.repoPath, + featureBranch: sprintFeatureBranch, + scope, + decisiveRuns, + maxTaskReviewRuns: qaSettings.maxTaskReviewRuns, + }); + } + + const triggerType = resolveTaskTriggerType(args.task, qaSettings); + const triggerSettings = triggerType === "completed_task_without_pr" + ? qaSettings.completedTaskWithoutPr + : qaSettings.taskCompletion; + const configuredReviewerCount = Math.max(1, triggerSettings.agentPresetIds?.length || 0); + const latestCycleHasChangesRequest = previousTaskCycleRuns.some((run) => ( + run.status === "completed" && run.outcome === "changes_requested" + )); + const latestCycleHasCompletedReviewer = previousTaskCycleRuns.some((run) => run.status === "completed"); + const recoveringPartialReviewerCycle = previousTaskCycleRuns.length > 0 + && !latestCycleHasChangesRequest + && ( + previousTaskCycleRuns.length < configuredReviewerCount + || previousTaskCycleRuns.some((run) => run.status === "running" || run.status === "cancelled") + || (latestCycleHasCompletedReviewer && previousTaskCycleRuns.some((run) => run.status === "failed" || run.status === "errored")) + || triggerSettings.agentPresetIds?.some((presetId) => ( + !previousTaskCycleRuns.some((run) => run.agentPresetId === presetId) + )) + ); + const requestExistingRuns = recoveringPartialReviewerCycle ? Math.max(0, existingRuns - 1) : existingRuns; + const requestDecisiveRuns = recoveringPartialReviewerCycle + && previousTaskCycleRuns.some((run) => run.status === "completed") + ? Math.max(0, decisiveRuns - 1) + : decisiveRuns; const requests = await buildQaReviewRequests({ task: args.task, taskRun, - project: this.deps.projectManagementRepository.getProject(args.projectId) || null, - sprint: this.deps.projectManagementRepository.getSprint(args.sprintId) || null, + project, + sprint, sprintRunId: args.sprintRunId || null, settings, budgetArgs: { - existingRuns, - decisiveRuns, + existingRuns: requestExistingRuns, + decisiveRuns: requestDecisiveRuns, latestRun, }, resolveAgent: (projectId, agentPresetId) => @@ -248,31 +300,45 @@ export class QualityAssuranceService { return { reviewed: false, reopenedTask: false, mergeBlocked: false, reportText: "" }; } - const project = this.deps.projectManagementRepository.getProject(args.projectId); - const sprint = this.deps.projectManagementRepository.getSprint(args.sprintId); - if (!project || !sprint) { + const effectiveRequests = recoveringPartialReviewerCycle + ? requests.filter((request) => { + return !previousTaskCycleRuns.some((run) => ( + run.agentPresetId === request.agentPresetId && run.status === "completed" + )); + }) + : requests; + if (effectiveRequests.length === 0) { return { reviewed: false, reopenedTask: false, mergeBlocked: false, reportText: "" }; } + const effectiveTriggerType = effectiveRequests[0]!.triggerType; + const runIndex = recoveringPartialReviewerCycle + ? previousTaskCycleRuns[0]!.runIndex + : existingRuns + 1; - const triggerType = requests[0]!.triggerType; - const sprintFeatureBranch = requests[0]!.sprintFeatureBranch; - const runIndex = existingRuns + 1; - - const runs = requests.map((request) => { - const run = this.deps.qaReviewRepository.createRun(request.runPayload); + const runs = effectiveRequests.map((request) => { + const resumeFromRun = this.findResumableQaReviewerRun(previousTaskCycleRuns, request.agentPresetId); + const run = this.deps.qaReviewRepository.createRun({ + ...request.runPayload, + runIndex, + payload: { + ...(request.runPayload.payload || {}), + runIndex, + reviewDispatchStatus: "pending", + }, + }); // Signal that the task has entered the QA stage so the live view advances // from coding-completed → QA and starts timing the review immediately // (the review itself can take minutes). Persisting the QA_PENDING indicator // makes the stage tag, boat race and stats reflect QA for the whole review, // not just the event-derived stage timeline. this.appendTaskEvent(taskRun, "qa_review_started", { - triggerType: request.triggerType, + triggerType: effectiveTriggerType, qaReviewRunId: run.id, runIndex, agentPresetId: request.agentPresetId, agentName: request.agentName, }); - return { request, run }; + return { request, run, resumeFromRun }; }); this.setTaskQaPending(args.task, true); @@ -318,11 +384,17 @@ export class QualityAssuranceService { caughtError?: unknown; }> = []; - for (const { request, run } of runs) { + for (const { request, run, resumeFromRun } of runs) { let resolvedReview: NormalizedQaReviewResult | undefined; let caughtError: unknown; try { + this.deps.qaReviewRepository.updateRun(run.id, { + payload: { + ...this.getLatestQaRunPayload(run), + reviewDispatchStatus: "running", + }, + }); resolvedReview = await this.runReview({ triggerType: request.triggerType, scope, @@ -335,6 +407,8 @@ export class QualityAssuranceService { taskRun, sprintRunId: args.sprintRunId || null, agentPresetId: request.agentPresetId, + qaRun: run, + resumeFromRun, reviewBranch, baseBranch: sprintFeatureBranch, }); @@ -356,7 +430,7 @@ export class QualityAssuranceService { outcome: "pass", summaryMarkdown: intentOutcome.summary, payload: { - ...run.payload, + ...this.getLatestQaRunPayload(run), ...resolvedReview!.raw, }, finishedAt: new Date().toISOString(), @@ -382,8 +456,10 @@ export class QualityAssuranceService { summaryMarkdown: intentOutcome.summary, fixInstructions: intentOutcome.fixInstructions, payload: { - ...run.payload, + ...this.getLatestQaRunPayload(run), ...resolvedReview!.raw, + continuationKey: `qa-followup:${run.id}`, + continuationStatus: intentOutcome.fixInstructions ? "pending" : "skipped", }, finishedAt: qaDecisionFinishedAt, }); @@ -407,7 +483,7 @@ export class QualityAssuranceService { status: "cancelled", summaryMarkdown: qaError.message, payload: { - ...run.payload, + ...this.getLatestQaRunPayload(run), error_code: qaError.code, }, finishedAt: new Date().toISOString(), @@ -438,7 +514,7 @@ export class QualityAssuranceService { status: "failed", summaryMarkdown: qaError.message, payload: { - ...run.payload, + ...this.getLatestQaRunPayload(run), error_code: qaError.code, }, finishedAt: new Date().toISOString(), @@ -467,91 +543,17 @@ export class QualityAssuranceService { const changesRequested = reviewResults.find((result) => result.intentOutcome.intent === "changes_requested"); if (changesRequested && changesRequested.intentOutcome.intent === "changes_requested") { const changesIntent = changesRequested.intentOutcome; - const qaDecisionFinishedAt = new Date().toISOString(); - let continued: QaFixContinuationResult; - try { - continued = changesIntent.fixInstructions - ? await this.requestFixesForTask({ - task: args.task, - taskRun, - repoPath: args.repoPath, - featureBranch: sprintFeatureBranch, - scope, - prompt: changesIntent.fixInstructions, - }) - : { applied: false, mode: "none" as const, noProgress: false, blocker: null }; - } catch (error) { - this.deps.qaReviewRepository.updateRun(changesRequested.run.id, { - payload: { - ...changesRequested.run.payload, - ...changesRequested.resolvedReview!.raw, - continued: false, - continuationMode: "failed", - continuationError: error instanceof Error ? error.message : String(error), - }, - finishedAt: qaDecisionFinishedAt, - }); - throw error; - } - - this.deps.qaReviewRepository.updateRun(changesRequested.run.id, { - payload: { - ...changesRequested.run.payload, - ...changesRequested.resolvedReview!.raw, - continued: continued.applied, - continuationMode: continued.mode, - followUpNoProgress: continued.noProgress, - followUpBlocker: continued.blocker, - postExhaustionVerificationEligible: continued.applied - && decisiveRuns + 1 === qaSettings.maxTaskReviewRuns, - }, - finishedAt: qaDecisionFinishedAt, - }); - - if (continued.noProgress) { - this.deps.projectManagementRepository.updateTask(taskId, { - status: "coding_completed", - mergeIndicator: null, - }); - args.task.status = "CODING_COMPLETED"; - } else if (continued.applied) { - this.deps.projectManagementRepository.updateTask(taskId, { - status: "in_progress", - ...MERGE_PROJECTION_RESET, - }); - args.task.status = "RUNNING"; - } else { - this.deps.projectManagementRepository.updateTask(taskId, { - status: "pending", - ...MERGE_PROJECTION_RESET, - }); - args.task.status = "PENDING"; - } - // Re-entering the coding stage: drop any stale CI / QA / MERGED indicator. - clearMergeProjectionForRerun(args.task); - - this.appendTaskEvent(taskRun, "qa_review_changes_requested", { - triggerType: changesRequested.request.triggerType, - summary: changesIntent.summary, + return await this.continuePendingTaskQaRun({ + run: this.deps.qaReviewRepository.getRun(changesRequested.run.id) || changesRequested.run, + task: args.task, + taskRun, + repoPath: args.repoPath, + featureBranch: sprintFeatureBranch, + scope, + decisiveRuns: decisiveRuns + 1, + maxTaskReviewRuns: qaSettings.maxTaskReviewRuns, findings: changesRequested.resolvedReview!.findings, - fixInstructions: changesIntent.fixInstructions, - qaReviewRunId: changesRequested.run.id, - continued: continued.applied, - continuationMode: continued.mode, - followUpNoProgress: continued.noProgress, - followUpBlocker: continued.blocker, - postExhaustionVerificationEligible: continued.applied - && decisiveRuns + 1 === qaSettings.maxTaskReviewRuns, - agentPresetId: changesRequested.request.agentPresetId, - agentName: changesRequested.request.agentName, }); - - return { - reviewed: true, - reopenedTask: true, - mergeBlocked: true, - reportText: renderQaChangesRequestedReport(args.task.id, changesIntent.summary, continued.applied), - }; } const failedReview = reviewResults.find((result) => result.intentOutcome.intent !== "pass"); @@ -652,6 +654,36 @@ export class QualityAssuranceService { const latestRuns = historicalLatestRuns.filter((run) => run.sprintRunId === args.sprintRunId); const latestRun = latestRuns[0] ?? null; const maxRuns = qaSettings.maxSprintReviewRuns; + const pendingSprintContinuation = latestRuns.find((run) => isPendingQaContinuation(run)) ?? null; + + if (pendingSprintContinuation) { + return await this.continuePendingSprintQaRun({ + run: pendingSprintContinuation, + repoPath: args.repoPath, + subtasks: args.subtasks, + featureBranch: sprintFeatureBranch, + scope, + maxRuns, + }); + } + + const sprintPresetIds = Array.isArray(qaSettings.sprintCompletion.agentPresetIds) + && qaSettings.sprintCompletion.agentPresetIds.length > 0 + ? qaSettings.sprintCompletion.agentPresetIds + : [null]; + const latestCycleHasChangesRequest = latestRuns.some((run) => ( + run.status === "completed" && run.outcome === "changes_requested" + )); + const latestCycleHasCompletedReviewer = latestRuns.some((run) => run.status === "completed"); + const potentiallyRecoveringPartialReviewerCycle = latestRuns.length > 0 + && !latestCycleHasChangesRequest + && ( + latestRuns.length < sprintPresetIds.length + || latestRuns.some((run) => run.status === "running" || run.status === "cancelled") + || (latestCycleHasCompletedReviewer && latestRuns.some((run) => run.status === "failed" || run.status === "errored")) + || sprintPresetIds.some((presetId) => presetId !== null && !latestRuns.some((run) => run.agentPresetId === presetId)) + ); + const currentTaskSnapshot = buildSprintQaSnapshot(args.subtasks); const latestTaskUpdatedAt = this.getLatestSprintTaskUpdatedAt(args.projectId, args.sprintId); const shouldRunReview = shouldRunSprintQaReview({ @@ -668,11 +700,47 @@ export class QualityAssuranceService { shouldRunReview, }); - if (sprintQaDecision.action === "skip_review") { + if (!potentiallyRecoveringPartialReviewerCycle && sprintQaDecision.action === "skip_review") { return { reviewed: false, blockedCompletion: false, mergeBlocked: false, reportText: "" }; } - if (sprintQaDecision.action === "block_completion") { + if (!potentiallyRecoveringPartialReviewerCycle && sprintQaDecision.action === "block_completion") { + this.openSprintQaHumanHandoffIfTerminal({ + projectId: args.projectId, + sprintId: args.sprintId, + sprintRunId: args.sprintRunId, + latestRuns, + maxRuns, + }); + return { + reviewed: false, + blockedCompletion: true, + mergeBlocked: true, + reportText: latestRun ? renderSprintQaPendingReport(latestRun) : "", + }; + } + + const sprintAgents = await Promise.all(sprintPresetIds.map((configuredAgentPresetId) => ( + this.deps.agentPresetSyncService.resolveTargetedQualityAssuranceAgent( + args.projectId, + configuredAgentPresetId, + ) + ))); + const recoveringPartialReviewerCycle = latestRuns.length > 0 + && !latestCycleHasChangesRequest + && sprintAgents.some((agent) => { + const agentRuns = latestRuns.filter((run) => run.agentPresetId === agent.id); + if (agentRuns.some((run) => run.status === "completed")) { + return false; + } + return agentRuns.length === 0 + || agentRuns.some((run) => run.status === "running" || run.status === "cancelled") + || (latestCycleHasCompletedReviewer && agentRuns.some((run) => run.status === "failed" || run.status === "errored")); + }); + if (!recoveringPartialReviewerCycle && sprintQaDecision.action === "skip_review") { + return { reviewed: false, blockedCompletion: false, mergeBlocked: false, reportText: "" }; + } + if (!recoveringPartialReviewerCycle && sprintQaDecision.action === "block_completion") { this.openSprintQaHumanHandoffIfTerminal({ projectId: args.projectId, sprintId: args.sprintId, @@ -688,27 +756,39 @@ export class QualityAssuranceService { }; } - const sprintPresetIds = Array.isArray(qaSettings.sprintCompletion.agentPresetIds) - && qaSettings.sprintCompletion.agentPresetIds.length > 0 - ? qaSettings.sprintCompletion.agentPresetIds - : [null]; const latestHistoricalRunIndex = historicalLatestRuns.reduce((maxRunIndex, run) => { return Math.max(maxRunIndex, typeof run.runIndex === "number" ? run.runIndex : 0); }, 0); - const runIndex = Math.max(latestRun?.runIndex || 0, latestHistoricalRunIndex) + 1; + const runIndex = recoveringPartialReviewerCycle + ? latestRuns[0]!.runIndex + : Math.max(latestRun?.runIndex || 0, latestHistoricalRunIndex) + 1; const sprintReviewResults: Array<{ agentPresetId: string; agentName: string; run: QaReviewRunRecord; review?: NormalizedQaReviewResult; error?: unknown; - }> = []; - - for (const configuredAgentPresetId of sprintPresetIds) { - const agent = await this.deps.agentPresetSyncService.resolveTargetedQualityAssuranceAgent( - args.projectId, - configuredAgentPresetId, - ); + }> = recoveringPartialReviewerCycle + ? latestRuns.flatMap((run) => { + if (run.status !== "completed" || run.outcome !== "pass") { + return []; + } + return [{ + agentPresetId: run.agentPresetId || "", + agentName: run.agentName || "QA", + run, + review: this.restoreSprintQaReview(run), + }]; + }) + : []; + + const agentsToRun = recoveringPartialReviewerCycle + ? sprintAgents.filter((agent) => { + return !latestRuns.some((run) => run.agentPresetId === agent.id && run.status === "completed"); + }) + : sprintAgents; + const preparedSprintRuns = agentsToRun.map((agent) => { + const resumeFromRun = this.findResumableQaReviewerRun(latestRuns, agent.id); const run = this.deps.qaReviewRepository.createRun({ projectId: args.projectId, sprintId: args.sprintId, @@ -722,10 +802,20 @@ export class QualityAssuranceService { taskSnapshot: currentTaskSnapshot, agentPresetId: agent.id, agentName: agent.name, + reviewDispatchStatus: "pending", }, }); + return { agent, run, resumeFromRun }; + }); + for (const { agent, run, resumeFromRun } of preparedSprintRuns) { try { + this.deps.qaReviewRepository.updateRun(run.id, { + payload: { + ...this.getLatestQaRunPayload(run), + reviewDispatchStatus: "running", + }, + }); const memoryInstructions = resolveAgentMemoryInstructions( agent, settings.memory?.workerLearningsInstruction @@ -744,6 +834,8 @@ export class QualityAssuranceService { taskRun: null, sprintRunId: args.sprintRunId, agentPresetId: agent.id, + qaRun: run, + resumeFromRun, // Sprint QA reviews the integrated base branch (where all task work is // merged), falling back to the configured default branch. reviewBranch: sprintFeatureBranch, @@ -756,7 +848,7 @@ export class QualityAssuranceService { outcome: "pass", summaryMarkdown: review.summary, payload: { - ...run.payload, + ...this.getLatestQaRunPayload(run), ...review.raw, taskSnapshot: currentTaskSnapshot, }, @@ -773,7 +865,7 @@ export class QualityAssuranceService { summaryMarkdown: review.summary, fixInstructions: review.fixInstructions, payload: { - ...run.payload, + ...this.getLatestQaRunPayload(run), ...review.raw, taskSnapshot: currentTaskSnapshot, }, @@ -786,7 +878,7 @@ export class QualityAssuranceService { status: qaError.code === "CANCELLED" || isQaReviewCancellationError(error) ? "cancelled" : "failed", summaryMarkdown: qaError.message, payload: { - ...run.payload, + ...this.getLatestQaRunPayload(run), error_code: qaError.code, }, finishedAt: new Date().toISOString(), @@ -822,37 +914,18 @@ export class QualityAssuranceService { // Creating more automatic work at the cap leaves no budget to verify it // and previously trapped the sprint in an invisible heartbeat loop. const canApplyAutomaticFollowUp = runIndex < maxRuns; - const continued = canApplyAutomaticFollowUp && targetTask && fixInstructions && canContinueTargetTask - ? await this.requestFixesForTask({ - task: targetTask, - taskRun: targetTaskRun, - repoPath: args.repoPath, - featureBranch: sprintFeatureBranch, - scope, - prompt: fixInstructions, - }) - : { applied: false, mode: "none" as const }; - const createdFollowUpTasks = canApplyAutomaticFollowUp - ? this.createSprintFollowUpTasks({ - projectId: args.projectId, - sprintId: args.sprintId, - targetTask, - fixInstructions, - review, - existingSubtasks: args.subtasks, - sourceRunId: changesRequested.run.id, - }) - : []; - this.deps.qaReviewRepository.updateRun(changesRequested.run.id, { targetTaskKey: targetTask?.id || review.targetTaskKey, targetSessionId: targetTask?.session_id || null, targetProvider: targetTask?.provider || null, payload: { - ...changesRequested.run.payload, + ...this.getLatestQaRunPayload(changesRequested.run), ...review.raw, - continued: continued.applied, - continuationMode: continued.mode, + continuationKey: `sprint-qa-followup:${changesRequested.run.id}`, + continuationStatus: canApplyAutomaticFollowUp && targetTask && fixInstructions && canContinueTargetTask + ? "pending" + : "skipped", + continuationTaskRunId: targetTaskRun?.id || null, continuationSkippedReason: !canApplyAutomaticFollowUp ? "sprint_qa_retry_budget_exhausted" : targetTask && fixInstructions && !canContinueTargetTask @@ -861,22 +934,43 @@ export class QualityAssuranceService { automaticFollowUpSuppressedReason: canApplyAutomaticFollowUp ? undefined : "sprint_qa_retry_budget_exhausted", - createdFollowUpTaskKeys: createdFollowUpTasks.map((task) => task.taskKey), taskSnapshot: currentTaskSnapshot, }, finishedAt: new Date().toISOString(), }); - if (continued.applied && targetTask?.record_id) { - this.deps.projectManagementRepository.updateTask(targetTask.record_id, { - status: "in_progress", - ...MERGE_PROJECTION_RESET, + if (canApplyAutomaticFollowUp && targetTask && fixInstructions && canContinueTargetTask) { + return await this.continuePendingSprintQaRun({ + run: this.deps.qaReviewRepository.getRun(changesRequested.run.id) || changesRequested.run, + repoPath: args.repoPath, + subtasks: args.subtasks, + featureBranch: sprintFeatureBranch, + scope, + maxRuns, + review, }); - targetTask.status = "RUNNING"; - // Re-entering the coding stage: drop any stale CI / QA / MERGED indicator. - clearMergeProjectionForRerun(targetTask); } + const createdFollowUpTasks = canApplyAutomaticFollowUp + ? this.createSprintFollowUpTasks({ + projectId: args.projectId, + sprintId: args.sprintId, + targetTask, + fixInstructions, + review, + existingSubtasks: args.subtasks, + sourceRunId: changesRequested.run.id, + }) + : []; + this.deps.qaReviewRepository.updateRun(changesRequested.run.id, { + payload: { + ...this.getLatestQaRunPayload(changesRequested.run), + continued: false, + continuationMode: "none", + createdFollowUpTaskKeys: createdFollowUpTasks.map((task) => task.taskKey), + }, + }); + if (!canApplyAutomaticFollowUp) { const terminalLatestRuns = this.deps.qaReviewRepository .listLatestSprintCycleRuns(args.sprintId) @@ -897,7 +991,7 @@ export class QualityAssuranceService { reportText: renderSprintQaChangesRequestedReport( review.summary, targetTask?.id || review.targetTaskKey, - continued.applied, + false, createdFollowUpTasks.map((task) => task.taskKey), ) + (!canApplyAutomaticFollowUp ? renderSprintQaBudgetExhaustedReport(maxRuns) : ""), }; @@ -1046,6 +1140,26 @@ export class QualityAssuranceService { }); } + private findResumableQaReviewerRun( + runs: QaReviewRunRecord[], + agentPresetId: string | null, + ): QaReviewRunRecord | null { + return runs.find((run) => ( + run.agentPresetId === agentPresetId + && (run.status === "cancelled" || run.status === "failed") + && typeof run.payload?.reviewLogicalSessionId === "string" + && run.payload.reviewLogicalSessionId.trim().length > 0 + )) || null; + } + + private getLatestQaRunPayload(run: QaReviewRunRecord): Record { + const repository = this.deps.qaReviewRepository as Partial; + const persistedRun = typeof repository.getRun === "function" + ? repository.getRun(run.id) + : null; + return persistedRun?.payload || run.payload || {}; + } + private async runReview(args: { triggerType: QaReviewTriggerType; scope: DashboardSettingsScope; @@ -1058,6 +1172,8 @@ export class QualityAssuranceService { taskRun: TaskRunRecord | null; sprintRunId: string | null; agentPresetId: string | null; + qaRun?: QaReviewRunRecord; + resumeFromRun?: QaReviewRunRecord | null; reviewBranch: string | undefined; baseBranch: string; }): Promise { @@ -1077,13 +1193,35 @@ export class QualityAssuranceService { is_independent: true, status: "COMPLETED", }; + const settings = this.deps.getDashboardSettings(args.scope); + const requestedResume = settings.restartInvocationPolicy === "continue" && Boolean(args.resumeFromRun); + const candidateResumePayload = requestedResume ? args.resumeFromRun?.payload : null; const route = this.deps.taskService.resolveInvocationProvider("qa_review", pseudoTask, { scope: args.scope, cliOnly: true, }); - const provider = route.provider as CliQaProvider; - const providerConfigId = route.providerConfigId || route.provider; + const savedProviderConfigId = typeof candidateResumePayload?.reviewProviderConfigId === "string" + ? candidateResumePayload.reviewProviderConfigId + : null; + const savedProvider = typeof candidateResumePayload?.reviewProvider === "string" + ? candidateResumePayload.reviewProvider as CliQaProvider + : null; + const hasSavedRoute = Boolean( + requestedResume + && savedProviderConfigId + && savedProvider + && route.providers[savedProviderConfigId], + ); + const provider = hasSavedRoute ? savedProvider! : route.provider as CliQaProvider; + const providerConfigId = hasSavedRoute + ? savedProviderConfigId! + : route.providerConfigId || route.provider; const providerSettings = route.providers[providerConfigId]; + const canResume = requestedResume && hasSavedRoute; + const resumePayload = canResume ? candidateResumePayload : null; + const savedModel = typeof resumePayload?.reviewModel === "string" && resumePayload.reviewModel.trim() + ? resumePayload.reviewModel + : providerSettings.model; const memoryContext = args.agentPresetId ? await this.buildMemoryContext(args.scope.projectId!, args.scope.sprintId || null, args.agentPresetId, args.sprintGoal) @@ -1093,7 +1231,6 @@ export class QualityAssuranceService { memoryContext, }); const providerPrompt = buildProviderPrompt(prompt, providerSettings.thinkingMode, provider); - const settings = this.deps.getDashboardSettings(args.scope); const workflowSettings = { ...DEFAULT_CLI_WORKFLOW_SETTINGS, ...settings.cliWorkflow, @@ -1104,7 +1241,83 @@ export class QualityAssuranceService { githubToken: settings.git.githubToken, gitlabToken: settings.git.gitlabToken, }); - const snapshotSessionId = `qa-review-${provider}-${randomUUID()}`; + const savedLogicalSessionId = typeof resumePayload?.reviewLogicalSessionId === "string" + ? resumePayload.reviewLogicalSessionId.trim() + : ""; + const logicalSessionId = savedLogicalSessionId + || `${args.qaRun ? "cli-" : ""}qa-review-${provider}-${args.qaRun?.id || Date.now().toString(36)}`; + const previousProviderInvocation = canResume + && typeof this.deps.executionRepository.getLatestProviderInvocationUsageBySession === "function" + ? this.deps.executionRepository.getLatestProviderInvocationUsageBySession(logicalSessionId, "qa_review") + : null; + const savedNativeSessionId = typeof resumePayload?.reviewNativeSessionId === "string" + ? resumePayload.reviewNativeSessionId.trim() + : ""; + const continueSessionId = canResume + ? previousProviderInvocation?.nativeSessionId + || savedNativeSessionId + || (provider === "claude-code" ? null : logicalSessionId) + : null; + const openCodeBaselineRawUsageJson = provider === "opencode" + ? previousProviderInvocation?.rawUsageJson + || (resumePayload?.reviewOpenCodeBaselineRawUsageJson as Record | null | undefined) + || null + : null; + const snapshotSessionId = typeof resumePayload?.reviewSnapshotSessionId === "string" + && resumePayload.reviewSnapshotSessionId.trim() + ? resumePayload.reviewSnapshotSessionId + : `${logicalSessionId}-workspace`; + const workspaceSessionId = typeof resumePayload?.reviewWorkspaceSessionId === "string" + && resumePayload.reviewWorkspaceSessionId.trim() + ? resumePayload.reviewWorkspaceSessionId + : snapshotSessionId; + const existingReviewInvocationId = typeof args.qaRun?.payload?.reviewExecutionInvocationId === "string" + ? args.qaRun.payload.reviewExecutionInvocationId + : null; + const reviewExecutionInvocationId = existingReviewInvocationId + || (args.qaRun && typeof this.deps.executionRepository.createExecutionInvocation === "function" + ? this.deps.executionRepository.createExecutionInvocation({ + projectId: args.scope.projectId!, + sprintId: args.scope.sprintId || null, + taskId: args.taskRun?.taskId || null, + sprintRunId: args.sprintRunId, + taskRunId: args.taskRun?.id || null, + type: "qa_review", + provider, + model: savedModel, + startedAt: new Date().toISOString(), + }).id + : undefined); + if (args.qaRun) { + this.deps.qaReviewRepository.updateRun(args.qaRun.id, { + payload: { + ...this.getLatestQaRunPayload(args.qaRun), + reviewLogicalSessionId: logicalSessionId, + reviewSnapshotSessionId: snapshotSessionId, + reviewWorkspaceSessionId: workspaceSessionId, + reviewProvider: provider, + reviewProviderConfigId: providerConfigId, + reviewModel: savedModel, + reviewExecutionInvocationId, + reviewContinuationSourceRunId: canResume ? args.resumeFromRun?.id : undefined, + }, + }); + } + if (typeof this.deps.sessionTracking.createSession === "function") { + for (const trackedSessionId of new Set([logicalSessionId, workspaceSessionId])) { + this.deps.sessionTracking.createSession({ + id: trackedSessionId, + provider, + taskId: args.taskRun?.taskId || undefined, + title: args.currentTask ? `QA review: ${args.currentTask.title}` : "Sprint QA review", + prompt, + state: "RUNNING", + featureBranch: args.baseBranch, + workerBranch: args.reviewBranch, + repoPath: args.repoPath, + }); + } + } let snapshotWorkspace = args.repoPath; let shouldCleanupSnapshot = false; if (workflowSettings.executionMode === "DOCKER") { @@ -1120,24 +1333,39 @@ export class QualityAssuranceService { sessionId: snapshotSessionId, checkout: invocationWorkspace.snapshotCheckout, gitPolicy: invocationWorkspace.gitPolicy, + reuseExisting: canResume, }); shouldCleanupSnapshot = true; } else if (args.reviewBranch) { // QA must inspect the requested worker/feature branch in HOST mode too. // The visible repository normally remains on the default branch, which // otherwise turns every QA check into a false missing-file rejection. - snapshotWorkspace = await this.invocationWorkspacePreparer.createHostSnapshotWorkspace({ - repoPath: args.repoPath, - sessionId: snapshotSessionId, - checkout: buildInvocationSnapshotCheckout(gitPolicy, { - branch: args.reviewBranch, - fallbackBranch: args.baseBranch, - useDefaultBranch: false, - }), - gitPolicy, - }); + const savedSnapshotWorkspace = typeof resumePayload?.reviewSnapshotWorkspace === "string" + ? resumePayload.reviewSnapshotWorkspace + : ""; + snapshotWorkspace = canResume && savedSnapshotWorkspace + && await this.workspaceManager.workspaceExists(savedSnapshotWorkspace) + ? savedSnapshotWorkspace + : await this.invocationWorkspacePreparer.createHostSnapshotWorkspace({ + repoPath: args.repoPath, + sessionId: snapshotSessionId, + checkout: buildInvocationSnapshotCheckout(gitPolicy, { + branch: args.reviewBranch, + fallbackBranch: args.baseBranch, + useDefaultBranch: false, + }), + gitPolicy, + }); shouldCleanupSnapshot = true; } + if (args.qaRun) { + this.deps.qaReviewRepository.updateRun(args.qaRun.id, { + payload: { + ...this.getLatestQaRunPayload(args.qaRun), + reviewSnapshotWorkspace: snapshotWorkspace, + }, + }); + } let result; try { @@ -1150,11 +1378,11 @@ export class QualityAssuranceService { purpose: "qa_review", type: "qa_review", provider, - ...buildProviderSettingsOverride(providerSettings.model, providerSettings), + ...buildProviderSettingsOverride(savedModel, providerSettings), providerPrompt, repoPath: args.repoPath, cwd: snapshotWorkspace, - workspaceSessionId: `${args.scope.projectId || "project"}-qa-snapshot`, + workspaceSessionId, settings: { ...settings, cliWorkflow: workflowSettings, @@ -1168,6 +1396,10 @@ export class QualityAssuranceService { ].join("\n"), providerLabel: "QA", sessionIdPrefix: "qa-review", + logicalSessionId, + continueSessionId, + openCodeBaselineRawUsageJson, + invocationId: reviewExecutionInvocationId, systemRoutingMessage: args.agentInstructions.trim(), agentMcpAccess: args.agentPresetId ? this.deps.agentPresetRepository?.getAgentPreset(args.agentPresetId)?.mcpAccess ?? null @@ -1177,7 +1409,19 @@ export class QualityAssuranceService { this.touchSprintRunHeartbeat(args.sprintRunId, args.scope.sprintId); }, }); + if (args.qaRun) { + this.deps.qaReviewRepository.updateRun(args.qaRun.id, { + payload: { + ...this.getLatestQaRunPayload(args.qaRun), + reviewNativeSessionId: result.nativeSessionId, + reviewOpenCodeBaselineRawUsageJson: result.openCodeBaselineRawUsageJson || openCodeBaselineRawUsageJson, + }, + }); + } } catch (error) { + for (const trackedSessionId of new Set([logicalSessionId, workspaceSessionId])) { + this.deps.sessionTracking.updateSession?.(trackedSessionId, { state: "FAILED" }); + } throw parseQaError(error); } finally { if (settings.memory?.enabled && settings.memory.autoCaptureSprint && this.deps.memoryService && result) { @@ -1192,9 +1436,14 @@ export class QualityAssuranceService { ); } } - if (shouldCleanupSnapshot) { + if (shouldCleanupSnapshot && result) { await this.workspaceManager.removeWorktree(args.repoPath, snapshotWorkspace).catch(() => undefined); } + if (result) { + for (const trackedSessionId of new Set([logicalSessionId, workspaceSessionId])) { + this.deps.sessionTracking.updateSession?.(trackedSessionId, { state: "COMPLETED" }); + } + } } return result.parsed; @@ -1249,12 +1498,27 @@ export class QualityAssuranceService { return this.deps.qaReviewRepository.updateRun(run.id, { status: "cancelled", summaryMarkdown: recoveryDecision.summaryMarkdown, + payload: { + ...this.getLatestQaRunPayload(run), + reviewNativeSessionId: providerInvocation?.nativeSessionId || run.payload?.reviewNativeSessionId, + reviewOpenCodeBaselineRawUsageJson: providerInvocation?.rawUsageJson + || run.payload?.reviewOpenCodeBaselineRawUsageJson, + }, finishedAt: recoveryDecision.finishedAt, }); } private findLatestQaExecutionInvocation(run: QaReviewRunRecord): ExecutionInvocationRecord | null { const executionRepository = this.deps.executionRepository as Partial; + const correlatedInvocationId = typeof run.payload?.reviewExecutionInvocationId === "string" + ? run.payload.reviewExecutionInvocationId + : null; + if (correlatedInvocationId && typeof executionRepository.getExecutionInvocation === "function") { + const correlatedInvocation = executionRepository.getExecutionInvocation(correlatedInvocationId); + if (correlatedInvocation?.type === "qa_review") { + return correlatedInvocation; + } + } if (typeof executionRepository.listExecutionInvocations !== "function") { return null; } @@ -1528,6 +1792,526 @@ export class QualityAssuranceService { } } + private async continuePendingSprintQaRun(args: { + run: QaReviewRunRecord; + repoPath: string; + subtasks: Subtask[]; + featureBranch: string; + scope: DashboardSettingsScope; + maxRuns: number; + review?: NormalizedQaReviewResult; + }): Promise { + const targetTask = args.run.targetTaskKey + ? args.subtasks.find((task) => task.id === args.run.targetTaskKey) ?? null + : null; + const review = args.review || this.restoreSprintQaReview(args.run); + if (!targetTask || !args.run.fixInstructions?.trim() || this.isMergedSubtask(targetTask)) { + this.deps.qaReviewRepository.updateRun(args.run.id, { + payload: { + ...this.getLatestQaRunPayload(args.run), + continuationStatus: "skipped", + continuationSkippedReason: !targetTask + ? "target_task_missing" + : this.isMergedSubtask(targetTask) + ? "target_task_already_merged" + : "fix_instructions_missing", + }, + }); + return { + reviewed: true, + blockedCompletion: true, + mergeBlocked: true, + reportText: renderSprintQaChangesRequestedReport( + args.run.summaryMarkdown || review.summary, + args.run.targetTaskKey, + false, + [], + ), + }; + } + + if (this.activeQaContinuationRunIds.has(args.run.id)) { + return { + reviewed: false, + blockedCompletion: true, + mergeBlocked: true, + reportText: renderSprintQaChangesRequestedReport( + args.run.summaryMarkdown || review.summary, + targetTask.id, + false, + [], + ), + }; + } + + if (targetTask.record_id) { + clearMergeProjectionForRerun(targetTask); + this.deps.projectManagementRepository.updateTask(targetTask.record_id, { + status: "coding_completed", + isMerged: false, + mergeIndicator: "QA_PENDING", + }); + } + targetTask.status = "CODING_COMPLETED"; + targetTask.is_merged = false; + targetTask.merge_indicator = "QA_PENDING"; + targetTask.intervention_owner = undefined; + targetTask.intervention_hint = undefined; + + const payload = this.getLatestQaRunPayload(args.run); + const storedAttemptCount = payload.continuationAttemptCount; + const previousAttemptCount = typeof storedAttemptCount === "number" && Number.isFinite(storedAttemptCount) + ? Math.max(0, Math.trunc(storedAttemptCount)) + : 0; + const continuationAttemptCount = payload.continuationStatus === "running" && previousAttemptCount > 0 + ? previousAttemptCount + : previousAttemptCount + 1; + this.deps.qaReviewRepository.updateRun(args.run.id, { + payload: { + ...payload, + continuationKey: typeof payload.continuationKey === "string" + ? payload.continuationKey + : `sprint-qa-followup:${args.run.id}`, + continuationStatus: "running", + continuationAttemptCount, + continuationStartedAt: new Date().toISOString(), + continuationError: undefined, + }, + }); + + const taskRunId = typeof payload.continuationTaskRunId === "string" + ? payload.continuationTaskRunId + : null; + const executionRepository = this.deps.executionRepository as Partial; + const targetTaskRun = taskRunId && typeof executionRepository.getTaskRun === "function" + ? executionRepository.getTaskRun(taskRunId) + : this.resolveTaskRunForSubtask(targetTask, args.run.sprintRunId || undefined); + + this.activeQaContinuationRunIds.add(args.run.id); + let continued: QaFixContinuationResult; + try { + continued = await this.requestFixesForTask({ + task: targetTask, + taskRun: targetTaskRun, + repoPath: args.repoPath, + featureBranch: args.featureBranch, + scope: args.scope, + prompt: args.run.fixInstructions, + qaContinuationRunId: args.run.id, + }); + } catch (error) { + const failedAt = new Date().toISOString(); + const retryBudgetExhausted = continuationAttemptCount >= QA_INFRA_FAILURE_GRACE; + this.deps.qaReviewRepository.updateRun(args.run.id, { + payload: { + ...this.getLatestQaRunPayload(args.run), + continued: false, + continuationMode: "failed", + continuationStatus: retryBudgetExhausted ? "failed" : "pending", + continuationAttemptCount, + continuationError: error instanceof Error ? error.message : String(error), + continuationFailedAt: failedAt, + continuationSettledAt: retryBudgetExhausted ? failedAt : undefined, + followUpNoProgress: retryBudgetExhausted, + }, + }); + if (targetTask.record_id) { + clearMergeProjectionForRerun(targetTask); + this.deps.projectManagementRepository.updateTask(targetTask.record_id, { + status: "coding_completed", + isMerged: false, + mergeIndicator: "QA_PENDING", + }); + } + targetTask.status = "CODING_COMPLETED"; + targetTask.is_merged = false; + targetTask.merge_indicator = "QA_PENDING"; + targetTask.intervention_owner = undefined; + targetTask.intervention_hint = undefined; + throw error; + } finally { + this.activeQaContinuationRunIds.delete(args.run.id); + } + + const createdFollowUpTasks = this.createSprintFollowUpTasks({ + projectId: args.run.projectId, + sprintId: args.run.sprintId, + targetTask, + fixInstructions: args.run.fixInstructions, + review, + existingSubtasks: args.subtasks, + sourceRunId: args.run.id, + }); + this.deps.qaReviewRepository.updateRun(args.run.id, { + payload: { + ...this.getLatestQaRunPayload(args.run), + continued: continued.applied, + continuationMode: continued.mode, + continuationStatus: continued.noProgress + ? "no_progress" + : continued.applied + ? "completed" + : "skipped", + continuationSettledAt: new Date().toISOString(), + continuationError: undefined, + followUpNoProgress: continued.noProgress, + followUpBlocker: continued.blocker, + createdFollowUpTaskKeys: createdFollowUpTasks.map((task) => task.taskKey), + }, + }); + + if (continued.applied && targetTask.record_id) { + clearMergeProjectionForRerun(targetTask); + this.deps.projectManagementRepository.updateTask(targetTask.record_id, { + status: "coding_completed", + isMerged: false, + mergeIndicator: "QA_PENDING", + }); + targetTask.status = "CODING_COMPLETED"; + targetTask.is_merged = false; + targetTask.merge_indicator = "QA_PENDING"; + } + + return { + reviewed: true, + blockedCompletion: true, + mergeBlocked: true, + reportText: renderSprintQaChangesRequestedReport( + args.run.summaryMarkdown || review.summary, + targetTask.id, + continued.applied, + createdFollowUpTasks.map((task) => task.taskKey), + ) + (args.run.runIndex >= args.maxRuns ? renderSprintQaBudgetExhaustedReport(args.maxRuns) : ""), + }; + } + + private restoreSprintQaReview(run: QaReviewRunRecord): NormalizedQaReviewResult { + try { + return normalizeQaReviewResult(JSON.stringify(run.payload || {})); + } catch { + return { + verdict: "changes_requested", + summary: run.summaryMarkdown || "Sprint QA requested follow-up work.", + findings: [], + fixInstructions: run.fixInstructions, + targetTaskKey: run.targetTaskKey, + shouldHavePr: null, + followUpTasks: [], + raw: run.payload || {}, + }; + } + } + + private isPendingTaskQaContinuation(run: QaReviewRunRecord): boolean { + return isPendingQaContinuation(run); + } + + private async continuePendingTaskQaRun(args: { + run: QaReviewRunRecord; + task: Subtask; + taskRun: TaskRunRecord | null; + repoPath: string; + featureBranch: string; + scope: DashboardSettingsScope; + decisiveRuns: number; + maxTaskReviewRuns: number; + findings?: string[]; + }): Promise { + if (this.activeQaContinuationRunIds.has(args.run.id)) { + return { + reviewed: false, + reopenedTask: false, + mergeBlocked: true, + reportText: renderQaChangesRequestedReport( + args.task.id, + args.run.summaryMarkdown || "QA requested follow-up changes.", + false, + ), + }; + } + + if (this.isLaterTaskRunStillActive(args.run, args.taskRun)) { + return { + reviewed: false, + reopenedTask: false, + mergeBlocked: true, + reportText: renderQaChangesRequestedReport( + args.task.id, + args.run.summaryMarkdown || "QA requested follow-up changes.", + false, + ), + }; + } + + if (this.hasCompletedSameSessionQaFollowUp( + args.run, + args.taskRun, + args.run.targetSessionId?.trim() || args.task.session_id, + )) { + const reconciledAt = new Date().toISOString(); + this.deps.qaReviewRepository.updateRun(args.run.id, { + payload: { + ...(this.deps.qaReviewRepository.getRun(args.run.id)?.payload || args.run.payload || {}), + continuationKey: typeof args.run.payload?.continuationKey === "string" + ? args.run.payload.continuationKey + : `qa-followup:${args.run.id}`, + continuationStatus: "completed", + continuationMode: "cli", + continuationSettledAt: reconciledAt, + continuationReconciled: true, + continued: true, + followUpNoProgress: false, + followUpBlocker: null, + postExhaustionVerificationEligible: args.decisiveRuns === args.maxTaskReviewRuns, + }, + }); + if (args.task.record_id) { + clearMergeProjectionForRerun(args.task); + this.deps.projectManagementRepository.updateTask(args.task.record_id, { + status: "coding_completed", + isMerged: false, + mergeIndicator: "QA_PENDING", + }); + } + args.task.status = "CODING_COMPLETED"; + args.task.merge_indicator = "QA_PENDING"; + return { + reviewed: true, + reopenedTask: true, + mergeBlocked: true, + reportText: renderQaChangesRequestedReport( + args.task.id, + args.run.summaryMarkdown || "QA requested follow-up changes.", + true, + ), + }; + } + + // A legacy failure may already have projected this task back to pending or + // running ordinary coding. Re-establish the durable QA stage before the + // provider side effect so a concurrent snapshot/restart cannot redispatch it + // as unrelated task work while this handoff is in flight. + if (args.task.record_id) { + this.deps.projectManagementRepository.updateTask(args.task.record_id, { + status: "coding_completed", + isMerged: false, + mergeIndicator: "QA_PENDING", + }); + } + args.task.status = "CODING_COMPLETED"; + args.task.is_merged = false; + args.task.merge_indicator = "QA_PENDING"; + args.task.intervention_owner = undefined; + args.task.intervention_hint = undefined; + + this.activeQaContinuationRunIds.add(args.run.id); + const continuationKey = typeof args.run.payload?.continuationKey === "string" + ? args.run.payload.continuationKey + : `qa-followup:${args.run.id}`; + const latestPayload = (): Record => ( + this.deps.qaReviewRepository.getRun(args.run.id)?.payload || args.run.payload || {} + ); + const checkpointPayload = latestPayload(); + const storedAttemptCount = checkpointPayload.continuationAttemptCount; + const previousAttemptCount = typeof storedAttemptCount === "number" && Number.isFinite(storedAttemptCount) + ? Math.max(0, Math.trunc(storedAttemptCount)) + : checkpointPayload.continuationStatus === "failed" + ? 1 + : 0; + // A process restart leaves the durable row at `running`. Resuming that same + // logical/native provider turn must not consume another failure allowance; + // only a fresh dispatch from pending/failed starts a new bounded attempt. + const continuationAttemptCount = checkpointPayload.continuationStatus === "running" + && previousAttemptCount > 0 + ? previousAttemptCount + : previousAttemptCount + 1; + + this.deps.qaReviewRepository.updateRun(args.run.id, { + payload: { + ...checkpointPayload, + continuationKey, + continuationStatus: "running", + continuationAttemptCount, + continuationStartedAt: new Date().toISOString(), + continuationError: undefined, + followUpNoProgress: false, + followUpBlocker: null, + }, + }); + + let continued: QaFixContinuationResult; + try { + const executionRepository = this.deps.executionRepository as Partial; + const continuationTaskRun = args.run.taskRunId && typeof executionRepository.getTaskRun === "function" + ? executionRepository.getTaskRun(args.run.taskRunId) || args.taskRun + : args.taskRun; + const continuationSessionId = args.run.targetSessionId?.trim() || args.task.session_id; + const continuationTask = continuationSessionId && continuationSessionId !== args.task.session_id + ? { ...args.task, session_id: continuationSessionId } + : args.task; + continued = args.run.fixInstructions?.trim() + ? await this.requestFixesForTask({ + task: continuationTask, + taskRun: continuationTaskRun, + repoPath: args.repoPath, + featureBranch: args.featureBranch, + scope: args.scope, + prompt: args.run.fixInstructions, + qaContinuationRunId: args.run.id, + }) + : { applied: false, mode: "none", noProgress: false, blocker: null }; + } catch (error) { + const continuationError = error instanceof Error ? error.message : String(error); + const retryBudgetExhausted = continuationAttemptCount >= QA_INFRA_FAILURE_GRACE; + const failedAt = new Date().toISOString(); + this.deps.qaReviewRepository.updateRun(args.run.id, { + payload: { + ...latestPayload(), + continued: false, + continuationMode: "failed", + continuationStatus: retryBudgetExhausted ? "failed" : "pending", + continuationAttemptCount, + continuationError, + continuationFailedAt: failedAt, + continuationSettledAt: retryBudgetExhausted ? failedAt : undefined, + followUpNoProgress: retryBudgetExhausted, + followUpBlocker: retryBudgetExhausted ? continuationError : null, + }, + }); + if (args.task.record_id) { + this.deps.projectManagementRepository.updateTask(args.task.record_id, { + status: "coding_completed", + isMerged: false, + mergeIndicator: "QA_PENDING", + }); + } + args.task.status = "CODING_COMPLETED"; + args.task.is_merged = false; + args.task.merge_indicator = "QA_PENDING"; + args.task.intervention_owner = undefined; + args.task.intervention_hint = undefined; + throw error; + } finally { + this.activeQaContinuationRunIds.delete(args.run.id); + } + + const continuationStatus = continued.noProgress + ? "no_progress" + : continued.applied + ? "completed" + : "skipped"; + const postExhaustionVerificationEligible = continued.applied + && args.decisiveRuns === args.maxTaskReviewRuns; + this.deps.qaReviewRepository.updateRun(args.run.id, { + payload: { + ...latestPayload(), + continued: continued.applied, + continuationMode: continued.mode, + continuationStatus, + continuationSettledAt: new Date().toISOString(), + continuationError: undefined, + followUpNoProgress: continued.noProgress, + followUpBlocker: continued.blocker, + postExhaustionVerificationEligible, + }, + }); + + const taskId = args.task.record_id?.trim(); + if (taskId) { + if (continued.noProgress) { + this.deps.projectManagementRepository.updateTask(taskId, { + status: "coding_completed", + mergeIndicator: null, + }); + args.task.status = "CODING_COMPLETED"; + } else if (continued.applied) { + clearMergeProjectionForRerun(args.task); + this.deps.projectManagementRepository.updateTask(taskId, { + status: "coding_completed", + isMerged: false, + mergeIndicator: "QA_PENDING", + }); + args.task.status = "CODING_COMPLETED"; + args.task.merge_indicator = "QA_PENDING"; + } else { + this.deps.projectManagementRepository.updateTask(taskId, { + status: "pending", + ...MERGE_PROJECTION_RESET, + }); + args.task.status = "PENDING"; + } + if (!continued.applied) { + clearMergeProjectionForRerun(args.task); + } + } + + this.appendTaskEvent(args.taskRun, "qa_review_changes_requested", { + triggerType: args.run.triggerType, + summary: args.run.summaryMarkdown, + findings: args.findings || [], + fixInstructions: args.run.fixInstructions, + qaReviewRunId: args.run.id, + continued: continued.applied, + continuationMode: continued.mode, + continuationStatus, + followUpNoProgress: continued.noProgress, + followUpBlocker: continued.blocker, + postExhaustionVerificationEligible, + agentPresetId: args.run.agentPresetId, + agentName: args.run.agentName, + }); + + return { + reviewed: true, + reopenedTask: true, + mergeBlocked: true, + reportText: renderQaChangesRequestedReport( + args.task.id, + args.run.summaryMarkdown || "QA requested follow-up changes.", + continued.applied, + ), + }; + } + + private isLaterTaskRunAfterQaVerdict(run: QaReviewRunRecord, taskRun: TaskRunRecord | null): boolean { + if (!taskRun || !run.taskRunId || taskRun.id === run.taskRunId) { + return false; + } + const qaFinishedAt = Date.parse(run.finishedAt || run.startedAt); + const taskRunStartedAt = Date.parse(taskRun.startedAt || ""); + return !Number.isFinite(qaFinishedAt) + || !Number.isFinite(taskRunStartedAt) + || taskRunStartedAt > qaFinishedAt; + } + + private isLaterTaskRunStillActive(run: QaReviewRunRecord, taskRun: TaskRunRecord | null): boolean { + return this.isLaterTaskRunAfterQaVerdict(run, taskRun) + && taskRun?.state !== "COMPLETED" + && taskRun?.state !== "FAILED"; + } + + private hasCompletedSameSessionQaFollowUp( + run: QaReviewRunRecord, + taskRun: TaskRunRecord | null, + taskSessionId: string | undefined, + ): boolean { + if (this.isLaterTaskRunAfterQaVerdict(run, taskRun) && taskRun?.state === "COMPLETED") { + const qaFinishedAt = Date.parse(run.finishedAt || run.startedAt); + const taskRunFinishedAt = Date.parse(taskRun.finishedAt || ""); + if (Number.isFinite(taskRunFinishedAt) + && (!Number.isFinite(qaFinishedAt) || taskRunFinishedAt > qaFinishedAt)) { + return true; + } + } + // A completed provider invocation is not a durable publication checkpoint: + // the runtime can still exit after the provider committed in its workspace + // but before that patch reached the host worker branch. Only a later task run + // that fully settled can reconcile the handoff without re-entering the + // publication path; same-run continuations must reuse their saved baseline. + void taskSessionId; + return false; + } + private async requestFixesForTask(args: { task: Subtask; taskRun: TaskRunRecord | null; @@ -1535,6 +2319,7 @@ export class QualityAssuranceService { featureBranch: string; scope: DashboardSettingsScope; prompt: string; + qaContinuationRunId?: string; }): Promise { const provider = args.task.provider; const sessionId = args.task.session_id?.trim(); @@ -1562,6 +2347,7 @@ export class QualityAssuranceService { featureBranch: args.featureBranch, scope: args.scope, followUpPrompt, + qaContinuationRunId: args.qaContinuationRunId, }); return { applied: result.producedMergeWork, @@ -1580,6 +2366,7 @@ export class QualityAssuranceService { featureBranch: string; scope: DashboardSettingsScope; followUpPrompt: string; + qaContinuationRunId?: string; }): Promise { const settings = this.deps.getDashboardSettings(args.scope); const workflowSettings = { @@ -1821,7 +2608,45 @@ export class QualityAssuranceService { const providerPrompt = buildProviderPrompt(`${promptBody}\n\n${workspaceGuidance}`, followUpProviderSettings.thinkingMode, args.provider); const previousInvocation = this.deps.executionRepository.getLatestProviderInvocationUsageBySession(args.sessionId, "task_coding"); - const initialHead = (await this.runWorkspaceCommand(worktreePath, "git", ["rev-parse", "HEAD"])).stdout.trim(); + const persistedContinuationPayload = args.qaContinuationRunId + ? this.deps.qaReviewRepository.getRun(args.qaContinuationRunId)?.payload + : null; + const persistedWorkspaceBaseRef = typeof persistedContinuationPayload?.continuationWorkspaceBaseRef === "string" + ? persistedContinuationPayload.continuationWorkspaceBaseRef.trim() + : ""; + const initialHead = persistedWorkspaceBaseRef + || (await this.runWorkspaceCommand(worktreePath, "git", ["rev-parse", "HEAD"])).stdout.trim(); + if (persistedWorkspaceBaseRef) { + await this.runWorkspaceCommand(worktreePath, "git", ["rev-parse", "--verify", `${persistedWorkspaceBaseRef}^{commit}`]); + } else if (args.qaContinuationRunId) { + const run = this.deps.qaReviewRepository.getRun(args.qaContinuationRunId); + if (run) { + this.deps.qaReviewRepository.updateRun(run.id, { + payload: { + ...(run.payload || {}), + continuationWorkspaceBaseRef: initialHead, + continuationWorkspaceBaseRecordedAt: new Date().toISOString(), + }, + }); + } + } + // The shared QA coding path is also used by sprint-completion handoffs. + // Persist the verification-ready projection immediately before dispatch so + // a hard process exit cannot expose this task as ordinary pending/running + // coding, regardless of which QA scope initiated the continuation. + if (args.task.record_id) { + clearMergeProjectionForRerun(args.task); + this.deps.projectManagementRepository.updateTask(args.task.record_id, { + status: "coding_completed", + isMerged: false, + mergeIndicator: "QA_PENDING", + }); + } + args.task.status = "CODING_COMPLETED"; + args.task.is_merged = false; + args.task.merge_indicator = "QA_PENDING"; + args.task.intervention_owner = undefined; + args.task.intervention_hint = undefined; this.deps.sessionTracking.updateSession(args.sessionId, { state: "RUNNING" }); this.deps.sessionTracking.appendActivity(args.sessionId, { originator: "system", @@ -1858,9 +2683,6 @@ export class QualityAssuranceService { }); if (!result.ok) { - this.deps.projectManagementRepository.updateTask(args.task.record_id!, { - status: "pending", - }); this.deps.sessionTracking.updateSession(args.sessionId, { state: "FAILED" }); throw new Error(result.stderr || result.stdout || "CLI QA follow-up failed."); } @@ -1917,7 +2739,9 @@ export class QualityAssuranceService { // Existing commits/PR state prove that the task has merge work, but they do // not prove that this follow-up addressed the latest QA request. Only a // patch produced from this invocation counts as continuation progress. - const producedMergeWork = applyResult.hasChanges; + const previouslyPublishedFromBaseline = !applyResult.hasChanges + && await this.workerBranchAdvancedFromBaseline(args.repoPath, workerBranch, initialHead); + const producedMergeWork = applyResult.hasChanges || previouslyPublishedFromBaseline; let prUrl = args.task.pr_url || args.taskRun?.prUrl || null; if (hasUnpushed || hasAhead) { @@ -1965,6 +2789,23 @@ export class QualityAssuranceService { state: "COMPLETED", prUrl: prUrl || undefined, }); + if (args.taskRun?.id) { + // A failed QA continuation can mark the original coding run/dispatch + // terminal-failed. Once a same-session continuation succeeds, clear that + // stale projection before terminal sprint evaluation; otherwise the + // healthy task can still make the whole sprint fail in this cycle. + this.deps.executionRepository.updateTaskRun(args.taskRun.id, { + state: "COMPLETED", + }); + args.taskRun.state = "COMPLETED"; + const executionRepository = this.deps.executionRepository as Partial; + if (args.taskRun.dispatchId && typeof executionRepository.updateTaskDispatch === "function") { + executionRepository.updateTaskDispatch(args.taskRun.dispatchId, { + status: "completed", + errorMessage: null, + }); + } + } if (!producedMergeWork) { const blocker = providerOutcome.kind === "blocked" ? providerOutcome.blocker : null; this.appendTaskEvent(args.taskRun, "qa_followup_no_progress", { @@ -2065,6 +2906,31 @@ export class QualityAssuranceService { return runCommandStrict(command, args, worktreePath, env ?? process.env); } + private async workerBranchAdvancedFromBaseline( + repoPath: string, + workerBranch: string, + baselineRef: string, + ): Promise { + try { + const currentHead = (await runCommandStrict( + "git", + ["rev-parse", `refs/heads/${workerBranch}`], + repoPath, + )).stdout.trim(); + if (!currentHead || currentHead === baselineRef) { + return false; + } + await runCommandStrict( + "git", + ["merge-base", "--is-ancestor", baselineRef, currentHead], + repoPath, + ); + return true; + } catch { + return false; + } + } + private async captureMemoriesFromWorkspace( projectId: string, sprintId: string | undefined, @@ -2139,6 +3005,13 @@ export class QualityAssuranceService { existingSubtasks: Subtask[]; sourceRunId: string; }) { + const existing = this.deps.projectManagementRepository + .listTasks(args.projectId, args.sprintId) + .filter((task) => task.sourceType === "qa_review" && task.sourcePath === args.sourceRunId); + if (existing.length > 0) { + return existing; + } + const tasksToCreate = args.review.followUpTasks.length > 0 ? args.review.followUpTasks : (!args.targetTask && !args.fixInstructions) diff --git a/src/services/runtime-recovery/qa-review-recovery.ts b/src/services/runtime-recovery/qa-review-recovery.ts index c6d24abb7d..cbc2c9b4fe 100644 --- a/src/services/runtime-recovery/qa-review-recovery.ts +++ b/src/services/runtime-recovery/qa-review-recovery.ts @@ -3,12 +3,17 @@ import type { ExecutionRepository } from "../../repositories/execution-repositor import type { QaReviewRepository } from "../../repositories/qa-review-repository.js"; import { RECOVERED_STALE_QA_SUMMARY_PREFIX } from "../../domain/qa-review/qa-review-budget.js"; import { calculateInvocationDurationMs } from "./recovery-utils.js"; +import type { DockerContainer } from "../../contracts/app-types.js"; const QA_RUN_START_TIMEOUT_MS = 60_000; interface QaReviewRecoveryServiceDeps { executionRepository: ExecutionRepository; qaReviewRepository?: QaReviewRepository; + dockerService?: { + listContainers: () => Promise; + removeContainers?: (containerIds: string[], options?: { removeVolumes?: boolean }) => Promise; + }; } export class QaReviewRecoveryService { @@ -55,6 +60,9 @@ export class QaReviewRecoveryService { ? this.deps.executionRepository.getProviderInvocationUsage(latestInvocation.providerInvocationId) : null; if (providerInvocation?.status === "running") { + if (providerInvocation.executionMode === "DOCKER") { + await this.removeProviderContainer(providerInvocation.sessionId); + } this.deps.executionRepository.updateProviderInvocationUsage(providerInvocation.id, { status: "cancelled", finishedAt: reconciledAt, @@ -65,6 +73,12 @@ export class QaReviewRecoveryService { this.deps.qaReviewRepository.updateRun(run.id, { status: "cancelled", summaryMarkdown: failureReason, + payload: { + ...run.payload, + reviewNativeSessionId: providerInvocation?.nativeSessionId || run.payload?.reviewNativeSessionId, + reviewOpenCodeBaselineRawUsageJson: providerInvocation?.rawUsageJson + || run.payload?.reviewOpenCodeBaselineRawUsageJson, + }, finishedAt: reconciledAt, }); reconciledRunIds.push(run.id); @@ -73,7 +87,30 @@ export class QaReviewRecoveryService { return reconciledRunIds; } + private async removeProviderContainer(sessionId: string): Promise { + if (!this.deps.dockerService?.removeContainers) { + return; + } + const containers = await this.deps.dockerService.listContainers().catch(() => []); + const containerIds = containers + .filter((container) => container.labels?.["code-ux.session-id"]?.trim() === sessionId) + .map((container) => container.id || container.names) + .filter((containerId): containerId is string => Boolean(containerId)); + if (containerIds.length > 0) { + await this.deps.dockerService.removeContainers(containerIds, { removeVolumes: false }).catch(() => undefined); + } + } + private findLatestQaExecutionInvocation(run: ReturnType[number]): ExecutionInvocationRecord | null { + const correlatedInvocationId = typeof run.payload?.reviewExecutionInvocationId === "string" + ? run.payload.reviewExecutionInvocationId + : null; + if (correlatedInvocationId) { + const correlatedInvocation = this.deps.executionRepository.getExecutionInvocation(correlatedInvocationId); + if (correlatedInvocation?.type === "qa_review") { + return correlatedInvocation; + } + } const invocations = run.taskRunId ? this.deps.executionRepository.listExecutionInvocations({ projectId: run.projectId, @@ -139,6 +176,6 @@ export class QaReviewRecoveryService { return `${RECOVERED_STALE_QA_SUMMARY_PREFIX} after its Docker container disappeared for session ${providerInvocation.sessionId}. Code UX will retry the review.`; } - return null; + return `${RECOVERED_STALE_QA_SUMMARY_PREFIX} after the runtime process restarted. Code UX will continue the preserved review session.`; } } diff --git a/src/services/runtime-startup-recovery-service.ts b/src/services/runtime-startup-recovery-service.ts index b9f1cfc529..86d363aa5c 100644 --- a/src/services/runtime-startup-recovery-service.ts +++ b/src/services/runtime-startup-recovery-service.ts @@ -74,6 +74,7 @@ export interface RuntimeStartupRecoveryResult { restartPolicySyncedPausedSprintIds: string[]; restartPolicySyncedOrphanedSprintIds: string[]; reconciledDuplicateDispatchIds: string[]; + requeuedInterruptedRepairAttentionItemIds: string[]; } interface RuntimeStartupRecoveryServiceDeps { @@ -82,7 +83,7 @@ interface RuntimeStartupRecoveryServiceDeps { sprintRunLifecycleService: SprintRunLifecycleService; qaReviewRepository?: QaReviewRepository; projectManagementRepository: ProjectManagementRepository; - projectAttentionService?: Pick; + projectAttentionService?: Pick; guardrailService?: Pick; sprintOrchestrator: SprintOrchestrator; dockerService?: Pick<{ listContainers: () => Promise; removeContainers: (containerIds: string[], options?: { removeVolumes?: boolean }) => Promise }, "listContainers"> & { @@ -104,6 +105,7 @@ export class RuntimeStartupRecoveryService { const qaReviewRecovery = new QaReviewRecoveryService({ executionRepository: this.deps.executionRepository, qaReviewRepository: this.deps.qaReviewRepository, + dockerService: this.deps.dockerService, }); const invocationRecovery = new InvocationRecoveryService({ executionRepository: this.deps.executionRepository, @@ -144,6 +146,10 @@ export class RuntimeStartupRecoveryService { const reconciledDuplicateDispatchIds = this.reconcileDuplicateActiveTaskDispatches(); const reconciledTaskRunIds = this.reconcileInterruptedTaskRuns(); const reconciledPausedSprintRunIds = this.reconcileStalePausedSprintRuns(); + const requeuedInterruptedRepairAttentionItemIds = restartPolicies.sprintPolicy === "continue" + && restartPolicies.invocationPolicy === "continue" + ? await this.requeueInterruptedVirtualRepairAttention() + : []; const { resumedSprintRunIds, supersededSprintRunIds } = restartPolicies.sprintPolicy === "continue" ? this.resumeRecoverableSprintRuns() : { resumedSprintRunIds: [], supersededSprintRunIds: [] }; @@ -163,6 +169,7 @@ export class RuntimeStartupRecoveryService { || reconciledTerminalProviderDispatchIds.length > 0 || reconciledTerminalDispatchIds.length > 0 || reconciledDuplicateDispatchIds.length > 0 + || requeuedInterruptedRepairAttentionItemIds.length > 0 || rehydratedSprintRunIds.length > 0 || reconciledTaskRunIds.length > 0 || reconciledPausedSprintRunIds.length > 0 @@ -188,6 +195,7 @@ export class RuntimeStartupRecoveryService { reconciledTerminalProviderDispatches: reconciledTerminalProviderDispatchIds.length, reconciledTerminalDispatches: reconciledTerminalDispatchIds.length, reconciledDuplicateDispatches: reconciledDuplicateDispatchIds.length, + requeuedInterruptedRepairAttentionItems: requeuedInterruptedRepairAttentionItemIds.length, rehydratedSprintRuns: rehydratedSprintRunIds.length, reconciledTaskRuns: reconciledTaskRunIds.length, reconciledPausedSprintRuns: reconciledPausedSprintRunIds.length, @@ -220,6 +228,7 @@ export class RuntimeStartupRecoveryService { restartPolicySyncedPausedSprintIds, restartPolicySyncedOrphanedSprintIds, reconciledDuplicateDispatchIds, + requeuedInterruptedRepairAttentionItemIds, restartPolicyPausedSprintRunIds: restartPolicyResult.pausedSprintRunIds, restartPolicyCancelledSprintRunIds: restartPolicyResult.cancelledSprintRunIds, resumedSprintRunIds, @@ -227,6 +236,36 @@ export class RuntimeStartupRecoveryService { }; } + private async requeueInterruptedVirtualRepairAttention(): Promise { + const projectAttentionService = this.deps.projectAttentionService; + if (!projectAttentionService) { + return []; + } + const sessionIds = new Set(); + for (const project of this.deps.projectManagementRepository.listProjects().projects) { + for (const item of projectAttentionService.listActiveProjectItems(project.id)) { + if ( + item.status !== "claimed" + || (item.attentionType !== "ci_fix_required" && item.attentionType !== "merge_conflict") + ) { + continue; + } + const runtime = item.payload?.repairRuntime; + if (runtime && typeof runtime === "object") { + const sessionId = (runtime as Record).sessionId; + if (typeof sessionId === "string" && sessionId.trim()) { + sessionIds.add(sessionId.trim()); + } + } + } + } + // The old process may have died while Docker kept the provider alive. Stop that + // container without deleting its workspace volume before scheduling continuation, + // otherwise two providers could mutate the same repair workspace concurrently. + await this.removeContainersForSessions(sessionIds); + return projectAttentionService.requeueInterruptedVirtualRepairItems().map((item) => item.id); + } + private async demotePrematureMergeConflictEscalations(): Promise { const projectAttentionService = this.deps.projectAttentionService; const guardrailService = this.deps.guardrailService; diff --git a/src/services/virtual-worker-service.ts b/src/services/virtual-worker-service.ts index b39548e429..9470311eee 100644 --- a/src/services/virtual-worker-service.ts +++ b/src/services/virtual-worker-service.ts @@ -2,7 +2,7 @@ import { buildProviderSettingsOverride } from "./provider-settings-override.js"; import { randomUUID } from "crypto"; import type { CliWorkflowSettings, DashboardSettings, GitCiRunStatus, JulesSession, ProviderId, ProviderSettings, QwenModelProviderSettings, ThinkingMode, WorkerExecutionMode, Subtask } from "../contracts/app-types.js"; import type { ProviderInvocationUsageRecord, TaskDispatchRecord, WorkerTaskDispatchClaim } from "../contracts/execution-types.js"; -import type { ProjectAttentionItemRecord } from "../contracts/project-attention-types.js"; +import type { ProjectAttentionItemRecord, RepairAttentionRuntimeState } from "../contracts/project-attention-types.js"; import type { SettingsRepository } from "../repositories/settings-repository.js"; import type { SessionTrackingRepository } from "../repositories/session-tracking-repository.js"; import type { ExecutionRepository } from "../repositories/execution-repository.js"; @@ -15,7 +15,10 @@ import { buildProviderPrompt, DEFAULT_CLI_WORKFLOW_SETTINGS, sanitizeToken } fro import { isReadFileNotFoundToolError, buildReadFileRetryPrompt } from "./cli-workflow-text-utils.js"; import { WorkspaceManager } from "../infrastructure/providers/cli/workspace-manager.js"; import { buildInvocationGitPolicy, InvocationWorkspacePreparer } from "../infrastructure/providers/cli/invocation-workspace-preparer.js"; -import { WorkspaceArtifactService } from "../infrastructure/providers/cli/workspace-artifact-service.js"; +import { + WorkspaceArtifactService, + type AppliedWorkspacePatchResult, +} from "../infrastructure/providers/cli/workspace-artifact-service.js"; import { CODE_UX_GIT_PATHSPEC_EXCLUDE, CODE_UX_REPO_DIR } from "../infrastructure/git/code-ux-gitignore.js"; import { ProviderRunner } from "../infrastructure/providers/cli/provider-runner.js"; import { DockerRunner } from "../infrastructure/providers/cli/docker-runner.js"; @@ -78,6 +81,7 @@ const VIRTUAL_WORKER_CLI_PROVIDER_POOL: ProviderId[] = [ interface TaskCiFixContinuation { provider: Exclude; + providerConfigId: string; providerSettings: ProviderSettings; sessionId: string; resumeSessionId: string; @@ -904,6 +908,7 @@ export class VirtualWorkerService { return { provider, + providerConfigId: providerConfigId!, providerSettings: { ...configured, model: codingInvocation?.model?.trim() || taskRecord.model?.trim() || configured.model, @@ -1044,9 +1049,217 @@ export class VirtualWorkerService { } } + private readRepairRuntime( + item: ProjectAttentionItemRecord, + purpose: RepairAttentionRuntimeState["purpose"], + ): RepairAttentionRuntimeState | null { + const value = this.asRecord(item.payload?.repairRuntime); + if ( + value?.purpose !== purpose + || typeof value.sessionId !== "string" + || typeof value.workspaceSessionId !== "string" + || typeof value.provider !== "string" + || typeof value.providerConfigId !== "string" + || typeof value.model !== "string" + ) { + return null; + } + return { + purpose, + sessionId: value.sessionId, + workspaceSessionId: value.workspaceSessionId, + provider: value.provider, + providerConfigId: value.providerConfigId, + model: value.model, + nativeSessionId: typeof value.nativeSessionId === "string" ? value.nativeSessionId : null, + activeAttemptId: typeof value.activeAttemptId === "string" ? value.activeAttemptId : null, + attemptRecorded: value.attemptRecorded === true, + phase: value.phase === "workspace_ready" || value.phase === "provider_running" || value.phase === "interrupted" + ? value.phase + : "claimed", + workspaceBaselineHead: typeof value.workspaceBaselineHead === "string" ? value.workspaceBaselineHead : null, + workspaceRepairHead: typeof value.workspaceRepairHead === "string" ? value.workspaceRepairHead : null, + publicationPhase: value.publicationPhase === "workspace_finalized" + || value.publicationPhase === "host_publishing" + || value.publicationPhase === "host_published" + ? value.publicationPhase + : "pending", + publishedHeadSha: typeof value.publishedHeadSha === "string" ? value.publishedHeadSha : null, + updatedAt: typeof value.updatedAt === "string" ? value.updatedAt : new Date().toISOString(), + }; + } + + private checkpointRepairRuntime( + item: ProjectAttentionItemRecord, + runtime: Omit, + ): RepairAttentionRuntimeState { + const checkpoint: RepairAttentionRuntimeState = { + ...runtime, + updatedAt: new Date().toISOString(), + }; + const updated = this.deps.projectAttentionService.patchItemPayload(item.id, { + repairRuntime: checkpoint, + }); + item.payload = updated.payload; + return checkpoint; + } + + private latestRepairInvocation(runtime: RepairAttentionRuntimeState): ProviderInvocationUsageRecord | null { + return this.deps.executionRepository.getLatestProviderInvocationUsageBySession(runtime.sessionId); + } + + private buildRepairPublicationCommitMessage(summary: string, workspaceRepairHead: string | null): string { + return workspaceRepairHead + ? `${summary}\n\nCode-UX-Repair-Head: ${workspaceRepairHead}` + : summary; + } + + private async findPublishedRepairCommit(args: { + repoPath: string; + workerBranch: string; + workspaceRepairHead: string | null; + githubMode: "REMOTE" | "LOCAL"; + gitAuth: GitHttpAuthOptions; + }): Promise { + if (!args.workspaceRepairHead) { + return null; + } + const marker = `Code-UX-Repair-Head: ${args.workspaceRepairHead}`; + let commitSha: string; + try { + commitSha = (await runCommandStrict( + "git", + [ + "log", + "-1", + "--format=%H", + "--fixed-strings", + `--grep=${marker}`, + `refs/heads/${args.workerBranch}`, + ], + args.repoPath, + )).stdout.trim(); + } catch { + return null; + } + if (!commitSha) { + return null; + } + await this.ensureRepairBranchPublished(args); + return commitSha; + } + + private async findTreeEquivalentPublishedRepair(args: { + repoPath: string; + worktreePath: string; + workerBranch: string; + workspaceBaselineHead: string | null; + workspaceRepairHead: string | null; + expectedCommitSubject: string; + requiredParentRefs: string[]; + githubMode: "REMOTE" | "LOCAL"; + gitAuth: GitHttpAuthOptions; + }): Promise { + if (!args.workspaceBaselineHead || !args.workspaceRepairHead) { + return null; + } + let workspaceRepairTree: string; + let workspaceBaselineTree: string; + let hostCandidates: Array<{ head: string; tree: string; subject: string }>; + try { + workspaceRepairTree = await this.workspaceArtifactService.resolveWorkspaceTree(args.worktreePath); + workspaceBaselineTree = (await this.runWorkspaceCommand( + args.worktreePath, + "git", + ["rev-parse", `${args.workspaceBaselineHead}^{tree}`], + )).stdout.trim(); + const hostLog = (await runCommandStrict( + "git", + [ + "log", + "--format=%H%x09%T%x09%s", + "--fixed-strings", + `--grep=${args.expectedCommitSubject}`, + `refs/heads/${args.workerBranch}`, + ], + args.repoPath, + )).stdout; + hostCandidates = hostLog.split("\n").map((line) => { + const [head = "", tree = "", subject = ""] = line.split("\t"); + return { head: head.trim(), tree: tree.trim(), subject: subject.trim() }; + }); + } catch { + return null; + } + if ( + !workspaceRepairTree + || !workspaceBaselineTree + || (workspaceRepairTree === workspaceBaselineTree + && (args.requiredParentRefs.length === 0 || args.workspaceRepairHead === args.workspaceBaselineHead)) + ) { + return null; + } + for (const candidate of hostCandidates) { + if ( + !candidate.head + || candidate.head === args.workspaceBaselineHead + || candidate.tree !== workspaceRepairTree + || candidate.subject !== args.expectedCommitSubject + ) { + continue; + } + try { + await runCommandStrict( + "git", + ["merge-base", "--is-ancestor", args.workspaceBaselineHead, candidate.head], + args.repoPath, + ); + } catch { + continue; + } + let hasRequiredParents = true; + for (const parentRef of args.requiredParentRefs) { + try { + await runCommandStrict( + "git", + ["merge-base", "--is-ancestor", parentRef, candidate.head], + args.repoPath, + ); + } catch { + hasRequiredParents = false; + break; + } + } + if (hasRequiredParents) { + await this.ensureRepairBranchPublished(args); + return candidate.head; + } + } + return null; + } + + private async ensureRepairBranchPublished(args: { + repoPath: string; + workerBranch: string; + githubMode: "REMOTE" | "LOCAL"; + gitAuth: GitHttpAuthOptions; + }): Promise { + if (args.githubMode === "LOCAL") { + return; + } + const pushEnv = await buildGitHttpAuthEnvForRepoWithFallbacks(args.repoPath, args.gitAuth); + await runCommandStrict( + "git", + ["push", "-u", "origin", `refs/heads/${args.workerBranch}:refs/heads/${args.workerBranch}`], + args.repoPath, + pushEnv ?? process.env, + ); + } + private async resolveMergeConflictAttention(workerEndpointId: string, item: ProjectAttentionItemRecord): Promise { const settings = this.resolveDashboardSettings(item.projectId, item.sprintId); const guardrailScope = { projectId: item.projectId, sprintId: item.sprintId }; + const savedRuntime = this.readRepairRuntime(item, "merge_conflict"); const workerAgent = await this.deps.agentPresetSyncService?.resolveTargetedCodingAgent( item.projectId, settings.agents?.routing?.mergeConflict?.agentPresetId ?? null, @@ -1069,9 +1282,20 @@ export class VirtualWorkerService { } : null, }); - const provider = route.provider as Exclude; - const providerConfigId = route.providerConfigId || route.provider; - const providerSettings = route.providers[providerConfigId]; + const savedProviderSettings = savedRuntime + && savedRuntime.provider !== "jules" + && settings.aiProvider.providers[savedRuntime.providerConfigId]?.provider === savedRuntime.provider + ? route.providers[savedRuntime.providerConfigId] || settings.aiProvider.providers[savedRuntime.providerConfigId] + : null; + const provider = (savedProviderSettings ? savedRuntime!.provider : route.provider) as Exclude; + const providerConfigId = savedProviderSettings + ? savedRuntime!.providerConfigId + : route.providerConfigId || route.provider; + const resolvedProviderSettings = savedProviderSettings || route.providers[providerConfigId]; + const providerSettings: ProviderSettings = { + ...resolvedProviderSettings, + model: savedRuntime?.model || resolvedProviderSettings.model, + }; const workflowSettings = { ...DEFAULT_CLI_WORKFLOW_SETTINGS, ...settings.cliWorkflow, @@ -1143,8 +1367,9 @@ export class VirtualWorkerService { return; } + const isRestartContinuation = Boolean(savedRuntime?.activeAttemptId && savedRuntime.attemptRecorded); const mergeConflictEval = this.evaluateMergeConflictGuardrail(settings, guardrailScope, item); - if (mergeConflictEval && !mergeConflictEval.allowed && mergeConflictEval.action !== "WARN_ONLY") { + if (!isRestartContinuation && mergeConflictEval && !mergeConflictEval.allowed && mergeConflictEval.action !== "WARN_ONLY") { this.escalateAttentionToHuman( workerEndpointId, item, @@ -1157,9 +1382,30 @@ export class VirtualWorkerService { // failures, crashes, and quota-exhausted runs all consume the retry budget. Recording // only on success (the previous behavior) meant a conflict that never resolved retried // indefinitely until the provider API limit was hit instead of escalating after `cap`. - this.recordMergeConflictAttempt(guardrailScope, item); + const sessionId = savedRuntime?.sessionId + || `virtual-merge-${provider}-${Date.now().toString(36)}-${randomUUID().slice(0, 8)}`; + let repairRuntime = this.checkpointRepairRuntime(item, { + purpose: "merge_conflict", + sessionId, + workspaceSessionId: savedRuntime?.workspaceSessionId || sessionId, + provider, + providerConfigId, + model: savedRuntime?.model || providerSettings.model, + nativeSessionId: savedRuntime?.nativeSessionId || null, + activeAttemptId: savedRuntime?.activeAttemptId || randomUUID(), + // Persist the marker before recording. A process exit in this tiny window can + // under-count once, but can never charge the same interrupted attempt twice. + attemptRecorded: true, + phase: "claimed", + workspaceBaselineHead: savedRuntime?.workspaceBaselineHead || null, + workspaceRepairHead: savedRuntime?.workspaceRepairHead || null, + publicationPhase: savedRuntime?.publicationPhase || "pending", + publishedHeadSha: savedRuntime?.publishedHeadSha || null, + }); + if (!isRestartContinuation) { + this.recordMergeConflictAttempt(guardrailScope, item); + } - const sessionId = `virtual-merge-${provider}-${Date.now().toString(36)}-${randomUUID().slice(0, 8)}`; let worktreePath = this.workspaceManager.buildWorktreePath(repoPath, sessionId, workflowSettings.executionMode); const title = item.title; let succeeded = false; @@ -1171,23 +1417,29 @@ export class VirtualWorkerService { ? resolveAgentMemoryInstructions(workerAgent || {}, settings.memory?.workerLearningsInstruction) : ""; - this.deps.sessionTracking.createSession({ - id: sessionId, - provider, - taskId: buildTaskRunKey(repoPath, 0, `attention-${item.id}`), - title, - prompt: item.summaryMarkdown, - state: "RUNNING", - featureBranch: sourceBranch, - workerBranch: sourceBranch, - repoPath, - }); + const trackedSession = this.deps.sessionTracking.getSession(sessionId); + if (trackedSession) { + this.deps.sessionTracking.updateSession(sessionId, { state: "RUNNING" }); + } else { + this.deps.sessionTracking.createSession({ + id: sessionId, + provider, + taskId: buildTaskRunKey(repoPath, 0, `attention-${item.id}`), + title, + prompt: item.summaryMarkdown, + state: "RUNNING", + featureBranch: sourceBranch, + workerBranch: sourceBranch, + repoPath, + }); + } this.deps.sessionTracking.appendActivity(sessionId, { originator: "system", description: `Virtual worker claimed merge conflict between ${sourceBranch} and ${targetBranch}.`, }); let cleanedUp = false; + let preserveWorkspace = false; try { const effectiveWorkflowSettings = await this.resolveVirtualWorkerWorkflowSettings({ workflowSettings, @@ -1197,9 +1449,10 @@ export class VirtualWorkerService { }); const prepared = await this.invocationWorkspacePreparer.prepareWorktree({ repoPath, - worktreePath: this.workspaceManager.buildWorktreePath(repoPath, sessionId, effectiveWorkflowSettings.executionMode), + worktreePath: this.workspaceManager.buildWorktreePath(repoPath, repairRuntime.workspaceSessionId, effectiveWorkflowSettings.executionMode), workerBranch: sourceBranch, featureBranch: targetBranch, + resumeSessionId: repairRuntime.workspaceSessionId, gitAuth, gitPolicy: buildInvocationGitPolicy({ githubMode: settings.git.githubMode, @@ -1210,96 +1463,188 @@ export class VirtualWorkerService { }); const finalWorktreePath = prepared.worktreePath; worktreePath = finalWorktreePath; - initialHead = (await this.runWorkspaceCommand(finalWorktreePath, "git", ["rev-parse", "HEAD"])).stdout.trim(); - const hasConflicts = await this.runMergeIntoSource(finalWorktreePath, targetRef, sessionId); - if (hasConflicts) { - const workspaceGuidance = await this.workspaceManager.buildWorkspaceGuidance(item.summaryMarkdown, finalWorktreePath); - const providerPrompt = buildProviderPrompt( - this.buildMergeConflictPrompt( - item, - sourceBranch, - targetBranch, - workspaceGuidance, - workerAgent?.instructionMarkdown, - memoryContext, - memoryInstructions, - ), - providerSettings.thinkingMode, - provider, - ); - await this.runProviderWithRetry({ - provider, - providerPrompt, - workflowSettings: effectiveWorkflowSettings, + repairRuntime = this.checkpointRepairRuntime(item, { + ...repairRuntime, + phase: "workspace_ready", + }); + const currentWorkspaceHead = (await this.runWorkspaceCommand(finalWorktreePath, "git", ["rev-parse", "HEAD"])).stdout.trim(); + if (!repairRuntime.workspaceBaselineHead) { + repairRuntime = this.checkpointRepairRuntime(item, { + ...repairRuntime, + workspaceBaselineHead: currentWorkspaceHead, + }); + } + initialHead = repairRuntime.workspaceBaselineHead || currentWorkspaceHead; + let workspaceFinalized = repairRuntime.publicationPhase !== "pending"; + if (!workspaceFinalized && prepared.resumed === true && currentWorkspaceHead !== initialHead) { + workspaceFinalized = await this.isWorkspaceMergeFinalized(finalWorktreePath, targetRef); + if (workspaceFinalized) { + repairRuntime = this.checkpointRepairRuntime(item, { + ...repairRuntime, + workspaceRepairHead: currentWorkspaceHead, + publicationPhase: "workspace_finalized", + }); + } + } + if (!workspaceFinalized) { + const resumedMerge = prepared.resumed === true && await this.workspaceHasMergeInProgress(finalWorktreePath); + const hasConflicts = resumedMerge + ? (await this.listUnresolvedFiles(finalWorktreePath)).length > 0 + : await this.runMergeIntoSource(finalWorktreePath, targetRef, sessionId); + if (hasConflicts) { + const workspaceGuidance = await this.workspaceManager.buildWorkspaceGuidance(item.summaryMarkdown, finalWorktreePath); + const providerPrompt = buildProviderPrompt( + this.buildMergeConflictPrompt( + item, + sourceBranch, + targetBranch, + workspaceGuidance, + workerAgent?.instructionMarkdown, + memoryContext, + memoryInstructions, + ), + providerSettings.thinkingMode, + provider, + ); + const previousInvocation = this.latestRepairInvocation(repairRuntime); + repairRuntime = this.checkpointRepairRuntime(item, { + ...repairRuntime, + nativeSessionId: previousInvocation?.nativeSessionId || repairRuntime.nativeSessionId, + phase: "provider_running", + }); + const nativeSessionId = await this.runProviderWithRetry({ + provider, + providerPrompt, + workflowSettings: effectiveWorkflowSettings, + repoPath, + worktreePath: finalWorktreePath, + sessionId, + attentionItem: item, + purpose: "merge_conflict", + model: providerSettings.model, + thinkingMode: providerSettings.thinkingMode, + apiKey: providerSettings.apiKey, + maxConcurrentTasks: providerSettings.maxConcurrentTasks, + qwenAuthMode: providerSettings.qwenAuthMode, + qwenRegion: providerSettings.qwenRegion, + qwenBaseUrl: providerSettings.qwenBaseUrl, + qwenEnvKey: providerSettings.qwenEnvKey, + qwenModelId: providerSettings.qwenModelId, + qwenProtocol: providerSettings.qwenProtocol, + qwenAdditionalModelProviders: providerSettings.qwenAdditionalModelProviders, + openCodeAuthMode: providerSettings.openCodeAuthMode, + openCodeProviderId: providerSettings.openCodeProviderId, + openCodeModelId: providerSettings.openCodeModelId, + openCodeBaseUrl: providerSettings.openCodeBaseUrl, + openCodeEnvKey: providerSettings.openCodeEnvKey, + openCodePackage: providerSettings.openCodePackage, + providerMountAuth: providerSettings.mountAuth, + providerAuthPath: providerSettings.authPath, + providerConfigMode: providerSettings.providerConfigMode, + providerConfigPath: providerSettings.providerConfigPath, + customBaseUrl: providerSettings.customBaseUrl, + customModel: providerSettings.customModel, + continueSessionId: repairRuntime.nativeSessionId, + openCodeBaselineRawUsageJson: provider === "opencode" + ? previousInvocation?.rawUsageJson ?? null + : null, + githubToken: settings.git.githubToken, + agentMcpAccess: workerAgent ? workerClarificationAgentMcpAccess(workerAgent.mcpAccess) : null, + mcpAgentId: workerAgent?.id ?? null, + }); + if (nativeSessionId) { + repairRuntime = this.checkpointRepairRuntime(item, { + ...repairRuntime, + nativeSessionId, + phase: "provider_running", + }); + } + } + await this.ensureMergeConflictResolved(finalWorktreePath); + await this.ensureMergeConflictPreservesPromptLiterals(finalWorktreePath, item); + await this.finalizeMergeCommit(finalWorktreePath, sourceBranch, targetBranch); + await this.ensureTargetMergedIntoSource(finalWorktreePath, targetRef); + if (settings.memory?.enabled && settings.memory.autoCaptureSprint) { + await this.captureMemoriesFromWorkspace( + item.projectId, + item.sprintId || undefined, + workerAgent?.id || null, + finalWorktreePath, + item.id, + ); + } + const workspaceRepairHead = (await this.runWorkspaceCommand(finalWorktreePath, "git", ["rev-parse", "HEAD"])).stdout.trim(); + repairRuntime = this.checkpointRepairRuntime(item, { + ...repairRuntime, + workspaceRepairHead, + publicationPhase: "workspace_finalized", + }); + } + const repairCommitSubject = `fix(merge): resolve ${targetBranch} into ${sourceBranch}`; + let recoveredPublishedHead = repairRuntime.publicationPhase === "host_publishing" + && repairRuntime.workspaceRepairHead !== repairRuntime.workspaceBaselineHead + ? await this.findPublishedRepairCommit({ + repoPath, + workerBranch: sourceBranch, + workspaceRepairHead: repairRuntime.workspaceRepairHead, + githubMode: settings.git.githubMode, + gitAuth, + }) + : null; + if (!recoveredPublishedHead && repairRuntime.publicationPhase === "host_publishing") { + recoveredPublishedHead = await this.findTreeEquivalentPublishedRepair({ repoPath, worktreePath: finalWorktreePath, - sessionId, - attentionItem: item, - purpose: "merge_conflict", - model: providerSettings.model, - thinkingMode: providerSettings.thinkingMode, - apiKey: providerSettings.apiKey, - maxConcurrentTasks: providerSettings.maxConcurrentTasks, - qwenAuthMode: providerSettings.qwenAuthMode, - qwenRegion: providerSettings.qwenRegion, - qwenBaseUrl: providerSettings.qwenBaseUrl, - qwenEnvKey: providerSettings.qwenEnvKey, - qwenModelId: providerSettings.qwenModelId, - qwenProtocol: providerSettings.qwenProtocol, - qwenAdditionalModelProviders: providerSettings.qwenAdditionalModelProviders, - openCodeAuthMode: providerSettings.openCodeAuthMode, - openCodeProviderId: providerSettings.openCodeProviderId, - openCodeModelId: providerSettings.openCodeModelId, - openCodeBaseUrl: providerSettings.openCodeBaseUrl, - openCodeEnvKey: providerSettings.openCodeEnvKey, - openCodePackage: providerSettings.openCodePackage, - providerMountAuth: providerSettings.mountAuth, - providerAuthPath: providerSettings.authPath, - providerConfigMode: providerSettings.providerConfigMode, - providerConfigPath: providerSettings.providerConfigPath, - customBaseUrl: providerSettings.customBaseUrl, - customModel: providerSettings.customModel, - githubToken: settings.git.githubToken, - agentMcpAccess: workerAgent ? workerClarificationAgentMcpAccess(workerAgent.mcpAccess) : null, - mcpAgentId: workerAgent?.id ?? null, + workerBranch: sourceBranch, + workspaceBaselineHead: repairRuntime.workspaceBaselineHead, + workspaceRepairHead: repairRuntime.workspaceRepairHead, + expectedCommitSubject: repairCommitSubject, + requiredParentRefs: settings.git.githubMode === "LOCAL" ? [targetBranch] : [`origin/${targetBranch}`], + githubMode: settings.git.githubMode, + gitAuth, }); } - await this.ensureMergeConflictResolved(finalWorktreePath); - await this.ensureMergeConflictPreservesPromptLiterals(finalWorktreePath, item); - await this.finalizeMergeCommit(finalWorktreePath, sourceBranch, targetBranch); - await this.ensureTargetMergedIntoSource(finalWorktreePath, targetRef); - if (settings.memory?.enabled && settings.memory.autoCaptureSprint) { - await this.captureMemoriesFromWorkspace( - item.projectId, - item.sprintId || undefined, - workerAgent?.id || null, + const alreadyPublished = Boolean(recoveredPublishedHead) + || (repairRuntime.publicationPhase === "host_published" && Boolean(repairRuntime.publishedHeadSha)); + let applyResult: AppliedWorkspacePatchResult = { + hasChanges: alreadyPublished, + commitSha: recoveredPublishedHead || repairRuntime.publishedHeadSha || undefined, + }; + if (!alreadyPublished) { + const patchText = await this.workspaceArtifactService.exportBinaryPatch( finalWorktreePath, - item.id, + repairRuntime.workspaceBaselineHead || initialHead, ); + repairRuntime = this.checkpointRepairRuntime(item, { + ...repairRuntime, + publicationPhase: "host_publishing", + }); + applyResult = await this.workspaceArtifactService.applyPatchToBranch({ + repoPath, + baseRef: repairRuntime.workspaceBaselineHead || initialHead, + workerBranch: sourceBranch, + patchText, + commitMessage: this.buildRepairPublicationCommitMessage( + repairCommitSubject, + repairRuntime.workspaceRepairHead, + ), + parentRefs: settings.git.githubMode === "LOCAL" ? [targetBranch] : [`origin/${targetBranch}`], + // A conflict resolved by keeping the source side leaves the tree unchanged but + // still needs a merge commit recording the target as a parent, otherwise the PR + // keeps reporting the conflict and the resolution loops forever. + forceMergeCommit: true, + gitAuth, + gitIdentity: effectiveWorkflowSettings.containerMountGitConfig + ? undefined + : { + name: effectiveWorkflowSettings.containerGitUserName, + email: effectiveWorkflowSettings.containerGitUserEmail, + }, + githubMode: settings.git.githubMode, + }); } - const patchText = await this.workspaceArtifactService.exportBinaryPatch(finalWorktreePath, initialHead); - const applyResult = await this.workspaceArtifactService.applyPatchToBranch({ - repoPath, - baseRef: initialHead, - workerBranch: sourceBranch, - patchText, - commitMessage: `fix(merge): resolve ${targetBranch} into ${sourceBranch}`, - parentRefs: settings.git.githubMode === "LOCAL" ? [targetBranch] : [`origin/${targetBranch}`], - // A conflict resolved by keeping the source side leaves the tree unchanged but - // still needs a merge commit recording the target as a parent, otherwise the PR - // keeps reporting the conflict and the resolution loops forever. - forceMergeCommit: true, - gitAuth, - gitIdentity: effectiveWorkflowSettings.containerMountGitConfig - ? undefined - : { - name: effectiveWorkflowSettings.containerGitUserName, - email: effectiveWorkflowSettings.containerGitUserEmail, - }, - githubMode: settings.git.githubMode, - }); - let hasUnpushed = applyResult.hasChanges; - let hasAhead = applyResult.hasChanges; + let hasUnpushed = alreadyPublished || applyResult.hasChanges; + let hasAhead = alreadyPublished || applyResult.hasChanges; if (!applyResult.hasChanges) { hasUnpushed = await this.prService.hasUnpushedCommits(repoPath, sourceBranch, targetBranch); hasAhead = await this.prService.hasWorkerBranchCommitsAgainstFeature(repoPath, sourceBranch, targetBranch); @@ -1323,6 +1668,11 @@ export class VirtualWorkerService { || ((hasUnpushed || hasAhead) ? (await runCommandStrict("git", ["rev-parse", `refs/heads/${sourceBranch}`], repoPath)).stdout.trim() : initialHead); + repairRuntime = this.checkpointRepairRuntime(item, { + ...repairRuntime, + publicationPhase: "host_published", + publishedHeadSha: headSha, + }); this.deps.sessionTracking.updateSession(sessionId, { state: "COMPLETED" }); this.deps.sessionTracking.appendActivity(sessionId, { originator: "system", @@ -1355,6 +1705,13 @@ export class VirtualWorkerService { } catch (error) { const message = error instanceof Error ? error.message : String(error); if (isProviderCancellationError(error)) { + preserveWorkspace = true; + const previousInvocation = this.latestRepairInvocation(repairRuntime); + repairRuntime = this.checkpointRepairRuntime(item, { + ...repairRuntime, + nativeSessionId: previousInvocation?.nativeSessionId || repairRuntime.nativeSessionId, + phase: "interrupted", + }); this.deps.sessionTracking.updateSession(sessionId, { state: "CANCELLED" }); this.deps.sessionTracking.appendActivity(sessionId, { originator: "system", @@ -1369,6 +1726,13 @@ export class VirtualWorkerService { provider, error, }); + this.deps.projectAttentionService.requeueItem(item.id, { + repairRuntime, + lastVirtualWorkerError: message, + lastVirtualWorkerInterruptedAt: new Date().toISOString(), + lastVirtualWorkerProvider: provider, + lastVirtualWorkerSessionId: sessionId, + }); return; } this.deps.sessionTracking.updateSession(sessionId, { state: "FAILED" }); @@ -1378,8 +1742,17 @@ export class VirtualWorkerService { }); const retryEval = this.evaluateMergeConflictGuardrail(settings, guardrailScope, item); if (!retryEval || retryEval.allowed || retryEval.action === "WARN_ONLY") { + preserveWorkspace = true; const now = new Date().toISOString(); this.deps.projectAttentionService.requeueItem(item.id, { + repairRuntime: { + ...repairRuntime, + nativeSessionId: this.latestRepairInvocation(repairRuntime)?.nativeSessionId || repairRuntime.nativeSessionId, + activeAttemptId: null, + attemptRecorded: false, + phase: "interrupted", + updatedAt: now, + }, lastVirtualWorkerError: message, lastVirtualWorkerFailedAt: now, lastVirtualWorkerProvider: provider, @@ -1410,11 +1783,9 @@ export class VirtualWorkerService { item.summaryMarkdown.trim(), ].join("\n")); } finally { - // Virtual merge worktrees are ephemeral — always clean up to prevent - // stale worktree references from poisoning subsequent git fetch operations. const shouldCleanup = succeeded ? workflowSettings.cleanupWorktreeOnSuccess - : true; + : !preserveWorkspace; if (shouldCleanup) { await this.workspaceManager.removeWorktree(repoPath, worktreePath).catch(() => undefined); cleanedUp = true; @@ -1473,6 +1844,7 @@ export class VirtualWorkerService { private async resolveCiFixAttention(workerEndpointId: string, item: ProjectAttentionItemRecord): Promise { const settings = this.resolveDashboardSettings(item.projectId, item.sprintId); + const savedRuntime = this.readRepairRuntime(item, "ci_fix"); const taskContinuation = await this.resolveTaskCiFixContinuation(item, settings); const workerAgent = taskContinuation?.workerAgent || await this.deps.agentPresetSyncService?.resolveTargetedCodingAgent( @@ -1498,10 +1870,22 @@ export class VirtualWorkerService { } : null, }); - const provider = taskContinuation?.provider - || ciFixRoute!.provider as Exclude; - const providerConfigId = ciFixRoute?.providerConfigId || ciFixRoute?.provider || provider; - const providerSettings = taskContinuation?.providerSettings || ciFixRoute!.providers[providerConfigId]; + const savedProviderSettings = savedRuntime + && savedRuntime.provider !== "jules" + && settings.aiProvider.providers[savedRuntime.providerConfigId]?.provider === savedRuntime.provider + ? settings.aiProvider.providers[savedRuntime.providerConfigId] + : null; + const provider = (savedProviderSettings + ? savedRuntime!.provider + : taskContinuation?.provider || ciFixRoute!.provider) as Exclude; + const providerConfigId = savedProviderSettings + ? savedRuntime!.providerConfigId + : taskContinuation?.providerConfigId || ciFixRoute?.providerConfigId || ciFixRoute?.provider || provider; + const resolvedProviderSettings = savedProviderSettings || taskContinuation?.providerSettings || ciFixRoute!.providers[providerConfigId]; + const providerSettings: ProviderSettings = { + ...resolvedProviderSettings, + model: savedRuntime?.model || resolvedProviderSettings.model, + }; const workflowSettings = { ...DEFAULT_CLI_WORKFLOW_SETTINGS, ...settings.cliWorkflow, @@ -1528,7 +1912,8 @@ export class VirtualWorkerService { const maxRetries = ciFixEval?.cap ?? 0; const capLabel = maxRetries > 0 ? String(maxRetries) : "∞"; - if (ciFixEval && !ciFixEval.allowed && ciFixEval.action !== "WARN_ONLY") { + const isRestartContinuation = Boolean(savedRuntime?.activeAttemptId && savedRuntime.attemptRecorded); + if (!isRestartContinuation && ciFixEval && !ciFixEval.allowed && ciFixEval.action !== "WARN_ONLY") { this.escalateAttentionToHuman(workerEndpointId, item, `Virtual worker reached the CI autofix guardrail (${retryCount}/${capLabel}). Escalating to human.`); return; } @@ -1536,9 +1921,7 @@ export class VirtualWorkerService { // Record the attempt up-front so failed/crashed CI-fix runs also consume the retry // budget — recording only on success let an unfixable failure retry until the // provider API limit instead of escalating after `cap` attempts. - this.deps.guardrailService?.record(guardrailScope, guardrailKey, "ci_fix"); - - const sessionId = taskContinuation?.sessionId + const sessionId = savedRuntime?.sessionId || taskContinuation?.sessionId || `virtual-cifix-${provider}-${Date.now().toString(36)}-${randomUUID().slice(0, 8)}`; const resumeTarget = taskContinuation ? { sessionId: taskContinuation.resumeSessionId } @@ -1547,7 +1930,26 @@ export class VirtualWorkerService { workerBranch: branchName, providers: [provider], }); - const workspaceOwnerSessionId = resumeTarget?.sessionId || sessionId; + const workspaceOwnerSessionId = savedRuntime?.workspaceSessionId || resumeTarget?.sessionId || sessionId; + let repairRuntime = this.checkpointRepairRuntime(item, { + purpose: "ci_fix", + sessionId, + workspaceSessionId: workspaceOwnerSessionId, + provider, + providerConfigId, + model: savedRuntime?.model || providerSettings.model, + nativeSessionId: savedRuntime?.nativeSessionId || taskContinuation?.continueSessionId || null, + activeAttemptId: savedRuntime?.activeAttemptId || randomUUID(), + attemptRecorded: true, + phase: "claimed", + workspaceBaselineHead: savedRuntime?.workspaceBaselineHead || null, + workspaceRepairHead: savedRuntime?.workspaceRepairHead || null, + publicationPhase: savedRuntime?.publicationPhase || "pending", + publishedHeadSha: savedRuntime?.publishedHeadSha || null, + }); + if (!isRestartContinuation) { + this.deps.guardrailService?.record(guardrailScope, guardrailKey, "ci_fix"); + } let worktreePath = this.workspaceManager.buildWorkspaceRef(repoPath, workspaceOwnerSessionId, workflowSettings.executionMode); const title = item.title; let succeeded = false; @@ -1559,7 +1961,7 @@ export class VirtualWorkerService { ? resolveAgentMemoryInstructions(workerAgent || {}, settings.memory?.workerLearningsInstruction) : ""; - if (taskContinuation) { + if (this.deps.sessionTracking.getSession(sessionId)) { this.deps.sessionTracking.updateSession(sessionId, { state: "RUNNING" }); } else { this.deps.sessionTracking.createSession({ @@ -1580,6 +1982,7 @@ export class VirtualWorkerService { }); let cleanedUp = false; + let preserveWorkspace = false; const gitAuth: GitHttpAuthOptions = { githubToken: settings.git.githubToken, gitlabToken: settings.git.gitlabToken, @@ -1596,7 +1999,7 @@ export class VirtualWorkerService { worktreePath: this.workspaceManager.buildWorkspaceRef(repoPath, workspaceOwnerSessionId, effectiveWorkflowSettings.executionMode), workerBranch: branchName, featureBranch: branchName, - resumeSessionId: resumeTarget?.sessionId, + resumeSessionId: repairRuntime.workspaceSessionId, gitAuth, gitPolicy: buildInvocationGitPolicy({ githubMode: settings.git.githubMode, @@ -1607,92 +2010,167 @@ export class VirtualWorkerService { }); const finalWorktreePath = prepared.worktreePath; worktreePath = finalWorktreePath; - initialHead = (await this.runWorkspaceCommand(finalWorktreePath, "git", ["rev-parse", "HEAD"])).stdout.trim(); - - const workspaceGuidance = await this.workspaceManager.buildWorkspaceGuidance(item.summaryMarkdown, finalWorktreePath); - const providerPrompt = buildProviderPrompt( - this.buildCiFixPrompt( - item, - branchName, - workspaceGuidance, - workerAgent?.instructionMarkdown, - memoryContext, - memoryInstructions, - ), - providerSettings.thinkingMode, - provider, - ); - await this.runProviderWithRetry({ - provider, - providerPrompt, - workflowSettings: effectiveWorkflowSettings, - repoPath, - worktreePath: finalWorktreePath, - sessionId, - attentionItem: item, - purpose: "ci_fix", - model: providerSettings.model, - thinkingMode: providerSettings.thinkingMode, - apiKey: providerSettings.apiKey, - maxConcurrentTasks: providerSettings.maxConcurrentTasks, - qwenAuthMode: providerSettings.qwenAuthMode, - - qwenRegion: providerSettings.qwenRegion, - qwenBaseUrl: providerSettings.qwenBaseUrl, - qwenEnvKey: providerSettings.qwenEnvKey, - qwenModelId: providerSettings.qwenModelId, - qwenProtocol: providerSettings.qwenProtocol, - qwenAdditionalModelProviders: providerSettings.qwenAdditionalModelProviders, - openCodeAuthMode: providerSettings.openCodeAuthMode, - openCodeProviderId: providerSettings.openCodeProviderId, - openCodeModelId: providerSettings.openCodeModelId, - openCodeBaseUrl: providerSettings.openCodeBaseUrl, - openCodeEnvKey: providerSettings.openCodeEnvKey, - openCodePackage: providerSettings.openCodePackage, - providerMountAuth: providerSettings.mountAuth, - providerAuthPath: providerSettings.authPath, - providerConfigMode: providerSettings.providerConfigMode, - providerConfigPath: providerSettings.providerConfigPath, - customBaseUrl: providerSettings.customBaseUrl, - customModel: providerSettings.customModel, - taskRunId: taskContinuation?.taskRunId || undefined, - continueSessionId: taskContinuation?.continueSessionId, - openCodeBaselineRawUsageJson: provider === "opencode" - ? taskContinuation?.previousInvocation?.rawUsageJson ?? null - : null, - githubToken: settings.git.githubToken, - agentMcpAccess: workerAgent ? workerClarificationAgentMcpAccess(workerAgent.mcpAccess) : null, - mcpAgentId: workerAgent?.id ?? null, + repairRuntime = this.checkpointRepairRuntime(item, { + ...repairRuntime, + phase: "workspace_ready", }); + const currentWorkspaceHead = (await this.runWorkspaceCommand(finalWorktreePath, "git", ["rev-parse", "HEAD"])).stdout.trim(); + if (!repairRuntime.workspaceBaselineHead) { + repairRuntime = this.checkpointRepairRuntime(item, { + ...repairRuntime, + workspaceBaselineHead: currentWorkspaceHead, + }); + } + initialHead = repairRuntime.workspaceBaselineHead || currentWorkspaceHead; - if (settings.memory?.enabled && settings.memory.autoCaptureSprint) { - await this.captureMemoriesFromWorkspace( - item.projectId, - item.sprintId || undefined, - workerAgent?.id || null, - finalWorktreePath, - item.id, + if (repairRuntime.publicationPhase === "pending") { + const workspaceGuidance = await this.workspaceManager.buildWorkspaceGuidance(item.summaryMarkdown, finalWorktreePath); + const providerPrompt = buildProviderPrompt( + this.buildCiFixPrompt( + item, + branchName, + workspaceGuidance, + workerAgent?.instructionMarkdown, + memoryContext, + memoryInstructions, + ), + providerSettings.thinkingMode, + provider, ); + const previousInvocation = this.latestRepairInvocation(repairRuntime) + || taskContinuation?.previousInvocation + || null; + repairRuntime = this.checkpointRepairRuntime(item, { + ...repairRuntime, + nativeSessionId: previousInvocation?.nativeSessionId || repairRuntime.nativeSessionId, + phase: "provider_running", + }); + const nativeSessionId = await this.runProviderWithRetry({ + provider, + providerPrompt, + workflowSettings: effectiveWorkflowSettings, + repoPath, + worktreePath: finalWorktreePath, + sessionId, + attentionItem: item, + purpose: "ci_fix", + model: providerSettings.model, + thinkingMode: providerSettings.thinkingMode, + apiKey: providerSettings.apiKey, + maxConcurrentTasks: providerSettings.maxConcurrentTasks, + qwenAuthMode: providerSettings.qwenAuthMode, + qwenRegion: providerSettings.qwenRegion, + qwenBaseUrl: providerSettings.qwenBaseUrl, + qwenEnvKey: providerSettings.qwenEnvKey, + qwenModelId: providerSettings.qwenModelId, + qwenProtocol: providerSettings.qwenProtocol, + qwenAdditionalModelProviders: providerSettings.qwenAdditionalModelProviders, + openCodeAuthMode: providerSettings.openCodeAuthMode, + openCodeProviderId: providerSettings.openCodeProviderId, + openCodeModelId: providerSettings.openCodeModelId, + openCodeBaseUrl: providerSettings.openCodeBaseUrl, + openCodeEnvKey: providerSettings.openCodeEnvKey, + openCodePackage: providerSettings.openCodePackage, + providerMountAuth: providerSettings.mountAuth, + providerAuthPath: providerSettings.authPath, + providerConfigMode: providerSettings.providerConfigMode, + providerConfigPath: providerSettings.providerConfigPath, + customBaseUrl: providerSettings.customBaseUrl, + customModel: providerSettings.customModel, + taskRunId: taskContinuation?.taskRunId || undefined, + continueSessionId: repairRuntime.nativeSessionId, + openCodeBaselineRawUsageJson: provider === "opencode" + ? previousInvocation?.rawUsageJson ?? null + : null, + githubToken: settings.git.githubToken, + agentMcpAccess: workerAgent ? workerClarificationAgentMcpAccess(workerAgent.mcpAccess) : null, + mcpAgentId: workerAgent?.id ?? null, + }); + if (nativeSessionId) { + repairRuntime = this.checkpointRepairRuntime(item, { + ...repairRuntime, + nativeSessionId, + phase: "provider_running", + }); + } + + if (settings.memory?.enabled && settings.memory.autoCaptureSprint) { + await this.captureMemoriesFromWorkspace( + item.projectId, + item.sprintId || undefined, + workerAgent?.id || null, + finalWorktreePath, + item.id, + ); + } + const workspaceRepairHead = (await this.runWorkspaceCommand(finalWorktreePath, "git", ["rev-parse", "HEAD"])).stdout.trim(); + repairRuntime = this.checkpointRepairRuntime(item, { + ...repairRuntime, + workspaceRepairHead, + publicationPhase: "workspace_finalized", + }); } - const patchText = await this.workspaceArtifactService.exportBinaryPatch(finalWorktreePath, initialHead); - const applyResult = await this.workspaceArtifactService.applyPatchToBranch({ - repoPath, - baseRef: initialHead, - workerBranch: branchName, - patchText, - commitMessage: `fix(ci): resolve failing checks on ${branchName}`, - gitAuth, - gitIdentity: effectiveWorkflowSettings.containerMountGitConfig - ? undefined - : { - name: effectiveWorkflowSettings.containerGitUserName, - email: effectiveWorkflowSettings.containerGitUserEmail, - }, - githubMode: settings.git.githubMode, - }); - let hasUnpushed = applyResult.hasChanges; - let hasAhead = applyResult.hasChanges; + const repairCommitSubject = `fix(ci): resolve failing checks on ${branchName}`; + let recoveredPublishedHead = repairRuntime.publicationPhase === "host_publishing" + && repairRuntime.workspaceRepairHead !== repairRuntime.workspaceBaselineHead + ? await this.findPublishedRepairCommit({ + repoPath, + workerBranch: branchName, + workspaceRepairHead: repairRuntime.workspaceRepairHead, + githubMode: settings.git.githubMode, + gitAuth, + }) + : null; + if (!recoveredPublishedHead && repairRuntime.publicationPhase === "host_publishing") { + recoveredPublishedHead = await this.findTreeEquivalentPublishedRepair({ + repoPath, + worktreePath: finalWorktreePath, + workerBranch: branchName, + workspaceBaselineHead: repairRuntime.workspaceBaselineHead, + workspaceRepairHead: repairRuntime.workspaceRepairHead, + expectedCommitSubject: repairCommitSubject, + requiredParentRefs: [], + githubMode: settings.git.githubMode, + gitAuth, + }); + } + const alreadyPublished = Boolean(recoveredPublishedHead) + || (repairRuntime.publicationPhase === "host_published" && Boolean(repairRuntime.publishedHeadSha)); + let applyResult: AppliedWorkspacePatchResult = { + hasChanges: alreadyPublished, + commitSha: recoveredPublishedHead || repairRuntime.publishedHeadSha || undefined, + }; + if (!alreadyPublished) { + const patchText = await this.workspaceArtifactService.exportBinaryPatch( + finalWorktreePath, + repairRuntime.workspaceBaselineHead || initialHead, + ); + repairRuntime = this.checkpointRepairRuntime(item, { + ...repairRuntime, + publicationPhase: "host_publishing", + }); + applyResult = await this.workspaceArtifactService.applyPatchToBranch({ + repoPath, + baseRef: repairRuntime.workspaceBaselineHead || initialHead, + workerBranch: branchName, + patchText, + commitMessage: this.buildRepairPublicationCommitMessage( + repairCommitSubject, + repairRuntime.workspaceRepairHead, + ), + gitAuth, + gitIdentity: effectiveWorkflowSettings.containerMountGitConfig + ? undefined + : { + name: effectiveWorkflowSettings.containerGitUserName, + email: effectiveWorkflowSettings.containerGitUserEmail, + }, + githubMode: settings.git.githubMode, + }); + } + let hasUnpushed = alreadyPublished || applyResult.hasChanges; + let hasAhead = alreadyPublished || applyResult.hasChanges; if (!applyResult.hasChanges) { hasUnpushed = await this.prService.hasUnpushedCommits(repoPath, branchName, compareBaseBranch); hasAhead = await this.prService.hasWorkerBranchCommitsAgainstFeature(repoPath, branchName, compareBaseBranch); @@ -1715,6 +2193,11 @@ export class VirtualWorkerService { || ((hasUnpushed || hasAhead) ? (await runCommandStrict("git", ["rev-parse", `refs/heads/${branchName}`], repoPath)).stdout.trim() : initialHead); + repairRuntime = this.checkpointRepairRuntime(item, { + ...repairRuntime, + publicationPhase: "host_published", + publishedHeadSha: headSha, + }); this.deps.sessionTracking.updateSession(sessionId, { state: "COMPLETED" }); this.deps.sessionTracking.appendActivity(sessionId, { originator: "system", @@ -1746,6 +2229,28 @@ export class VirtualWorkerService { succeeded = true; } catch (error) { const message = error instanceof Error ? error.message : String(error); + if (isProviderCancellationError(error)) { + preserveWorkspace = true; + const previousInvocation = this.latestRepairInvocation(repairRuntime); + repairRuntime = this.checkpointRepairRuntime(item, { + ...repairRuntime, + nativeSessionId: previousInvocation?.nativeSessionId || repairRuntime.nativeSessionId, + phase: "interrupted", + }); + this.deps.sessionTracking.updateSession(sessionId, { state: "CANCELLED" }); + this.deps.sessionTracking.appendActivity(sessionId, { + originator: "system", + description: `Virtual worker CI-fix run cancelled before completion: ${message}`, + }); + this.deps.projectAttentionService.requeueItem(item.id, { + repairRuntime, + lastVirtualWorkerError: message, + lastVirtualWorkerInterruptedAt: new Date().toISOString(), + lastVirtualWorkerProvider: provider, + lastVirtualWorkerSessionId: sessionId, + }); + return; + } this.deps.sessionTracking.updateSession(sessionId, { state: "FAILED" }); this.deps.sessionTracking.appendActivity(sessionId, { originator: "system", @@ -1753,8 +2258,17 @@ export class VirtualWorkerService { }); const retryEval = this.deps.guardrailService?.evaluate(guardrailScope, guardrailKey, "ci_fix") ?? null; if (retryEval && (retryEval.allowed || retryEval.action === "WARN_ONLY")) { + preserveWorkspace = true; const now = new Date().toISOString(); this.deps.projectAttentionService.requeueItem(item.id, { + repairRuntime: { + ...repairRuntime, + nativeSessionId: this.latestRepairInvocation(repairRuntime)?.nativeSessionId || repairRuntime.nativeSessionId, + activeAttemptId: null, + attemptRecorded: false, + phase: "interrupted", + updatedAt: now, + }, lastVirtualWorkerError: message, lastVirtualWorkerFailedAt: now, lastVirtualWorkerProvider: provider, @@ -1785,7 +2299,7 @@ export class VirtualWorkerService { item.summaryMarkdown.trim(), ].join("\n")); } finally { - const shouldCleanup = taskContinuation + const shouldCleanup = taskContinuation || preserveWorkspace ? false : succeeded ? workflowSettings.cleanupWorktreeOnSuccess @@ -1952,6 +2466,15 @@ export class VirtualWorkerService { ].filter(Boolean).join("\n"); } + private async workspaceHasMergeInProgress(worktreePath: string): Promise { + try { + const result = await this.runWorkspaceCommand(worktreePath, "git", ["rev-parse", "--verify", "MERGE_HEAD"]); + return result.stdout.trim().length > 0; + } catch { + return false; + } + } + private async runMergeIntoSource(worktreePath: string, targetRef: string, sessionId: string): Promise { try { await this.runWorkspaceCommand(worktreePath, "git", ["merge", "--no-ff", "--no-commit", targetRef]); @@ -2020,7 +2543,7 @@ export class VirtualWorkerService { githubToken: string; agentMcpAccess?: AgentMcpAccessConfig | null; mcpAgentId?: string | null; - }): Promise { + }): Promise { const effectiveModel = resolveEffectiveModel({ provider: args.provider, model: args.model, @@ -2083,6 +2606,7 @@ export class VirtualWorkerService { if (!result.ok) { throw new Error(result.stderr || result.stdout || "Provider failed without output."); } + return result.nativeSessionId || null; } private async isMergeConflictResolvedOnRemote( @@ -2326,6 +2850,18 @@ export class VirtualWorkerService { } } + private async isWorkspaceMergeFinalized(worktreePath: string, targetRef: string): Promise { + if (await this.workspaceHasMergeInProgress(worktreePath)) { + return false; + } + try { + await this.ensureTargetMergedIntoSource(worktreePath, targetRef); + return true; + } catch { + return false; + } + } + private async hasMergeHead(worktreePath: string): Promise { try { await this.runWorkspaceCommand(worktreePath, "git", ["rev-parse", "-q", "--verify", "MERGE_HEAD"]); diff --git a/tests/backend/domain/qa-review/qa-review-budget.test.ts b/tests/backend/domain/qa-review/qa-review-budget.test.ts index 220c3db3c3..2e45bd7c18 100644 --- a/tests/backend/domain/qa-review/qa-review-budget.test.ts +++ b/tests/backend/domain/qa-review/qa-review-budget.test.ts @@ -1,6 +1,7 @@ import { describe, expect, it } from "vitest"; import { evaluateQaReviewBudget, + isPendingQaContinuation, isRecoveredStaleQaRun, shouldVerifyContinuedQaFix, RECOVERED_STALE_QA_SUMMARY_PREFIX, @@ -63,6 +64,30 @@ describe("QA Review Budget", () => { }); }); + describe("isPendingQaContinuation", () => { + const changesRequestedRun = (payload: Record): QaReviewRunRecord => makeRun({ + outcome: "changes_requested", + fixInstructions: "Apply the requested fix.", + payload, + }); + + it("retries a legacy failed continuation that was not marked terminal", () => { + expect(isPendingQaContinuation(changesRequestedRun({ + continuationStatus: "failed", + continuationMode: "failed", + continued: false, + }))).toBe(true); + }); + + it("does not retry a failed continuation after the infrastructure grace ceiling", () => { + expect(isPendingQaContinuation(changesRequestedRun({ + continuationStatus: "failed", + continuationAttemptCount: QA_INFRA_FAILURE_GRACE, + followUpNoProgress: true, + }))).toBe(false); + }); + }); + describe("evaluateQaReviewBudget", () => { it("rejects immediately if QA is disabled (maxTaskReviewRuns <= 0)", () => { const result = evaluateQaReviewBudget({ diff --git a/tests/backend/domain/sprint/orchestrator/cycle-runner.test.ts b/tests/backend/domain/sprint/orchestrator/cycle-runner.test.ts index 352ef37ce3..3f2f6883a3 100644 --- a/tests/backend/domain/sprint/orchestrator/cycle-runner.test.ts +++ b/tests/backend/domain/sprint/orchestrator/cycle-runner.test.ts @@ -3040,6 +3040,85 @@ describe("CycleRunner attention sync", () => { ); }); + it("replays a legacy failed QA fix handoff after restart even when old code changed the task state", async () => { + const deps = buildDeps(); + const reviewCompletedTask = vi.fn().mockResolvedValue({ + reviewed: true, + reopenedTask: true, + mergeBlocked: true, + reportText: "Resumed the pending QA follow-up.", + }); + deps.qualityAssuranceService = { + getTaskMergeGateStatus: vi.fn().mockReturnValue({ + mergeAllowed: false, + reason: "changes_requested", + summary: "QA requested fixes.", + latestRun: { + id: "qa-run-pending-followup", + status: "completed", + outcome: "changes_requested", + fixInstructions: "Address the review findings.", + payload: { + continuationStatus: "failed", + continuationMode: "failed", + continued: false, + }, + }, + runsUsed: 1, + maxRuns: 2, + }), + reviewCompletedTask, + } as any; + deps.getDashboardSettings = vi.fn().mockReturnValue({ + ...DEFAULT_DASHBOARD_SETTINGS, + agents: { + ...DEFAULT_DASHBOARD_SETTINGS.agents, + qualityAssurance: { + ...DEFAULT_DASHBOARD_SETTINGS.agents.qualityAssurance, + enabled: true, + }, + }, + }); + + const runner = new CycleRunner(deps); + const task = { + id: "T1", + record_id: "task-1", + title: "Restarted task", + prompt: "Finish implementation", + depends_on: [], + is_independent: true, + status: "RUNNING", + merge_indicator: "QA_PENDING", + provider: "codex", + session_id: "cli-codex-session-1", + }; + + await (runner as any).reviewCompletedTasks( + [task], + new Map([["T1", "RUNNING"]]), + { + executionContext: { + project: { id: "project-1", name: "Project 1" } as any, + sprint: { id: "sprint-1", name: "Sprint 1" } as any, + sprintNumber: 1, + repoPath: "/repo/project-1", + featureBranch: "feature/sprint-1", + defaultBranch: "main", + }, + repoPath: "/repo/project-1", + sprintRunId: "run-1", + } as any, + deps.getDashboardSettings(), + ); + + expect(reviewCompletedTask).toHaveBeenCalledTimes(1); + expect(reviewCompletedTask).toHaveBeenCalledWith(expect.objectContaining({ + task: expect.objectContaining({ id: "T1", status: "RUNNING" }), + })); + expect(deps.startTask).not.toHaveBeenCalled(); + }); + describe("QA exhaustion policy", () => { const runExhaustedPolicy = async ( exhaustionPolicy: "ESCALATE_TO_HUMAN" | "FAIL_TASK" | "FINISH_TASK", diff --git a/tests/backend/infrastructure/providers/cli/workspace-artifact-service.test.ts b/tests/backend/infrastructure/providers/cli/workspace-artifact-service.test.ts index 53135bf3ce..841622f5e8 100644 --- a/tests/backend/infrastructure/providers/cli/workspace-artifact-service.test.ts +++ b/tests/backend/infrastructure/providers/cli/workspace-artifact-service.test.ts @@ -125,13 +125,54 @@ describe("WorkspaceArtifactService", () => { }, } as IWorkspaceManager; - const patchText = await new WorkspaceArtifactService(workspaceManager) - .exportBinaryPatch("docker-volume://workspace", baseRef); + const service = new WorkspaceArtifactService(workspaceManager); + const patchText = await service.exportBinaryPatch("docker-volume://workspace", baseRef); + const workspaceTree = await service.resolveWorkspaceTree("docker-volume://workspace"); - expect(calls).toHaveLength(1); + expect(calls).toHaveLength(2); expect(calls[0]).toMatchObject({ command: "sh", options: { trimOutput: false } }); + expect(calls[1]).toMatchObject({ command: "sh" }); expect(patchText).toContain("diff --git a/new.txt b/new.txt"); expect(patchText).toContain("diff --git a/tracked.txt b/tracked.txt"); + await runGit(repoPath, ["add", "tracked.txt", "new.txt"]); + expect(workspaceTree).toBe((await runGit(repoPath, ["write-tree"])).trim()); + expect(workspaceTree).not.toBe((await runGit(repoPath, ["rev-parse", `${baseRef}^{tree}`])).trim()); + expect((await fs.readdir(repoPath)).some((entry) => entry.startsWith(".code-ux-export-"))).toBe(false); + }); + + it("resolves the effective host workspace tree from uncommitted tracked and untracked edits", async () => { + const repoPath = await fs.mkdtemp(path.join(os.tmpdir(), "workspace-artifact-tree-")); + cleanupPaths.push(repoPath); + await runGit(repoPath, ["init"]); + await runGit(repoPath, ["config", "user.name", "Code UX Test"]); + await runGit(repoPath, ["config", "user.email", "code-ux@example.com"]); + await fs.writeFile(path.join(repoPath, "tracked.txt"), "base\n", "utf8"); + await runGit(repoPath, ["add", "tracked.txt"]); + await runGit(repoPath, ["commit", "-m", "base"]); + const baselineTree = (await runGit(repoPath, ["rev-parse", "HEAD^{tree}"])).trim(); + await fs.writeFile(path.join(repoPath, "tracked.txt"), "updated\n", "utf8"); + await fs.writeFile(path.join(repoPath, "untracked.txt"), "new\n", "utf8"); + + const workspaceManager = { + runWorkspaceCommand: async ( + _worktreePath: string, + command: string, + args: string[], + options: WorkspaceCommandOptions = {}, + ) => await runCommandStrict(command, args, repoPath, options.env ?? process.env, { + trimOutput: options.trimOutput, + signal: options.signal, + stdinFile: options.stdinFile, + }), + } as IWorkspaceManager; + + const workspaceTree = await new WorkspaceArtifactService(workspaceManager) + .resolveWorkspaceTree(repoPath); + await runGit(repoPath, ["add", "tracked.txt", "untracked.txt"]); + const expectedTree = (await runGit(repoPath, ["write-tree"])).trim(); + + expect(workspaceTree).toBe(expectedTree); + expect(workspaceTree).not.toBe(baselineTree); expect((await fs.readdir(repoPath)).some((entry) => entry.startsWith(".code-ux-export-"))).toBe(false); }); diff --git a/tests/backend/repositories/session-tracking-repository.test.ts b/tests/backend/repositories/session-tracking-repository.test.ts index bfb5de64dc..cd2bed1a16 100644 --- a/tests/backend/repositories/session-tracking-repository.test.ts +++ b/tests/backend/repositories/session-tracking-repository.test.ts @@ -102,6 +102,31 @@ describe("SessionTrackingRepository", () => { expect(repo.getSession("cli-codex-running")?.state).toBe("CANCELLED"); }); + it("tracks and recovers virtual repair sessions without a cli id prefix", async () => { + const repo = await createRepo(); + repo.createSession({ + id: "virtual-cifix-codex-repair-1", + provider: "codex", + state: "RUNNING", + prompt: "repair CI", + title: "CI repair", + workerBranch: "fix/ci", + repoPath: "/tmp/repo-repair", + }); + + expect(repo.listTrackedCliSessions()).toEqual([ + expect.objectContaining({ id: "virtual-cifix-codex-repair-1", state: "RUNNING" }), + ]); + expect(repo.findLatestCliSessionForBranch({ + repoPath: "/tmp/repo-repair", + workerBranch: "fix/ci", + providers: ["codex"], + })).toEqual(expect.objectContaining({ sessionId: "virtual-cifix-codex-repair-1" })); + + expect(repo.recoverInterruptedCliSessions().sessionIds).toEqual(["virtual-cifix-codex-repair-1"]); + expect(repo.getSession("virtual-cifix-codex-repair-1")?.state).toBe("CANCELLED"); + }); + it("finds latest failed cli session for task resume target", async () => { const repo = await createRepo(); diff --git a/tests/backend/scripts/mockup-sprint-pentest-runner.test.ts b/tests/backend/scripts/mockup-sprint-pentest-runner.test.ts index 3e817b3160..6c639fb745 100644 --- a/tests/backend/scripts/mockup-sprint-pentest-runner.test.ts +++ b/tests/backend/scripts/mockup-sprint-pentest-runner.test.ts @@ -1,4 +1,9 @@ import { describe, expect, it } from "vitest"; +import * as fs from "fs/promises"; +import * as os from "os"; +import * as path from "path"; +import { runCommandStrict } from "../../../src/services/cli-process-runner.js"; +import { getScenario } from "../../../scripts/e2e/mockup-sprint-pentest-scenarios.mjs"; type RunnerModule = { isMockupPollStateProgress: (input: { @@ -42,6 +47,19 @@ type RunnerModule = { invocations: Array<{ taskKey: string | null; purpose: string; status: string }>, expectedCounts: Record, ) => void; + selectDuringOrchestrationBranch: ( + branches: string[], + defaultBranch: string, + requireTaskBranch?: boolean, + ) => string | null; + mutateDefaultBranch: ( + repoDir: string, + mutation: { defaultBranch: string; commitMessage: string; files: Array<{ path: string; content: string }> }, + ) => Promise; + assertExpectedMergeConflictInvocation: ( + invocations: Array<{ purpose: string; status: string }>, + createsMergeConflict: boolean, + ) => void; }; const runner = await import("../../../scripts/e2e/run-mockup-sprint-pentest.mjs") as RunnerModule; @@ -62,6 +80,67 @@ describe("mockup sprint pentest runner polling", () => { })).toBe(false); }); + it("waits for a task branch before applying completion-conflict mutations", () => { + expect(runner.selectDuringOrchestrationBranch([ + "main", + "feature/sprint-completion-conflict", + ], "main", true)).toBeNull(); + expect(runner.selectDuringOrchestrationBranch([ + "main", + "feature/sprint-completion-conflict", + "task/feature-sprint-completion-output", + ], "main", true)).toBe("task/feature-sprint-completion-output"); + expect(runner.selectDuringOrchestrationBranch([ + "main", + "feature/sprint-completion-conflict", + ], "main")).toBe("feature/sprint-completion-conflict"); + }); + + it("wires the completion-conflict scenario to a running task and merge invocation assertion", () => { + const scenario = getScenario("completion-merge-conflict"); + expect(scenario?.projectRuns[0]).toMatchObject({ + duringOrchestration: { + waitForRunningTaskCoding: true, + }, + expected: { createsMergeConflict: true }, + }); + }); + + it("synchronizes the checked-out default worktree after fixture mutation", async () => { + const repoDir = await fs.mkdtemp(path.join(os.tmpdir(), "code-ux-mockup-mutation-")); + try { + await runCommandStrict("git", ["init", "-b", "main"], repoDir); + await runCommandStrict("git", ["config", "user.name", "Code UX Test"], repoDir); + await runCommandStrict("git", ["config", "user.email", "code-ux@example.com"], repoDir); + await fs.writeFile(path.join(repoDir, "base.txt"), "base\n", "utf8"); + await runCommandStrict("git", ["add", "."], repoDir); + await runCommandStrict("git", ["commit", "-m", "base"], repoDir); + + await runner.mutateDefaultBranch(repoDir, { + defaultBranch: "main", + commitMessage: "fixture mutation", + files: [{ path: "src/conflict.js", content: "export const value = 'default';\n" }], + }); + + expect((await runCommandStrict("git", ["status", "--porcelain"], repoDir)).stdout.trim()).toBe(""); + expect(await fs.readFile(path.join(repoDir, "src/conflict.js"), "utf8")) + .toBe("export const value = 'default';\n"); + expect((await runCommandStrict("git", ["show", "HEAD:src/conflict.js"], repoDir)).stdout) + .toContain("default"); + } finally { + await fs.rm(repoDir, { recursive: true, force: true }); + } + }); + + it("requires a completed merge-conflict provider invocation when the scenario promises one", () => { + expect(() => runner.assertExpectedMergeConflictInvocation([ + { purpose: "task_coding", status: "completed" }, + ], true)).toThrow("expected at least one completed merge_conflict invocation"); + expect(() => runner.assertExpectedMergeConflictInvocation([ + { purpose: "merge_conflict", status: "completed" }, + ], true)).not.toThrow(); + }); + it("treats task/sprint changes and output readiness changes as observable progress", () => { expect(runner.isMockupPollStateProgress({ progressChanged: true, diff --git a/tests/backend/services/quality-assurance-service.test.ts b/tests/backend/services/quality-assurance-service.test.ts index 062eebe343..ec2bb7c0d6 100644 --- a/tests/backend/services/quality-assurance-service.test.ts +++ b/tests/backend/services/quality-assurance-service.test.ts @@ -14,6 +14,7 @@ import { StructuredProviderResponseService } from "../../../src/services/structu import { StructuredAgentRequestService } from "../../../src/services/structured-agent-request-service.js"; import { DEFAULT_DASHBOARD_SETTINGS } from "../../../src/repositories/settings-defaults.js"; import { buildSprintQaSnapshot } from "../../../src/domain/qa-review/sprint-qa-snapshot.js"; +import { QA_INFRA_FAILURE_GRACE } from "../../../src/domain/qa-review/qa-review-budget.js"; /** Permissive guardrail stub: QA review runs are always allowed unless a test overrides it. */ const qaGuardrailStub = () => ({ @@ -197,6 +198,153 @@ describe("QualityAssuranceService", () => { expect(removeWorktree).toHaveBeenCalledWith("/repo/project", "/repo/project/.worktrees/qa-review-snapshot"); }); + it.each(["task_completion", "sprint_completion"] as const)( + "resumes an interrupted %s reviewer with its correlated session, route, and snapshot", + async (triggerType) => { + const previousRun = { + id: `previous-${triggerType}`, + projectId: "project-1", + sprintId: "sprint-1", + sprintRunId: "sprint-run-1", + taskId: triggerType === "task_completion" ? "task-1" : null, + taskRunId: triggerType === "task_completion" ? "task-run-1" : null, + triggerType, + status: "cancelled", + outcome: null, + runIndex: 1, + agentPresetId: "qa-agent", + agentName: "QA", + targetTaskKey: null, + targetSessionId: null, + targetProvider: null, + summaryMarkdown: "Interrupted", + fixInstructions: null, + payload: { + reviewLogicalSessionId: "cli-qa-review-codex-stable", + reviewSnapshotSessionId: "cli-qa-review-codex-stable-workspace", + reviewWorkspaceSessionId: "cli-qa-review-codex-stable-workspace", + reviewSnapshotWorkspace: "docker-volume://qa-preserved", + reviewProvider: "codex", + reviewProviderConfigId: "codex", + reviewModel: "gpt-stable", + reviewNativeSessionId: "native-stable", + }, + startedAt: "2026-07-14T08:00:00.000Z", + finishedAt: "2026-07-14T08:01:00.000Z", + createdAt: "2026-07-14T08:00:00.000Z", + updatedAt: "2026-07-14T08:01:00.000Z", + } as any; + let currentRun = { + ...previousRun, + id: `current-${triggerType}`, + status: "running", + runIndex: 2, + payload: {}, + finishedAt: null, + } as any; + const qaReviewRepository = { + getRun: vi.fn(() => currentRun), + updateRun: vi.fn((_id: string, update: any) => { + currentRun = { + ...currentRun, + ...update, + payload: update.payload === undefined ? currentRun.payload : update.payload, + }; + return currentRun; + }), + }; + const createExecutionInvocation = vi.fn(() => ({ id: `invocation-${triggerType}` })); + const executeRequest = vi.fn().mockResolvedValue({ + parsed: { + verdict: "pass", + summary: "Looks good.", + findings: [], + fixInstructions: null, + targetTaskKey: null, + shouldHavePr: true, + followUpTasks: [], + raw: {}, + }, + sessionId: "cli-qa-review-codex-stable", + nativeSessionId: "native-resumed", + invocationId: `invocation-${triggerType}`, + openCodeBaselineRawUsageJson: null, + }); + const createSession = vi.fn(); + const service = new QualityAssuranceService({ + projectManagementRepository: {} as any, + executionRepository: { + createExecutionInvocation, + getLatestProviderInvocationUsageBySession: vi.fn().mockReturnValue({ + nativeSessionId: "native-stable", + rawUsageJson: null, + }), + } as any, + guardrailService: qaGuardrailStub(), + sessionTracking: { createSession, updateSession: vi.fn() } as any, + qaReviewRepository: qaReviewRepository as any, + taskService: { + resolveInvocationProvider: () => ({ + provider: "codex", + providerConfigId: "codex", + providers: { codex: { model: "gpt-new", apiKey: "key", thinkingMode: "HIGH" } }, + }), + } as any, + agentPresetSyncService: {} as any, + providerRunner: {} as any, + structuredAgentRequestService: { executeRequest } as any, + getDashboardSettings: () => DEFAULT_DASHBOARD_SETTINGS, + getGithubToken: () => undefined, + sendSessionMessage: async () => ({}), + }); + const createOrReuseSnapshotWorkspace = vi.spyOn( + (service as any).workspaceManager, + "createOrReuseSnapshotWorkspace", + ).mockResolvedValue("docker-volume://qa-preserved"); + vi.spyOn((service as any).workspaceManager, "removeWorktree").mockResolvedValue(undefined); + + await (service as any).runReview({ + triggerType, + scope: { projectId: "project-1", sprintId: "sprint-1" }, + projectName: "QA Project", + sprintGoal: "Ship safely", + repoPath: "/repo/project", + agentInstructions: "Review carefully.", + subtasks: [], + currentTask: triggerType === "task_completion" + ? { id: "T1", title: "Task", prompt: "Prompt", depends_on: [], status: "COMPLETED", is_independent: true } + : null, + taskRun: triggerType === "task_completion" ? { id: "task-run-1", taskId: "task-1" } : null, + sprintRunId: "sprint-run-1", + agentPresetId: "qa-agent", + qaRun: currentRun, + resumeFromRun: previousRun, + reviewBranch: "feature/sprint-1", + baseBranch: "dev", + }); + + expect(createOrReuseSnapshotWorkspace).toHaveBeenCalled(); + expect(createExecutionInvocation).toHaveBeenCalledTimes(1); + expect(executeRequest).toHaveBeenCalledWith(expect.objectContaining({ + invocationId: `invocation-${triggerType}`, + logicalSessionId: "cli-qa-review-codex-stable", + continueSessionId: "native-stable", + workspaceSessionId: "cli-qa-review-codex-stable-workspace", + model: "gpt-stable", + cwd: "docker-volume://qa-preserved", + })); + expect(currentRun.payload).toMatchObject({ + reviewExecutionInvocationId: `invocation-${triggerType}`, + reviewLogicalSessionId: "cli-qa-review-codex-stable", + reviewNativeSessionId: "native-resumed", + reviewContinuationSourceRunId: previousRun.id, + }); + expect(createSession).toHaveBeenCalledWith(expect.objectContaining({ + id: "cli-qa-review-codex-stable-workspace", + })); + }, + ); + it("builds sprint review prompts with the full task instructions", async () => { const service = new QualityAssuranceService({ projectManagementRepository: {} as any, @@ -394,6 +542,179 @@ describe("QualityAssuranceService", () => { expect(tasks[1]?.dependsOnTaskIds).toEqual([task.id]); }); + it("resumes a checkpointed sprint QA coding handoff without rerunning the reviewer", async () => { + const dir = await fs.mkdtemp(path.join(os.tmpdir(), "qa-service-sprint-resume-")); + tempDirs.push(dir); + const storage = new AppDbStorage(path.join(dir, "app.db")); + const projectRepository = new ProjectManagementRepository(storage); + const executionRepository = new ExecutionRepository(storage); + const qaReviewRepository = new QaReviewRepository(storage); + const project = projectRepository.createProject({ name: "QA Project", sourceType: "local", sourceRef: dir }); + const sprint = projectRepository.createSprint(project.id, { + name: "Sprint 1", + goal: "Ship safely", + status: "running", + featureBranch: "feature/sprint-1", + }); + const task = projectRepository.createTask(project.id, { + sprintId: sprint.id, + taskKey: "T1", + title: "Initial task", + promptMarkdown: "Implement the initial feature.", + status: "completed", + isIndependent: true, + provider: "codex", + sessionId: "cli-codex-task-1", + }); + const sprintRun = executionRepository.createSprintRun({ projectId: project.id, sprintId: sprint.id, status: "running" }); + const taskRun = executionRepository.createTaskRun({ + projectId: project.id, + sprintId: sprint.id, + sprintRunId: sprintRun.id, + taskId: task.id, + state: "COMPLETED", + provider: "codex", + sessionId: "cli-codex-task-1", + startedAt: new Date(Date.now() - 60_000).toISOString(), + finishedAt: new Date(Date.now() - 30_000).toISOString(), + }); + const passingPeerRun = qaReviewRepository.createRun({ + projectId: project.id, + sprintId: sprint.id, + sprintRunId: sprintRun.id, + triggerType: "sprint_completion", + runIndex: 1, + payload: { verdict: "pass", summary: "Peer reviewer passed." }, + }); + qaReviewRepository.updateRun(passingPeerRun.id, { + status: "completed", + outcome: "pass", + summaryMarkdown: "Peer reviewer passed.", + finishedAt: new Date().toISOString(), + }); + const qaRun = qaReviewRepository.createRun({ + projectId: project.id, + sprintId: sprint.id, + sprintRunId: sprintRun.id, + triggerType: "sprint_completion", + runIndex: 1, + targetTaskKey: "T1", + targetSessionId: "cli-codex-task-1", + targetProvider: "codex", + payload: { + verdict: "changes_requested", + summary: "One integration fix is required.", + findings: ["Missing integration guard."], + fixInstructions: "Add the integration guard.", + targetTaskKey: "T1", + followUpTasks: [], + continuationStatus: "running", + continuationAttemptCount: 1, + continuationTaskRunId: taskRun.id, + }, + }); + qaReviewRepository.updateRun(qaRun.id, { + status: "completed", + outcome: "changes_requested", + targetTaskKey: "T1", + targetSessionId: "cli-codex-task-1", + targetProvider: "codex", + summaryMarkdown: "One integration fix is required.", + fixInstructions: "Add the integration guard.", + finishedAt: new Date().toISOString(), + }); + const service = new QualityAssuranceService({ + projectManagementRepository: projectRepository, + executionRepository, + guardrailService: qaGuardrailStub(), + sessionTracking: {} as any, + qaReviewRepository, + taskService: {} as any, + agentPresetSyncService: {} as any, + providerRunner: {} as any, + getDashboardSettings: () => ({ + ...DEFAULT_DASHBOARD_SETTINGS, + agents: { + ...DEFAULT_DASHBOARD_SETTINGS.agents, + qualityAssurance: { + ...DEFAULT_DASHBOARD_SETTINGS.agents.qualityAssurance, + enabled: true, + sprintCompletion: { enabled: true, agentPresetId: null }, + maxSprintReviewRuns: 3, + }, + }, + }), + getGithubToken: () => undefined, + sendSessionMessage: async () => ({}), + }); + const runReview = vi.spyOn(service as any, "runReview"); + const requestFixes = vi.spyOn(service as any, "requestFixesForTask") + .mockImplementationOnce(async () => { + expect(qaReviewRepository.getRun(qaRun.id)?.payload).toMatchObject({ + continuationStatus: "running", + continuationAttemptCount: 1, + }); + throw new Error("Command spawner host exited (code=null, signal=SIGHUP)"); + }) + .mockResolvedValueOnce({ applied: true, mode: "cli", noProgress: false, blocker: null }); + const subtasks = [{ + record_id: task.id, + project_id: project.id, + sprint_id: sprint.id, + id: "T1", + title: task.title, + prompt: task.promptMarkdown, + depends_on: [], + is_independent: true, + status: "COMPLETED", + provider: "codex", + session_id: "cli-codex-task-1", + }] as any; + + await expect(service.reviewSprintCompletion({ + projectId: project.id, + sprintId: sprint.id, + sprintRunId: sprintRun.id, + repoPath: dir, + subtasks, + })).rejects.toThrow("Command spawner host exited"); + expect(qaReviewRepository.getRun(qaRun.id)?.payload).toMatchObject({ + continuationStatus: "pending", + continuationAttemptCount: 1, + }); + expect(projectRepository.getTask(task.id)).toMatchObject({ + status: "coding_completed", + isMerged: false, + mergeIndicator: "QA_PENDING", + }); + + const resumed = await service.reviewSprintCompletion({ + projectId: project.id, + sprintId: sprint.id, + sprintRunId: sprintRun.id, + repoPath: dir, + subtasks, + }); + + expect(resumed.blockedCompletion).toBe(true); + expect(runReview).not.toHaveBeenCalled(); + expect(requestFixes).toHaveBeenCalledTimes(2); + expect(requestFixes).toHaveBeenLastCalledWith(expect.objectContaining({ + qaContinuationRunId: qaRun.id, + taskRun: expect.objectContaining({ id: taskRun.id }), + })); + expect(qaReviewRepository.getRun(qaRun.id)?.payload).toMatchObject({ + continuationStatus: "completed", + continuationAttemptCount: 2, + continued: true, + }); + expect(projectRepository.getTask(task.id)).toMatchObject({ + status: "coding_completed", + isMerged: false, + mergeIndicator: "QA_PENDING", + }); + }); + it("does not continue an already merged task during sprint completion QA", async () => { const dir = await fs.mkdtemp(path.join(os.tmpdir(), "qa-service-merged-sprint-followup-")); tempDirs.push(dir); @@ -1610,7 +1931,7 @@ describe("QualityAssuranceService", () => { getGithubToken: () => undefined, sendSessionMessage: async () => ({}), }); - vi.spyOn(service as any, "runReview").mockResolvedValue({ + const runReviewSpy = vi.spyOn(service as any, "runReview").mockResolvedValue({ verdict: "changes_requested", summary: "The task needs one follow-up fix.", findings: [], @@ -1620,12 +1941,13 @@ describe("QualityAssuranceService", () => { followUpTasks: [], raw: { verdict: "changes_requested", summary: "The task needs one follow-up fix." }, }); - vi.spyOn(service as any, "requestFixesForTask").mockImplementation(async () => { + const requestFixesSpy = vi.spyOn(service as any, "requestFixesForTask").mockImplementation(async () => { const latestRun = qaReviewRepository.getLatestTaskRun(task.id); expect(latestRun?.status).toBe("completed"); expect(latestRun?.outcome).toBe("changes_requested"); expect(latestRun?.finishedAt).toBeTruthy(); - return { applied: true, mode: "cli" }; + expect(latestRun?.payload?.continuationStatus).toBe("running"); + return { applied: true, mode: "cli", noProgress: false, blocker: null }; }); const outcome = await service.reviewCompletedTask({ @@ -1652,7 +1974,297 @@ describe("QualityAssuranceService", () => { expect(outcome.reopenedTask).toBe(true); expect(qaReviewRepository.getLatestTaskRun(task.id)?.payload?.continued).toBe(true); + expect(qaReviewRepository.getLatestTaskRun(task.id)?.payload?.continuationStatus).toBe("completed"); expect(qaReviewRepository.getLatestTaskRun(task.id)?.payload?.postExhaustionVerificationEligible).toBe(false); + expect(projectRepository.getTask(task.id)).toMatchObject({ + status: "coding_completed", + mergeIndicator: "QA_PENDING", + }); + + const completedRun = qaReviewRepository.getLatestTaskRun(task.id)!; + qaReviewRepository.updateRun(completedRun.id, { + payload: { + verdict: "changes_requested", + summary: "The task needs one follow-up fix.", + }, + }); + runReviewSpy.mockClear(); + requestFixesSpy.mockClear(); + + const replayOutcome = await service.reviewCompletedTask({ + projectId: project.id, + sprintId: sprint.id, + sprintRunId: sprintRun.id, + repoPath: dir, + task: { + record_id: task.id, + project_id: project.id, + sprint_id: sprint.id, + id: "T1", + title: "Initial task", + prompt: "Implement the initial feature.", + depends_on: [], + is_independent: true, + status: "CODING_COMPLETED", + provider: "opencode", + session_id: "session-1", + pr_url: "https://example.com/pr/1", + }, + subtasks: [], + }); + + expect(replayOutcome.reopenedTask).toBe(true); + expect(runReviewSpy).not.toHaveBeenCalled(); + expect(requestFixesSpy).toHaveBeenCalledTimes(1); + expect(qaReviewRepository.getLatestTaskRun(task.id)?.id).toBe(completedRun.id); + expect(qaReviewRepository.getLatestTaskRun(task.id)?.payload?.continuationStatus).toBe("completed"); + }); + + it("checkpoints failed QA continuations for bounded same-session retry and restores the QA stage", async () => { + const updateTask = vi.fn(); + const originalTaskRun = { + id: "task-run-original", + taskId: "task-1", + sprintRunId: "sprint-run-1", + sessionId: "cli-codex-original", + state: "COMPLETED", + startedAt: "2026-07-14T08:00:00.000Z", + finishedAt: "2026-07-14T08:05:00.000Z", + } as any; + const replacementTaskRun = { + ...originalTaskRun, + id: "task-run-replacement", + sessionId: "cli-codex-replacement", + state: "FAILED", + startedAt: "2026-07-14T08:10:00.000Z", + finishedAt: "2026-07-14T08:11:00.000Z", + } as any; + let storedRun = { + id: "qa-run-1", + projectId: "project-1", + sprintId: "sprint-1", + sprintRunId: "sprint-run-1", + taskId: "task-1", + taskRunId: originalTaskRun.id, + triggerType: "task_completion", + status: "completed", + outcome: "changes_requested", + runIndex: 1, + targetSessionId: "cli-codex-original", + summaryMarkdown: "The task needs a repair.", + fixInstructions: "Repair the implementation.", + payload: { continuationStatus: "pending" }, + startedAt: "2026-07-14T08:06:00.000Z", + finishedAt: "2026-07-14T08:07:00.000Z", + } as any; + const qaReviewRepository = { + getRun: vi.fn(() => storedRun), + updateRun: vi.fn((_id: string, update: Record) => { + storedRun = { ...storedRun, ...update }; + return storedRun; + }), + } as any; + const executionRepository = { + getTaskRun: vi.fn((id: string) => id === originalTaskRun.id ? originalTaskRun : null), + listExecutionInvocations: vi.fn().mockReturnValue([]), + appendTaskRunEvent: vi.fn(), + } as any; + const service = new QualityAssuranceService({ + projectManagementRepository: { updateTask } as any, + executionRepository, + guardrailService: qaGuardrailStub(), + sessionTracking: {} as any, + qaReviewRepository, + taskService: {} as any, + agentPresetSyncService: {} as any, + providerRunner: {} as any, + getDashboardSettings: () => DEFAULT_DASHBOARD_SETTINGS, + getGithubToken: () => undefined, + sendSessionMessage: async () => ({}), + }); + const requestFixesForTask = vi.spyOn(service as any, "requestFixesForTask") + .mockRejectedValue(new Error("transient provider failure")); + const task = { + record_id: "task-1", + id: "T1", + title: "Task", + prompt: "Implement the task.", + depends_on: [], + is_independent: true, + status: "CODING_COMPLETED", + provider: "codex", + session_id: "cli-codex-original", + } as any; + const continuationArgs = { + run: storedRun, + task, + taskRun: originalTaskRun, + repoPath: "/repo/project", + featureBranch: "feature/sprint-1", + scope: { projectId: "project-1", sprintId: "sprint-1" }, + decisiveRuns: 1, + maxTaskReviewRuns: 3, + }; + + await expect((service as any).continuePendingTaskQaRun(continuationArgs)) + .rejects.toThrow("transient provider failure"); + + expect(storedRun.payload).toMatchObject({ + continuationStatus: "pending", + continuationAttemptCount: 1, + continuationMode: "failed", + followUpNoProgress: false, + }); + expect(task).toMatchObject({ status: "CODING_COMPLETED", merge_indicator: "QA_PENDING" }); + expect(updateTask).toHaveBeenLastCalledWith("task-1", expect.objectContaining({ + status: "coding_completed", + mergeIndicator: "QA_PENDING", + })); + + storedRun = { + ...storedRun, + payload: { + ...storedRun.payload, + continuationStatus: "running", + continuationAttemptCount: 1, + }, + }; + task.status = "PENDING"; + await expect((service as any).continuePendingTaskQaRun({ + ...continuationArgs, + run: storedRun, + task, + })).rejects.toThrow("transient provider failure"); + expect(storedRun.payload).toMatchObject({ + continuationStatus: "pending", + continuationAttemptCount: 1, + followUpNoProgress: false, + }); + expect(task).toMatchObject({ status: "CODING_COMPLETED", merge_indicator: "QA_PENDING" }); + + storedRun = { + ...storedRun, + payload: { + ...storedRun.payload, + continuationStatus: "failed", + continuationAttemptCount: QA_INFRA_FAILURE_GRACE - 1, + followUpNoProgress: false, + }, + }; + task.status = "RUNNING"; + task.session_id = "cli-codex-replacement"; + await expect((service as any).continuePendingTaskQaRun({ + ...continuationArgs, + run: storedRun, + task, + taskRun: replacementTaskRun, + })).rejects.toThrow("transient provider failure"); + + expect(requestFixesForTask).toHaveBeenCalledTimes(3); + expect(requestFixesForTask).toHaveBeenLastCalledWith(expect.objectContaining({ + task: expect.objectContaining({ session_id: "cli-codex-original" }), + taskRun: expect.objectContaining({ id: "task-run-original" }), + })); + expect(storedRun.payload).toMatchObject({ + continuationStatus: "failed", + continuationAttemptCount: QA_INFRA_FAILURE_GRACE, + continuationMode: "failed", + followUpNoProgress: true, + followUpBlocker: "transient provider failure", + }); + }); + + it("defers a legacy QA continuation while newer coding is active and reconciles it after completion", async () => { + let storedRun = { + id: "qa-run-legacy", + projectId: "project-1", + sprintId: "sprint-1", + sprintRunId: "sprint-run-1", + taskId: "task-1", + taskRunId: "task-run-original", + triggerType: "task_completion", + status: "completed", + outcome: "changes_requested", + runIndex: 1, + targetSessionId: "cli-codex-original", + summaryMarkdown: "The task needs a repair.", + fixInstructions: "Repair the implementation.", + payload: { continuationStatus: "failed", continuationMode: "failed", continued: false }, + startedAt: "2026-07-14T08:06:00.000Z", + finishedAt: "2026-07-14T08:07:00.000Z", + } as any; + const qaReviewRepository = { + getRun: vi.fn(() => storedRun), + updateRun: vi.fn((_id: string, update: Record) => { + storedRun = { ...storedRun, ...update }; + return storedRun; + }), + } as any; + const updateTask = vi.fn(); + const service = new QualityAssuranceService({ + projectManagementRepository: { updateTask } as any, + executionRepository: { listExecutionInvocations: vi.fn().mockReturnValue([]) } as any, + guardrailService: qaGuardrailStub(), + sessionTracking: {} as any, + qaReviewRepository, + taskService: {} as any, + agentPresetSyncService: {} as any, + providerRunner: {} as any, + getDashboardSettings: () => DEFAULT_DASHBOARD_SETTINGS, + getGithubToken: () => undefined, + sendSessionMessage: async () => ({}), + }); + const requestFixesForTask = vi.spyOn(service as any, "requestFixesForTask"); + const task = { + record_id: "task-1", + id: "T1", + title: "Task", + prompt: "Implement the task.", + depends_on: [], + is_independent: true, + status: "RUNNING", + provider: "codex", + session_id: "cli-codex-replacement", + } as any; + const laterTaskRun = { + id: "task-run-replacement", + taskId: "task-1", + sessionId: "cli-codex-replacement", + state: "RUNNING", + startedAt: "2026-07-14T08:10:00.000Z", + finishedAt: null, + } as any; + const args = { + run: storedRun, + task, + taskRun: laterTaskRun, + repoPath: "/repo/project", + featureBranch: "feature/sprint-1", + scope: { projectId: "project-1", sprintId: "sprint-1" }, + decisiveRuns: 1, + maxTaskReviewRuns: 3, + }; + + const deferred = await (service as any).continuePendingTaskQaRun(args); + expect(deferred).toMatchObject({ reviewed: false, reopenedTask: false, mergeBlocked: true }); + expect(requestFixesForTask).not.toHaveBeenCalled(); + expect(qaReviewRepository.updateRun).not.toHaveBeenCalled(); + + laterTaskRun.state = "COMPLETED"; + laterTaskRun.finishedAt = "2026-07-14T08:15:00.000Z"; + const reconciled = await (service as any).continuePendingTaskQaRun({ ...args, run: storedRun }); + expect(reconciled).toMatchObject({ reviewed: true, reopenedTask: true, mergeBlocked: true }); + expect(requestFixesForTask).not.toHaveBeenCalled(); + expect(storedRun.payload).toMatchObject({ + continuationStatus: "completed", + continuationReconciled: true, + continued: true, + }); + expect(task).toMatchObject({ status: "CODING_COMPLETED", merge_indicator: "QA_PENDING" }); + expect(updateTask).toHaveBeenLastCalledWith("task-1", expect.objectContaining({ + status: "coding_completed", + mergeIndicator: "QA_PENDING", + })); }); it("recovers a running task QA review when the execution invocation never linked provider runtime", async () => { @@ -3236,6 +3848,7 @@ describe("QualityAssuranceService", () => { const updateTaskMock = vi.fn(); const updateTaskRunMock = vi.fn(); + const updateTaskDispatchMock = vi.fn(); const service = new QualityAssuranceService({ projectManagementRepository: { @@ -3250,6 +3863,7 @@ describe("QualityAssuranceService", () => { createProviderInvocationUsage: vi.fn().mockReturnValue({ id: "usage-followup" }), updateProviderInvocationUsage: vi.fn(), updateTaskRun: updateTaskRunMock, + updateTaskDispatch: updateTaskDispatchMock, appendTaskRunEvent: vi.fn(), } as any, guardrailService: qaGuardrailStub(), @@ -3312,6 +3926,8 @@ describe("QualityAssuranceService", () => { const taskRunShape = { id: "task-run-123", + dispatchId: "dispatch-123", + state: "FAILED", workerBranch: null, prUrl: null, }; @@ -3340,6 +3956,12 @@ describe("QualityAssuranceService", () => { { remoteOnly: true }, ); expect(updateTaskRunMock).toHaveBeenCalledWith("task-run-123", { workerBranch: "task/feature-sprint-1-T1-gemini-recovered" }); + expect(updateTaskRunMock).toHaveBeenCalledWith("task-run-123", { state: "COMPLETED" }); + expect(updateTaskDispatchMock).toHaveBeenCalledWith("dispatch-123", { + status: "completed", + errorMessage: null, + }); + expect(taskRunShape.state).toBe("COMPLETED"); expect(taskShape.worker_branch).toBe("task/feature-sprint-1-T1-gemini-recovered"); expect(taskRunShape.workerBranch).toBe("task/feature-sprint-1-T1-gemini-recovered"); }); @@ -3615,6 +4237,107 @@ describe("QualityAssuranceService", () => { expect((service as any).workspaceArtifactService.exportBinaryPatch).toHaveBeenCalledWith("/worktree", "pushed-worker-tip"); }); + it("reuses the durable QA patch baseline when restart happens after the provider committed", async () => { + let storedRun = { + id: "qa-run-publish-crash", + payload: { + continuationStatus: "running", + continuationWorkspaceBaseRef: "original-worker-tip", + }, + } as any; + const qaReviewRepository = { + getRun: vi.fn(() => storedRun), + updateRun: vi.fn((_id: string, update: Record) => { + storedRun = { ...storedRun, ...update }; + return storedRun; + }), + } as any; + const updateTask = vi.fn(); + const service = new QualityAssuranceService({ + projectManagementRepository: { updateTask, getSprint: vi.fn().mockReturnValue(null) } as any, + executionRepository: { + getLatestProviderInvocationUsageBySession: vi.fn().mockReturnValue({ nativeSessionId: "native-followup" }), + updateTaskRun: vi.fn(), + } as any, + guardrailService: qaGuardrailStub(), + sessionTracking: { updateSession: vi.fn(), appendActivity: vi.fn() } as any, + qaReviewRepository, + taskService: {} as any, + agentPresetSyncService: { getOptionalWorkerAgentForRepoPath: vi.fn().mockResolvedValue(undefined) } as any, + providerRunner: {} as any, + getDashboardSettings: () => ({ + ...DEFAULT_DASHBOARD_SETTINGS, + git: { ...DEFAULT_DASHBOARD_SETTINGS.git, autoCreatePr: false }, + memory: { ...DEFAULT_DASHBOARD_SETTINGS.memory, enabled: false }, + }), + getGithubToken: () => undefined, + sendSessionMessage: async () => ({}), + }); + vi.spyOn((service as any).invocationWorkspacePreparer, "resolveContinuationWorkspace").mockResolvedValue({ + worktreePath: "/worktree", + hasPreservedWorkspace: true, + currentBranch: "feature/task-1", + }); + vi.spyOn(service as any, "syncExistingCliFollowUpWorkspace").mockResolvedValue(undefined); + vi.spyOn((service as any).workspaceManager, "buildWorkspaceGuidance").mockResolvedValue(""); + const runWorkspaceCommand = vi.spyOn(service as any, "runWorkspaceCommand").mockImplementation( + async (_path: string, _command: string, args: string[]) => ({ + ok: true, + stdout: args.includes("--verify") ? "original-worker-tip\n" : "provider-created-commit\n", + stderr: "", + code: 0, + }), + ); + vi.spyOn((service as any).providerExecutionService, "executeProvider").mockImplementation(async () => { + expect(updateTask).toHaveBeenLastCalledWith("task-record-1", { + status: "coding_completed", + isMerged: false, + mergeIndicator: "QA_PENDING", + }); + return { + ok: true, + stdout: "", + stderr: "", + text: "already completed before restart", + usageTelemetry: { conversation: [], rawUsageJson: null }, + }; + }); + const exportPatch = vi.spyOn((service as any).workspaceArtifactService, "exportBinaryPatch").mockResolvedValue(""); + vi.spyOn((service as any).workspaceArtifactService, "applyPatchToBranch").mockResolvedValue({ hasChanges: false }); + vi.spyOn((service as any).prService, "hasUnpushedCommits").mockResolvedValue(false); + vi.spyOn((service as any).prService, "hasWorkerBranchCommitsAgainstFeature").mockResolvedValue(true); + vi.spyOn(service as any, "workerBranchAdvancedFromBaseline").mockResolvedValue(true); + + const result = await (service as any).continueCliTaskSession({ + provider: "codex", + sessionId: "cli-codex-task-1", + task: { + id: "T1", + record_id: "task-record-1", + title: "Fix thing", + prompt: "Implement the fix", + depends_on: [], + is_independent: true, + status: "CODING_COMPLETED", + worker_branch: "feature/task-1", + }, + taskRun: null, + repoPath: "/repo", + featureBranch: "feature/sprint-1", + scope: { projectId: "project-1", sprintId: "sprint-1" }, + followUpPrompt: "Address QA findings", + qaContinuationRunId: storedRun.id, + }); + + expect(result.producedMergeWork).toBe(true); + expect(exportPatch).toHaveBeenCalledWith("/worktree", "original-worker-tip"); + expect(runWorkspaceCommand).toHaveBeenCalledWith( + "/worktree", + "git", + ["rev-parse", "--verify", "original-worker-tip^{commit}"], + ); + }); + it("resets stale merged state when a CLI QA follow-up opens a new PR", async () => { const runProvider = vi.fn().mockResolvedValue({ ok: true, @@ -4484,7 +5207,13 @@ describe("QualityAssuranceService", () => { sendSessionMessage: async () => ({}), }); vi.spyOn(service as any, "runReview") - .mockResolvedValueOnce({ verdict: "pass", summary: "A passed.", findings: [], fixInstructions: null, targetTaskKey: null, shouldHavePr: null, followUpTasks: [], raw: { reviewer: "a" } }) + .mockImplementationOnce(async () => { + const precreated = qaReviewRepository.listLatestSprintCycleRuns(sprint.id); + expect(precreated).toHaveLength(2); + expect(precreated.find((run) => run.agentPresetId === qaA.id)?.payload?.reviewDispatchStatus).toBe("running"); + expect(precreated.find((run) => run.agentPresetId === qaB.id)?.payload?.reviewDispatchStatus).toBe("pending"); + return { verdict: "pass", summary: "A passed.", findings: [], fixInstructions: null, targetTaskKey: null, shouldHavePr: null, followUpTasks: [], raw: { reviewer: "a" } }; + }) .mockResolvedValueOnce({ verdict: "pass", summary: "B passed.", findings: [], fixInstructions: null, targetTaskKey: null, shouldHavePr: null, followUpTasks: [], raw: { reviewer: "b" } }); const outcome = await service.reviewSprintCompletion({ @@ -4504,6 +5233,74 @@ describe("QualityAssuranceService", () => { expect(runs.every((run) => run.runIndex === 1 && run.outcome === "pass")).toBe(true); }); + it("continues only the interrupted task reviewer in a partially completed cycle", async () => { + const dir = await fs.mkdtemp(path.join(os.tmpdir(), "qa-service-task-reviewer-recovery-")); + tempDirs.push(dir); + const storage = new AppDbStorage(path.join(dir, "app.db")); + const projectRepository = new ProjectManagementRepository(storage); + const executionRepository = new ExecutionRepository(storage); + const qaReviewRepository = new QaReviewRepository(storage); + const agentPresetRepository = new AgentPresetRepository(storage); + const project = projectRepository.createProject({ name: "QA Project", sourceType: "local", sourceRef: dir }); + const sprint = projectRepository.createSprint(project.id, { name: "Sprint", goal: "Ship", status: "running", featureBranch: "feature/sprint" }); + const task = projectRepository.createTask(project.id, { sprintId: sprint.id, taskKey: "T1", title: "Task", promptMarkdown: "Ship it", status: "coding_completed", isIndependent: true }); + const qaA = agentPresetRepository.createAgentPreset(project.id, { name: "QA A", presetId: "qa-a-recovery", instructionMarkdown: "A" }); + const qaB = agentPresetRepository.createAgentPreset(project.id, { name: "QA B", presetId: "qa-b-recovery", instructionMarkdown: "B" }); + const completed = qaReviewRepository.createRun({ projectId: project.id, sprintId: sprint.id, taskId: task.id, triggerType: "task_completion", runIndex: 1, agentPresetId: qaA.id, agentName: "QA A" }); + qaReviewRepository.updateRun(completed.id, { status: "completed", outcome: "pass", summaryMarkdown: "A passed", payload: { verdict: "pass", summary: "A passed" }, finishedAt: new Date().toISOString() }); + const interrupted = qaReviewRepository.createRun({ projectId: project.id, sprintId: sprint.id, taskId: task.id, triggerType: "task_completion", runIndex: 1, agentPresetId: qaB.id, agentName: "QA B", payload: { reviewLogicalSessionId: "qa-b-session" } }); + qaReviewRepository.updateRun(interrupted.id, { status: "cancelled", summaryMarkdown: "Restarted", finishedAt: new Date().toISOString() }); + const resolveAgent = async (_projectId: string, id: string | null) => ({ id: id!, name: id === qaA.id ? "QA A" : "QA B", instructionMarkdown: "Review" }); + const service = new QualityAssuranceService({ + projectManagementRepository: projectRepository, executionRepository, guardrailService: qaGuardrailStub(), sessionTracking: {} as any, + qaReviewRepository, taskService: {} as any, agentPresetSyncService: { resolveTargetedQualityAssuranceAgent: resolveAgent } as any, + providerRunner: {} as any, getDashboardSettings: () => ({ ...DEFAULT_DASHBOARD_SETTINGS, agents: { ...DEFAULT_DASHBOARD_SETTINGS.agents, qualityAssurance: { ...DEFAULT_DASHBOARD_SETTINGS.agents.qualityAssurance, enabled: true, taskCompletion: { enabled: true, agentPresetIds: [qaA.id, qaB.id], agentPresetId: qaA.id }, maxTaskReviewRuns: 1 } } }), + getGithubToken: () => undefined, sendSessionMessage: async () => ({}), + }); + const runReview = vi.spyOn(service as any, "runReview").mockResolvedValue({ verdict: "pass", summary: "B passed", findings: [], fixInstructions: null, targetTaskKey: null, shouldHavePr: true, followUpTasks: [], raw: { verdict: "pass", summary: "B passed" } }); + vi.spyOn(service as any, "cleanupCliWorkspaceIfNeeded").mockResolvedValue(undefined); + const outcome = await service.reviewCompletedTask({ projectId: project.id, sprintId: sprint.id, repoPath: dir, task: { record_id: task.id, id: "T1", title: "Task", prompt: "Ship it", depends_on: [], is_independent: true, status: "CODING_COMPLETED", pr_url: "https://example.test/1" } as any, subtasks: [] }); + + expect(outcome.reviewed).toBe(true); + expect(runReview).toHaveBeenCalledTimes(1); + expect(runReview).toHaveBeenCalledWith(expect.objectContaining({ agentPresetId: qaB.id, resumeFromRun: expect.objectContaining({ id: interrupted.id }) })); + expect(qaReviewRepository.listLatestTaskCycleRuns(task.id).filter((run) => run.agentPresetId === qaA.id)).toHaveLength(1); + expect(qaReviewRepository.countTaskRuns(task.id)).toBe(1); + }); + + it("fills a missing legacy sprint reviewer without rerunning the completed reviewer", async () => { + const dir = await fs.mkdtemp(path.join(os.tmpdir(), "qa-service-sprint-reviewer-recovery-")); + tempDirs.push(dir); + const storage = new AppDbStorage(path.join(dir, "app.db")); + const projectRepository = new ProjectManagementRepository(storage); + const executionRepository = new ExecutionRepository(storage); + const qaReviewRepository = new QaReviewRepository(storage); + const agentPresetRepository = new AgentPresetRepository(storage); + const project = projectRepository.createProject({ name: "QA Project", sourceType: "local", sourceRef: dir }); + const sprint = projectRepository.createSprint(project.id, { name: "Sprint", goal: "Ship", status: "running", featureBranch: "feature/sprint" }); + const sprintRun = executionRepository.createSprintRun({ projectId: project.id, sprintId: sprint.id, status: "running" }); + const qaA = agentPresetRepository.createAgentPreset(project.id, { name: "QA A", presetId: "sprint-a-recovery", instructionMarkdown: "A" }); + const qaB = agentPresetRepository.createAgentPreset(project.id, { name: "QA B", presetId: "sprint-b-recovery", instructionMarkdown: "B" }); + const legacy = qaReviewRepository.createRun({ projectId: project.id, sprintId: sprint.id, sprintRunId: sprintRun.id, triggerType: "sprint_completion", runIndex: 1, agentPresetId: qaA.id, agentName: "QA A", payload: { verdict: "pass", summary: "A passed", taskSnapshot: [] } }); + qaReviewRepository.updateRun(legacy.id, { status: "completed", outcome: "pass", summaryMarkdown: "A passed", finishedAt: new Date().toISOString() }); + const resolveAgent = async (_projectId: string, id: string | null) => ({ id: id!, name: id === qaA.id ? "QA A" : "QA B", instructionMarkdown: "Review" }); + const service = new QualityAssuranceService({ + projectManagementRepository: projectRepository, executionRepository, guardrailService: qaGuardrailStub(), sessionTracking: {} as any, + qaReviewRepository, taskService: {} as any, agentPresetSyncService: { resolveTargetedQualityAssuranceAgent: resolveAgent } as any, + providerRunner: {} as any, getDashboardSettings: () => ({ ...DEFAULT_DASHBOARD_SETTINGS, agents: { ...DEFAULT_DASHBOARD_SETTINGS.agents, qualityAssurance: { ...DEFAULT_DASHBOARD_SETTINGS.agents.qualityAssurance, enabled: true, sprintCompletion: { enabled: true, agentPresetIds: [qaA.id, qaB.id], agentPresetId: qaA.id }, maxSprintReviewRuns: 1 } } }), + getGithubToken: () => undefined, sendSessionMessage: async () => ({}), + }); + const runReview = vi.spyOn(service as any, "runReview").mockResolvedValue({ verdict: "pass", summary: "B passed", findings: [], fixInstructions: null, targetTaskKey: null, shouldHavePr: null, followUpTasks: [], raw: { verdict: "pass", summary: "B passed" } }); + const outcome = await service.reviewSprintCompletion({ projectId: project.id, sprintId: sprint.id, sprintRunId: sprintRun.id, repoPath: dir, subtasks: [] }); + + expect(outcome.blockedCompletion).toBe(false); + expect(runReview).toHaveBeenCalledTimes(1); + expect(runReview).toHaveBeenCalledWith(expect.objectContaining({ agentPresetId: qaB.id })); + const latest = qaReviewRepository.listLatestSprintCycleRuns(sprint.id); + expect(latest).toHaveLength(2); + expect(latest.every((run) => run.runIndex === 1 && run.outcome === "pass")).toBe(true); + }); + it("resolves one default QA reviewer when no trigger agent IDs are configured", async () => { const dir = await fs.mkdtemp(path.join(os.tmpdir(), "qa-service-default-reviewer-")); tempDirs.push(dir); diff --git a/tests/backend/services/runtime-startup-recovery-service.test.ts b/tests/backend/services/runtime-startup-recovery-service.test.ts index 4de563d8c9..6910cddde5 100644 --- a/tests/backend/services/runtime-startup-recovery-service.test.ts +++ b/tests/backend/services/runtime-startup-recovery-service.test.ts @@ -8,6 +8,7 @@ import { ExecutionRepository } from "../../../src/repositories/execution-reposit import { GuardrailRepository } from "../../../src/repositories/guardrail-repository.js"; import { ProjectAttentionRepository } from "../../../src/repositories/project-attention-repository.js"; import { ProjectWorkerAssignmentRepository } from "../../../src/repositories/project-worker-assignment-repository.js"; +import { WorkerEndpointRepository } from "../../../src/repositories/worker-endpoint-repository.js"; import { QaReviewRepository } from "../../../src/repositories/qa-review-repository.js"; import { SessionTrackingRepository } from "../../../src/repositories/session-tracking-repository.js"; import { DEFAULT_DASHBOARD_SETTINGS } from "../../../src/repositories/settings-defaults.js"; @@ -44,6 +45,7 @@ async function createFixture(options?: { const guardrailRepository = new GuardrailRepository(storage); const projectAttentionRepository = new ProjectAttentionRepository(storage); const projectWorkerAssignmentRepository = new ProjectWorkerAssignmentRepository(storage); + const workerEndpointRepository = new WorkerEndpointRepository(storage); const projectAttentionService = new ProjectAttentionService( projectAttentionRepository, projectWorkerAssignmentRepository, @@ -84,6 +86,7 @@ async function createFixture(options?: { guardrailRepository, projectAttentionRepository, projectAttentionService, + workerEndpointRepository, guardrailService, qaReviewRepository, sessionTracking, @@ -126,6 +129,79 @@ describe("RuntimeStartupRecoveryService", () => { expect(taskCodingOrder).toBeLessThan(providerOrder); }); + it("stops and requeues interrupted virtual repair attention while preserving its checkpoint", async () => { + const removeContainers = vi.fn().mockResolvedValue(undefined); + const { + projectRepository, + projectAttentionService, + workerEndpointRepository, + service, + } = await createFixture({ + dockerService: { + listContainers: vi.fn().mockResolvedValue([{ + id: "repair-container", + labels: { "code-ux.session-id": "virtual-merge-codex-repair-1" }, + }]), + removeContainers, + }, + }); + const project = projectRepository.createProject({ + name: "Repair Recovery Project", + sourceType: "local", + sourceRef: "/workspace/repair-recovery", + }); + const endpoint = workerEndpointRepository.createVirtualEndpoint({ + endpointKey: "virtual:repair-recovery", + displayName: "Virtual repair worker", + status: "connected", + transport: "internal", + capabilities: {}, + }); + const item = projectAttentionService.openItem({ + projectId: project.id, + attentionType: "merge_conflict", + severity: "high", + ownerType: "worker", + title: "Repair conflict", + summaryMarkdown: "Continue the interrupted repair.", + payload: { + repoPath: "/workspace/repair-recovery", + repairRuntime: { + purpose: "merge_conflict", + sessionId: "virtual-merge-codex-repair-1", + workspaceSessionId: "virtual-merge-codex-repair-1", + provider: "codex", + providerConfigId: "codex", + model: "codex-model", + nativeSessionId: "native-repair-1", + activeAttemptId: "attempt-1", + attemptRecorded: true, + phase: "provider_running", + updatedAt: new Date().toISOString(), + }, + }, + }); + projectAttentionService.claimItem(item.id, endpoint.id, "virtual_worker_test"); + + const result = await service.recover(); + + expect(removeContainers).toHaveBeenCalledWith(["repair-container"], { removeVolumes: false }); + expect(result.requeuedInterruptedRepairAttentionItemIds).toEqual([item.id]); + expect(projectAttentionService.getItem(item.id)).toMatchObject({ + status: "open", + assignedWorkerEndpointId: null, + payload: expect.objectContaining({ + repairRuntime: expect.objectContaining({ + sessionId: "virtual-merge-codex-repair-1", + nativeSessionId: "native-repair-1", + activeAttemptId: "attempt-1", + attemptRecorded: true, + }), + repairRecoveryReason: "startup_interrupted_virtual_repair", + }), + }); + }); + it("demotes premature virtual merge-conflict human escalations back to automatic worker attention", async () => { const { projectRepository, diff --git a/tests/backend/services/virtual-worker-service.test.ts b/tests/backend/services/virtual-worker-service.test.ts index 1e2d0a90c3..2595b3cfe0 100644 --- a/tests/backend/services/virtual-worker-service.test.ts +++ b/tests/backend/services/virtual-worker-service.test.ts @@ -1680,6 +1680,8 @@ describe("VirtualWorkerService", () => { hasWorkerBranchCommitsAgainstFeature: vi.fn().mockResolvedValue(true), }; const runCommandSpy = vi.spyOn(cliProcessRunner, "runCommandStrict") + .mockResolvedValueOnce({ ok: true, stdout: "initial-head\n", stderr: "", code: 0 }) + .mockResolvedValueOnce({ ok: true, stdout: "repair-head\n", stderr: "", code: 0 }) .mockResolvedValueOnce({ ok: true, stdout: "", stderr: "", code: 0 }) .mockResolvedValueOnce({ ok: true, stdout: "cafebabe\n", stderr: "", code: 0 }); @@ -2127,7 +2129,7 @@ describe("VirtualWorkerService", () => { "/tmp/wt", "src", "tgt", - undefined, + expect.stringMatching(/^virtual-merge-codex-/), expect.anything(), { remoteOnly: true }, ); @@ -2805,13 +2807,23 @@ describe("VirtualWorkerService", () => { vi.spyOn((virtualWorkerService as any), "isMergeConflictResolvedOnRemote").mockResolvedValue(false); vi.spyOn((virtualWorkerService as any).workspaceManager, "prepareWorktree") .mockRejectedValue(new Error("Command spawner host exited (code=null, signal=SIGHUP)")); - vi.spyOn((virtualWorkerService as any).workspaceManager, "removeWorktree").mockResolvedValue(undefined); + const removeWorktree = vi.spyOn((virtualWorkerService as any).workspaceManager, "removeWorktree").mockResolvedValue(undefined); await (virtualWorkerService as any).handleAttentionItem(endpoint.id, item, "test"); const updatedItem = projectAttentionService.getItem(item.id); - expect(updatedItem?.status).toBe("claimed"); + expect(updatedItem?.status).toBe("open"); + expect(updatedItem?.assignedWorkerEndpointId).toBeNull(); expect(updatedItem?.payload?.workerOutcome).toBeUndefined(); + expect(updatedItem?.payload?.repairRuntime).toEqual(expect.objectContaining({ + purpose: "merge_conflict", + sessionId: expect.stringMatching(/^virtual-merge-codex-/), + workspaceSessionId: expect.stringMatching(/^virtual-merge-codex-/), + activeAttemptId: expect.any(String), + attemptRecorded: true, + phase: "interrupted", + })); + expect(removeWorktree).not.toHaveBeenCalled(); const activeItems = projectAttentionService.listActiveProjectItems(project.id); expect(activeItems.some(i => i.attentionType === "human_escalation_required")).toBe(false); @@ -2820,6 +2832,727 @@ describe("VirtualWorkerService", () => { expect(sessions.find(session => session.id.startsWith("virtual-merge-codex-"))?.state).toBe("CANCELLED"); }); + it("continues a checkpointed worker-owned CI fix without charging the interrupted attempt twice", async () => { + const { + virtualWorkerService, + projectAttentionService, + project, + workerEndpointRepository, + settingsRepository, + } = await setupServiceWithProject(); + settingsRepository.saveProjectSettings(project.id, { + aiProvider: { + providers: { + codex: { model: "gpt-current-after-restart" }, + }, + }, + } as any); + const evaluate = vi.fn(() => ({ + allowed: true, + count: 1, + cap: 5, + action: "BLOCK_AND_ESCALATE" as const, + })); + const record = vi.fn(); + (virtualWorkerService as any).deps.guardrailService = { + evaluate, + record, + reset: vi.fn(), + getCounts: vi.fn(() => ({})), + }; + const endpoint = workerEndpointRepository.createVirtualEndpoint({ + endpointKey: "virtual:continued-ci-fix", + displayName: "Virtual Worker", + status: "connected", + transport: "internal", + capabilities: {}, + }); + const item = projectAttentionService.openItem({ + projectId: project.id, + attentionType: "ci_fix_required", + severity: "high", + ownerType: "worker", + title: "CI Fix", + summaryMarkdown: "Continue the interrupted repair", + payload: { + repoPath: "/test", + branchName: "fix/checkpointed-ci", + repairRuntime: { + purpose: "ci_fix", + sessionId: "virtual-cifix-codex-stable", + workspaceSessionId: "virtual-cifix-codex-workspace", + provider: "codex", + providerConfigId: "codex", + model: "gpt-saved-ci-model", + nativeSessionId: "native-ci-stable", + activeAttemptId: "ci-attempt-stable", + attemptRecorded: true, + phase: "provider_running", + updatedAt: new Date().toISOString(), + }, + }, + }); + vi.spyOn((virtualWorkerService as any).dockerService, "isAvailable").mockResolvedValue(true); + const buildWorkspaceRef = vi.spyOn((virtualWorkerService as any).workspaceManager, "buildWorkspaceRef") + .mockReturnValue("/tmp/continued-ci-fix"); + const prepareWorktree = vi.spyOn((virtualWorkerService as any).workspaceManager, "prepareWorktree") + .mockResolvedValue({ worktreePath: "/tmp/continued-ci-fix", resumed: true }); + vi.spyOn((virtualWorkerService as any).workspaceManager, "buildWorkspaceGuidance").mockResolvedValue("guidance"); + vi.spyOn((virtualWorkerService as any), "runWorkspaceCommand").mockResolvedValue({ + ok: true, + stdout: "initial-head\n", + stderr: "", + code: 0, + }); + const runProvider = vi.spyOn((virtualWorkerService as any), "runProviderWithRetry") + .mockRejectedValue(new Error("Command spawner host exited (code=null, signal=SIGHUP)")); + + await (virtualWorkerService as any).handleAttentionItem(endpoint.id, item, "test"); + + expect(evaluate).toHaveBeenCalledWith( + { projectId: project.id, sprintId: null }, + `main-merge-ci-fix:${item.id}`, + "ci_fix", + ); + expect(record).not.toHaveBeenCalled(); + expect(buildWorkspaceRef).toHaveBeenCalledWith( + "/test", + "virtual-cifix-codex-workspace", + expect.anything(), + ); + expect(prepareWorktree.mock.calls[0]?.[4]).toBe("virtual-cifix-codex-workspace"); + expect(runProvider).toHaveBeenCalledWith(expect.objectContaining({ + provider: "codex", + sessionId: "virtual-cifix-codex-stable", + continueSessionId: "native-ci-stable", + model: "gpt-saved-ci-model", + })); + expect(projectAttentionService.getItem(item.id)).toMatchObject({ + status: "open", + assignedWorkerEndpointId: null, + payload: { + repairRuntime: expect.objectContaining({ + purpose: "ci_fix", + sessionId: "virtual-cifix-codex-stable", + workspaceSessionId: "virtual-cifix-codex-workspace", + provider: "codex", + providerConfigId: "codex", + model: "gpt-saved-ci-model", + nativeSessionId: "native-ci-stable", + activeAttemptId: "ci-attempt-stable", + attemptRecorded: true, + phase: "interrupted", + }), + }, + }); + }); + + it("publishes a checkpointed CI-fix workspace from its original baseline without rerunning the provider", async () => { + const { virtualWorkerService, projectAttentionService, project, workerEndpointRepository } = await setupServiceWithProject(); + const endpoint = workerEndpointRepository.createVirtualEndpoint({ + endpointKey: "virtual:publish-checkpointed-ci-fix", + displayName: "Virtual Worker", + status: "connected", + transport: "internal", + capabilities: {}, + }); + const item = projectAttentionService.openItem({ + projectId: project.id, + attentionType: "ci_fix_required", + severity: "high", + ownerType: "worker", + title: "CI Fix", + summaryMarkdown: "Publish the completed repair", + payload: { + repoPath: "/test", + branchName: "fix/checkpointed-publication", + repairRuntime: { + purpose: "ci_fix", + sessionId: "virtual-cifix-codex-publish", + workspaceSessionId: "virtual-cifix-codex-publish-workspace", + provider: "codex", + providerConfigId: "codex", + model: "gpt-5", + nativeSessionId: "native-ci-publish", + activeAttemptId: "ci-publish-attempt", + attemptRecorded: true, + phase: "interrupted", + workspaceBaselineHead: "ci-original-head", + workspaceRepairHead: "ci-repair-head", + publicationPhase: "workspace_finalized", + publishedHeadSha: null, + updatedAt: new Date().toISOString(), + }, + }, + }); + vi.spyOn((virtualWorkerService as any).dockerService, "isAvailable").mockResolvedValue(true); + vi.spyOn((virtualWorkerService as any).workspaceManager, "buildWorkspaceRef") + .mockReturnValue("/tmp/checkpointed-ci-publication"); + vi.spyOn((virtualWorkerService as any).workspaceManager, "prepareWorktree") + .mockResolvedValue({ worktreePath: "/tmp/checkpointed-ci-publication", resumed: true }); + vi.spyOn((virtualWorkerService as any), "runWorkspaceCommand").mockResolvedValue({ + ok: true, + stdout: "ci-repair-head\n", + stderr: "", + code: 0, + }); + const runProvider = vi.spyOn((virtualWorkerService as any), "runProviderWithRetry"); + const exportBinaryPatch = vi.spyOn((virtualWorkerService as any).workspaceArtifactService, "exportBinaryPatch") + .mockResolvedValue("ci patch"); + const applyPatchToBranch = vi.spyOn((virtualWorkerService as any).workspaceArtifactService, "applyPatchToBranch") + .mockResolvedValue({ hasChanges: true, commitSha: "ci-host-head" }); + + await (virtualWorkerService as any).handleAttentionItem(endpoint.id, item, "test"); + + expect(runProvider).not.toHaveBeenCalled(); + expect(exportBinaryPatch).toHaveBeenCalledWith( + "/tmp/checkpointed-ci-publication", + "ci-original-head", + ); + expect(applyPatchToBranch).toHaveBeenCalledWith(expect.objectContaining({ + baseRef: "ci-original-head", + workerBranch: "fix/checkpointed-publication", + commitMessage: expect.stringContaining("Code-UX-Repair-Head: ci-repair-head"), + })); + expect(projectAttentionService.getItem(item.id)).toMatchObject({ + status: "resolved", + payload: { + repairRuntime: expect.objectContaining({ + workspaceBaselineHead: "ci-original-head", + workspaceRepairHead: "ci-repair-head", + publicationPhase: "host_published", + publishedHeadSha: "ci-host-head", + }), + }, + }); + }); + + it("settles a CI-fix restart from host_publishing when the marked repair commit already exists", async () => { + const { virtualWorkerService, projectAttentionService, project, workerEndpointRepository } = await setupServiceWithProject(); + const endpoint = workerEndpointRepository.createVirtualEndpoint({ + endpointKey: "virtual:ci-host-publishing", + displayName: "Virtual Worker", + status: "connected", + transport: "internal", + capabilities: {}, + }); + const item = projectAttentionService.openItem({ + projectId: project.id, + attentionType: "ci_fix_required", + severity: "high", + ownerType: "worker", + title: "CI Fix", + summaryMarkdown: "Settle the published repair", + payload: { + repoPath: "/test", + branchName: "fix/already-published-ci", + repairRuntime: { + purpose: "ci_fix", + sessionId: "virtual-cifix-host-publishing", + workspaceSessionId: "virtual-cifix-host-publishing-workspace", + provider: "codex", + providerConfigId: "codex", + model: "gpt-5", + nativeSessionId: "native-ci-host-publishing", + activeAttemptId: "ci-host-publishing-attempt", + attemptRecorded: true, + phase: "interrupted", + workspaceBaselineHead: "ci-host-publishing-base", + workspaceRepairHead: "ci-host-publishing-repair", + publicationPhase: "host_publishing", + publishedHeadSha: null, + updatedAt: new Date().toISOString(), + }, + }, + }); + vi.spyOn((virtualWorkerService as any).dockerService, "isAvailable").mockResolvedValue(true); + vi.spyOn((virtualWorkerService as any).workspaceManager, "buildWorkspaceRef") + .mockReturnValue("/tmp/ci-host-publishing"); + vi.spyOn((virtualWorkerService as any).workspaceManager, "prepareWorktree") + .mockResolvedValue({ worktreePath: "/tmp/ci-host-publishing", resumed: true }); + vi.spyOn((virtualWorkerService as any), "runWorkspaceCommand").mockResolvedValue({ + ok: true, + stdout: "ci-host-publishing-repair\n", + stderr: "", + code: 0, + }); + const findPublishedRepairCommit = vi.spyOn((virtualWorkerService as any), "findPublishedRepairCommit") + .mockResolvedValue("ci-already-published-head"); + const runProvider = vi.spyOn((virtualWorkerService as any), "runProviderWithRetry"); + const exportBinaryPatch = vi.spyOn((virtualWorkerService as any).workspaceArtifactService, "exportBinaryPatch"); + const applyPatchToBranch = vi.spyOn((virtualWorkerService as any).workspaceArtifactService, "applyPatchToBranch"); + + await (virtualWorkerService as any).handleAttentionItem(endpoint.id, item, "test"); + + expect(findPublishedRepairCommit).toHaveBeenCalledWith(expect.objectContaining({ + workerBranch: "fix/already-published-ci", + workspaceRepairHead: "ci-host-publishing-repair", + })); + expect(runProvider).not.toHaveBeenCalled(); + expect(exportBinaryPatch).not.toHaveBeenCalled(); + expect(applyPatchToBranch).not.toHaveBeenCalled(); + expect(projectAttentionService.getItem(item.id)).toMatchObject({ + status: "resolved", + payload: { + repairRuntime: expect.objectContaining({ + publicationPhase: "host_published", + publishedHeadSha: "ci-already-published-head", + }), + }, + }); + }); + + it("settles a legacy unmarked equal-head CI publication after the host branch advances", async () => { + const { + virtualWorkerService, + projectAttentionService, + project, + workerEndpointRepository, + settingsRepository, + } = await setupServiceWithProject(); + settingsRepository.saveProjectSettings(project.id, { + git: { githubMode: "LOCAL" }, + } as any); + const endpoint = workerEndpointRepository.createVirtualEndpoint({ + endpointKey: "virtual:legacy-ci-host-publishing", + displayName: "Virtual Worker", + status: "connected", + transport: "internal", + capabilities: {}, + }); + const item = projectAttentionService.openItem({ + projectId: project.id, + attentionType: "ci_fix_required", + severity: "high", + ownerType: "worker", + title: "CI Fix", + summaryMarkdown: "Settle the legacy publication", + payload: { + repoPath: "/test", + branchName: "fix/legacy-published-ci", + repairRuntime: { + purpose: "ci_fix", + sessionId: "virtual-cifix-legacy-host-publishing", + workspaceSessionId: "virtual-cifix-legacy-host-publishing-workspace", + provider: "codex", + providerConfigId: "codex", + model: "gpt-5", + nativeSessionId: "native-ci-legacy-host-publishing", + activeAttemptId: "ci-legacy-host-publishing-attempt", + attemptRecorded: true, + phase: "interrupted", + workspaceBaselineHead: "ci-legacy-base", + workspaceRepairHead: "ci-legacy-base", + publicationPhase: "host_publishing", + publishedHeadSha: null, + updatedAt: new Date().toISOString(), + }, + }, + }); + vi.spyOn((virtualWorkerService as any).dockerService, "isAvailable").mockResolvedValue(true); + vi.spyOn((virtualWorkerService as any).workspaceManager, "buildWorkspaceRef") + .mockReturnValue("/tmp/ci-legacy-host-publishing"); + vi.spyOn((virtualWorkerService as any).workspaceManager, "prepareWorktree") + .mockResolvedValue({ worktreePath: "/tmp/ci-legacy-host-publishing", resumed: true }); + vi.spyOn((virtualWorkerService as any), "runWorkspaceCommand").mockImplementation( + async (_path: string, _command: string, args: string[]) => ({ + ok: true, + stdout: args[1]?.endsWith("^{tree}") ? "ci-baseline-tree\n" : "ci-legacy-base\n", + stderr: "", + code: 0, + }), + ); + vi.spyOn((virtualWorkerService as any).workspaceArtifactService, "resolveWorkspaceTree") + .mockResolvedValue("ci-repair-tree"); + vi.spyOn((virtualWorkerService as any), "findPublishedRepairCommit").mockResolvedValue(null); + const runCommandSpy = vi.spyOn(cliProcessRunner, "runCommandStrict").mockImplementation( + async (_command: string, args: string[]) => { + if (args[0] === "log") { + return { + ok: true, + stdout: [ + "ci-newer-host-head\tci-newer-host-tree\tchore: advance branch", + "ci-unsafe-old-head\tci-repair-tree\tfix(ci): resolve failing checks on fix/legacy-published-ci", + "ci-legacy-host-head\tci-repair-tree\tfix(ci): resolve failing checks on fix/legacy-published-ci", + ].join("\n"), + stderr: "", + code: 0, + }; + } + if (args[0] === "merge-base") { + if (args[3] === "ci-unsafe-old-head") { + throw new Error("candidate predates the saved baseline"); + } + return { ok: true, stdout: "", stderr: "", code: 0 }; + } + throw new Error(`Unexpected Git command: ${args.join(" ")}`); + }, + ); + const runProvider = vi.spyOn((virtualWorkerService as any), "runProviderWithRetry"); + const exportBinaryPatch = vi.spyOn((virtualWorkerService as any).workspaceArtifactService, "exportBinaryPatch"); + const applyPatchToBranch = vi.spyOn((virtualWorkerService as any).workspaceArtifactService, "applyPatchToBranch"); + + await (virtualWorkerService as any).handleAttentionItem(endpoint.id, item, "test"); + runCommandSpy.mockRestore(); + + expect(runProvider).not.toHaveBeenCalled(); + expect(exportBinaryPatch).not.toHaveBeenCalled(); + expect(applyPatchToBranch).not.toHaveBeenCalled(); + expect(projectAttentionService.getItem(item.id)).toMatchObject({ + status: "resolved", + payload: { + repairRuntime: expect.objectContaining({ + publicationPhase: "host_published", + publishedHeadSha: "ci-legacy-host-head", + }), + }, + }); + }); + + it("continues a checkpointed merge repair without charging another attempt", async () => { + const { virtualWorkerService, projectAttentionService, project, workerEndpointRepository } = await setupServiceWithProject(); + const endpoint = workerEndpointRepository.createVirtualEndpoint({ + endpointKey: "virtual:continued-merge", + displayName: "Virtual Worker", + status: "connected", + transport: "internal", + capabilities: {}, + }); + const item = projectAttentionService.openItem({ + projectId: project.id, + attentionType: "merge_conflict", + severity: "high", + ownerType: "worker", + title: "Merge Conflict", + summaryMarkdown: "Continue it", + payload: { + repoPath: "/test", + conflictingBranches: { source: "src", target: "tgt" }, + mergeConflictResolutionAttempts: 1, + repairRuntime: { + purpose: "merge_conflict", + sessionId: "virtual-merge-codex-stable", + workspaceSessionId: "virtual-merge-codex-workspace", + provider: "codex", + providerConfigId: "codex", + model: "gpt-5", + nativeSessionId: "native-merge-stable", + activeAttemptId: "attempt-stable", + attemptRecorded: true, + phase: "provider_running", + updatedAt: new Date().toISOString(), + }, + }, + }); + vi.spyOn((virtualWorkerService as any).dockerService, "isAvailable").mockResolvedValue(true); + vi.spyOn((virtualWorkerService as any), "isMergeConflictResolvedOnRemote").mockResolvedValue(false); + const prepareWorktree = vi.spyOn((virtualWorkerService as any).workspaceManager, "prepareWorktree") + .mockResolvedValue({ worktreePath: "/tmp/continued-merge", resumed: true }); + vi.spyOn((virtualWorkerService as any), "workspaceHasMergeInProgress").mockResolvedValue(false); + vi.spyOn((virtualWorkerService as any), "runMergeIntoSource").mockResolvedValue(true); + vi.spyOn((virtualWorkerService as any), "runWorkspaceCommand").mockResolvedValue({ + ok: true, stdout: "initial-head\n", stderr: "", code: 0, + }); + vi.spyOn((virtualWorkerService as any).workspaceManager, "buildWorkspaceGuidance").mockResolvedValue("guidance"); + const runProvider = vi.spyOn((virtualWorkerService as any), "runProviderWithRetry") + .mockRejectedValue(new Error("Command spawner host exited (code=null, signal=SIGHUP)")); + + await (virtualWorkerService as any).handleAttentionItem(endpoint.id, item, "test"); + + expect(projectAttentionService.getItem(item.id)?.payload?.mergeConflictResolutionAttempts).toBe(1); + expect(prepareWorktree.mock.calls[0]?.[4]).toBe("virtual-merge-codex-workspace"); + expect(runProvider).toHaveBeenCalledWith(expect.objectContaining({ + sessionId: "virtual-merge-codex-stable", + continueSessionId: "native-merge-stable", + model: "gpt-5", + })); + expect(projectAttentionService.getItem(item.id)?.payload?.repairRuntime).toEqual(expect.objectContaining({ + sessionId: "virtual-merge-codex-stable", + workspaceSessionId: "virtual-merge-codex-workspace", + nativeSessionId: "native-merge-stable", + activeAttemptId: "attempt-stable", + attemptRecorded: true, + })); + }); + + it("detects and publishes a merge commit created before restart from the original workspace baseline", async () => { + const { virtualWorkerService, projectAttentionService, project, workerEndpointRepository } = await setupServiceWithProject(); + const endpoint = workerEndpointRepository.createVirtualEndpoint({ + endpointKey: "virtual:publish-checkpointed-merge", + displayName: "Virtual Worker", + status: "connected", + transport: "internal", + capabilities: {}, + }); + const item = projectAttentionService.openItem({ + projectId: project.id, + attentionType: "merge_conflict", + severity: "high", + ownerType: "worker", + title: "Merge Conflict", + summaryMarkdown: "Publish the completed merge", + payload: { + repoPath: "/test", + conflictingBranches: { source: "fix/merge-publication", target: "dev" }, + mergeConflictResolutionAttempts: 1, + repairRuntime: { + purpose: "merge_conflict", + sessionId: "virtual-merge-codex-publish", + workspaceSessionId: "virtual-merge-codex-publish-workspace", + provider: "codex", + providerConfigId: "codex", + model: "gpt-5", + nativeSessionId: "native-merge-publish", + activeAttemptId: "merge-publish-attempt", + attemptRecorded: true, + phase: "provider_running", + workspaceBaselineHead: "merge-original-head", + workspaceRepairHead: null, + publicationPhase: "pending", + publishedHeadSha: null, + updatedAt: new Date().toISOString(), + }, + }, + }); + vi.spyOn((virtualWorkerService as any).dockerService, "isAvailable").mockResolvedValue(true); + vi.spyOn((virtualWorkerService as any), "isMergeConflictResolvedOnRemote").mockResolvedValue(false); + vi.spyOn((virtualWorkerService as any).workspaceManager, "prepareWorktree") + .mockResolvedValue({ worktreePath: "/tmp/checkpointed-merge-publication", resumed: true }); + vi.spyOn((virtualWorkerService as any), "runWorkspaceCommand").mockResolvedValue({ + ok: true, + stdout: "merge-repair-head\n", + stderr: "", + code: 0, + }); + vi.spyOn((virtualWorkerService as any), "workspaceHasMergeInProgress").mockResolvedValue(false); + const ensureTargetMergedIntoSource = vi.spyOn((virtualWorkerService as any), "ensureTargetMergedIntoSource") + .mockResolvedValue(undefined); + const runMergeIntoSource = vi.spyOn((virtualWorkerService as any), "runMergeIntoSource"); + const runProvider = vi.spyOn((virtualWorkerService as any), "runProviderWithRetry"); + const finalizeMergeCommit = vi.spyOn((virtualWorkerService as any), "finalizeMergeCommit"); + const exportBinaryPatch = vi.spyOn((virtualWorkerService as any).workspaceArtifactService, "exportBinaryPatch") + .mockResolvedValue("merge patch"); + const applyPatchToBranch = vi.spyOn((virtualWorkerService as any).workspaceArtifactService, "applyPatchToBranch") + .mockResolvedValue({ hasChanges: true, commitSha: "merge-host-head" }); + + await (virtualWorkerService as any).handleAttentionItem(endpoint.id, item, "test"); + + expect(runMergeIntoSource).not.toHaveBeenCalled(); + expect(runProvider).not.toHaveBeenCalled(); + expect(finalizeMergeCommit).not.toHaveBeenCalled(); + expect(ensureTargetMergedIntoSource).toHaveBeenCalledWith( + "/tmp/checkpointed-merge-publication", + "origin/dev", + ); + expect(exportBinaryPatch).toHaveBeenCalledWith( + "/tmp/checkpointed-merge-publication", + "merge-original-head", + ); + expect(applyPatchToBranch).toHaveBeenCalledWith(expect.objectContaining({ + baseRef: "merge-original-head", + workerBranch: "fix/merge-publication", + forceMergeCommit: true, + commitMessage: expect.stringContaining("Code-UX-Repair-Head: merge-repair-head"), + })); + expect(projectAttentionService.getItem(item.id)).toMatchObject({ + status: "resolved", + payload: { + repairRuntime: expect.objectContaining({ + workspaceBaselineHead: "merge-original-head", + workspaceRepairHead: "merge-repair-head", + publicationPhase: "host_published", + publishedHeadSha: "merge-host-head", + }), + }, + }); + }); + + it("settles a merge restart from host_publishing when the marked repair commit already exists", async () => { + const { virtualWorkerService, projectAttentionService, project, workerEndpointRepository } = await setupServiceWithProject(); + const endpoint = workerEndpointRepository.createVirtualEndpoint({ + endpointKey: "virtual:merge-host-publishing", + displayName: "Virtual Worker", + status: "connected", + transport: "internal", + capabilities: {}, + }); + const item = projectAttentionService.openItem({ + projectId: project.id, + attentionType: "merge_conflict", + severity: "high", + ownerType: "worker", + title: "Merge Conflict", + summaryMarkdown: "Settle the published merge repair", + payload: { + repoPath: "/test", + conflictingBranches: { source: "fix/already-published-merge", target: "dev" }, + mergeConflictResolutionAttempts: 1, + repairRuntime: { + purpose: "merge_conflict", + sessionId: "virtual-merge-host-publishing", + workspaceSessionId: "virtual-merge-host-publishing-workspace", + provider: "codex", + providerConfigId: "codex", + model: "gpt-5", + nativeSessionId: "native-merge-host-publishing", + activeAttemptId: "merge-host-publishing-attempt", + attemptRecorded: true, + phase: "interrupted", + workspaceBaselineHead: "merge-host-publishing-base", + workspaceRepairHead: "merge-host-publishing-repair", + publicationPhase: "host_publishing", + publishedHeadSha: null, + updatedAt: new Date().toISOString(), + }, + }, + }); + vi.spyOn((virtualWorkerService as any).dockerService, "isAvailable").mockResolvedValue(true); + vi.spyOn((virtualWorkerService as any), "isMergeConflictResolvedOnRemote").mockResolvedValue(false); + vi.spyOn((virtualWorkerService as any).workspaceManager, "prepareWorktree") + .mockResolvedValue({ worktreePath: "/tmp/merge-host-publishing", resumed: true }); + vi.spyOn((virtualWorkerService as any), "runWorkspaceCommand").mockResolvedValue({ + ok: true, + stdout: "merge-host-publishing-repair\n", + stderr: "", + code: 0, + }); + const findPublishedRepairCommit = vi.spyOn((virtualWorkerService as any), "findPublishedRepairCommit") + .mockResolvedValue("merge-already-published-head"); + const runMergeIntoSource = vi.spyOn((virtualWorkerService as any), "runMergeIntoSource"); + const runProvider = vi.spyOn((virtualWorkerService as any), "runProviderWithRetry"); + const finalizeMergeCommit = vi.spyOn((virtualWorkerService as any), "finalizeMergeCommit"); + const exportBinaryPatch = vi.spyOn((virtualWorkerService as any).workspaceArtifactService, "exportBinaryPatch"); + const applyPatchToBranch = vi.spyOn((virtualWorkerService as any).workspaceArtifactService, "applyPatchToBranch"); + + await (virtualWorkerService as any).handleAttentionItem(endpoint.id, item, "test"); + + expect(findPublishedRepairCommit).toHaveBeenCalledWith(expect.objectContaining({ + workerBranch: "fix/already-published-merge", + workspaceRepairHead: "merge-host-publishing-repair", + })); + expect(runMergeIntoSource).not.toHaveBeenCalled(); + expect(runProvider).not.toHaveBeenCalled(); + expect(finalizeMergeCommit).not.toHaveBeenCalled(); + expect(exportBinaryPatch).not.toHaveBeenCalled(); + expect(applyPatchToBranch).not.toHaveBeenCalled(); + expect(projectAttentionService.getItem(item.id)).toMatchObject({ + status: "resolved", + payload: { + repairRuntime: expect.objectContaining({ + publicationPhase: "host_published", + publishedHeadSha: "merge-already-published-head", + }), + }, + }); + }); + + it("settles a legacy unmarked merge publication only when its tree and merge parent match", async () => { + const { + virtualWorkerService, + projectAttentionService, + project, + workerEndpointRepository, + settingsRepository, + } = await setupServiceWithProject(); + settingsRepository.saveProjectSettings(project.id, { + git: { githubMode: "LOCAL" }, + } as any); + const endpoint = workerEndpointRepository.createVirtualEndpoint({ + endpointKey: "virtual:legacy-merge-host-publishing", + displayName: "Virtual Worker", + status: "connected", + transport: "internal", + capabilities: {}, + }); + const item = projectAttentionService.openItem({ + projectId: project.id, + attentionType: "merge_conflict", + severity: "high", + ownerType: "worker", + title: "Merge Conflict", + summaryMarkdown: "Settle the legacy merge publication", + payload: { + repoPath: "/test", + conflictingBranches: { source: "fix/legacy-published-merge", target: "dev" }, + mergeConflictResolutionAttempts: 1, + repairRuntime: { + purpose: "merge_conflict", + sessionId: "virtual-merge-legacy-host-publishing", + workspaceSessionId: "virtual-merge-legacy-host-publishing-workspace", + provider: "codex", + providerConfigId: "codex", + model: "gpt-5", + nativeSessionId: "native-merge-legacy-host-publishing", + activeAttemptId: "merge-legacy-host-publishing-attempt", + attemptRecorded: true, + phase: "interrupted", + workspaceBaselineHead: "merge-legacy-base", + workspaceRepairHead: "merge-legacy-repair", + publicationPhase: "host_publishing", + publishedHeadSha: null, + updatedAt: new Date().toISOString(), + }, + }, + }); + vi.spyOn((virtualWorkerService as any).dockerService, "isAvailable").mockResolvedValue(true); + vi.spyOn((virtualWorkerService as any), "isMergeConflictResolvedOnRemote").mockResolvedValue(false); + vi.spyOn((virtualWorkerService as any).workspaceManager, "prepareWorktree") + .mockResolvedValue({ worktreePath: "/tmp/merge-legacy-host-publishing", resumed: true }); + vi.spyOn((virtualWorkerService as any), "runWorkspaceCommand").mockImplementation( + async (_path: string, _command: string, args: string[]) => ({ + ok: true, + stdout: args[1]?.endsWith("^{tree}") ? "merge-baseline-tree\n" : "merge-legacy-repair\n", + stderr: "", + code: 0, + }), + ); + vi.spyOn((virtualWorkerService as any).workspaceArtifactService, "resolveWorkspaceTree") + .mockResolvedValue("merge-repair-tree"); + vi.spyOn((virtualWorkerService as any), "findPublishedRepairCommit").mockResolvedValue(null); + const runCommandSpy = vi.spyOn(cliProcessRunner, "runCommandStrict").mockImplementation( + async (_command: string, args: string[]) => { + if (args[0] === "log") { + return { + ok: true, + stdout: "merge-legacy-host-head\tmerge-repair-tree\tfix(merge): resolve dev into fix/legacy-published-merge\n", + stderr: "", + code: 0, + }; + } + if (args[0] === "merge-base") { + if (args[2] === "fix/legacy-published-merge") { + throw new Error("source is not yet contained in target"); + } + return { ok: true, stdout: "", stderr: "", code: 0 }; + } + throw new Error(`Unexpected Git command: ${args.join(" ")}`); + }, + ); + const runMergeIntoSource = vi.spyOn((virtualWorkerService as any), "runMergeIntoSource"); + const runProvider = vi.spyOn((virtualWorkerService as any), "runProviderWithRetry"); + const finalizeMergeCommit = vi.spyOn((virtualWorkerService as any), "finalizeMergeCommit"); + const exportBinaryPatch = vi.spyOn((virtualWorkerService as any).workspaceArtifactService, "exportBinaryPatch"); + const applyPatchToBranch = vi.spyOn((virtualWorkerService as any).workspaceArtifactService, "applyPatchToBranch"); + + await (virtualWorkerService as any).handleAttentionItem(endpoint.id, item, "test"); + runCommandSpy.mockRestore(); + + expect(runMergeIntoSource).not.toHaveBeenCalled(); + expect(runProvider).not.toHaveBeenCalled(); + expect(finalizeMergeCommit).not.toHaveBeenCalled(); + expect(exportBinaryPatch).not.toHaveBeenCalled(); + expect(applyPatchToBranch).not.toHaveBeenCalled(); + expect(projectAttentionService.getItem(item.id)).toMatchObject({ + status: "resolved", + payload: { + repairRuntime: expect.objectContaining({ + publicationPhase: "host_published", + publishedHeadSha: "merge-legacy-host-head", + }), + }, + }); + }); + it("resolveActionRequiredAttention covers auto-approve plan path", async () => { const { virtualWorkerService, projectAttentionService, project, workerEndpointRepository, settingsRepository } = await setupServiceWithProject(); From 86aa102e9470dbfd37725fa811c0e634920d18c5 Mon Sep 17 00:00:00 2001 From: Pierre Date: Tue, 14 Jul 2026 12:38:57 +0200 Subject: [PATCH 2/3] fix: release repair capacity during recovery --- .../docs/settings-restart-behavior.mdx | 1 + docs-web/settings/restart-behavior.md | 1 + docs/settings/restart-behavior.md | 1 + .../runtime-recovery/qa-review-recovery.ts | 18 +- .../runtime-startup-recovery-service.ts | 79 +++++++- .../runtime-startup-recovery-service.test.ts | 182 +++++++++++++++++- 6 files changed, 264 insertions(+), 18 deletions(-) diff --git a/docs-web/content/docs/settings-restart-behavior.mdx b/docs-web/content/docs/settings-restart-behavior.mdx index bbdd5070f8..78c8261bcc 100644 --- a/docs-web/content/docs/settings-restart-behavior.mdx +++ b/docs-web/content/docs/settings-restart-behavior.mdx @@ -36,6 +36,7 @@ When `restartSprintPolicy = continue` and `restartInvocationPolicy = continue`, - records the original worker-branch baseline before invoking a QA coding follow-up and reuses it after restart, so provider commits made before host-branch publication are still exported and published instead of being mistaken for an empty follow-up - reconciles the recovered coding task-run and dispatch after a successful handoff, preventing an earlier transient failure marker from incorrectly failing the sprint during terminal evaluation - requeues interrupted worker-owned CI-fix and merge-conflict attention, clearing ownership left by the stopped virtual worker +- closes the stopped repair attempt's provider-usage row before requeueing it, so a hard restart cannot leave a stale invocation occupying the provider concurrency limit - resumes those repair workers with the same logical session, native provider session when available, and preserved workspace, so uncommitted repair progress survives the process boundary Recovery closes the interrupted invocation row for auditability and creates a correlated continuation invocation. That terminal audit row does not mean the logical work was abandoned. diff --git a/docs-web/settings/restart-behavior.md b/docs-web/settings/restart-behavior.md index bbdd5070f8..78c8261bcc 100644 --- a/docs-web/settings/restart-behavior.md +++ b/docs-web/settings/restart-behavior.md @@ -36,6 +36,7 @@ When `restartSprintPolicy = continue` and `restartInvocationPolicy = continue`, - records the original worker-branch baseline before invoking a QA coding follow-up and reuses it after restart, so provider commits made before host-branch publication are still exported and published instead of being mistaken for an empty follow-up - reconciles the recovered coding task-run and dispatch after a successful handoff, preventing an earlier transient failure marker from incorrectly failing the sprint during terminal evaluation - requeues interrupted worker-owned CI-fix and merge-conflict attention, clearing ownership left by the stopped virtual worker +- closes the stopped repair attempt's provider-usage row before requeueing it, so a hard restart cannot leave a stale invocation occupying the provider concurrency limit - resumes those repair workers with the same logical session, native provider session when available, and preserved workspace, so uncommitted repair progress survives the process boundary Recovery closes the interrupted invocation row for auditability and creates a correlated continuation invocation. That terminal audit row does not mean the logical work was abandoned. diff --git a/docs/settings/restart-behavior.md b/docs/settings/restart-behavior.md index 52fadf312f..1cf8b9824c 100644 --- a/docs/settings/restart-behavior.md +++ b/docs/settings/restart-behavior.md @@ -36,6 +36,7 @@ When `restartSprintPolicy = continue` and `restartInvocationPolicy = continue`, - records the original worker-branch baseline before invoking a QA coding follow-up and reuses it after restart, so provider commits made before host-branch publication are still exported and published instead of being mistaken for an empty follow-up - reconciles the recovered coding task-run and dispatch after a successful handoff, preventing an earlier transient failure marker from incorrectly failing the sprint during terminal evaluation - requeues interrupted worker-owned CI-fix and merge-conflict attention, clearing ownership left by the stopped virtual worker +- closes the stopped repair attempt's provider-usage row before requeueing it, so a hard restart cannot leave a stale invocation occupying the provider concurrency limit - resumes those repair workers with the same logical session, native provider session when available, and preserved workspace, so uncommitted repair progress survives the process boundary Recovery closes the interrupted invocation row for auditability and creates a correlated continuation invocation. That terminal audit row does not mean the logical work was abandoned. diff --git a/src/services/runtime-recovery/qa-review-recovery.ts b/src/services/runtime-recovery/qa-review-recovery.ts index cbc2c9b4fe..249dea2bac 100644 --- a/src/services/runtime-recovery/qa-review-recovery.ts +++ b/src/services/runtime-recovery/qa-review-recovery.ts @@ -17,6 +17,9 @@ interface QaReviewRecoveryServiceDeps { } export class QaReviewRecoveryService { + private providerContainerInventory: Promise | null = null; + private readonly removedProviderContainerIds = new Set(); + constructor(private readonly deps: QaReviewRecoveryServiceDeps) {} async reconcileInterruptedQaReviewRuns(activeContainerSessionIds: ReadonlySet): Promise { @@ -91,13 +94,22 @@ export class QaReviewRecoveryService { if (!this.deps.dockerService?.removeContainers) { return; } - const containers = await this.deps.dockerService.listContainers().catch(() => []); + this.providerContainerInventory ??= this.deps.dockerService.listContainers().catch(() => []); + const containers = await this.providerContainerInventory; const containerIds = containers .filter((container) => container.labels?.["code-ux.session-id"]?.trim() === sessionId) .map((container) => container.id || container.names) - .filter((containerId): containerId is string => Boolean(containerId)); + .filter((containerId): containerId is string => ( + Boolean(containerId) && !this.removedProviderContainerIds.has(containerId as string) + )); if (containerIds.length > 0) { - await this.deps.dockerService.removeContainers(containerIds, { removeVolumes: false }).catch(() => undefined); + await this.deps.dockerService.removeContainers(containerIds, { removeVolumes: false }) + .then(() => { + for (const containerId of containerIds) { + this.removedProviderContainerIds.add(containerId); + } + }) + .catch(() => undefined); } } diff --git a/src/services/runtime-startup-recovery-service.ts b/src/services/runtime-startup-recovery-service.ts index 86d363aa5c..6b8aa5a208 100644 --- a/src/services/runtime-startup-recovery-service.ts +++ b/src/services/runtime-startup-recovery-service.ts @@ -74,6 +74,7 @@ export interface RuntimeStartupRecoveryResult { restartPolicySyncedPausedSprintIds: string[]; restartPolicySyncedOrphanedSprintIds: string[]; reconciledDuplicateDispatchIds: string[]; + reconciledInterruptedRepairProviderInvocationIds: string[]; requeuedInterruptedRepairAttentionItemIds: string[]; } @@ -146,10 +147,12 @@ export class RuntimeStartupRecoveryService { const reconciledDuplicateDispatchIds = this.reconcileDuplicateActiveTaskDispatches(); const reconciledTaskRunIds = this.reconcileInterruptedTaskRuns(); const reconciledPausedSprintRunIds = this.reconcileStalePausedSprintRuns(); - const requeuedInterruptedRepairAttentionItemIds = restartPolicies.sprintPolicy === "continue" + const interruptedRepairRecovery = restartPolicies.sprintPolicy === "continue" && restartPolicies.invocationPolicy === "continue" ? await this.requeueInterruptedVirtualRepairAttention() - : []; + : { attentionItemIds: [], providerInvocationIds: [] }; + const requeuedInterruptedRepairAttentionItemIds = interruptedRepairRecovery.attentionItemIds; + const reconciledInterruptedRepairProviderInvocationIds = interruptedRepairRecovery.providerInvocationIds; const { resumedSprintRunIds, supersededSprintRunIds } = restartPolicies.sprintPolicy === "continue" ? this.resumeRecoverableSprintRuns() : { resumedSprintRunIds: [], supersededSprintRunIds: [] }; @@ -169,6 +172,7 @@ export class RuntimeStartupRecoveryService { || reconciledTerminalProviderDispatchIds.length > 0 || reconciledTerminalDispatchIds.length > 0 || reconciledDuplicateDispatchIds.length > 0 + || reconciledInterruptedRepairProviderInvocationIds.length > 0 || requeuedInterruptedRepairAttentionItemIds.length > 0 || rehydratedSprintRunIds.length > 0 || reconciledTaskRunIds.length > 0 @@ -195,6 +199,7 @@ export class RuntimeStartupRecoveryService { reconciledTerminalProviderDispatches: reconciledTerminalProviderDispatchIds.length, reconciledTerminalDispatches: reconciledTerminalDispatchIds.length, reconciledDuplicateDispatches: reconciledDuplicateDispatchIds.length, + reconciledInterruptedRepairProviderInvocations: reconciledInterruptedRepairProviderInvocationIds.length, requeuedInterruptedRepairAttentionItems: requeuedInterruptedRepairAttentionItemIds.length, rehydratedSprintRuns: rehydratedSprintRunIds.length, reconciledTaskRuns: reconciledTaskRunIds.length, @@ -228,6 +233,7 @@ export class RuntimeStartupRecoveryService { restartPolicySyncedPausedSprintIds, restartPolicySyncedOrphanedSprintIds, reconciledDuplicateDispatchIds, + reconciledInterruptedRepairProviderInvocationIds, requeuedInterruptedRepairAttentionItemIds, restartPolicyPausedSprintRunIds: restartPolicyResult.pausedSprintRunIds, restartPolicyCancelledSprintRunIds: restartPolicyResult.cancelledSprintRunIds, @@ -236,12 +242,21 @@ export class RuntimeStartupRecoveryService { }; } - private async requeueInterruptedVirtualRepairAttention(): Promise { + private async requeueInterruptedVirtualRepairAttention(): Promise<{ + attentionItemIds: string[]; + providerInvocationIds: string[]; + }> { const projectAttentionService = this.deps.projectAttentionService; if (!projectAttentionService) { - return []; + return { attentionItemIds: [], providerInvocationIds: [] }; } const sessionIds = new Set(); + const repairs: Array<{ + attentionItemId: string; + projectId: string; + purpose: "ci_fix" | "merge_conflict"; + sessionId: string | null; + }> = []; for (const project of this.deps.projectManagementRepository.listProjects().projects) { for (const item of projectAttentionService.listActiveProjectItems(project.id)) { if ( @@ -251,19 +266,67 @@ export class RuntimeStartupRecoveryService { continue; } const runtime = item.payload?.repairRuntime; + let sessionId: string | null = null; if (runtime && typeof runtime === "object") { - const sessionId = (runtime as Record).sessionId; - if (typeof sessionId === "string" && sessionId.trim()) { - sessionIds.add(sessionId.trim()); + const runtimeSessionId = (runtime as Record).sessionId; + if (typeof runtimeSessionId === "string" && runtimeSessionId.trim()) { + sessionId = runtimeSessionId.trim(); + sessionIds.add(sessionId); } } + repairs.push({ + attentionItemId: item.id, + projectId: item.projectId, + purpose: item.attentionType === "ci_fix_required" ? "ci_fix" : "merge_conflict", + sessionId, + }); } } // The old process may have died while Docker kept the provider alive. Stop that // container without deleting its workspace volume before scheduling continuation, // otherwise two providers could mutate the same repair workspace concurrently. await this.removeContainersForSessions(sessionIds); - return projectAttentionService.requeueInterruptedVirtualRepairItems().map((item) => item.id); + + // A hard process stop can leave the old attempt's usage row marked `running` + // even though its virtual worker owner is gone. Close that audit attempt before + // requeueing the attention item so it cannot consume provider capacity forever. + // The logical/native session identifiers remain in repairRuntime and are reused + // by the continuation invocation. + const reconciledAt = new Date().toISOString(); + const providerInvocationIds: string[] = []; + for (const invocation of this.deps.executionRepository.listRunningProviderInvocationUsages()) { + const matchesInterruptedRepair = repairs.some((repair) => ( + invocation.projectId === repair.projectId + && invocation.purpose === repair.purpose + && ( + invocation.attentionItemId === repair.attentionItemId + || ( + !invocation.attentionItemId + && repair.sessionId !== null + && invocation.sessionId === repair.sessionId + ) + ) + )); + if (!matchesInterruptedRepair) { + continue; + } + cancelStaleProviderInvocation( + this.deps.executionRepository, + invocation, + this.deps.executionRepository.listExecutionInvocationsByProviderInvocationId(invocation.id), + { + reconciledAt, + recoveryReason: "startup_interrupted_virtual_repair", + systemMessage: `Closed interrupted ${invocation.purpose} attempt during startup recovery; Code UX will continue its preserved repair session.`, + }, + ); + providerInvocationIds.push(invocation.id); + } + + return { + attentionItemIds: projectAttentionService.requeueInterruptedVirtualRepairItems().map((item) => item.id), + providerInvocationIds, + }; } private async demotePrematureMergeConflictEscalations(): Promise { diff --git a/tests/backend/services/runtime-startup-recovery-service.test.ts b/tests/backend/services/runtime-startup-recovery-service.test.ts index 6910cddde5..354b27ee6b 100644 --- a/tests/backend/services/runtime-startup-recovery-service.test.ts +++ b/tests/backend/services/runtime-startup-recovery-service.test.ts @@ -129,10 +129,26 @@ describe("RuntimeStartupRecoveryService", () => { expect(taskCodingOrder).toBeLessThan(providerOrder); }); - it("stops and requeues interrupted virtual repair attention while preserving its checkpoint", async () => { + it.each([ + { + attentionType: "ci_fix_required" as const, + purpose: "ci_fix" as const, + sessionId: "virtual-cifix-codex-repair-1", + }, + { + attentionType: "merge_conflict" as const, + purpose: "merge_conflict" as const, + sessionId: "virtual-merge-codex-repair-1", + }, + ])("stops and requeues interrupted $attentionType attention while preserving its checkpoint and releasing capacity", async ({ + attentionType, + purpose, + sessionId, + }) => { const removeContainers = vi.fn().mockResolvedValue(undefined); const { projectRepository, + executionRepository, projectAttentionService, workerEndpointRepository, service, @@ -140,7 +156,7 @@ describe("RuntimeStartupRecoveryService", () => { dockerService: { listContainers: vi.fn().mockResolvedValue([{ id: "repair-container", - labels: { "code-ux.session-id": "virtual-merge-codex-repair-1" }, + labels: { "code-ux.session-id": sessionId }, }]), removeContainers, }, @@ -159,7 +175,7 @@ describe("RuntimeStartupRecoveryService", () => { }); const item = projectAttentionService.openItem({ projectId: project.id, - attentionType: "merge_conflict", + attentionType, severity: "high", ownerType: "worker", title: "Repair conflict", @@ -167,9 +183,9 @@ describe("RuntimeStartupRecoveryService", () => { payload: { repoPath: "/workspace/repair-recovery", repairRuntime: { - purpose: "merge_conflict", - sessionId: "virtual-merge-codex-repair-1", - workspaceSessionId: "virtual-merge-codex-repair-1", + purpose, + sessionId, + workspaceSessionId: sessionId, provider: "codex", providerConfigId: "codex", model: "codex-model", @@ -182,17 +198,81 @@ describe("RuntimeStartupRecoveryService", () => { }, }); projectAttentionService.claimItem(item.id, endpoint.id, "virtual_worker_test"); + const providerInvocation = executionRepository.createProviderInvocationUsage({ + projectId: project.id, + attentionItemId: item.id, + sessionId, + provider: "codex", + purpose, + status: "running", + startedAt: "2026-07-14T10:00:00.000Z", + }); + const legacyProviderInvocation = executionRepository.createProviderInvocationUsage({ + projectId: project.id, + sessionId, + provider: "codex", + purpose, + status: "running", + startedAt: "2026-07-14T10:00:01.000Z", + }); + const unrelatedProject = projectRepository.createProject({ + name: "Unrelated Repair Recovery Project", + sourceType: "local", + sourceRef: "/workspace/unrelated-repair-recovery", + }); + const unrelatedProviderInvocation = executionRepository.createProviderInvocationUsage({ + projectId: unrelatedProject.id, + sessionId, + provider: "codex", + purpose, + status: "running", + startedAt: "2026-07-14T10:00:02.000Z", + }); + const executionInvocation = executionRepository.createExecutionInvocation({ + projectId: project.id, + attentionItemId: item.id, + providerInvocationId: providerInvocation.id, + type: purpose, + provider: "codex", + status: "running", + startedAt: "2026-07-14T10:00:00.000Z", + }); const result = await service.recover(); expect(removeContainers).toHaveBeenCalledWith(["repair-container"], { removeVolumes: false }); + expect(result.reconciledInterruptedRepairProviderInvocationIds).toHaveLength(2); + expect(result.reconciledInterruptedRepairProviderInvocationIds).toEqual(expect.arrayContaining([ + providerInvocation.id, + legacyProviderInvocation.id, + ])); expect(result.requeuedInterruptedRepairAttentionItemIds).toEqual([item.id]); + expect(executionRepository.getProviderInvocationUsage(providerInvocation.id)).toMatchObject({ + status: "cancelled", + finishedAt: expect.any(String), + }); + expect(executionRepository.getProviderInvocationUsage(legacyProviderInvocation.id)).toMatchObject({ + status: "cancelled", + finishedAt: expect.any(String), + }); + expect(executionRepository.getExecutionInvocation(executionInvocation.id)).toMatchObject({ + status: "cancelled", + finishedAt: expect.any(String), + errorMessage: null, + }); + expect(executionRepository.getProviderInvocationUsage(unrelatedProviderInvocation.id)).toMatchObject({ + status: "running", + finishedAt: null, + }); + expect(executionRepository.listRunningProviderInvocationUsages(["codex"]).map((invocation) => invocation.id)).toEqual([ + unrelatedProviderInvocation.id, + ]); expect(projectAttentionService.getItem(item.id)).toMatchObject({ status: "open", assignedWorkerEndpointId: null, payload: expect.objectContaining({ repairRuntime: expect.objectContaining({ - sessionId: "virtual-merge-codex-repair-1", + sessionId, nativeSessionId: "native-repair-1", activeAttemptId: "attempt-1", attemptRecorded: true, @@ -601,6 +681,94 @@ describe("RuntimeStartupRecoveryService", () => { }); }); + it("reuses one Docker inventory while cancelling multiple interrupted QA reviewers", async () => { + const listContainers = vi.fn().mockResolvedValue([ + { id: "qa-container-1", labels: { "code-ux.session-id": "qa-review-session-1" } }, + { id: "qa-container-2", labels: { "code-ux.session-id": "qa-review-session-2" } }, + ]); + const removeContainers = vi.fn().mockResolvedValue(undefined); + const { + projectRepository, + executionRepository, + qaReviewRepository, + } = await createFixture(); + const project = projectRepository.createProject({ + name: "Batched QA Recovery Project", + sourceType: "local", + sourceRef: "/workspace/batched-qa-recovery", + }); + const sprint = projectRepository.createSprint(project.id, { + name: "Batched QA Recovery Sprint", + number: 9, + status: "running", + }); + const sprintRun = executionRepository.createSprintRun({ + projectId: project.id, + sprintId: sprint.id, + executorMode: "docker_cli", + status: "running", + }); + + const qaRunIds: string[] = []; + const providerInvocationIds: string[] = []; + for (const index of [1, 2]) { + const sessionId = `qa-review-session-${index}`; + const providerInvocation = executionRepository.createProviderInvocationUsage({ + projectId: project.id, + sprintId: sprint.id, + sprintRunId: sprintRun.id, + sessionId, + provider: "codex", + purpose: "qa_review", + executionMode: "DOCKER", + status: "running", + startedAt: `2026-07-14T10:00:0${index}.000Z`, + }); + const executionInvocation = executionRepository.createExecutionInvocation({ + projectId: project.id, + sprintId: sprint.id, + sprintRunId: sprintRun.id, + providerInvocationId: providerInvocation.id, + type: "qa_review", + provider: "codex", + status: "running", + startedAt: `2026-07-14T10:00:0${index}.000Z`, + }); + const qaRun = qaReviewRepository.createRun({ + projectId: project.id, + sprintId: sprint.id, + sprintRunId: sprintRun.id, + triggerType: "sprint_completion", + runIndex: index, + payload: { reviewExecutionInvocationId: executionInvocation.id }, + startedAt: `2026-07-14T10:00:0${index}.000Z`, + }); + qaRunIds.push(qaRun.id); + providerInvocationIds.push(providerInvocation.id); + } + + const qaRecovery = new QaReviewRecoveryService({ + executionRepository, + qaReviewRepository, + dockerService: { listContainers, removeContainers }, + }); + const reconciledRunIds = await qaRecovery.reconcileInterruptedQaReviewRuns(new Set([ + "qa-review-session-1", + "qa-review-session-2", + ])); + + expect(reconciledRunIds).toHaveLength(2); + expect(reconciledRunIds).toEqual(expect.arrayContaining(qaRunIds)); + expect(listContainers).toHaveBeenCalledTimes(1); + expect(removeContainers).toHaveBeenCalledTimes(2); + expect(removeContainers).toHaveBeenCalledWith(["qa-container-1"], { removeVolumes: false }); + expect(removeContainers).toHaveBeenCalledWith(["qa-container-2"], { removeVolumes: false }); + expect(providerInvocationIds.map((id) => executionRepository.getProviderInvocationUsage(id)?.status)).toEqual([ + "cancelled", + "cancelled", + ]); + }); + it("immediately cancels a fresh QA review row with no backing invocation after restart", async () => { const { projectRepository, From 3c70767028f919bad200819ab5724d1587f739a3 Mon Sep 17 00:00:00 2001 From: Pierre Date: Tue, 14 Jul 2026 12:55:10 +0200 Subject: [PATCH 3/3] fix: reconcile completed repairs during restart --- .../docs/settings-restart-behavior.mdx | 4 +- docs-web/settings/restart-behavior.md | 4 +- docs/settings/restart-behavior.md | 4 +- .../runtime/provider-invocation-recovery.ts | 20 ++- src/services/provider-execution-service.ts | 12 +- .../runtime-startup-recovery-service.ts | 57 ++++++-- .../provider-execution-service.test.ts | 30 +++++ .../runtime-startup-recovery-service.test.ts | 123 ++++++++++++++++++ 8 files changed, 229 insertions(+), 25 deletions(-) diff --git a/docs-web/content/docs/settings-restart-behavior.mdx b/docs-web/content/docs/settings-restart-behavior.mdx index 78c8261bcc..c7b5cd4c62 100644 --- a/docs-web/content/docs/settings-restart-behavior.mdx +++ b/docs-web/content/docs/settings-restart-behavior.mdx @@ -36,10 +36,10 @@ When `restartSprintPolicy = continue` and `restartInvocationPolicy = continue`, - records the original worker-branch baseline before invoking a QA coding follow-up and reuses it after restart, so provider commits made before host-branch publication are still exported and published instead of being mistaken for an empty follow-up - reconciles the recovered coding task-run and dispatch after a successful handoff, preventing an earlier transient failure marker from incorrectly failing the sprint during terminal evaluation - requeues interrupted worker-owned CI-fix and merge-conflict attention, clearing ownership left by the stopped virtual worker -- closes the stopped repair attempt's provider-usage row before requeueing it, so a hard restart cannot leave a stale invocation occupying the provider concurrency limit +- closes the stopped repair attempt's provider-usage row before requeueing it, so a hard restart cannot leave a stale invocation occupying the provider concurrency limit. A durable `workspace_finalized`, `host_publishing`, or `host_published` checkpoint proves that the provider returned successfully, so recovery records that attempt as completed; an attempt interrupted before that boundary is recorded as cancelled. - resumes those repair workers with the same logical session, native provider session when available, and preserved workspace, so uncommitted repair progress survives the process boundary -Recovery closes the interrupted invocation row for auditability and creates a correlated continuation invocation. That terminal audit row does not mean the logical work was abandoned. +Recovery creates a correlated continuation invocation only when provider work was interrupted. When provider work already completed, recovery continues publication or attention finalization from the durable checkpoint without calling the provider again. A cancelled audit attempt therefore does not mean the logical work was abandoned, while a completed attempt remains visible as completed across the restart. ## Recommended Configuration diff --git a/docs-web/settings/restart-behavior.md b/docs-web/settings/restart-behavior.md index 78c8261bcc..c7b5cd4c62 100644 --- a/docs-web/settings/restart-behavior.md +++ b/docs-web/settings/restart-behavior.md @@ -36,10 +36,10 @@ When `restartSprintPolicy = continue` and `restartInvocationPolicy = continue`, - records the original worker-branch baseline before invoking a QA coding follow-up and reuses it after restart, so provider commits made before host-branch publication are still exported and published instead of being mistaken for an empty follow-up - reconciles the recovered coding task-run and dispatch after a successful handoff, preventing an earlier transient failure marker from incorrectly failing the sprint during terminal evaluation - requeues interrupted worker-owned CI-fix and merge-conflict attention, clearing ownership left by the stopped virtual worker -- closes the stopped repair attempt's provider-usage row before requeueing it, so a hard restart cannot leave a stale invocation occupying the provider concurrency limit +- closes the stopped repair attempt's provider-usage row before requeueing it, so a hard restart cannot leave a stale invocation occupying the provider concurrency limit. A durable `workspace_finalized`, `host_publishing`, or `host_published` checkpoint proves that the provider returned successfully, so recovery records that attempt as completed; an attempt interrupted before that boundary is recorded as cancelled. - resumes those repair workers with the same logical session, native provider session when available, and preserved workspace, so uncommitted repair progress survives the process boundary -Recovery closes the interrupted invocation row for auditability and creates a correlated continuation invocation. That terminal audit row does not mean the logical work was abandoned. +Recovery creates a correlated continuation invocation only when provider work was interrupted. When provider work already completed, recovery continues publication or attention finalization from the durable checkpoint without calling the provider again. A cancelled audit attempt therefore does not mean the logical work was abandoned, while a completed attempt remains visible as completed across the restart. ## Recommended Configuration diff --git a/docs/settings/restart-behavior.md b/docs/settings/restart-behavior.md index 1cf8b9824c..c5ab48cea1 100644 --- a/docs/settings/restart-behavior.md +++ b/docs/settings/restart-behavior.md @@ -36,10 +36,10 @@ When `restartSprintPolicy = continue` and `restartInvocationPolicy = continue`, - records the original worker-branch baseline before invoking a QA coding follow-up and reuses it after restart, so provider commits made before host-branch publication are still exported and published instead of being mistaken for an empty follow-up - reconciles the recovered coding task-run and dispatch after a successful handoff, preventing an earlier transient failure marker from incorrectly failing the sprint during terminal evaluation - requeues interrupted worker-owned CI-fix and merge-conflict attention, clearing ownership left by the stopped virtual worker -- closes the stopped repair attempt's provider-usage row before requeueing it, so a hard restart cannot leave a stale invocation occupying the provider concurrency limit +- closes the stopped repair attempt's provider-usage row before requeueing it, so a hard restart cannot leave a stale invocation occupying the provider concurrency limit. A durable `workspace_finalized`, `host_publishing`, or `host_published` checkpoint proves that the provider returned successfully, so recovery records that attempt as completed; an attempt interrupted before that boundary is recorded as cancelled. - resumes those repair workers with the same logical session, native provider session when available, and preserved workspace, so uncommitted repair progress survives the process boundary -Recovery closes the interrupted invocation row for auditability and creates a correlated continuation invocation. That terminal audit row does not mean the logical work was abandoned. +Recovery creates a correlated continuation invocation only when provider work was interrupted. When provider work already completed, recovery continues publication or attention finalization from the durable checkpoint without calling the provider again. A cancelled audit attempt therefore does not mean the logical work was abandoned, while a completed attempt remains visible as completed across the restart. ## Recommended Configuration diff --git a/src/domain/runtime/provider-invocation-recovery.ts b/src/domain/runtime/provider-invocation-recovery.ts index 950d973404..52e59171f9 100644 --- a/src/domain/runtime/provider-invocation-recovery.ts +++ b/src/domain/runtime/provider-invocation-recovery.ts @@ -48,13 +48,29 @@ export function cancelStaleProviderInvocation( ); } +export function completeStaleProviderInvocation( + executionRepository: ExecutionRepository, + providerInvocation: ProviderInvocationUsageRecord, + linkedInvocations: ExecutionInvocationRecord[], + context: ProviderInvocationRecoveryContext +): void { + finalizeStaleProviderInvocation( + executionRepository, + providerInvocation, + linkedInvocations, + context, + "completed", + "completed", + ); +} + function finalizeStaleProviderInvocation( executionRepository: ExecutionRepository, providerInvocation: ProviderInvocationUsageRecord, linkedInvocations: ExecutionInvocationRecord[], context: ProviderInvocationRecoveryContext, - providerStatus: Extract, - executionStatus: Extract, + providerStatus: Extract, + executionStatus: Extract, ): void { const durationMs = calculateInvocationDurationMs(providerInvocation, context.reconciledAt); diff --git a/src/services/provider-execution-service.ts b/src/services/provider-execution-service.ts index c45fbf9862..6297ea84fe 100644 --- a/src/services/provider-execution-service.ts +++ b/src/services/provider-execution-service.ts @@ -720,11 +720,17 @@ export class ProviderExecutionService { if (invocation && this.deps.executionRepository) { const finishedAt = new Date().toISOString(); const durationMs = Date.now() - startedMs; + // A successful runner result is authoritative even if shutdown began while + // the result was being returned. Preserve `running` for startup recovery only + // when shutdown interrupts without a terminal result; otherwise a completed + // repair can be published while its provider audit row remains stuck running. const shouldPersistTerminalUsage = this.isProviderWorkStillRunning(invocation.id, execInvocationId) - && !isServerShutdownAbort(args.signal); + && (result.ok || !isServerShutdownAbort(args.signal)); if (shouldPersistTerminalUsage) { this.deps.executionRepository.updateProviderInvocationUsage(invocation.id, { - status: (args.signal?.aborted || isRuntimeShutdownInProgress()) ? "cancelled" : (result.ok ? "completed" : "failed"), + status: result.ok + ? "completed" + : (args.signal?.aborted || isRuntimeShutdownInProgress()) ? "cancelled" : "failed", model: effectiveModel, nativeSessionId: result.nativeSessionId ? args.redactTextForPersistence?.(result.nativeSessionId) ?? result.nativeSessionId @@ -827,7 +833,7 @@ export class ProviderExecutionService { } if (providerResult.ok) { - if (execInvocationId && this.isExecutionInvocationStillRunning(execInvocationId) && !isRuntimeShutdownInProgress()) { + if (execInvocationId && this.isExecutionInvocationStillRunning(execInvocationId)) { if (args.finalizeExecutionInvocation !== false) { this.deps.executionRepository?.updateExecutionInvocation(execInvocationId, { status: "completed", diff --git a/src/services/runtime-startup-recovery-service.ts b/src/services/runtime-startup-recovery-service.ts index 6b8aa5a208..20ffb1b331 100644 --- a/src/services/runtime-startup-recovery-service.ts +++ b/src/services/runtime-startup-recovery-service.ts @@ -20,7 +20,11 @@ import { sanitizeToken } from "./cli-workflow-utils.js"; import { QaReviewRecoveryService } from "./runtime-recovery/qa-review-recovery.js"; import { InvocationRecoveryService } from "./runtime-recovery/invocation-recovery.js"; import { calculateInvocationDurationMs, isTerminalTaskRunState } from "./runtime-recovery/recovery-utils.js"; -import { cancelStaleProviderInvocation, failStaleProviderInvocation } from "../domain/runtime/provider-invocation-recovery.js"; +import { + cancelStaleProviderInvocation, + completeStaleProviderInvocation, + failStaleProviderInvocation, +} from "../domain/runtime/provider-invocation-recovery.js"; import type { GuardrailService } from "./guardrail-service.js"; import type { SprintRunLifecycleService } from "./sprint-run-lifecycle-service.js"; import { runCommandStrict } from "./cli-process-runner.js"; @@ -256,6 +260,8 @@ export class RuntimeStartupRecoveryService { projectId: string; purpose: "ci_fix" | "merge_conflict"; sessionId: string | null; + providerWorkCompleted: boolean; + publicationPhase: string | null; }> = []; for (const project of this.deps.projectManagementRepository.listProjects().projects) { for (const item of projectAttentionService.listActiveProjectItems(project.id)) { @@ -267,18 +273,27 @@ export class RuntimeStartupRecoveryService { } const runtime = item.payload?.repairRuntime; let sessionId: string | null = null; + let publicationPhase: string | null = null; if (runtime && typeof runtime === "object") { - const runtimeSessionId = (runtime as Record).sessionId; + const runtimeRecord = runtime as Record; + const runtimeSessionId = runtimeRecord.sessionId; if (typeof runtimeSessionId === "string" && runtimeSessionId.trim()) { sessionId = runtimeSessionId.trim(); sessionIds.add(sessionId); } + publicationPhase = typeof runtimeRecord.publicationPhase === "string" + ? runtimeRecord.publicationPhase + : null; } repairs.push({ attentionItemId: item.id, projectId: item.projectId, purpose: item.attentionType === "ci_fix_required" ? "ci_fix" : "merge_conflict", sessionId, + providerWorkCompleted: publicationPhase === "workspace_finalized" + || publicationPhase === "host_publishing" + || publicationPhase === "host_published", + publicationPhase, }); } } @@ -295,7 +310,7 @@ export class RuntimeStartupRecoveryService { const reconciledAt = new Date().toISOString(); const providerInvocationIds: string[] = []; for (const invocation of this.deps.executionRepository.listRunningProviderInvocationUsages()) { - const matchesInterruptedRepair = repairs.some((repair) => ( + const matchingRepair = repairs.find((repair) => ( invocation.projectId === repair.projectId && invocation.purpose === repair.purpose && ( @@ -307,19 +322,33 @@ export class RuntimeStartupRecoveryService { ) ) )); - if (!matchesInterruptedRepair) { + if (!matchingRepair) { continue; } - cancelStaleProviderInvocation( - this.deps.executionRepository, - invocation, - this.deps.executionRepository.listExecutionInvocationsByProviderInvocationId(invocation.id), - { - reconciledAt, - recoveryReason: "startup_interrupted_virtual_repair", - systemMessage: `Closed interrupted ${invocation.purpose} attempt during startup recovery; Code UX will continue its preserved repair session.`, - }, - ); + const linkedInvocations = this.deps.executionRepository.listExecutionInvocationsByProviderInvocationId(invocation.id); + if (matchingRepair.providerWorkCompleted) { + completeStaleProviderInvocation( + this.deps.executionRepository, + invocation, + linkedInvocations, + { + reconciledAt, + recoveryReason: "startup_completed_virtual_repair_provider", + systemMessage: `Recovered the completed ${invocation.purpose} provider attempt from its durable ${matchingRepair.publicationPhase} checkpoint; Code UX will continue repair publication and finalization.`, + }, + ); + } else { + cancelStaleProviderInvocation( + this.deps.executionRepository, + invocation, + linkedInvocations, + { + reconciledAt, + recoveryReason: "startup_interrupted_virtual_repair", + systemMessage: `Closed interrupted ${invocation.purpose} attempt during startup recovery; Code UX will continue its preserved repair session.`, + }, + ); + } providerInvocationIds.push(invocation.id); } diff --git a/tests/backend/services/provider-execution-service.test.ts b/tests/backend/services/provider-execution-service.test.ts index 32d7767a58..c728f03fa5 100644 --- a/tests/backend/services/provider-execution-service.test.ts +++ b/tests/backend/services/provider-execution-service.test.ts @@ -11,6 +11,7 @@ import type { ProviderInvocationPurpose } from "../../../src/contracts/execution import type { AppendExecutionInvocationMessageInput } from "../../../src/contracts/invocation-types.js"; import { MAX_TOOL_PAYLOAD_CHARS } from "../../../src/services/invocation-message-limits.js"; import { SERVER_SHUTDOWN_STOP_REASON } from "../../../src/services/active-dispatch-registry.js"; +import { beginRuntimeShutdown, resetRuntimeShutdownForTests } from "../../../src/services/shutdown-state.js"; import { DEFAULT_DASHBOARD_SETTINGS } from "../../../src/repositories/settings-defaults.js"; import { GOOGLE_DRIVE_PROMPT_SECTION_MARKER } from "../../../src/services/google-drive-mount-service.js"; import * as fs from "node:fs/promises"; @@ -128,6 +129,7 @@ describe("ProviderExecutionService", () => { afterEach(() => { vi.useRealTimers(); + resetRuntimeShutdownForTests(); }); it("Happy path: returns ok: true, creates invocation and usage", async () => { @@ -813,6 +815,34 @@ describe("ProviderExecutionService", () => { ); }); + it("records successful provider completion when shutdown races with the terminal result", async () => { + const controller = new AbortController(); + providerRunner.runProvider.mockImplementation(async () => { + beginRuntimeShutdown(); + controller.abort(SERVER_SHUTDOWN_STOP_REASON); + return mockResult; + }); + + const result = await service.executeProvider({ + ...defaultArgs, + signal: controller.signal, + workflowSettings: { + ...defaultArgs.workflowSettings, + executionMode: "DOCKER", + }, + }); + + expect(result).toBe(mockResult); + expect(executionRepository.updateProviderInvocationUsage).toHaveBeenCalledWith( + "prov-inv-1", + expect.objectContaining({ status: "completed" }), + ); + expect(executionRepository.updateExecutionInvocation).toHaveBeenCalledWith( + "exec-inv-1", + expect.objectContaining({ status: "completed" }), + ); + }); + it("Text output mode: calls runProviderForText when expectTextOutput is true", async () => { const textMockResult = { ...mockResult, text: "text output" }; providerRunner.runProviderForText.mockResolvedValue(textMockResult); diff --git a/tests/backend/services/runtime-startup-recovery-service.test.ts b/tests/backend/services/runtime-startup-recovery-service.test.ts index 354b27ee6b..884716ebde 100644 --- a/tests/backend/services/runtime-startup-recovery-service.test.ts +++ b/tests/backend/services/runtime-startup-recovery-service.test.ts @@ -282,6 +282,129 @@ describe("RuntimeStartupRecoveryService", () => { }); }); + it.each([ + { attentionType: "ci_fix_required" as const, purpose: "ci_fix" as const, publicationPhase: "workspace_finalized" }, + { attentionType: "ci_fix_required" as const, purpose: "ci_fix" as const, publicationPhase: "host_publishing" }, + { attentionType: "ci_fix_required" as const, purpose: "ci_fix" as const, publicationPhase: "host_published" }, + { attentionType: "merge_conflict" as const, purpose: "merge_conflict" as const, publicationPhase: "workspace_finalized" }, + { attentionType: "merge_conflict" as const, purpose: "merge_conflict" as const, publicationPhase: "host_publishing" }, + { attentionType: "merge_conflict" as const, purpose: "merge_conflict" as const, publicationPhase: "host_published" }, + ])("finalizes $purpose provider audit as completed from a durable $publicationPhase checkpoint", async ({ + attentionType, + purpose, + publicationPhase, + }) => { + const { + projectRepository, + executionRepository, + projectAttentionService, + workerEndpointRepository, + service, + } = await createFixture(); + const project = projectRepository.createProject({ + name: "Completed Repair Recovery Project", + sourceType: "local", + sourceRef: "/workspace/completed-repair-recovery", + }); + const endpoint = workerEndpointRepository.createVirtualEndpoint({ + endpointKey: `virtual:completed-repair-${purpose}-${publicationPhase}`, + displayName: "Virtual completed repair worker", + status: "connected", + transport: "internal", + capabilities: {}, + }); + const sessionId = `virtual-${purpose}-${publicationPhase}`; + const item = projectAttentionService.openItem({ + projectId: project.id, + attentionType, + severity: "high", + ownerType: "worker", + title: "Finish checkpointed repair", + summaryMarkdown: "Recover provider completion and finish publication.", + payload: { + repoPath: "/workspace/completed-repair-recovery", + repairRuntime: { + purpose, + sessionId, + workspaceSessionId: `${sessionId}-workspace`, + provider: "codex", + providerConfigId: "codex", + model: "codex-model", + nativeSessionId: "native-completed-repair", + activeAttemptId: "completed-attempt", + attemptRecorded: true, + phase: "provider_running", + workspaceBaselineHead: "baseline-head", + workspaceRepairHead: "repair-head", + publicationPhase, + publishedHeadSha: publicationPhase === "host_published" ? "published-head" : null, + updatedAt: new Date().toISOString(), + }, + }, + }); + projectAttentionService.claimItem(item.id, endpoint.id, "virtual_worker_test"); + const providerInvocation = executionRepository.createProviderInvocationUsage({ + projectId: project.id, + attentionItemId: item.id, + sessionId, + provider: "codex", + purpose, + status: "running", + startedAt: "2026-07-14T10:00:00.000Z", + }); + const executionInvocation = executionRepository.createExecutionInvocation({ + projectId: project.id, + attentionItemId: item.id, + providerInvocationId: providerInvocation.id, + type: purpose, + provider: "codex", + status: "running", + startedAt: "2026-07-14T10:00:00.000Z", + }); + + const result = await service.recover(); + + expect(result.reconciledInterruptedRepairProviderInvocationIds).toEqual([providerInvocation.id]); + expect(result.requeuedInterruptedRepairAttentionItemIds).toEqual([item.id]); + expect(executionRepository.getProviderInvocationUsage(providerInvocation.id)).toMatchObject({ + status: "completed", + finishedAt: expect.any(String), + }); + expect(executionRepository.getExecutionInvocation(executionInvocation.id)).toMatchObject({ + status: "completed", + finishedAt: expect.any(String), + errorMessage: null, + }); + expect(executionRepository.listExecutionInvocationMessages(executionInvocation.id)).toEqual( + expect.arrayContaining([ + expect.objectContaining({ + role: "system", + contentMarkdown: expect.stringContaining(`durable ${publicationPhase} checkpoint`), + metadata: expect.objectContaining({ + recovery: "startup_completed_virtual_repair_provider", + providerInvocationId: providerInvocation.id, + sessionId, + }), + }), + ]), + ); + expect(executionRepository.listRunningProviderInvocationUsages(["codex"])).toEqual([]); + expect(projectAttentionService.getItem(item.id)).toMatchObject({ + status: "open", + assignedWorkerEndpointId: null, + payload: expect.objectContaining({ + repairRuntime: expect.objectContaining({ + sessionId, + nativeSessionId: "native-completed-repair", + workspaceSessionId: `${sessionId}-workspace`, + workspaceBaselineHead: "baseline-head", + workspaceRepairHead: "repair-head", + publicationPhase, + }), + }), + }); + }); + it("demotes premature virtual merge-conflict human escalations back to automatic worker attention", async () => { const { projectRepository,