Skip to content

fix: harden orchestration restart recovery - #2328

Merged
numnx merged 10 commits into
devfrom
fix/orchestration-monitoring-hardening
Jul 6, 2026
Merged

fix: harden orchestration restart recovery#2328
numnx merged 10 commits into
devfrom
fix/orchestration-monitoring-hardening

Conversation

@numnx

@numnx numnx commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator

Summary

This PR hardens orchestration recovery across app restarts and restart-edge cases.

  • Adds General -> Restart Behavior settings for sprint and invocation recovery policy.
  • Applies startup recovery policies before watch-loop recovery so active runs can continue, pause, or cancel coherently.
  • Preserves Docker workspace/runtime volumes while allowing restart/cancel policy to remove stale labelled containers.
  • Repairs paused sprint projection drift so paused runs do not keep appearing as running after restart.
  • Prevents virtual merge-conflict workers from falsely resolving an item when no merge evidence was recorded.
  • Keeps checked-out worker branches clean after patch materialization updates refs.

Root Cause

Restart recovery had several independent paths for sprint runs, dispatches, provider invocations, QA rows, leases, Docker containers, and sprint projection rows. Those paths could disagree after a restart, especially around paused runs, interrupted local invocations, preserved Docker volumes, and virtual merge-conflict repair.

Validation

  • pnpm run typecheck
  • pnpm vitest run tests/backend/services/runtime-startup-recovery-service.test.ts tests/backend/infrastructure/providers/cli/workspace-artifact-service.test.ts tests/backend/repositories/settings-repository.test.ts
  • pnpm vitest run tests/backend/services/virtual-worker-service.test.ts
  • pnpm run build
  • Local rapid restart stress: graceful restart, repeated graceful restart, hard-kill restart, delayed restart.
  • Runtime readiness verified at http://127.0.0.1:4444/ready after stress.

Risk / Rollback

Risk is concentrated in startup recovery and virtual-worker merge-conflict handling. Rollback is reverting this PR; existing persisted sprint/task state remains compatible because the new restart policy settings default to continue and sanitize unknown values back to defaults.

@numnx
numnx marked this pull request as ready for review July 6, 2026 13:27
@numnx
numnx merged commit d96b117 into dev Jul 6, 2026
8 checks passed
@numnx numnx mentioned this pull request Jul 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant