Skip to content

fix(server): do not block command readiness on recovery turns - #313

Merged
patroza merged 1 commit into
fork/changesfrom
nonblocking-startup-recovery
Aug 4, 2026
Merged

fix(server): do not block command readiness on recovery turns#313
patroza merged 1 commit into
fork/changesfrom
nonblocking-startup-recovery

Conversation

@patroza

@patroza patroza commented Aug 4, 2026

Copy link
Copy Markdown
Owner

Summary

  • Why the design blocked readiness: restart recovery (reconcileStartupstartSession + sendTurn) ran inside ProviderCommandReactor.start(), which is on the critical path before orphan settle and before signalCommandReady. That ordering exists so (1) recovery can claim/resume before orphan audit clears “running” zombies, and (2) clients do not race half-started provider state. The cost: a hung agent on recovery made all HTTP (including Discord /oauth/token) wait forever.
  • What we change (safely):
    1. Claim phase (sync, still before orphan settle): write durable restartRecovery marker, demote binding to stopped, settle projection as interrupted. Orphan audit will not wipe recovery intent.
    2. Continuation (parked on ServerActivation): startSession + recovery sendTurn run after activation / readiness boundary, each with a 45s timeout. Hung agents fail closed with recovery-failed state instead of pinning boot.
    3. Bootstrap exemptions: /oauth/token, /.well-known/t3/environment, and connect health paths skip the global command-readiness middleware so Discord/bootstrap stay up even if other startup work is slow.
  • Unit tests cover claim-before-continue parking and bootstrap path exemptions.

Risks if we had only “moved recovery later”

Risk Mitigation in this PR
Orphan audit clears activeTurnId / kills mid-start process Claim writes marker + stops live-claiming status before orphan settle; continuation is after orphan
Clients issue turns while recovery is mid-flight Continuations still park until activation; product APIs still wait for command ready
Recovery hangs forever in background 45s timeouts on startSession/sendTurn + failure persistence
Drain/shutdown during recovery drain still awaits startupReconciliationDone after continuations

Test plan

  • vp test run apps/server/src/orchestration/Layers/ProviderCommandReactor.test.ts
  • Readiness exemption unit assertions in server.test.ts
  • CI Fork Check/Test on the PR
  • After merge: recompose integration; optional t3vm restart with a stuck recovery candidate and confirm Discord oauth succeeds within seconds

Split restart recovery into a durable claim phase (before orphan settle)
and a parked startSession/sendTurn continuation (after activation), with
provider timeouts. Exempt oauth/environment bootstrap paths from the
global readiness gate so Discord and health probes stay available when a
hung agent would otherwise stall the whole HTTP surface.
@patroza
patroza marked this pull request as ready for review August 4, 2026 13:17
@patroza
patroza merged commit c1d5d69 into fork/changes Aug 4, 2026
9 checks passed
patroza added a commit that referenced this pull request Aug 4, 2026
Split restart recovery into a durable claim phase (before orphan settle)
and a parked startSession/sendTurn continuation (after activation), with
provider timeouts. Exempt oauth/environment bootstrap paths from the
global readiness gate so Discord and health probes stay available when a
hung agent would otherwise stall the whole HTTP surface.

Co-authored-by: T3 Code PR Stack <41898282+github-actions[bot]@users.noreply.github.com>
patroza added a commit that referenced this pull request Aug 4, 2026
Split restart recovery into a durable claim phase (before orphan settle)
and a parked startSession/sendTurn continuation (after activation), with
provider timeouts. Exempt oauth/environment bootstrap paths from the
global readiness gate so Discord and health probes stay available when a
hung agent would otherwise stall the whole HTTP surface.

Co-authored-by: T3 Code PR Stack <41898282+github-actions[bot]@users.noreply.github.com>
patroza added a commit that referenced this pull request Aug 4, 2026
…313)

fixup! fix(server): do not block command readiness on recovery turns (#313)
patroza added a commit that referenced this pull request Aug 5, 2026
Split restart recovery into a durable claim phase (before orphan settle)
and a parked startSession/sendTurn continuation (after activation), with
provider timeouts. Exempt oauth/environment bootstrap paths from the
global readiness gate so Discord and health probes stay available when a
hung agent would otherwise stall the whole HTTP surface.

Co-authored-by: T3 Code PR Stack <41898282+github-actions[bot]@users.noreply.github.com>
patroza added a commit that referenced this pull request Aug 5, 2026
…313)

fixup! fix(server): do not block command readiness on recovery turns (#313)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant