Skip to content

Fix rollout health-check timeouts from transient container states#201

Merged
arjunkomath merged 3 commits into
mainfrom
fix/transient-container-state-race
Jul 24, 2026
Merged

Fix rollout health-check timeouts from transient container states#201
arjunkomath merged 3 commits into
mainfrom
fix/transient-container-state-race

Conversation

@arjunkomath

Copy link
Copy Markdown
Member

Problem

Rollouts intermittently failed with health_check_timeout even though every container was running. Verified against rollout 4e8afd45: during a fast deploy burst, an agent status report snapshotted containers still in podman's created state (podman ps -a includes them). The agent mapped any non-running/non-exited state to "stopped", and the control plane moved those deployments to observedPhase: "stopped" — a phase non-serverless deployments never recovered from, and one that emits no resource/status-changed event. The rollout's waitForEvent then timed out after 10 minutes and rolled back.

Fix

Agent (reporting.go): skip containers in transient states when building the status report, mirroring the existing log-collector filter. A created container is simply reported on a later tick once it settles.

Control plane (agent-status.ts): the stale-stopped restore path is no longer serverless-only — any runtimeDesiredState: "running" deployment stuck in stopped/sleeping gets promoted back to ready when its container reports running. When the promotion reaches a ready phase and the deployment belongs to a rollout, it emits resource/status-changed and a rollout log so the waiting workflow unblocks immediately instead of timing out. Helper renamed getStaleStoppedServerlessReportUpdategetStaleStoppedReportUpdate.

Testing

  • Added a regression test: a non-serverless stopped-phase deployment with a running container is promoted and notifies its rollout via Inngest.
  • bun run test: 288 tests pass (44 files)
  • go build ./... && go vet clean, bunx tsc --noEmit clean

🤖 Generated with Claude Code

During fast deploys, a status report could snapshot a container in
podman's 'created' state. The agent mapped any non-running/non-exited
state to 'stopped', and the control plane moved the deployment to an
observedPhase of 'stopped' that non-serverless deployments never
recovered from, so rollouts timed out after 10 minutes despite the
container running fine.

Agent: skip containers in transient states in the status report,
matching the existing log-collector filter; they get reported on a
later tick once they settle.

Control plane: allow any desired-running deployment stuck in a
stopped/sleeping phase (previously serverless-only) to be promoted
back to ready when its container reports running, and when it belongs
to a rollout, emit resource/status-changed and a rollout log so the
waiting workflow unblocks immediately.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Jul 24, 2026

Copy link
Copy Markdown

React Doctor found no issues. 🎉

⚠️ Warning: .github/workflows/react-doctor.yml is configured incorrectly. See below to fix.

React Doctor compares against main to report only the issues this pull request introduces. This run couldn't complete that comparison (usually a shallow CI checkout with no merge base), so it listed every issue in the changed files, including ones that already existed on main.

Add fetch-depth: 0 to the actions/checkout step in .github/workflows/react-doctor.yml so the checkout includes the history React Doctor needs:

 jobs:
   react-doctor:
     steps:
       - uses: actions/checkout@v5
+        with:
+          fetch-depth: 0

       - uses: millionco/react-doctor@v2

To silence this warning, set silence-missing-baseline-warning: true on the React Doctor action.

Reviewed by React Doctor for commit df568c5.

arjunkomath and others added 2 commits July 24, 2026 22:17
Omitting transient-state containers from the status report made the
control plane treat them as absent: attached deployments were marked
unknown and removed deployments were deleted while their container
still existed. The unknown -> running recovery then emitted no
resource/status-changed event, so a rollout could still stall for the
full waitForEvent timeout.

The agent now reports these containers with status "transient"; the
control plane counts them as reported (no unknown-marking, no early
deletion) but skips all phase and health transitions until the state
settles. The unknown -> running restore path now also sets the
ready-transition flag so rollouts are notified after persistence.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The transient default swallowed settled states: a paused, stopped, or
unknown-state container would be reported presence-only forever,
leaving a previously healthy deployment marked ready and routable.

Reserve "transient" for intended intermediates (created, configured,
initialized, stopping, removing). Settled non-running states (stopped,
paused) and unknown/unrecognized states now map to stopped — the
deployment leaves routing and drift reconciliation can repair it, with
unexpected states logged for visibility.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@arjunkomath
arjunkomath merged commit 9e63f88 into main Jul 24, 2026
10 checks passed
@arjunkomath
arjunkomath deleted the fix/transient-container-state-race branch July 24, 2026 12:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant