Fix rollout health-check timeouts from transient container states#201
Merged
Conversation
During fast deploys, a status report could snapshot a container in podman's 'created' state. The agent mapped any non-running/non-exited state to 'stopped', and the control plane moved the deployment to an observedPhase of 'stopped' that non-serverless deployments never recovered from, so rollouts timed out after 10 minutes despite the container running fine. Agent: skip containers in transient states in the status report, matching the existing log-collector filter; they get reported on a later tick once they settle. Control plane: allow any desired-running deployment stuck in a stopped/sleeping phase (previously serverless-only) to be promoted back to ready when its container reports running, and when it belongs to a rollout, emit resource/status-changed and a rollout log so the waiting workflow unblocks immediately. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
React Doctor found no issues. 🎉
|
Omitting transient-state containers from the status report made the control plane treat them as absent: attached deployments were marked unknown and removed deployments were deleted while their container still existed. The unknown -> running recovery then emitted no resource/status-changed event, so a rollout could still stall for the full waitForEvent timeout. The agent now reports these containers with status "transient"; the control plane counts them as reported (no unknown-marking, no early deletion) but skips all phase and health transitions until the state settles. The unknown -> running restore path now also sets the ready-transition flag so rollouts are notified after persistence. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The transient default swallowed settled states: a paused, stopped, or unknown-state container would be reported presence-only forever, leaving a previously healthy deployment marked ready and routable. Reserve "transient" for intended intermediates (created, configured, initialized, stopping, removing). Settled non-running states (stopped, paused) and unknown/unrecognized states now map to stopped — the deployment leaves routing and drift reconciliation can repair it, with unexpected states logged for visibility. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Rollouts intermittently failed with
health_check_timeouteven though every container was running. Verified against rollout4e8afd45: during a fast deploy burst, an agent status report snapshotted containers still in podman'screatedstate (podman ps -aincludes them). The agent mapped any non-running/non-exitedstate to"stopped", and the control plane moved those deployments toobservedPhase: "stopped"— a phase non-serverless deployments never recovered from, and one that emits noresource/status-changedevent. The rollout'swaitForEventthen timed out after 10 minutes and rolled back.Fix
Agent (
reporting.go): skip containers in transient states when building the status report, mirroring the existing log-collector filter. Acreatedcontainer is simply reported on a later tick once it settles.Control plane (
agent-status.ts): the stale-stopped restore path is no longer serverless-only — anyruntimeDesiredState: "running"deployment stuck instopped/sleepinggets promoted back to ready when its container reports running. When the promotion reaches a ready phase and the deployment belongs to a rollout, it emitsresource/status-changedand a rollout log so the waiting workflow unblocks immediately instead of timing out. Helper renamedgetStaleStoppedServerlessReportUpdate→getStaleStoppedReportUpdate.Testing
stopped-phase deployment with a running container is promoted and notifies its rollout via Inngest.bun run test: 288 tests pass (44 files)go build ./... && go vetclean,bunx tsc --noEmitclean🤖 Generated with Claude Code