Skip to content

Release converge: fail fast on terminal resource failures instead of burning the deploy timeout #53

Description

@ashishmax31

Summary

The hub's converge reconciler (pkg/worker/release/converge.go in Stackdome/stackdome) has no fail-fast path. A release that can never converge sits until the deploy timeout expires (EffectiveSettings().DeployTimeoutMinutes, default 15m) and then fails with the generic message:

timed out waiting for convergence after 15m0s

Two agent-reported states are already terminal per revision and can be recognised immediately:

  1. Build failure — agent sets Stalled=True with reason BuildFailed. The imagebuild controller already records the real failure on the release timeline; the release itself still waits out the clock.
  2. Port readiness failure (new in v0.6.8-alpha, StackResource port readiness gate + branded Traefik error pages #52) — once a declared port outlives the 3-minute port-check grace window, the agent condemns the revision: WorkloadAvailable=False and Converged=False, both with reason PortNotListening, plus a readiness_failure entry in status.lastFailureDetail. Condemnation is terminal for that revision — the verdict is cached and never re-dialled until a new rollout. Nothing about waiting longer changes the outcome.

In both cases the release should be marked failed as soon as the terminal signal is observed, carrying the agent's reason/message instead of the generic timeout text.

Scope note

This is hub-side work — the agent already reports everything needed. Filed here alongside the port-readiness feature it follows from; transfer to Stackdome/stackdome if that fits the backlog better.

Not in scope

Surfacing the readiness failure on the release timeline (typed readiness_failure mapping + resource_failed event) is being done separately as part of the v0.6.8-alpha integration. This issue is only about ending the release early rather than waiting for the timeout.

Priority

P2 — degrades deploy DX (15 minutes of dead waiting on a diagnosable failure), no correctness impact; the timeline already names the cause. Estimate: ~2-3 hrs.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions