Blue/Green active service selector becomes empty during CrashLoopBackOff Progressing state, bypassing prePromotionAnalysis and autoPromotionEnabled gate (EKS + Fargate + Flux GitOps) #4749
Unanswered
gprabuheins
asked this question in
Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Environment
Argo Rollouts version: v1.9.0 (Helm chart 2.40.9)
Kubernetes: EKS + Fargate
GitOps: Flux v2
Strategy: Blue/Green (autoPromotionEnabled: false)
Load Balancer: AWS NLB (internal, nlb-target-type: ip)
Application: Spring Boot with /actuator/health probes
Summary
During UAT validation of blue/green deployments, we observed a failure pattern where the active service selector (rollouts-pod-template-hash) became unstable or empty during prolonged Progressing states triggered by bad image deployments (CrashLoopBackOff / readiness failures).
This resulted in a complete traffic blackout on the active service, requiring manual intervention to restore stable routing.
Observed Behavior
Trigger Condition
Deploying a bad image (CrashLoopBackOff) as a second or later failed revision, with a prior successful revision already stable and active.
Failure Sequence
Bad image deploys → new ReplicaSet spins up
→ Pods enter CrashLoopBackOff
→ Rollout enters Progressing state
→ Active service selector (rollouts-pod-template-hash) becomes empty
→ NLB has no valid endpoint target
→ Full traffic blackout on active service
→ progressDeadlineAbort does not fire (referenced in #3988)
→ Rollout remains stuck in Progressing indefinitely
→ Manual intervention required:
Observed Events
SwitchService: Switched selector for service 'my-service' from '' to ''
RolloutCompleted: Initial deploy ← bypass of prePromotionAnalysis
SwitchService: Switched selector for service 'my-service' from '' to ''
Key Observation
When the active service selector was empty (''), subsequent deploys were classified as "Initial deploy" by the controller, bypassing:
prePromotionAnalysis
autoPromotionEnabled: false gate
All configured safety gates
This caused the bad image to be promoted directly to stable,active without any analysis or manual approval.
Configuration
yamlstrategy:
blueGreen:
activeService: my-service
previewService: my-service-preview
autoPromotionEnabled: false
scaleDownDelaySeconds: 3600
abortScaleDownDelaySeconds: 30
prePromotionAnalysis:
templates:
- templateName: my-health-check
args:
- name: service-name
value: my-service-preview
spec:
progressDeadlineSeconds: 1200
progressDeadlineAbort: true
Questions
Is the active service selector becoming empty ('') during prolonged Progressing states a known condition? We believe this may be related to issue Rollout stuck in infinite "Progressing" even after progressDeadlineSeconds exceeds #3988 where progressDeadlineAbort does not fire when CrashLoopBackOff restarts reset the progress deadline timer.
Are there any recommended configurations or patterns for preserving stable traffic routing during startup-failure scenarios in blue/green deployments? Specifically around ensuring the active service selector is never left in an empty state during controller recovery.
Is the "Initial deploy" classification triggered intentionally when the active service selector is empty? If so, is there a way to prevent this from bypassing prePromotionAnalysis and autoPromotionEnabled: false when a stable revision already exists?
Are there any planned improvements around:
Selector stability during prolonged Progressing states?
Recovery behavior when progressDeadlineAbort fails to fire?
Related to issues #3988 and #4453?
Additional Context
We applied SSA merge semantics (kustomize.toolkit.fluxcd.io/ssa: merge) to the active and preview services to prevent Flux from overwriting Argo-managed selector fields. This resolved the Flux-side selector conflict but did not address the Argo-internal selector clearing behavior.
We are currently evaluating whether Kubernetes-native rolling deployments with strict capacity-preservation settings (maxUnavailable: 0, minReadySeconds, progressDeadlineSeconds) may provide a simpler and more deterministic operational recovery model for these specific failure modes, while we await upstream resolution.
Any guidance from the maintainers or community on recommended patterns for this scenario would be greatly appreciated.
All reactions