Rollout confused after deploying during a canary phase #1372
Comments
|
Additionally, the 'Steps' currently all show as complete. So I have the new Revision 7, the previous upgrade at Revision 6 and the previous stable at Revision 5. What seems to be happening now is that all pods are being moved from Revision 6 to Revision 7 very quickly, without pausing. Revision 5 pods are not being touched yet. |
|
Did you happen to go back to the previously deployed version or all these pod specs different from each other? |
|
No... it was going forward..
…On Mon, Jul 26, 2021, at 4:31 PM, Jesse Suen wrote:
Did you happen to go back to the previously deployed version?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub <#1372 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAD4PP3OZQNYNLOQPBXCTPLTZXA2LANCNFSM5BAVRREA>.
|
|
I can reliable reproduce a similar issues with replicaset using the rollout + template below. Getting the following error: apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
name: success-rate-qal-usw2
spec:
args:
- name: analysis-interval
value: "300s"
- name: analysis-runs
value: "36"
- name: failure-limit
value: "4"
- name: inconclusive-limit
value: "4"
- name: secret-url
valueFrom:
secretKeyRef:
name: example-secret
key: secretUrl
metrics:
- name: webmetric
successCondition: result == 'It worked!'
count: "{{args.analysis-runs}}"
interval: "{{args.analysis-interval}}"
failureLimit: "{{args.failure-limit}}"
provider:
web:
# placeholders are resolved when an AnalysisRun is created
url: "{{args.secret-url}}"
jsonPath: "{$.message}"
---
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
name: rollout-analysis-step
spec:
replicas: 4
revisionHistoryLimit: 2
selector:
matchLabels:
app: rollout-analysis-step
template:
metadata:
labels:
app: rollout-analysis-step
spec:
containers:
- name: rollouts-demo
image: argoproj/rollouts-demo:blue
imagePullPolicy: Always
ports:
- containerPort: 8080
strategy:
canary:
steps:
- setWeight: 25
# An AnalysisTemplate is referenced at the second step, which starts an AnalysisRun after
# the setWeight step. The rollout will not progress to the following step until the
# AnalysisRun is complete. A failure/error of the analysis will cause the rollout's update to
# abort, and set the canary weight to zero.
- analysis:
templates:
- templateName: success-rate-qal-usw2Issue is reproducible locally with k3s and rollout controller version: a601a0c |
|
This issue is stale because it has been open 60 days with no activity. |
|
I faced same issue in our deployments. argo rollout status being "more replicas need to be updated" and in controller logs "Skip scale down of older RS 'service-123asd': still referenced" say this error |
|
Hit this error too. The rollout stuck at progressing waiting for more replicas |
Summary
I had a slow rollout progressing through about 8 steps, with a 30 min wait between each. Before it was finished, I pushed another change. This change create a new Revision, but then it got stuck. No pods were upgraded to the new revision, the previous revision and stable versions stopped progressing.
I unstuck it by running
promotevia the CLI tool. I ran it once, and the UI showed that it started the step over again. I then had to promote it through the entire workflow. No changes were being made to the newest revision. Once I went through the entire Rollout like this, the new revision started to upgrade.Diagnostics
1.0.2
Logs for when in broken state:
Message from the maintainers:
Impacted by this bug? Give it a 👍. We prioritize the issues with the most 👍.
The text was updated successfully, but these errors were encountered: