-
Notifications
You must be signed in to change notification settings - Fork 839
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Rollout confused after deploying during a canary phase #1372
Comments
Additionally, the 'Steps' currently all show as complete. So I have the new Revision 7, the previous upgrade at Revision 6 and the previous stable at Revision 5. What seems to be happening now is that all pods are being moved from Revision 6 to Revision 7 very quickly, without pausing. Revision 5 pods are not being touched yet.
|
Did you happen to go back to the previously deployed version or all these pod specs different from each other? |
No... it was going forward..
…On Mon, Jul 26, 2021, at 4:31 PM, Jesse Suen wrote:
Did you happen to go back to the previously deployed version?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub <#1372 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAD4PP3OZQNYNLOQPBXCTPLTZXA2LANCNFSM5BAVRREA>.
|
I can reliable reproduce a similar issues with replicaset using the rollout + template below. Getting the following error:
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
name: success-rate-qal-usw2
spec:
args:
- name: analysis-interval
value: "300s"
- name: analysis-runs
value: "36"
- name: failure-limit
value: "4"
- name: inconclusive-limit
value: "4"
- name: secret-url
valueFrom:
secretKeyRef:
name: example-secret
key: secretUrl
metrics:
- name: webmetric
successCondition: result == 'It worked!'
count: "{{args.analysis-runs}}"
interval: "{{args.analysis-interval}}"
failureLimit: "{{args.failure-limit}}"
provider:
web:
# placeholders are resolved when an AnalysisRun is created
url: "{{args.secret-url}}"
jsonPath: "{$.message}"
---
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
name: rollout-analysis-step
spec:
replicas: 4
revisionHistoryLimit: 2
selector:
matchLabels:
app: rollout-analysis-step
template:
metadata:
labels:
app: rollout-analysis-step
spec:
containers:
- name: rollouts-demo
image: argoproj/rollouts-demo:blue
imagePullPolicy: Always
ports:
- containerPort: 8080
strategy:
canary:
steps:
- setWeight: 25
# An AnalysisTemplate is referenced at the second step, which starts an AnalysisRun after
# the setWeight step. The rollout will not progress to the following step until the
# AnalysisRun is complete. A failure/error of the analysis will cause the rollout's update to
# abort, and set the canary weight to zero.
- analysis:
templates:
- templateName: success-rate-qal-usw2 Issue is reproducible locally with k3s and rollout controller version: a601a0c |
This issue is stale because it has been open 60 days with no activity. |
I faced same issue in our deployments. argo rollout status being "more replicas need to be updated" and in controller logs "Skip scale down of older RS 'service-123asd': still referenced" say this error |
Hit this error too. The rollout stuck at progressing waiting for more replicas |
Summary
I had a slow rollout progressing through about 8 steps, with a 30 min wait between each. Before it was finished, I pushed another change. This change create a new Revision, but then it got stuck. No pods were upgraded to the new revision, the previous revision and stable versions stopped progressing.
I unstuck it by running
promote
via the CLI tool. I ran it once, and the UI showed that it started the step over again. I then had to promote it through the entire workflow. No changes were being made to the newest revision. Once I went through the entire Rollout like this, the new revision started to upgrade.Diagnostics
1.0.2
Logs for when in broken state:
Message from the maintainers:
Impacted by this bug? Give it a 👍. We prioritize the issues with the most 👍.
The text was updated successfully, but these errors were encountered: