pod hpa would create extra pods during deployment rolling update when there is no load at all during the rolling upgrade #72775
Labels
kind/bug
Categorizes issue or PR as related to a bug.
lifecycle/rotten
Denotes an issue or PR that has aged beyond stale and will be auto-closed.
needs-triage
Indicates an issue or PR lacks a `triage/foo` label and requires one.
sig/autoscaling
Categorizes an issue or PR as relevant to SIG Autoscaling.
Comments
k8s-ci-robot
added
the
needs-sig
|
@kubernetes/sig-autoscalling |
|
/sig autoscalling |
|
It seems that there is several issue in hpa code, the calculation is based on current replica, but during rolling upgrade, the max surge would create extra replica containing different versions of replica set, but after calculation the desired replica is written all to the current version of replica set via deployment spec |
|
/sig autoscaling |
k8s-ci-robot
added
sig/autoscaling
|
Seeing some very similar behaviour here with some of our deployments. We recently had a deployment scale up to 20 pods during an update even though it was running stable at 3 replicas, had a MinReplicas of 3, and was well within its target average value for scaling. We have the HPA configured with MaxReplicas of 40, MinReplicas of 3. The deployment itself has RollingUpdate with 0% unavailable and 100% surge configured as well which may or may not be contributing to the issue. |
|
I am seeing the same behavior. and HPA When I change image it creates another pod and kills old one, than HPA decides to scale up and spin another pod. It might happen up to 3 times, so at the end I might find 4 pods running when I need only one. And even with |
|
I can confirm... I am having exact same issue as above running on GKE. Runs stable as 1 pod but on a new deployment i get some crazy amounts of pods that spin up... currently they are higher than my current max and i have 2 versions of the pod deploying... which is very dangerous for us. version: 1.12.6-gke.10 deployment yaml hpa One thing I have noticed tho... it does eventually get resolved... sometimes within 10 minutes... other times I have come across it and it has been same for almost 2 hours |
|
The same problem here. I am wondering is there any workaround for this issue? |
|
+1 We have seen something on the order of 2x the pods --exceeding the HPA max-- maybe more during rolling updates with a surge of only 25%. At our scale that's anywhere from 250 to 1000+ pods. |
|
I'm seeing that the desired count in the HPA is set to the number of pods with surge. For example, with a starting replica count of 100 and surge of 25%, the new desired count is set to 125. That count is sustained well after the rollout is done until the HPA cooldown elapses and scales the deployment back down. I think what's happening is that HPA locks in a desired count that includes the surging pods. The deployment controller sees the new number of pods and surges some more above that. On the next HPA cycle, it sets that number as the new desired and the cycle continues until the rollout completes faster than the HPA has a chance to bump the desired count. Below is an example from our live environment that shows this escalation. Our surge is set to 25%. Note that several of the increases are about 25% apart and that many of the increases are taking place while the deployments controller is cleaning up old pods. Summary: |
|
What if you set the surge to lower than 25% (considering the lagging cooling behavior) ? |
|
The amount of increase in the HPA feedback loop decreases. Also it's more likely that we break out of the feedback loop sooner as the rolling update completes faster. However, there is still a positive feedback loop. |
|
Any workaround at all? Maybe some specific |
|
The workaround we're using is to delete the HPA during a rolling update and to apply it back after it is done. |
|
+1 |
6 tasks
|
Hi All I am facing the same issue on EKS did anyone found a solution ? |
|
Hi @max-rocket-internet I couldn't find the solution in these documents can you please guide me to it |
|
There is no solution. A fix has been merged that might fix it. You need to wait until EKS has been updated with a k8s version that includes this fix. |
|
@max-rocket-internet Thanks alot |
zq-david-wang
changed the title
|
Someone needs to open a new issue. @sbocinec? We are not on 1.14 yet. |
|
This bug actually did take down a production cluster just recently. One of our deployments scaled out to 120 PODs in a very short time during a rolling update, saturating the nodes. To make thing worse the pods did stuck at terminating state when we started to clean up manually. So this is at least for us a critical bug, present in k8s 1.14.6 (on AWS deployed with kube-aws) |
|
@max-rocket-internet , why is is better to open a new issue rather then reopening this issue? |
|
I think I have learned something very important. The bug ONLY occurs if you are using the new HPA API This will trigger the bug And this config will NOT trigger the bug |
|
Another observation is that if you have upgraded the HPA object to the new API version, you cannot downgrade again. You will have to delete the |
|
Unfortunately, The bug is reproducable on |
|
I think there are multiple issues. I fixed the problem for new pods that are really busy at startup by delaying readiness, but I've discovered that if I delete a pod and the graceful shutdown uses a lot cpu that the hpa appears to make a scaling decision based on the terminating pod. I don't know if it is a race condition between the pod going from running/ready to terminating and the TERM signal being sent or something else. I tried to use a sleep 30 in the preStop hook in hopes of breaking the "presumed" race, but no luck. I could just set the grace period for pods to 0 and murder them without a chance to spike CPU but that seems sub optimal. I guess I'll need to go read the hpa code or something. :P |
|
/reopen |
|
@djjayeeta: You can't reopen an issue/PR unless you authored it or you are a collaborator. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
I am facing the same issue. My version It seems to happen only with CPU metrics, I tried using memory metrics to reproduce it, but was unable to. My deployment maxSurge is 25% and minimum replicas for hpa is 2. So it creates 1 Pod during rolling update. During the rolling update non of the PODS(creating, running or terminating) is using a huge CPU. I continuously checked for metrics. In fact it was missing for some time, My HPA events are Type Reason Age From Message Normal SuccessfulRescale 76s (x6 over 144m) horizontal-pod-autoscaler New size: 3; reason: cpu resource utilization (percentage of request) above target I did a watch on the annotations of HPA. The reported utilization is also less than target autoscaling.alpha.kubernetes.io/current-metrics:[{"type":"Resource","resource":{"name":"cpu","currentAverageUtilization":3,"currentAverageValue":"4m"}}] Each time I do a rollout restart my replica count increases. |
|
/reopen |
|
@serathius: Reopened this issue. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
@zq-david-wang: This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the The Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
k8s-ci-robot
added
the
needs-triage
|
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale |
k8s-ci-robot
added
the
lifecycle/stale
|
/remove-lifecycle stale |
k8s-ci-robot
removed
the
lifecycle/stale
|
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale |
k8s-ci-robot
added
the
lifecycle/stale
|
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle rotten |
k8s-ci-robot
added
lifecycle/rotten
|
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /close |
|
@k8s-triage-robot: Closing this issue. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
and others
What happened:
I have deployment with following upgrade strategy
and hpa:
When sending patch command to update container image, (current replicas is 1), following happended
What you expected to happen:
How to reproduce it (as minimally and precisely as possible):
Anything else we need to know?:
Environment:
kubectl version):uname -a):The text was updated successfully, but these errors were encountered: