Description
Which component are you using?: Cluster Autoscaler
What version of the component are you using?: v1.22
Component version:
What k8s version are you using (kubectl version)?: v1.22.17
kubectl version Output
$ kubectl version
What environment is this in?: Forked Kubernetes
What did you expect to happen?: Once unneeded nodes have sat idle longer than --scale-down-unneeded-time + --scale-down-delay-after-add, CA should proceed with scale-down even if a scale-up is still in flight.
What happened instead?: CA will not scale down any nodes in a group if it believes a scale-up for that group is still in progress. It waits indefinitely until the pending node registers or until the ASG maxSize is reached and CA cancels the scale-up.
How to reproduce it (as minimally and precisely as possible):
- Submit a pod request (with custom node selector) that triggers scale-up of a Node group (e.g. node-group-A)
- CA scales up 1 node in node-group-A
- Scheduler fails to schedule the pod in the node as Node selector constraint was not satisfied and keeps pod in pending stage
- Ensure that pod remains Pending until a new node is added, and the new node takes >5 min to register.
- During next scan, CA scales up another node in node-group-A resulting in a continuous scale up loop that lasts several hours.
Anything else we need to know?:
Other Findings:
- The scale-up loop for our unschedulable pod takes ~5 min to complete (three 90 s scheduler retries at 30 s intervals).
- Every time CA attempts to satisfy the pod, it increments CloudProviderTarget on node-group “nodes-4…”, leaving Ready < Target, so CA holds deletes.
- Scale-down cooldown timers (--scale-down-unneeded-time, --scale-down-delay-after-add, etc.) have all elapsed, but CA’s group-level state remains “ScaleUp: InProgress,” so ScaleDown stays forbidden.
- There is no current flag to allow scale-down while a scale-up is still pending.