Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
cluster-autoscaler not scaling down #2552
The title is a little misleading, but I didn't know how to name it.
Anyway, it does scale down, to a certain point, than it doesn't.
Usually, it stops scaling down at 2 nodes, even if I set the minimum to 1 and there are very few pods running.
The allocated resources are OK too:
The cluster autoscaler logs only say
Is it intended and I'm not understanding it right? How can I improve this?
Not sure if this helps, but one thing I noticed was that the min/max on my ASG itself didn't allow the cluster-autoscaler to go any further, despite the autoscaler having different numbers.
e.g. My ASG as deployed by kops stated min 10 max 20, but the cluster autoscaler said min 2, max 25, and the cluster only scaled to 20.
@caarlos0 By design cluster-autoscaler won't scale down nodes running non-daemon, non-mirrored kube-system pods. The reasoning is that we don't want to cause failures by stopping critical parts of cluster infrastructure.
So for example the fact that ip-172-16-33-98.ec2.internal is running heapster is enough reason for CA not to scale-down this node.
I Increased and log level, and yah, you're right:
Can I force all those pods to run on a single node?
@caarlos0 I can't think of any way to do that without either modifying the config of all those pods or just doing it manually (setting ASG size to 1 and letting them all schedule on a single node before scaling up again). You can try asking on stack overflow, maybe someone knows a better way to do it.