New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cluster-autoscaler not scaling down #2552

Closed
caarlos0 opened this Issue Apr 17, 2017 · 8 comments

Comments

Projects
None yet
5 participants
@caarlos0

caarlos0 commented Apr 17, 2017

The title is a little misleading, but I didn't know how to name it.

Anyway, it does scale down, to a certain point, than it doesn't.

Usually, it stops scaling down at 2 nodes, even if I set the minimum to 1 and there are very few pods running.

Example:

$ kubectl get pods --all-namespaces -o wide
NAMESPACE     NAME                                                   READY     STATUS    RESTARTS   AGE       IP             NODE
default       ingress-nginx-3737565091-xz68c                         1/1       Running   0          5d        10.39.224.4    ip-172-16-33-40.ec2.internal
default       nginx-default-backend-623339674-z93xc                  1/1       Running   0          5d        10.39.224.2    ip-172-16-33-40.ec2.internal
kube-system   cluster-autoscaler-888265702-ztn61                     1/1       Running   0          4d        10.39.248.1    ip-172-16-53-56.ec2.internal
kube-system   dns-controller-489065303-t1589                         1/1       Running   0          4d        172.16.53.56   ip-172-16-53-56.ec2.internal
kube-system   etcd-server-events-ip-172-16-53-56.ec2.internal        1/1       Running   0          4d        172.16.53.56   ip-172-16-53-56.ec2.internal
kube-system   etcd-server-ip-172-16-53-56.ec2.internal               1/1       Running   0          4d        172.16.53.56   ip-172-16-53-56.ec2.internal
kube-system   heapster-564189836-36l0p                               1/1       Running   0          5d        10.39.160.1    ip-172-16-33-98.ec2.internal
kube-system   kube-apiserver-ip-172-16-53-56.ec2.internal            1/1       Running   0          4d        172.16.53.56   ip-172-16-53-56.ec2.internal
kube-system   kube-controller-manager-ip-172-16-53-56.ec2.internal   1/1       Running   0          4d        172.16.53.56   ip-172-16-53-56.ec2.internal
kube-system   kube-dns-782804071-wgbdm                               4/4       Running   0          5d        10.39.160.4    ip-172-16-33-98.ec2.internal
kube-system   kube-dns-autoscaler-2813114833-k7m8r                   1/1       Running   0          5d        10.39.224.3    ip-172-16-33-40.ec2.internal
kube-system   kube-proxy-ip-172-16-33-40.ec2.internal                1/1       Running   0          5d        172.16.33.40   ip-172-16-33-40.ec2.internal
kube-system   kube-proxy-ip-172-16-33-98.ec2.internal                1/1       Running   0          5d        172.16.33.98   ip-172-16-33-98.ec2.internal
kube-system   kube-proxy-ip-172-16-53-56.ec2.internal                1/1       Running   0          4d        172.16.53.56   ip-172-16-53-56.ec2.internal
kube-system   kube-scheduler-ip-172-16-53-56.ec2.internal            1/1       Running   0          4d        172.16.53.56   ip-172-16-53-56.ec2.internal
kube-system   kubernetes-dashboard-3203831700-nt5s0                  1/1       Running   0          4d        10.39.248.2    ip-172-16-53-56.ec2.internal
kube-system   monitoring-influxdb-grafana-v4-t31zw                   2/2       Running   0          5d        10.39.160.2    ip-172-16-33-98.ec2.internal
kube-system   weave-net-cv3hj                                        2/2       Running   1          5d        172.16.33.98   ip-172-16-33-98.ec2.internal
kube-system   weave-net-gmvnv                                        2/2       Running   0          5d        172.16.33.40   ip-172-16-33-40.ec2.internal
kube-system   weave-net-mvr9n                                        2/2       Running   0          4d        172.16.53.56   ip-172-16-53-56.ec2.internal

And yet:

$ kubectl get nodes
NAME                           STATUS         AGE       VERSION
ip-172-16-33-40.ec2.internal   Ready          5d        v1.5.6
ip-172-16-33-98.ec2.internal   Ready          5d        v1.5.6
ip-172-16-53-56.ec2.internal   Ready,master   4d        v1.5.6

The allocated resources are OK too:

Name:			ip-172-16-33-40.ec2.internal
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  CPU Requests	CPU Limits	Memory Requests	Memory Limits
  ------------	----------	---------------	-------------
  330m (16%)	210m (10%)	430Mi (2%)	420Mi (2%)


Name:			ip-172-16-33-98.ec2.internal
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  CPU Requests	CPU Limits	Memory Requests	Memory Limits
  ------------	----------	---------------	-------------
  760m (38%)	400m (20%)	1140Mi (7%)	1220Mi (7%)



Name:			ip-172-16-53-56.ec2.internal
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  CPU Requests	CPU Limits	Memory Requests	Memory Limits
  ------------	----------	---------------	-------------
  1100m (55%)	300m (15%)	750Mi (19%)	700Mi (18%)

The cluster autoscaler logs only say I0417 11:52:11.065164 1 scale_down.go:163] No candidates for scale down.

Is it intended and I'm not understanding it right? How can I improve this?

PS1: ip-172-16-53-56.ec2.internal is the master.
PS2: I'm using this in a dev cluster, that's why I want to downscale a lot

@ohaiwalt

This comment has been minimized.

Show comment
Hide comment
@ohaiwalt

ohaiwalt Apr 17, 2017

Not sure if this helps, but one thing I noticed was that the min/max on my ASG itself didn't allow the cluster-autoscaler to go any further, despite the autoscaler having different numbers.

e.g. My ASG as deployed by kops stated min 10 max 20, but the cluster autoscaler said min 2, max 25, and the cluster only scaled to 20.

ohaiwalt commented Apr 17, 2017

Not sure if this helps, but one thing I noticed was that the min/max on my ASG itself didn't allow the cluster-autoscaler to go any further, despite the autoscaler having different numbers.

e.g. My ASG as deployed by kops stated min 10 max 20, but the cluster autoscaler said min 2, max 25, and the cluster only scaled to 20.

@caarlos0

This comment has been minimized.

Show comment
Hide comment
@caarlos0

caarlos0 Apr 17, 2017

@ohaiwalt thanks, but I checked that already, min is 1 and max is 6, the same as the cluster-autoscaler config :/

caarlos0 commented Apr 17, 2017

@ohaiwalt thanks, but I checked that already, min is 1 and max is 6, the same as the cluster-autoscaler config :/

@MaciekPytel

This comment has been minimized.

Show comment
Hide comment
@MaciekPytel

MaciekPytel Apr 19, 2017

Contributor

@caarlos0 By design cluster-autoscaler won't scale down nodes running non-daemon, non-mirrored kube-system pods. The reasoning is that we don't want to cause failures by stopping critical parts of cluster infrastructure.

So for example the fact that ip-172-16-33-98.ec2.internal is running heapster is enough reason for CA not to scale-down this node.
In cluster-autoscaler logs you should be able to see why it doesn't want to scale each node:
I0419 12:20:05.737806 5 cluster.go:88] Fast evaluation: node mynode-lg73 cannot be removed: non-deamons set, non-mirrored, kube-system pod present: heapster-v1.3.0-3806988011-h6lxj

Contributor

MaciekPytel commented Apr 19, 2017

@caarlos0 By design cluster-autoscaler won't scale down nodes running non-daemon, non-mirrored kube-system pods. The reasoning is that we don't want to cause failures by stopping critical parts of cluster infrastructure.

So for example the fact that ip-172-16-33-98.ec2.internal is running heapster is enough reason for CA not to scale-down this node.
In cluster-autoscaler logs you should be able to see why it doesn't want to scale each node:
I0419 12:20:05.737806 5 cluster.go:88] Fast evaluation: node mynode-lg73 cannot be removed: non-deamons set, non-mirrored, kube-system pod present: heapster-v1.3.0-3806988011-h6lxj

@caarlos0

This comment has been minimized.

Show comment
Hide comment
@caarlos0

caarlos0 Apr 19, 2017

@MaciekPytel figured it could be that, thanks!

caarlos0 commented Apr 19, 2017

@MaciekPytel figured it could be that, thanks!

@caarlos0

This comment has been minimized.

Show comment
Hide comment
@caarlos0

caarlos0 Apr 19, 2017

I Increased and log level, and yah, you're right:

$ kubectl logs -n kube-system --since=4h cluster-autoscaler-776613730-kqgk2 | grep "cannot be removed"| cut -d' ' -f6- | sort | uniq
   1 cluster.go:88] Fast evaluation: node ip-172-16-38-51.ec2.internal cannot be removed: non-deamons set, non-mirrored, kube-system pod present: dns-controller-3586597043-531v5
   1 cluster.go:88] Fast evaluation: node ip-172-16-49-207.ec2.internal cannot be removed: non-deamons set, non-mirrored, kube-system pod present: heapster-564189836-3h2ts
   1 cluster.go:88] Fast evaluation: node ip-172-16-49-207.ec2.internal cannot be removed: non-deamons set, non-mirrored, kube-system pod present: kube-dns-1321724180-c0rjr
   1 cluster.go:88] Fast evaluation: node ip-172-16-49-207.ec2.internal cannot be removed: non-deamons set, non-mirrored, kube-system pod present: kube-dns-autoscaler-265231812-dv17j
   1 cluster.go:88] Fast evaluation: node ip-172-16-49-207.ec2.internal cannot be removed: non-deamons set, non-mirrored, kube-system pod present: kubernetes-dashboard-2396447444-6bwtq
   1 cluster.go:88] Fast evaluation: node ip-172-16-49-207.ec2.internal cannot be removed: non-deamons set, non-mirrored, kube-system pod present: monitoring-influxdb-grafana-v4-50v9d
   1 cluster.go:88] Fast evaluation: node ip-172-16-51-146.ec2.internal cannot be removed: non-deamons set, non-mirrored, kube-system pod present: cluster-autoscaler-776613730-kqgk2

Can I force all those pods to run on a single node?

caarlos0 commented Apr 19, 2017

I Increased and log level, and yah, you're right:

$ kubectl logs -n kube-system --since=4h cluster-autoscaler-776613730-kqgk2 | grep "cannot be removed"| cut -d' ' -f6- | sort | uniq
   1 cluster.go:88] Fast evaluation: node ip-172-16-38-51.ec2.internal cannot be removed: non-deamons set, non-mirrored, kube-system pod present: dns-controller-3586597043-531v5
   1 cluster.go:88] Fast evaluation: node ip-172-16-49-207.ec2.internal cannot be removed: non-deamons set, non-mirrored, kube-system pod present: heapster-564189836-3h2ts
   1 cluster.go:88] Fast evaluation: node ip-172-16-49-207.ec2.internal cannot be removed: non-deamons set, non-mirrored, kube-system pod present: kube-dns-1321724180-c0rjr
   1 cluster.go:88] Fast evaluation: node ip-172-16-49-207.ec2.internal cannot be removed: non-deamons set, non-mirrored, kube-system pod present: kube-dns-autoscaler-265231812-dv17j
   1 cluster.go:88] Fast evaluation: node ip-172-16-49-207.ec2.internal cannot be removed: non-deamons set, non-mirrored, kube-system pod present: kubernetes-dashboard-2396447444-6bwtq
   1 cluster.go:88] Fast evaluation: node ip-172-16-49-207.ec2.internal cannot be removed: non-deamons set, non-mirrored, kube-system pod present: monitoring-influxdb-grafana-v4-50v9d
   1 cluster.go:88] Fast evaluation: node ip-172-16-51-146.ec2.internal cannot be removed: non-deamons set, non-mirrored, kube-system pod present: cluster-autoscaler-776613730-kqgk2

Can I force all those pods to run on a single node?

@MaciekPytel

This comment has been minimized.

Show comment
Hide comment
@MaciekPytel

MaciekPytel Apr 19, 2017

Contributor

@caarlos0 I can't think of any way to do that without either modifying the config of all those pods or just doing it manually (setting ASG size to 1 and letting them all schedule on a single node before scaling up again). You can try asking on stack overflow, maybe someone knows a better way to do it.

Contributor

MaciekPytel commented Apr 19, 2017

@caarlos0 I can't think of any way to do that without either modifying the config of all those pods or just doing it manually (setting ASG size to 1 and letting them all schedule on a single node before scaling up again). You can try asking on stack overflow, maybe someone knows a better way to do it.

@francoran

This comment has been minimized.

Show comment
Hide comment
@francoran

francoran Apr 24, 2017

@caarlos0 did you find a way to overcome this issue? I have the exact same problem, cluster is only scaling up

francoran commented Apr 24, 2017

@caarlos0 did you find a way to overcome this issue? I have the exact same problem, cluster is only scaling up

@caarlos0

This comment has been minimized.

Show comment
Hide comment
@caarlos0

caarlos0 Apr 24, 2017

@francoran in my case I created a scheduled action to scale down to 1 instance at night... but maybe pod affinity can help you...

caarlos0 commented Apr 24, 2017

@francoran in my case I created a scheduled action to scale down to 1 instance at night... but maybe pod affinity can help you...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment