Insufficient pods stops scaling-in confusing message-AWS EKS

[prageethw]

logs below

I0113 03:43:30.105993 1 cluster.go:222] Evaluation ip-192-168-75-46.us-east-2.compute.internal for test/tiller-deploy-7b584bf689-vzbn4 -> PodFitsResources predicate mismatch, reason: Insufficient pods
I0113 03:43:30.106018 1 cluster.go:222] Evaluation ip-192-168-84-242.us-east-2.compute.internal for test/tiller-deploy-7b584bf689-vzbn4 -> PodFitsResources predicate mismatch, reason: Insufficient pods
I0113 03:43:30.106038 1 cluster.go:222] Evaluation ip-192-168-50-191.us-east-2.compute.internal for test/tiller-deploy-7b584bf689-vzbn4 -> PodFitsResources predicate mismatch, reason: Insufficient pods
I0113 03:43:30.106085 1 cluster.go:222] Evaluation ip-192-168-41-162.us-east-2.compute.internal for test/tiller-deploy-7b584bf689-vzbn4 -> PodFitsResources predicate mismatch, reason: Insufficient pods
I0113 03:43:30.106106 1 cluster.go:126] Fast evaluation: node ip-192-168-2-214.us-east-2.compute.internal is not suitable for removal: failed to find place for test/tiller-deploy-7b584bf689-vzbn4

[Jeffwan]

@prageethw could you provide more information. like your autoscaler configuration. nodes in your EKS cluster?

[MaciekPytel]

There is a limit of how many pods you can run per node (regardless of resources), you're hitting that limit. You can find that limit in the output of kubectl get node <nodename> -o yaml.

The message is coming from Kubernetes default scheduler code and there is not much we can do to improve it. Autoscaler works by importing scheduler code and treating it as black box to check if pod fits. It doesn't understand why the pod doesn't fit, it just forwards message from scheduler.

[aleksandra-malinowska]

IIRC, some other issue mentioned EKS deployments have lower number of pods per node than allowed by open-source Kubernetes, due to limited IPs for pods. You may want to double check that when estimating how many nodes are needed.

[Jeffwan]

That's true. That's because the CNI implementation on EKS. Max pods number is limited by the network interfaces attached to instance multiplied by the number of ips per ENI - which varies depending on the size of instance. It's apparent for small instances, this number can be quite a low number.

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html#AvailableIpPerENI

[prageethw]

@Jeffwan @aleksandra-malinowska @MaciekPytel

sorry guys , i'm not sure whether it is gonna be too much info below, but i feel like something not quite right here

my findings:

I think since I have not put any hard limit on my cluster as the number of instances in any particular AZ CA should automatically spin up instances in away maximize resource usage, also as I instructed CA to balance similar node groups it should spin 1 each instance in each node group (in this specific case as I only get 3 instances created by CA), but in reality it has rather spinup 2 instances in us-east-2a , I think that has lead to the imbalance in volume zones.

kubectl get nodes -o yaml |grep zone

      failure-domain.beta.kubernetes.io/zone: us-east-2b
      failure-domain.beta.kubernetes.io/zone: us-east-2a
      failure-domain.beta.kubernetes.io/zone: us-east-2a

CA logs:

[aws-cluster-autoscaler-788f4bb5df-2hfhs] I0127 05:17:21.967400       1 cluster.go:225] Pod kube-ingress/nginx-default-backend-d5ffff847-gnvmd can be moved to ip-192-168-30-209.us-east-2.compute.internal
[aws-cluster-autoscaler-788f4bb5df-2hfhs] I0127 05:17:21.967420       1 cluster.go:225] Pod aws-cluster-autoscaler/aws-cluster-autoscaler-788f4bb5df-t6wcr can be moved to ip-192-168-30-209.us-east-2.compute.internal
[aws-cluster-autoscaler-788f4bb5df-2hfhs] I0127 05:17:21.967489       1 scheduler_binder.go:532] All bound volumes for Pod "metrics/prometheus-alertmanager-0" match with Node "ip-192-168-30-209.us-east-2.compute.internal"
[aws-cluster-autoscaler-788f4bb5df-2hfhs] I0127 05:17:21.967510       1 cluster.go:222] Evaluation ip-192-168-30-209.us-east-2.compute.internal for metrics/prometheus-alertmanager-0 -> NoVolumeZoneConflict predicate mismatch, reason: node(s) had no available volume zone
[aws-cluster-autoscaler-788f4bb5df-2hfhs] I0127 05:17:21.967522       1 cluster.go:222] Evaluation ip-192-168-35-168.us-east-2.compute.internal for metrics/prometheus-alertmanager-0 -> PodFitsResources predicate mismatch, reason: Insufficient pods
[aws-cluster-autoscaler-788f4bb5df-2hfhs] I0127 05:17:21.967528       1 cluster.go:126] Fast evaluation: node ip-192-168-54-250.us-east-2.compute.internal is not suitable for removal: failed to find place for metrics/prometheus-alertmanager-0
[aws-cluster-autoscaler-788f4bb5df-2hfhs] I0127 05:17:21.967537       1 scale_down.go:493] 2 nodes found to be unremovable in simulation, will re-check them at 2019-01-27 05:22:21.782775908 +0000 UTC m=+338.653465891
[aws-cluster-autoscaler-788f4bb5df-2hfhs] I0127 05:17:21.967552       1 static_autoscaler.go:360] Scale down status: unneededOnly=true lastScaleUpTime=2019-01-27 05:17:01.767617473 +0000 UTC m=+18.638307385 lastScaleDownDeleteTime=2019-01-27 05:17:01.767618274 +0000 UTC m=+18.638308190 lastScaleDownFailTime=2019-01-27 05:17:01.767619078 +0000 UTC m=+18.638308999 scaleDownForbidden=false isDeleteInProgress=false

the node that complains about above...
kubectl get node ip-192-168-54-250.us-east-2.compute.internal -o yaml

apiVersion: v1
kind: Node
metadata:
  annotations:
    node.alpha.kubernetes.io/ttl: "0"
    volumes.kubernetes.io/controller-managed-attach-detach: "true"
  creationTimestamp: "2019-01-27T04:52:54Z"
  labels:
    alpha.eksctl.io/cluster-name: prageethw-1548562376
    alpha.eksctl.io/nodegroup-name: ng1
    beta.kubernetes.io/arch: amd64
    beta.kubernetes.io/instance-type: t2.medium
    beta.kubernetes.io/os: linux
    failure-domain.beta.kubernetes.io/region: us-east-2
    failure-domain.beta.kubernetes.io/zone: us-east-2a
    kubernetes.io/hostname: ip-192-168-54-250.us-east-2.compute.internal
  name: ip-192-168-54-250.us-east-2.compute.internal
  resourceVersion: "11240"
  selfLink: /api/v1/nodes/ip-192-168-54-250.us-east-2.compute.internal
  uid: 685bcfde-21ef-11e9-adb0-02d4d8a022de
spec:
  providerID: aws:///us-east-2a/i-0d3400cef10bf183b
status:
  addresses:
  - address: 192.168.54.250
    type: InternalIP
  - address: 18.188.158.178
    type: ExternalIP
  - address: ip-192-168-54-250.us-east-2.compute.internal
    type: InternalDNS
  - address: ec2-18-188-158-178.us-east-2.compute.amazonaws.com
    type: ExternalDNS
  - address: ip-192-168-54-250.us-east-2.compute.internal
    type: Hostname
  allocatable:
    cpu: "2"
    ephemeral-storage: "19316009748"
    hugepages-2Mi: "0"
    memory: 3937628Ki
    pods: "17"
  capacity:
    cpu: "2"
    ephemeral-storage: 20959212Ki
    hugepages-2Mi: "0"
    memory: 4040028Ki
    pods: "17"
  conditions:
  - lastHeartbeatTime: "2019-01-27T05:28:07Z"
    lastTransitionTime: "2019-01-27T04:52:53Z"
    message: kubelet has sufficient disk space available
    reason: KubeletHasSufficientDisk
    status: "False"
    type: OutOfDisk
  - lastHeartbeatTime: "2019-01-27T05:28:07Z"
    lastTransitionTime: "2019-01-27T04:52:53Z"
    message: kubelet has sufficient memory available
    reason: KubeletHasSufficientMemory
    status: "False"
    type: MemoryPressure
  - lastHeartbeatTime: "2019-01-27T05:28:07Z"
    lastTransitionTime: "2019-01-27T04:52:53Z"
    message: kubelet has no disk pressure
    reason: KubeletHasNoDiskPressure
    status: "False"
    type: DiskPressure
  - lastHeartbeatTime: "2019-01-27T05:28:07Z"
    lastTransitionTime: "2019-01-27T04:52:53Z"
    message: kubelet has sufficient PID available
    reason: KubeletHasSufficientPID
    status: "False"
    type: PIDPressure
  - lastHeartbeatTime: "2019-01-27T05:28:07Z"
    lastTransitionTime: "2019-01-27T04:53:24Z"
    message: kubelet is posting ready status
    reason: KubeletReady
    status: "True"
    type: Ready
  daemonEndpoints:
    kubeletEndpoint:
      Port: 10250
  images:
  - names:
    - 602401143452.dkr.ecr.us-east-2.amazonaws.com/amazon-k8s-cni@sha256:1fafb3a9a12feef7105986c0ed5e3a6b327f2b5a5356e64f026509614dab921d
    - 602401143452.dkr.ecr.us-east-2.amazonaws.com/amazon-k8s-cni:1.2.1
    sizeBytes: 355178570
  - names:
    - k8s.gcr.io/cluster-autoscaler@sha256:7ff5a60304b344f2f29c804c7253632bbc818794f6932236a56db107a6a8f5af
    - k8s.gcr.io/cluster-autoscaler:v1.13.1
    sizeBytes: 136618018
  - names:
    - 602401143452.dkr.ecr.us-east-2.amazonaws.com/eks/kube-proxy@sha256:45be43d0609c92aad89cb8607a0452a59db46133cf9fede314367eb16fac5750
    - 602401143452.dkr.ecr.us-east-2.amazonaws.com/eks/kube-proxy:v1.11.5
    sizeBytes: 97950829
  - names:
    - registry.opensource.zalan.do/teapot/external-dns@sha256:3238aaf10240d6322c87073766f67d8cf24dc2db4f0ef01b62d18890573f8075
    - registry.opensource.zalan.do/teapot/external-dns:v0.5.9
    sizeBytes: 51942746
  - names:
    - gcr.io/google_containers/metrics-server-amd64@sha256:78938f933822856f443e6827fe5b37d6cc2f74ae888ac8b33d06fdbe5f8c658b
    - gcr.io/google_containers/metrics-server-amd64:v0.3.1
    sizeBytes: 40767713
  - names:
    - prom/alertmanager@sha256:196af0317d3449c1300aa26ff0366f68c67d04581d2c9f8609cbb227424e226c
    - prom/alertmanager:v0.15.3
    sizeBytes: 34526872
  - names:
    - jimmidyson/configmap-reload@sha256:befec9f23d2a9da86a298d448cc9140f56a457362a7d9eecddba192db1ab489e
    - jimmidyson/configmap-reload:v0.2.2
    sizeBytes: 22381476
  - names:
    - prom/node-exporter@sha256:c390c8fea4cd362a28ad5070aedd6515aacdfdffd21de6db42ead05e332be5a9
    - prom/node-exporter:v0.17.0
    sizeBytes: 20982005
  - names:
    - k8s.gcr.io/defaultbackend@sha256:865b0c35e6da393b8e80b7e3799f777572399a4cff047eb02a81fa6e7a48ed4b
    - k8s.gcr.io/defaultbackend:1.4
    sizeBytes: 4844064
  - names:
    - 602401143452.dkr.ecr.us-east-2.amazonaws.com/eks/pause-amd64@sha256:bea77c323c47f7b573355516acf927691182d1333333d1f41b7544012fab7adf
    - 602401143452.dkr.ecr.us-east-2.amazonaws.com/eks/pause-amd64:3.1
    sizeBytes: 742472
  nodeInfo:
    architecture: amd64
    bootID: 27694b06-e7f2-4feb-8ffc-282739eba230
    containerRuntimeVersion: docker://17.6.2
    kernelVersion: 4.14.88-88.76.amzn2.x86_64
    kubeProxyVersion: v1.11.5
    kubeletVersion: v1.11.5
    machineID: 850b402de8cc415cad301d83360ba7ab
    operatingSystem: linux
    osImage: Amazon Linux 2
    systemUUID: EC220C39-D10F-9D7B-2732-C8F492335AFF
  volumesAttached:
  - devicePath: /dev/xvdcr
    name: kubernetes.io/aws-ebs/aws://us-east-2a/vol-0c94aa75e03f863b2
  volumesInUse:
  - kubernetes.io/aws-ebs/aws://us-east-2a/vol-0c94aa75e03f863b2

cluster config 1 cluster with 3 node groups each in AZ, One AZ as set minimum to 0.

#----------ng1
    eksctl create nodegroup \
    --cluster $NAME \
    --name $NG1_NAME \
    --node-zones $ZONE1 \
    --node-type ${NODE_TYPE:-t2.small} \
    --nodes ${DESIRED_NODE_COUNT:-3} \
    --nodes-max ${NODE_COUNT:-5}  \
    --nodes-min ${MIN_NODE_COUNT:-3} \
    --asg-access \
    --external-dns-access \
    --ssh-access --ssh-public-key ${SSH_PUBLIC_KEY:-keys/k8s-eks.pub}
#----------ng2
    eksctl create nodegroup \
    --cluster $NAME \
    --name $NG2_NAME \
    --node-zones $ZONE2 \
    --node-type ${NODE_TYPE:-t2.small} \
    --nodes ${DESIRED_NODE_COUNT:-3} \
    --nodes-max ${NODE_COUNT:-5}  \
    --nodes-min ${MIN_NODE_COUNT:-3} \
    --asg-access \
    --external-dns-access \
    --ssh-access --ssh-public-key ${SSH_PUBLIC_KEY:-keys/k8s-eks.pub}
# --node-zones $ZONE2 \
#----------ng3
    eksctl create nodegroup \
    --cluster $NAME \
    --name $NG3_NAME \
    --node-zones $ZONE3 \
    --node-type ${NODE_TYPE:-t2.small} \
    --nodes ${DESIRED_NODE_COUNT:-3} \
    --nodes-max ${NODE_COUNT:-5}  \
    --nodes-min 0 \
    --asg-access \
    --external-dns-access \
    --ssh-access --ssh-public-key ${SSH_PUBLIC_KEY:-keys/k8s-eks.pub}

CA helm:

helm install stable/cluster-autoscaler \
    --name aws-cluster-autoscaler \
    --namespace aws-cluster-autoscaler \
    --version 0.11.0 \
    --set autoDiscovery.clusterName=$NAME \
    --set awsRegion=$AWS_DEFAULT_REGION \
    --set sslCertPath=/etc/kubernetes/pki/ca.crt \
    --set rbac.create=true \
    --set autoscalingGroupsnamePrefix[0].maxSize=10 \
    --set autoscalingGroupsnamePrefix[0].minSize=2 \
    --set autoscalingGroupsnamePrefix[0].name=$NAME \
    --set extraArgs.scale-down-delay-after-add="5m0s" \
    --set extraArgs.scale-down-unneeded-time="5m0s" \
    --set extraArgs.scale-down-unready-time="10m0s" \
    --set extraArgs.scale-down-utilization-threshold=0.8 \
    --set extraArgs.scan-interval="20s" \
    --set extraArgs.skip-nodes-with-system-pods=0 \
    --set extraArgs.skip-nodes-with-local-storage=0 \
    --set extraArgs.balance-similar-node-groups="true" \
    --set replicaCount=2 \
    --set podDisruptionBudget="minAvailable: 1" \
    --set resources.limits.cpu="200m",resources.limits.memory="100Mi"
kubectl -n aws-cluster-autoscaler rollout status deployment aws-cluster-autoscaler

[prageethw]

update:
I can say this behavior is random and I have successfully able to re-run with same configs and this time CA successfully scaled-in to 2 nodes which is what I have expected and spread instances rationally across AZ., My guess is CA in above occasion felt into a condition that it could not come out.

kubectl get nodes -o yaml |grep zone

      failure-domain.beta.kubernetes.io/zone: us-east-2b
      failure-domain.beta.kubernetes.io/zone: us-east-2a

[Jeffwan]

@prageethw As I know, CA doesn't have native support for zone distribution but only --balance-similar-node-groups=true. The way you are doing is kind of hack. Honestly, I have not tested this feature and based on the source codes, looks like it firstly looks for similar nodegroup and then try to balancer loads. The problem is when CA compare similarity, node labels are also considered. In EKS,
you will see following labels on the node, failure-domain.beta.kubernetes.io/zone=us-west-2c is not ignored which means even you use same node group instance with exact same resource, node group is considered not similar. I am not sure if my guest is wrong or not, wait for @aleksandra-malinowska and @MaciekPytel to confirm.

alpha.eksctl.io/cluster-name=cluster-name
alpha.eksctl.io/nodegroup-name=ng-4ff9b808
beta.kubernetes.io/arch=amd64
beta.kubernetes.io/instance-type=p2.xlarge
beta.kubernetes.io/os=linux
failure-domain.beta.kubernetes.io/region=us-west-2
failure-domain.beta.kubernetes.io/zone=us-west-2c
kubernetes.io/hostname=ip-192xx.us-west-2.compute.internal

The logs you attach is for scale down, could you also attach scale up logs? That would be helpful.

[aleksandra-malinowska]

The problem is when CA compare similarity, node labels are also considered.

Node labels affect scheduling (via node selectors and node affinity). I agree a "smarter" way may be to disregard them if no pod is using this feature, or just rely on binpacking to take care of it (although it's costlier), but there's a reason it was considered in the first place.

failure-domain.beta.kubernetes.io/zone=us-west-2c is not ignored which means even you use same node group

Both failure-domain.beta.kubernetes.io/zone and failure-domain.beta.kubernetes.io/region are ignored by default. See https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/processors/nodegroupset/compare_nodegroups.go#L97

Most likely the label preventing balancing in your case is alpha.eksctl.io/nodegroup-name=ng-4ff9b808 (I'm assuming its value is different for different node groups).

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close