Scale-down causes downtime for the pods to be moved in another node (aws EKS)

[niteshsakhiya]

I have set up K8S cluster using EKS. CA has been configured to increase/decrease the number of nodes based on resources availability for pods. During scale-down, the CA terminates a node before moving pods in the node on another node. So, the pods get scheduled on another node after the node gets terminated. Hence, There is some downtime until the re-scheduled pods become healthy on another node.

How can I avoid the downtime by ensuring that the pods get scheduled on another node before the node gets terminated?

Deployment :

apiVersion: apps/v1
kind: Deployment
metadata:
  name: cluster-autoscaler
  namespace: kube-system
  labels:
    app: cluster-autoscaler
spec:
  replicas: 1
  selector:
    matchLabels:
      app: cluster-autoscaler
  template:
    metadata:
      labels:
        app: cluster-autoscaler
    spec:
      serviceAccountName: cluster-autoscaler
      containers:
        - image: k8s.gcr.io/cluster-autoscaler:v1.12.3
          name: cluster-autoscaler
          resources:
            limits:
              cpu: 100m
              memory: 300Mi
            requests:
              cpu: 100m
              memory: 300Mi
          command:
            - ./cluster-autoscaler
            - --v=4
            - --stderrthreshold=info
            - --cloud-provider=aws
            - --skip-nodes-with-local-storage=false
            - --expander=least-waste
            - --node-group-auto-discovery=asg:tag=k8s.io/cluster-autoscaler/enabled,k8s.io/cluster-autoscaler/production
            - --balance-similar-node-groups=true
          env:
            - name: AWS_REGION
              value: eu-central-1
          volumeMounts:
            - name: ssl-certs
              mountPath: /etc/ssl/certs/ca-certificates.crt
              readOnly: true
          imagePullPolicy: "Always"
      volumes:
        - name: ssl-certs
          hostPath:
            path: "/etc/kubernetes/pki/ca.crt"

[Jeffwan]

@niteshsakhiya can you take a look at this issue?
#1907

We have some solutions but not elegant optimized.

1. make node unschedule.
2. add timeout since upstream service controller takes up to 100s to get node status.

Jeffwan@0c02d5b#diff-5f36cd12baa8998cd7f2ba7c4a00cbc6

[Jeffwan]

/sig aws

[niteshsakhiya]

@Jeffwan
I checked the issue.

1. Sorry, I am not sure about how making the node unSchedulable would help here.
2. The timeout is already set to 10 minutes(default).

[Jeffwan]

kubernetes service controller will fetch node changes. For every iteration, it will check node changes, if this node is unSchedulable, then it will call cloudprovider method to remove node from load balancer. If node just have taint like CA grants now, aws cloud provider won't remove it from load balancer which is not what we expect. That's the reason I make node unschedule. We have more discussion in the thread I show you, talking about elegant long term support.

[gustavosoares]

This is something that interest me. Can’t CA drain a node before actually terminating it?

[aleksandra-malinowska]

This is something that interest me. Can’t CA drain a node before actually terminating it?

CA drains (or at least, should drain) a node before terminating. It uses eviction API, and if eviction can't be granted, backs off from deleting that node.

[Constantin07]

Please, can someone explain how CA can drain a specific node upfront before AWS ASG decides which node exactly to terminate on scale-down event ?

e.g. when CA changes the desired nodes, e.g. from 3 to 2, and then ASG starts to terminate a random node - how does CA know which node to drain ?

[aleksandra-malinowska]

Please, can someone explain how CA can drain a specific node upfront before AWS ASG decides which node exactly to terminate on scale-down event ?

e.g. when CA changes the desired nodes, e.g. from 3 to 2, and then ASG starts to terminate a random node - how does CA know which node to drain ?

CA always requests deleting a specific node. This is a basic requirement for implementing support for a new cloud provider. My understanding is that so far it worked on AWS as well.

[aleksandra-malinowska]

Relevant API call: https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/auto_scaling_groups.go#L242

@Jeffwan can you explain what's going on here? This doesn't sound like a Cluster Autoscaler issue, right?

[Constantin07]

@aleksandra-malinowska OK, so it looks like upon termination of specific instance we do decrement the desired number. Now it makes sense. Thanks.

[Jeffwan]

Please, can someone explain how CA can drain a specific node upfront before AWS ASG decides which node exactly to terminate on scale-down event ?

e.g. when CA changes the desired nodes, e.g. from 3 to 2, and then ASG starts to terminate a random node - how does CA know which node to drain ?

I got few days off and just see the message.
As Alexa said, "CA always requests deleting a specific node." ASG won't randomly terminate node. Instead, CA drain specific node and ASG terminate node by node id.

The root problem of this issue is CA use evict API rather than Drain API but service controller uses a NotScheduable label to remove a node from load balancer endpoints. Only drain API will label node, evict API will not do that.

[Constantin07]

@Jeffwan thanks for explanation. Makes sense.