The `safe-to-evict` annotation prevent worker-nodes scale-down continously

[cdlliuy]

First of all, I used safe-to-evict annotation to prevent my existing pod be broken by scale down . But with this annotation, cluster autoscaler simply jump to evaluate the worker node , and then NEW workloads with safe-to-evict come to that worker node again, so the worker node can't be removed at all ...

Below is the detail experiments:

First of all, I added a LARGE workload to trigger scale-out with job workload. Annotation is added, and once the job is done, the pod is shown as completed.

cat <<EOF | kubectl apply -f -
apiVersion: batch/v1
kind: Job
metadata:
  name: busybox-job-600
spec:
  template:
    metadata:
      annotations:
        cluster-autoscaler.kubernetes.io/safe-to-evict: "false"  
    spec:
      restartPolicy: Never
      containers:
      - name: busybox
        image: busybox
        imagePullPolicy: IfNotPresent
        resources:
          requests:
            memory: "1Gi"
            cpu: "200m"
          limits:
            memory: "1Gi"
            cpu: "200m"
        command: ['sh', '-c', 'echo Container 1 is Running ; sleep 600']
  backoffLimit: 4
  parallelism: 50
EOF

Then, I created a smaller workload with cronjob to simulate the continuous job jumped in with the evict annotation.

cat <<EOF | kubectl apply -f -
apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: busybox-cronjob
spec:
  schedule: "*/2 * * * *"
  jobTemplate:
    spec:
      parallelism: 10
      template:
        metadata:
          annotations:
            cluster-autoscaler.kubernetes.io/safe-to-evict: "false"        
        spec:
          restartPolicy: Never
          containers:
          - name: busybox
            image: busybox
            args:
            - /bin/sh
            - -c
            - date; echo Hello from the Kubernetes cluster; sleep 30
            resources:
              requests:
                memory: "1Gi"
                cpu: "200m"
              limits:
                memory: "1Gi"
                cpu: "200m"
EOF

The cluster autoscaler setting is

      - command:
        - ./cluster-autoscaler
        - --v=4
        - --balance-similar-node-groups=true
        - --alsologtostderr=true
        - --stderrthreshold=info
        - --cloud-provider=IKS
        - --skip-nodes-with-local-storage=true
        - --skip-nodes-with-system-pods=true
        - --scale-down-unneeded-time=10m
        - --scale-down-delay-after-add=10m
        - --scale-down-delay-after-delete=10m
        - --scale-down-utilization-threshold=0.5
        - --scan-interval=1m
        - --expander=random
        - --max-inactivity=10m
        - --max-failing-time=15m
        - --leader-elect=false
        - --max-node-provision-time=120m
        - --ignore-daemonsets-utilization=false
        - --max-bulk-soft-taint-count=0
        - --max-bulk-soft-taint-time=10m

====================

Observations:

• Once The BIG job workload was ended, the cpu/memory utilization is reduced, but the worker node can't be removed due to the annotation (expected behavior)

I0603 07:04:25.173566       1 scale_down.go:462] Node 10.243.128.9 - cpu utilization 0.556266
I0603 07:04:25.173579       1 scale_down.go:466] Node 10.243.128.9 is not suitable for removal - cpu utilization too big (0.556266)
I0603 07:04:25.173592       1 scale_down.go:462] Node 10.243.128.16 - cpu utilization 0.121483
I0603 07:04:25.173602       1 scale_down.go:462] Node 10.243.128.17 - memory utilization 0.175274
I0603 07:04:25.173624       1 scale_down.go:462] Node 10.243.128.4 - cpu utilization 0.588747
I0603 07:04:25.173634       1 scale_down.go:466] Node 10.243.128.4 is not suitable for removal - cpu utilization too big (0.588747)
I0603 07:04:25.173819       1 scale_down.go:511] Finding additional 2 candidates for scale down.
I0603 07:04:25.173899       1 cluster.go:93] Fast evaluation: 10.243.128.16 for removal
I0603 07:04:25.173911       1 cluster.go:107] Fast evaluation: node 10.243.128.16 cannot be removed: pod annotated as not safe to evict present: busybox-cronjob-1591167840-qzrfr
I0603 07:04:25.173916       1 cluster.go:93] Fast evaluation: 10.243.128.17 for removal
I0603 07:04:25.173922       1 cluster.go:107] Fast evaluation: node 10.243.128.17 cannot be removed: pod annotated as not safe to evict present: busybox-cronjob-1591167840-nwdvl
I0603 07:04:25.173931       1 scale_down.go:548] 2 nodes found to be unremovable in simulation, will re-check them at 2020-06-03 07:09:23.562749096 +0000 UTC m=+22534.296804927

• But given the cronjob workload jumped in continuously with the annotation, so all the worker nodes will have running pods with evict:false annotation ... Finally, cluster autoscaler can't scale down any worker node at all

=============

I expected the cluster autoscaler can mark the worker node unschedule to prevent further workload jumped in when it sees evict annotation attached.

Any comment?

[MaciekPytel]

I think what you describe is WAI - the annotation is preventing scale-down as expected. I don't think it should automatically block scheduling of other pods as soon as one pod with safe-to-evict: false is scheduled - there are plenty of cases where running multiple pods with safe-to-evict: false on a single node is desired.

If you have multiple non-evictable pods with different resource requirements my recommendation would be to run those on separate, specialized nodeGroups (node-pools, ASGs, VMSS, etc) and use combination of nodeSelector and taints to control which pods can schedule to which nodes.

[cdlliuy]

@MaciekPytel , Not automatically block scheduling ..
Just block unscheduling if the resource utilization is lower and there are still safe-to-evict: false running on it ..

Even to assign the safe-to-evict: false workloads to a separate nodeGroups, then I think this specific node group can't be scale-down at all if there is continuously safe-to-evict: false going on.

[bitfactory-henno-schooljan]

There really should be a way to prevent new workloads entering a node at some point, else you can end up with nodes that always have at least one non-evictable workload so it never reaches the point where it gets tainted. Apparently the cluster autoscaler does not taint a node until there are no workloads preventing scale-down, which may never happen.

Couldn't it just examine a node group to see if the current workloads would fit on less nodes, and then taint the least busy node? That would allow any existing workloads to run to completion on that node without allowing new workloads, so it may scale down eventually.

I think what you describe is WAI - the annotation is preventing scale-down as expected. I don't think it should automatically block scheduling of other pods as soon as one pod with safe-to-evict: false is scheduled - there are plenty of cases where running multiple pods with safe-to-evict: false on a single node is desired.

True, but I think the case where all nodes are under-utilized but never completely free of non-evictable workloads should be handled as well. At some point you want it to scale down because it fits on fewer nodes, and the only way to do that is to block scheduling while allowing existing workloads to finish.

[elmiko]

I expected the cluster autoscaler can mark the worker node unschedule to prevent further workload jumped in when it sees evict annotation attached.

Any comment?

i see that you have the setting --scale-down-unneeded-time=10m, wouldn't this prevent the autoscaler from removing an under-used node for 10 minutes?

if so, then during that 10 minute window it would be possible for new pods to be added on that node, iiuc.

mind you, this doesn't address some of the deeper questions, such as:

Couldn't it just examine a node group to see if the current workloads would fit on less nodes, and then taint the least busy node?

imo, this seems like something that could be possible a flag. whether it's this specific feature or not, it seems like having a flag that can taint a node after the last pod is completed seems like it would be useful, at least in this case.

[MaciekPytel]

I think it's difficult to have a simple solution that works for general case. In principle CA doesn't know that a pod with safe-to-evict: false will end soon (or at all). It may just as well run for days (ex. if it's a database). In that case simply tainting a node is obviously pretty bad, as it will lead to all that space being inaccessible for other pods, some of which may not even have space to run.
Doing some sort of simulated binpacking as described by @bitfactory-henno-schooljan may be the solution, but it would require re-implementing scale-down from scratch (current scale-down algorithm treats removing each node as an independent problem, it doesn't look at the whole NodeGroup). And even than you'd have an issue when new pod is created that doesn't schedule because of this taint - you'd need to also integrate this logic into scale-up.

I think an easier alternative may be to use scheduler profiles to switch from NodeResourcesLeastAllocated to NodeResourcesMostAllocated. This way scheduler will prefer placing pods on nodes that already have other pods running. Any node that has just a single pod with safe-to-evict is unlikely to be picked by scheduler, unless there is no other option.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[carlosjgp]

/reopen

I agree that ClusterAutoscaler should ignore Completed and Failed PODs when looking for the annotation these PODs are not relevant anymore

[k8s-ci-robot]

@carlosjgp: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen

I agree that ClusterAutoscaler should ignore Completed and Failed PODs when looking for the annotation these PODs are not relevant anymore

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.