Cluster Autoscaler does not scale down with Topology Spread (despite max skew being satisfied when scaled down)

[kevinkrp93]

Which component are you using?:
Cluster Autoscaler

What version of the component are you using?:

Component version:

What k8s version are you using (kubectl version)?:

1.30.7

What environment is this in?:

AKS

What did you expect to happen?:

Cluster Autoscaler scale down to 2 nodes from 4

What happened instead?:

Cluster Autoscaler does not scale down nodes

How to reproduce it (as minimally and precisely as possible):

1. create nodepool with w/ 4 nodes, min/max set to 4
2. Running Deployment spec

 apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-app
  labels:
    app: test-app
spec:
  replicas: 5
  selector:
    matchLabels:
      app: test-app
  template:
    metadata:
      labels:
        app: test-app
    spec:
      nodeSelector:
        agentpool: kubefabric
      topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: "kubernetes.io/hostname"
        whenUnsatisfiable: DoNotSchedule
        minDomains: 2
        labelSelector:
          matchLabels:
            app: test-app
        matchLabelKeys:
            - pod-template-hash
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80
        resources:
          requests:
            memory: "256Mi"
            cpu: "500m"  # 500m = 0.5 CPU core
          limits:
            memory: "512Mi"
            cpu: "1"  # 1 CPU core

3. set min to 2, and wait for scale down. (There will be no scaledown)
4. Update Deployment spec to remove topology spread constraints, observe scale down

Anything else we need to know?:

Related to Azure/AKS#4201

[adrianmoisey]

/area cluster-autoscaler

[MenD32]

It looks like it is ignoring nodes because they were recently considered unremovable.

Cluster autoscaler does not check a node for removability if it was considered unremovable fairly recently (in your example in the last 5 minutes).

can you provide an earlier log suggesting why they might unremovable or a later log showing the next attempt to remove them?

[kevinkrp93]

@MenD32 - scale down unneeded time in my CAS profile is 3 min. Also the log line always says 5m always. This is all I see as the previous log. My point is it is not even attempting to delete them

other log line

ignore log line

[MenD32]

can you show the log that shows when the nodes became unremovable?