Priority Expander sometimes picks all groups as equal #3956
Description
Which component are you using?:
cluster-autoscaler with the Helm Chart
What version of the component are you using?:
Component version: v1.19.1
What k8s version are you using (kubectl version)?: v1.19.6-eks-49a6c0
kubectl version Output
$ kubectl version
Server Version: version.Info{Major:"1", Minor:"19+", GitVersion:"v1.19.6-eks-49a6c0", GitCommit:"49a6c0bf091506e7bafcdb1b142351b69363355a", GitTreeState:"clean", BuildDate:"2020-12-23T22:10:21Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}
What environment is this in?:
AWS EKS using Managed Node Groups
What did you expect to happen?:
With this configuration for priority expander:
data:
priorities: |
19:
- .*
20:
- eks-88bc2311-7b73-c393-bc9f-eac7e9fda14d
- eks-a0bc2311-7b5e-7c9f-8556-411634f5a6d6
- eks-cebc2311-7b5f-3205-366e-9234f4ddf72a
30:
- eks-c0bc2311-7b89-23cc-8af4-af21ada330fd
- eks-1cbc2311-7bde-88d7-5d71-9235bbdefa25
- eks-b4bc2311-7b7a-daa7-bd9f-0aa2d1450f25
40:
- eks-b4bc2311-7b63-a611-77ac-8a38f308948a
- eks-c4bc2311-7b67-0ee5-2ce6-b4ea06543894
- eks-4abc2311-7b63-796d-f5e2-6a5c4bdec5e3
And these flags:
Command:
./cluster-autoscaler
--cloud-provider=aws
--namespace=kube-system
--node-group-auto-discovery=asg:tag=k8s.io/cluster-autoscaler/enabled,kubernetes.io/cluster/pulseplatform-prod
--balance-similar-node-groups=true
--expander=priority
--logtostderr=true
--max-node-provision-time=10m
--scale-down-delay-after-delete=10m
--skip-nodes-with-local-storage=false
--skip-nodes-with-system-pods=false
--stderrthreshold=info
--v=4
I expected the groups with priority 30 to be the highest priority because the ones marked 40 are ARM nodes and tainted so not applicable to most of my workloads and also have a set size as you'll see in the logs.
What happened instead?:
Sometimes, but I can't figure out what triggers it, all 6 of the non-arm ASGs are marked as highest priority.
How to reproduce it (as minimally and precisely as possible):
It seems to be intermittent but essentially have 3 sets of ASGs (in this case managed by managed node groups, but that shouldn't matter here) for ARM, Spot and On Demand and use my configuration. Sometimes the spot and on demand will be considered the same priority for some reason.
Anything else we need to know?:
Here's the logs of the problem happening:
I0319 11:41:44.034420 1 aws_manager.go:269] Refreshed ASG list, next refresh after 2021-03-19 11:42:44.03440351 +0000 UTC m=+1234.584471812
I0319 11:41:44.105351 1 filter_out_schedulable.go:65] Filtering out schedulables
I0319 11:41:44.114934 1 filter_out_schedulable.go:132] Filtered out 0 pods using hints
I0319 11:41:44.116261 1 filter_out_schedulable.go:170] 0 pods were kept as unschedulable based on caching
I0319 11:41:44.116339 1 filter_out_schedulable.go:171] 0 pods marked as unschedulable can be scheduled.
I0319 11:41:44.116399 1 filter_out_schedulable.go:82] No schedulable pods
I0319 11:41:44.116454 1 klogx.go:86] Pod pulseplatform-production/football-data-read-service-66bf4c7c47-hd4mk is unschedulable
I0319 11:41:44.116769 1 scale_up.go:364] Upcoming 0 nodes
I0319 11:41:44.118827 1 scale_up.go:400] Skipping node group eks-4abc2311-7b63-796d-f5e2-6a5c4bdec5e3 - max size reached
I0319 11:41:44.128759 1 scale_up.go:400] Skipping node group eks-b4bc2311-7b63-a611-77ac-8a38f308948a - max size reached
I0319 11:41:44.130961 1 scale_up.go:400] Skipping node group eks-c4bc2311-7b67-0ee5-2ce6-b4ea06543894 - max size reached
I0319 11:41:44.132179 1 priority.go:118] Successfully loaded priority configuration from configmap.
I0319 11:41:44.132209 1 priority.go:167] priority expander: eks-1cbc2311-7bde-88d7-5d71-9235bbdefa25 chosen as the highest available
I0319 11:41:44.132218 1 priority.go:167] priority expander: eks-88bc2311-7b73-c393-bc9f-eac7e9fda14d chosen as the highest available
I0319 11:41:44.132225 1 priority.go:167] priority expander: eks-a0bc2311-7b5e-7c9f-8556-411634f5a6d6 chosen as the highest available
I0319 11:41:44.132232 1 priority.go:167] priority expander: eks-b4bc2311-7b7a-daa7-bd9f-0aa2d1450f25 chosen as the highest available
I0319 11:41:44.132239 1 priority.go:167] priority expander: eks-c0bc2311-7b89-23cc-8af4-af21ada330fd chosen as the highest available
I0319 11:41:44.132246 1 priority.go:167] priority expander: eks-cebc2311-7b5f-3205-366e-9234f4ddf72a chosen as the highest available
I0319 11:41:44.132262 1 scale_up.go:456] Best option to resize: eks-a0bc2311-7b5e-7c9f-8556-411634f5a6d6
I0319 11:41:44.132276 1 scale_up.go:460] Estimated 1 nodes needed in eks-a0bc2311-7b5e-7c9f-8556-411634f5a6d6
I0319 11:41:44.132497 1 scale_up.go:566] Splitting scale-up between 3 similar node groups: {eks-a0bc2311-7b5e-7c9f-8556-411634f5a6d6, eks-88bc2311-7b73-c393-bc9f-eac7e9fda14d, eks-cebc2311-7b5f-3205-366e-9234f4ddf72a}
I0319 11:41:44.132518 1 scale_up.go:574] Final scale-up plan: [{eks-a0bc2311-7b5e-7c9f-8556-411634f5a6d6 1->2 (max: 8)}]
I0319 11:41:44.132545 1 scale_up.go:663] Scale-up: setting group eks-a0bc2311-7b5e-7c9f-8556-411634f5a6d6 size to 2
I0319 11:41:44.132581 1 auto_scaling_groups.go:219] Setting asg eks-a0bc2311-7b5e-7c9f-8556-411634f5a6d6 size to 2
Mostly it picks the 3 I expect but the configuration is consistent between times where it picks all 6 and picking the correct 3 and ruling out the other 3 because they are lower priority.