Description
Which component are you using?:
/area cluster-autoscaler
What version of the component are you using?:
Component version:
cluster-autoscaler-chart-9.46.0
cluster-autoscaler-chart-9.36.0
cluster-autoscaler-chart-9.29.0
What k8s version are you using (kubectl version
)?:
kubectl version
Output
$ kubectl version Client Version: v1.30.5 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.31.9-eks-5d4a308
What environment is this in?:
AWS EKS
What did you expect to happen?:
The autoscaler should trigger a scale up once the volume attachment limit is reached on the node where the pod is scheduled
What happened instead?:
Cluster didn't scale up and instead the following error is thrown:
Warning FailedAttachVolume 103s attachdetach-controller AttachVolume.Attach failed for volume "pvc-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" : rpc error: code = Internal desc = Could not attach volume "vol-xxxxxxxxxxxxxxxxx" to node "i-xxxxxxxxxxxxxxxx": could not attach volume "vol-058a94147ae79105c" to node "i-xxxxxxxxxxxxxxxx": operation error EC2: AttachVolume, https response error StatusCode: 400, RequestID: 1e78534e-6ecc-44f3-9b75-226dc81d13ed, api error AttachmentLimitExceeded: You may attach up to 27 devices for this instance type. Please use a larger instance size to attach more volumes. For a detailed list of limits please see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/volume_limits.html
How to reproduce it (as minimally and precisely as possible):
Deploy any of the above mentioned Cluster Autoscaler helm charts on EKS 1.31 using c6i.8xlarge
node type , create a dummy PVC and attach it to a pod, repeat that until a node has 25 volumes attached and the next pod will be stuck on ContainerCreating with the above error.
Anything else we need to know?:
This used to work on older versions of Autoscaler and EKS when the CSIMigration feature gates had to be added but since then the feature gates have been deprecated and this should have worked out of the box but it doesn't.