Skip to content

Cluster Autoscaler doesn't scale up based on volume attachment limits anymore on EKS #8241

Open
@sok1234

Description

@sok1234

Which component are you using?:

/area cluster-autoscaler

What version of the component are you using?:

Component version:

cluster-autoscaler-chart-9.46.0
cluster-autoscaler-chart-9.36.0
cluster-autoscaler-chart-9.29.0

What k8s version are you using (kubectl version)?:

kubectl version Output
$ kubectl version
Client Version: v1.30.5
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.31.9-eks-5d4a308

What environment is this in?:

AWS EKS

What did you expect to happen?:

The autoscaler should trigger a scale up once the volume attachment limit is reached on the node where the pod is scheduled

What happened instead?:

Cluster didn't scale up and instead the following error is thrown:

Warning FailedAttachVolume 103s attachdetach-controller AttachVolume.Attach failed for volume "pvc-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" : rpc error: code = Internal desc = Could not attach volume "vol-xxxxxxxxxxxxxxxxx" to node "i-xxxxxxxxxxxxxxxx": could not attach volume "vol-058a94147ae79105c" to node "i-xxxxxxxxxxxxxxxx": operation error EC2: AttachVolume, https response error StatusCode: 400, RequestID: 1e78534e-6ecc-44f3-9b75-226dc81d13ed, api error AttachmentLimitExceeded: You may attach up to 27 devices for this instance type. Please use a larger instance size to attach more volumes. For a detailed list of limits please see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/volume_limits.html

How to reproduce it (as minimally and precisely as possible):

Deploy any of the above mentioned Cluster Autoscaler helm charts on EKS 1.31 using c6i.8xlarge node type , create a dummy PVC and attach it to a pod, repeat that until a node has 25 volumes attached and the next pod will be stuck on ContainerCreating with the above error.

Anything else we need to know?:

This used to work on older versions of Autoscaler and EKS when the CSIMigration feature gates had to be added but since then the feature gates have been deprecated and this should have worked out of the box but it doesn't.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions