NodeGetInfo: MaxVolumesPerNode is wrong for a lot of nitro instance types

[ialidzhikov]

/sig storage
/kind bug
/area provider/aws
/priority critical-urgent

What happened?
The current handling for volume limits is copied from the in-tree plugin:

aws-ebs-csi-driver/pkg/driver/node.go

Lines 691 to 702 in f298d57

	// getVolumesLimit returns the limit of volumes that the node supports
	func (d *nodeService) getVolumesLimit() int64 {
	if d.driverOptions.volumeAttachLimit >= 0 {
	return d.driverOptions.volumeAttachLimit
	}
	ebsNitroInstanceTypeRegex := "^[cmr]5.*|t3|z1d"
	instanceType := d.metadata.GetInstanceType()
	if ok, _ := regexp.MatchString(ebsNitroInstanceTypeRegex, instanceType); ok {
	return defaultMaxEBSNitroVolumes
	}
	return defaultMaxEBSVolumes
	}

The problems I see with the current handling:

• The nitro instance types regex ^[cmr]5.*|t3|z1d is not good enough to cover all nitro instance types.
  From the AWS docs:

The following instances are built on the Nitro System:

Virtualized: A1, C5, C5a, C5ad, C5d, C5n, C6g, C6gd, C6gn, C6i, D3, D3en, DL1, G4, G4ad, G5, G5g, I3en, Im4gn, Inf1, Is4gen, M5, M5a, M5ad, M5d, M5dn, M5n, M5zn, M6a, M6g, M6gd, M6i, p3dn.24xlarge, P4, R5, R5a, R5ad, R5b, R5d, R5dn, R5n, R6g, R6gd, R6i, T3, T3a, T4g, high memory (u-*), VT1, X2gd, and z1d

Bare metal: a1.metal, c5.metal, c5d.metal, c5n.metal, c6g.metal, c6gd.metal, i3.metal, i3en.metal, m5.metal, m5d.metal, m5dn.metal, m5n.metal, m5zn.metal, m6g.metal, m6gd.metal, mac1.metal, r5.metal, r5b.metal, r5d.metal, r5dn.metal, r5n.metal, r6g.metal, r6gd.metal, r6i.metal, u-6tb1.metal, u-9tb1.metal, u-12tb1.metal, u-18tb1.metal, u-24tb1.metal, x2gd.metal, and z1d.metal

I will leave to the reader to discover how many of these instance types are not covered by the current regex.

• aws-ebs-csi-driver does not consider attached network interfaces or instance store volumes:
  From AWS docs:

Instances built on the Nitro System support a maximum number of attachments, which are shared between network interfaces, EBS volumes, and NVMe instance store volumes. Every instance has at least one network interface attachment. NVMe instance store volumes are automatically attached. For more information, see Elastic network interfaces and Instance store volumes.

The issue with the networking interfaces and instance store volumes is pretty well described in kubernetes/kubernetes#80967 and #427.

• Many nitro instance types have custom EBS volume limits:
  From the From AWS docs:

For other instances, the following limits apply:
- d3.8xlarge and d3en.12xlarge instances support a maximum of 3 EBS volumes.
- inf1.xlarge and inf1.2xlarge instances support a maximum of 26 EBS volumes.
- inf1.6xlarge instances support a maximum of 23 EBS volumes.
- inf1.24xlarge instances support a maximum of 11 EBS volumes.
- Most bare metal instances support a maximum of 31 EBS volumes.
- mac1.metal instances support a maximum of 16 EBS volumes.
- High memory virtualized instances support a maximum of 27 EBS volumes.
- High memory bare metal instances support a maximum of 19 EBS volumes.
If you launched a u-6tb1.metal, u-9tb1.metal, or u-12tb1.metal high memory bare metal instance before March 12, 2020, it supports a maximum of 14 EBS volumes. To attach up to 19 EBS volumes to these instances, contact your account team to upgrade the instance at no additional cost.

• (?) How is the root EBS volume handled? Is it always required to specify it when an instance is launched? Does it also contribute to the EBS attachments limit?

With all of the above issues combined, this leads to a very poor user experience when using various nitro instance types.

What you expected to happen?
The volume attachment limit to reflect the reality.

How to reproduce it (as minimally and precisely as possible)?

1. Create a Kubernetes cluster with a worker Node of type m6i.8xlarge

2. Check the corresponding CSINode object and make sure that it wrongly has 39 as limit

  spec:
    drivers:
    - allocatable:
        count: 39
      name: ebs.csi.aws.com

3. Make sure that the 27th volume on the Node cannot be attached.

Corresponding Pods fails to start with:

$ k describe po web-26

  Warning  FailedAttachVolume  5s (x5 over 14s)  attachdetach-controller  AttachVolume.Attach failed for volume "pv-shoot--foo--bar-<uuid>" : rpc error: code = Internal desc = Could not attach volume "vol-1234" to node "i-1234": attachment of disk "vol-1234" failed, expected device to be attached but was attaching

Anything else we need to know?:
Please do not close this issue as duplicate of kubernetes/kubernetes#80967 or #427.

Environment

• Kubernetes version (use kubectl version): v1.22.2
• Driver version: v1.1.4

[k8s-ci-robot]

@ialidzhikov: The label(s) area/storage cannot be applied, because the repository doesn't have them.

Details

In response to this:

/area storage
/kind bug
/area provider/aws
/priority critical-urgent

What happened?
The current handling for volume limits is copied from the in-tree plugin:

aws-ebs-csi-driver/pkg/driver/node.go

Lines 691 to 702 in f298d57

	// getVolumesLimit returns the limit of volumes that the node supports
	func (d *nodeService) getVolumesLimit() int64 {
	if d.driverOptions.volumeAttachLimit >= 0 {
	return d.driverOptions.volumeAttachLimit
	}
	ebsNitroInstanceTypeRegex := "^[cmr]5.*|t3|z1d"
	instanceType := d.metadata.GetInstanceType()
	if ok, _ := regexp.MatchString(ebsNitroInstanceTypeRegex, instanceType); ok {
	return defaultMaxEBSNitroVolumes
	}
	return defaultMaxEBSVolumes
	}

The problems I see with the current handling:

• The nitro instance types regex ^[cmr]5.*|t3|z1d is not good enough to cover all nitro instance types.
  From the AWS docs:

The following instances are built on the Nitro System:

Virtualized: A1, C5, C5a, C5ad, C5d, C5n, C6g, C6gd, C6gn, C6i, D3, D3en, DL1, G4, G4ad, G5, G5g, I3en, Im4gn, Inf1, Is4gen, M5, M5a, M5ad, M5d, M5dn, M5n, M5zn, M6a, M6g, M6gd, M6i, p3dn.24xlarge, P4, R5, R5a, R5ad, R5b, R5d, R5dn, R5n, R6g, R6gd, R6i, T3, T3a, T4g, high memory (u-*), VT1, X2gd, and z1d

Bare metal: a1.metal, c5.metal, c5d.metal, c5n.metal, c6g.metal, c6gd.metal, i3.metal, i3en.metal, m5.metal, m5d.metal, m5dn.metal, m5n.metal, m5zn.metal, m6g.metal, m6gd.metal, mac1.metal, r5.metal, r5b.metal, r5d.metal, r5dn.metal, r5n.metal, r6g.metal, r6gd.metal, r6i.metal, u-6tb1.metal, u-9tb1.metal, u-12tb1.metal, u-18tb1.metal, u-24tb1.metal, x2gd.metal, and z1d.metal

I will leave to the reader to discover how many of these instance types are not covered by the current regex.

• aws-ebs-csi-driver does not consider attached network interfaces or instance store volumes:
  From AWS docs:

Instances built on the Nitro System support a maximum number of attachments, which are shared between network interfaces, EBS volumes, and NVMe instance store volumes. Every instance has at least one network interface attachment. NVMe instance store volumes are automatically attached. For more information, see Elastic network interfaces and Instance store volumes.

The issue with the networking interfaces and instance store volumes is pretty well described in kubernetes/kubernetes#80967 and #427.

• Many nitro instance types have custom EBS volume limits:
  From the From AWS docs:

For other instances, the following limits apply:
- d3.8xlarge and d3en.12xlarge instances support a maximum of 3 EBS volumes.
- inf1.xlarge and inf1.2xlarge instances support a maximum of 26 EBS volumes.
- inf1.6xlarge instances support a maximum of 23 EBS volumes.
- inf1.24xlarge instances support a maximum of 11 EBS volumes.
- Most bare metal instances support a maximum of 31 EBS volumes.
- mac1.metal instances support a maximum of 16 EBS volumes.
- High memory virtualized instances support a maximum of 27 EBS volumes.
- High memory bare metal instances support a maximum of 19 EBS volumes.
If you launched a u-6tb1.metal, u-9tb1.metal, or u-12tb1.metal high memory bare metal instance before March 12, 2020, it supports a maximum of 14 EBS volumes. To attach up to 19 EBS volumes to these instances, contact your account team to upgrade the instance at no additional cost.

• (?) How is the root EBS volume handled? Is it always required to specify it when an instance is launched? Does it also contribute to the EBS attachments limit?

With all of the above issues combined, this leads to a very poor user experience when using various nitro instance types.

What you expected to happen?
The volume attachment limit to reflect the reality.

How to reproduce it (as minimally and precisely as possible)?

1. Create a Kubernetes cluster with a worker Node of type m6i.8xlarge

2. Check the corresponding CSINode object and make sure that it wrongly has 39 as limit

 spec:
   drivers:
   - allocatable:
       count: 39
     name: ebs.csi.aws.com

3. Make sure that the 27th volume on the Node cannot be attached.

Corresponding Pods fails to start with:

$ k describe po web-26

 Warning  FailedAttachVolume  5s (x5 over 14s)  attachdetach-controller  AttachVolume.Attach failed for volume "pv-shoot--foo--bar-<uuid>" : rpc error: code = Internal desc = Could not attach volume "vol-1234" to node "i-1234": attachment of disk "vol-1234" failed, expected device to be attached but was attaching

Anything else we need to know?:
Please do not close this issue as duplicate of kubernetes/kubernetes#80967 or #427.

Environment

• Kubernetes version (use kubectl version): v1.22.2
• Driver version: v1.1.4

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[ialidzhikov]

/sig storage

[ialidzhikov]

/priority critical-urgent

[stevehipwell]

Isn't this even more of an issue when we consider that the attachment limit also impacts ENIs and NVMe drives? So should this value be dynamic based on external considerations?

Out of interest does anyone know where the 25 comes from, considering 28 is documented as the Nitro default for total attachments.

[stevehipwell]

It looks like #1075 addresses my concern, I didn't see a PR linked so I commented before looking.

[gtxu]

We are now working on this issue together with the previous effort of #1075, a new PR will be posted to address this.

[stevehipwell]

@gtxu what would you expect the failure behaviour to be? We've seen a volume stuck attaching which causes a pod to be un-schedulable. It's possible that the cluster in question was using v1.5.0 when the following error was observed.

E0124 14:40:34.297032 10 csi_attacher.go:631] kubernetes.io/csi: attachment for vol-025c36d2cda510ab7 failed: rpc error: code = Internal desc = Could not attach volume "vol-025c36d2cda510ab7" to node "i-0c4e3e1f0a1915061": attachment of disk "vol-025c36d2cda510ab7" failed, expected device to be attached but was attaching