[EKS] [request]: enable support for provisioning gp3 volumes

[iamsudip]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request

Tell us about your request
Add support for provisioning gp3 volume: https://github.com/kubernetes-sigs/aws-ebs-csi-driver

Which service(s) is this request for?
EKS

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
I have provisioned "gp3" storageclass and tried adding a pvc/pv. But it failed with Warning ProvisioningFailed 9s (x5 over 59s) persistentvolume-controller Failed to provision volume with StorageClass "gp3": invalid AWS VolumeType "gp3". The underlying volume-provisioner managed by controlplane needs an upgrade.

Are you currently working around this issue?
Still using gp2

[mikestef9]

Hi @iamsudip

We just recently released gp3 support in the EBS CSI driver as part of the most recent release

https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/CHANGELOG-0.x.md

Are you sure you are using the latest version?

[iamsudip]

@mikestef9 How shall we upgrade the driver? We are using kubernetes version 1.18 EKS.

[forsberg]

I guess one has to manually install the CSI driver in the cluster, and use the CSI driver to provision gp3 volumes.

Described here: https://docs.aws.amazon.com/eks/latest/userguide/ebs-csi.html
And here: https://www.stacksimplify.com/aws-eks/kubernetes-storage/install-aws-ebs-csi-driver-on-aws-eks-for-persistent-storage/

It certainly would help if there was a simple way of having this done automatically.

[ffjia]

Any way to support gp3 without using EBS CSI driver?

[dudicoco]

Can this please be added to the standard EBS storage class provisioner? I believe a lot of us prefer not to migrate to the CSI driver just for this functionality.
Thanks!

[Pluies]

I'd like to see out-of-the-box gp3 support as well, but I'm not sure this will happen – the official AWS documentation on Storage Classes seems to indicate that the CSI driver is the way to go:

This topic uses the in-tree Amazon EBS storage provisioner. The existing in-tree Amazon EBS plugin is still supported, but by using a CSI driver, you benefit from the decoupling of Kubernetes upstream release cycle and CSI driver release cycle. Eventually, the in-tree plugin will be deprecated in favor of the CSI driver.

[mikestef9]

Closing as latest release of EBS CSI driver now supports gp3

https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/CHANGELOG-0.x.md#v080

We do not have any plans to backport gp3 support to the deprecated in-tree provisioner. Our focus right now is on getting the EBS CSI driver to 1.0 release in preparation for enabling the CSIMigration flag on EKS clusters.

[nodesocket]

@mikestef9 how do I even go about using the EBS CSI driver? Running EKS Kubernetes version 1.20 and platform version eks.2. Would another option be using annotations on the deployment to signal using gp3 volumes?

[mikestef9]

You can find documentation for using the EBS CSI driver here

https://docs.aws.amazon.com/eks/latest/userguide/ebs-csi.html

and more information in the GitHub project

https://github.com/kubernetes-sigs/aws-ebs-csi-driver

[nodesocket]

@mikestef9 does having to do all this manual work to get the EBS CSI driver up and running kind of defeat the purpose of paying for EKS? I.E. since I don't have the EBS CSI driver setup, what is my EKS cluster using currently?

[mikestef9]

We are very close to launching EKS add-ons support for the EBS CSI driver, you can follow this issue for progress #247. This will provide a one click installation experience for the EBS CSI driver.

[nodesocket]

@mikestef9 thanks for the information and link to the issue. I do wonder though, why is EBS CSI driver a plugin and opt-in? Shouldn't it just replace the current driver starting at maybe Kubernetes version 1.22 or something? Why keep the legacy default installed driver around if the EBS CSI is superior and supports gp3?

[bozerkins]

I am also curious, why CSI driver isn't the default in AWS EKS. Also, more curious why gp3 isn't the default choice in EKS.

[winter7]

Hi @mikestef9 I'm currently using CSI driver to apply gp3 volume.
My further question is, do you support provisioning IOPS and throughput for gp3 in storageClass? Since parameter iopsPerGB seemed to be not working with gp3 type according to my test. It would be very grateful if you have any idea.

[aldredb]

Is there any impact of modifying the gp2 EBS volume created by the EBS in-tree driver to gp3 via AWS console?

[anitakrueger]

Is there any impact of modifying the gp2 EBS volume created by the EBS in-tree driver to gp3 via AWS console?

I would like to know this as well. All our prometheus volumes have been created as gp2 and just updating the helm values to gp3 the provisioning of the pvc failed. So I thought, I could just provision as gp2 and then update on the AWS console...

[nodesocket]

Is there any impact of modifying the gp2 EBS volume created by the EBS in-tree driver to gp3 via AWS console?

I think you should be fine making this change in the AWS console in EBS volumes, but of course it would NOT persist if the volumes are delete and re-created by Kubernetes or future new Kubernetes volumes will still default to gp2.

[nodesocket]

Hi @mikestef9 I'm currently using CSI driver to apply gp3 volume. My further question is, do you support provisioning IOPS and throughput for gp3 in storageClass? Since parameter iopsPerGB seemed to be not working with gp3 type according to my test. It would be very grateful if you have any idea.

Can you give a quick and dirty cliff notes version how you go this working? I am using the opt-in add-on VPC-CNI v1.10.1-eksbuild.1 but still:

ubuntu@bt-vpn1:~$ kubectl get sc
NAME            PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
gp2 (default)   kubernetes.io/aws-ebs   Delete          WaitForFirstConsumer   false                  3d1h

Don't see gp3 as a valid storage class in my EKS cluster.

[winter7]

Hi @nodesocket , you should manually install CSI driver into EKS cluster by following above mentioned documents like https://docs.aws.amazon.com/eks/latest/userguide/ebs-csi.html or https://github.com/kubernetes-sigs/aws-ebs-csi-driver .
The official helm chart would also create a StorageClass named ebs-csi whose type is gp3.
Hi @aldredb @anitakrueger , like @nodesocket said, modifying volume type via AWS console is totally OK. Even I use csi-driver to create gp3 type storageClass, I still have to call AWS api (or AWS console) to modify provisioned IOPS and throughput. What you might be concerned about is like AWS maximum volume modification rate per EBS type and PV's reclaimPolicy.

[RaJiska]

Hi @winter7 , @nodesocket ,

Experimented with upgrading the type from gp2 to gp3 via console. Please note that if you are using the console to change the type, you may break the feature allowVolumeExpansion from the storageclass, where trying to expand your volume won't work anymore.