Out of tree EBS CSI driver needed for EKS v1.23

[richardcase]

With EKS v1.23 and higher it requires the use of the out of tree EBS CSI driver. If this isn't available then and PVC will not be bound and pods using the volumes will stay in pending.

This is a problem for creating new clusters and for upgrading clusters from 1.22

Docs: https://docs.amazonaws.cn/en_us/eks/latest/userguide/ebs-csi.html

PR's

• [v2.9] Set information about CSI EBS driver #725
• Set information about CSI EBS driver #724

[sanjay920]

the aws docs are quite confusing.I added a wiki in the opni repo that our engineers use to get around this problem if it comes in handy https://github.com/rancher/opni/wiki/Install-AWS-EBS-CSI-driver

[richardcase]

Thanks @sanjay920

[richardcase]

A support utility has been created to handle this until we can include the functionality in eks-operator.

When including in the operator should we automatically enable the EBS CSI addon? Doing this will require that the IAM permissions that Rancher uses will need to be updated (including the docs). Or should it be an opt-in by annotating the namespace?

[richardcase]

Wanting on feedback from rancherlabs/support-tools#217 before implementing in eks-operator

[kkaempf]

Wontfix for 2023-Q2-2.6.x, workaround (script) is available.

Planned for 2023-Q2-2.7.x

[salasberryfin]

/assign

[richardcase]

From discussion with @salasberryfin & @mjura we thought this would be the best approach:

• Installation of the EBS CSI driver will be optional (as is the case when using the AWS console)
• We will add a new field to EKSClusterConfigSpec to indicate that the driver should be installed
• If the field is true we will do all the steps to enable the driver
• Update the EKS create cluster UI to have an option to set this new field. (optionally) In the UI we can also warn the user that additional permissions are required.
• Update the Rancher docs around the minimum EKS permissions: https://ranchermanager.docs.rancher.com/v2.5/reference-guides/amazon-eks-permissions/minimum-eks-permissions

[salasberryfin]

On the eks-operator side, the issue can be divided in the following tasks:

• Add new boolean field to EKSClusterConfigSpec
• Implement logic to create OIDC provider and IAM role and install EKS addon
• Tests

[salasberryfin]

PR was merged and this is now waiting for rancher/dashboard#9043 to be implemented.

[richardcase]

@salasberryfin - we'll also need to make an update to the Rancher Docs so that the new permissions are included.

[salasberryfin]

Added rancher/rancher-docs#704 to update Rancher docs with the permissions required to enable the add-on.

[mjura]

UI and docs were merged, back to testing

[cpinjani]

Adjusting the milestone, as the UI/Doc issues are slated for v2.9
cc: @kkaempf @mjura

[cpinjani]

Validated on build:

v2.9-a57f7025abdd80a2912f7fe53247ef7e1d4993de-head
eks-operator:v1.9.0-rc.9

Enabled Out of tree EBS CSI driver while provisioning cluster, its set to ebsCSIDriver: true in config, and the cluster in EKS has the Add-On enabled:

time="2024-07-02T10:02:15Z" level=info msg="enabling [ebs csi driver add-on] for cluster [cpinjani-eks29]"
time="2024-07-02T10:02:22Z" level=info msg="cluster [c-t7tm4] finished updating"

---

However, after cluster becomes Active its set to ebsCSIDriver: null in spec, hence UI reports it as not set.
Expected value: ebsCSIDriver: true

@mjura @salasberryfin Can you please have a look ?

[salasberryfin]

Thanks for reporting @cpinjani. Could you confirm if this is null in upstreamSpec, eksConfig or both?

[cpinjani]

null in both, upstreamSpec & eksConfig

[cpinjani]

Validation passed on below build, able to set Out of tree EBS CSI driver while creation/edit of cluster.
ebsCSIDriver: true is set in cluster config.

v2.9-92f15949d71efb11baf63c878cc2f64bcd25b1e8-head
eks-operator:v1.9.1-rc.6