fix(eks*) define versions, IRSA(when required) and configuration policies for addons

[dduportal]

That one was pretty painful. No thanks to the awfull EKS documentation :'(

• Subtly hidden in https://docs.aws.amazon.com/eks/latest/userguide/csi-iam-role.html (tab "AWS CLI", section "2."), it appears that the IAM Role needs to be configured with a "OIDC trusted audience". Haven't tried without this, but better sticking to the documentation. As per https://registry.terraform.io/modules/terraform-aws-modules/iam/aws/latest/submodules/iam-assumable-role-with-oidc#inputs we can use oidc_fully_qualified_audiences for this.

• Absent from EKS documentation: there was some IAM permissions missing to Create/Delete EC2 volumes (yes....). Found the correct permissions in the AWS EBS CSI helm chart: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/docs/install.md#set-up-driver-permission

• Addons configuration: specify versions, policy for all, but the real 🔥 was that we did not specified the csi-controller-sa IAM role when installing the driver (neither from GUI or with Terraform).

  • Found the correct way to configure EKS Terraform's modules addons from the source which leads to the documentation of the Terraform AWS resource "eks_addon". Once validated manually in the AWS GUI console, I was able to transplant to Terraform (along with versions).

Protip: aws sts decode-authorization-message --encoded-message 'xxx' is quite useful to track the missing permissions from the CSI controller logs (kubectl -n kube-system logs -l app=ebs-csi-controller -c csi-provisioner -f) which prints the 403 IAM error with an encoded message. The decoded messages shows which IRSA is used, along with the missing permission.