source-controller: permission denied reading eks.amazonaws.com service account token (EKS/IRSA)

[bashims]

On an EKS cluster with IRSA enabled the source-controller cannot access AWS resources via the /var/run/secrets/eks.amazonaws.com/serviceaccount/token as provided by the AWS_WEB_IDENTITY_TOKEN_FILE environment. The service account is created by annotating the default service account eks.amazonaws.com/role-arn.

This issue seems to be due to the fact that the securityContext: fsGroup is not being set to the GID of the controller's user.

/ $ id
uid=100(controller) gid=65533(nogroup) groups=65533(nogroup)

/ $ env|grep AWS
AWS_ROLE_ARN=arn:aws:iam::000000000000000:role/cpt-479-flux-system-irsa
AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token
AWS_DEFAULT_REGION=us-east-1
AWS_REGION=us-east-1

/ $ cat /var/run/secrets/eks.amazonaws.com/serviceaccount/token
cat: can't open '/var/run/secrets/eks.amazonaws.com/serviceaccount/token': Permission denied

/ $ stat $(readlink -f /var/run/secrets/eks.amazonaws.com/serviceaccount/token)
  File: /run/secrets/eks.amazonaws.com/serviceaccount/..2021_02_07_03_32_59.713689454/token
  Size: 1009            Blocks: 8          IO Block: 4096   regular file
Device: 4dh/77d Inode: 4695092     Links: 1
Access: (0600/-rw-------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2021-02-07 03:33:03.000000000
Modify: 2021-02-07 03:32:59.000000000
Change: 2021-02-07 03:32:59.000000000

[stefanprodan]

The fsGroup value is irrelevant, can be anything but not left unspecified, see https://github.com/fluxcd/kustomize-controller/blob/main/config/manager/deployment.yaml#L21-L24