EKS 1.16 missing pod-identity-webhook

[edbighead]

What happened:
After upgrading from EKS 1.15 to EKS 1.16, podsare missing the following env vars:

AWS_ROLE_ARN=arn:aws:iam::xxx:role/oidc-my-service-api-qa
AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token

Found that pod-identity-webhook (which is responsible of injecting those) is missing in 1.16. It's still present on our 1.15 clusters.

EKS 1.15

kubectl get mutatingwebhookconfigurations pod-identity-webhook
NAME                   CREATED AT
pod-identity-webhook   2020-01-11T17:01:52Z

EKS 1.16

kubectl get mutatingwebhookconfigurations pod-identity-webhook
Error from server (NotFound): mutatingwebhookconfigurations.admissionregistration.k8s.io "pod-identity-webhook" not found

What you expected to happen:
pod-identity-webhook to be present after upgrade.

How to reproduce it (as minimally and precisely as possible):
Upgrade EKS 1.15 to 1.16 and execute kubectl get mutatingwebhookconfigurations pod-identity-webhook

Environment:

• AWS Region: ap-southeast-1
• EKS Platform version: "eks.2"
• Kubernetes version: "1.16"
• Webhook Version: -

[edbighead]

I was able to re-install amazon-eks-pod-identity-webhook in my EKS 1.16 cluster by building docker image and pushing it to my ECR repo, and running

make cluster-up IMAGE=my-account-id.dkr.ecr.my-region.amazonaws.com/pod-identity-webhook:latest

[amitkarpe]

Hi @edbighead,

How to recreate mutatingwebhookconfigurations pod-identity-webhook?
Where I can find its manifest file?

[edbighead]

Hi @edbighead,

How to recreate mutatingwebhookconfigurations pod-identity-webhook?
Where I can find its manifest file?

@amitkarpe clone the repo, build the image and push it to your repo, and run the make command i mentioned above (replacing the IMAGE), manifests can be found here https://github.com/aws/amazon-eks-pod-identity-webhook/tree/master/deploy

[jqmichael]

Hey @edbighead and @amitkarpe, do you mind ping us on Slack with the cluster ARN and timeframe so that we can look into on our side?

[amitkarpe]

Hi @jqmichael,
I was not able to connect to this slack channel. Which slack channel it is?
I am already a member of kubernetes/EKS channel. You can ping me there: https://kubernetes.slack.com/archives/C8SH2GSL9