Setting the env. variables (AWS_ROLE_ARN & AWS_WEB_IDENTITY_TOKEN_FILE) on a pod/container running in EKS. The go application will still try to use the IAM role specified by the EC2 metadata.

[steven-cherry]

Please fill out the sections below to help us address your issue.

Version of AWS SDK for Go?

v1.28.2

Version of Go (go version)?

1.13.5

What issue did you see?

When running an application in an EKS cluster and specifying the following environment variables on the pod/container
AWS_ROLE_ARN=......
AWS_WEB_IDENTITY_TOKEN_FILE=......
the IAM role that the EC2 metadata refers to is still used by the application and not the role specified by the AWS_ROLE_ARN environment variable.

Steps to reproduce

If you have an runnable example, please include it.

Run the following 'cloud controller' in a EKS cluster,
https://github.com/kubernetes/cloud-provider-aws
try and deploy for example an 'ELB' using the appropriate annotations listed in,
https://github.com/kubernetes/kubernetes/blob/e4b0a935fa393944b6322fa6ef0970d858ad70f6/pkg/cloudprovider/providers/aws/aws.go#L98
whilst having the following environment variables set to appropriate values
AWS_ROLE_ARN=......
AWS_WEB_IDENTITY_TOKEN_FILE=......
the application will still try and deploy the ELB using the IAM role attached to the underlying EC2 instance.

[dlaidlaw]

Also happening in newer aws-sdk-go: 1.28.11 with go version go1.13.5

[kelvingl]

+1

[xanderflood]

I ran into this issue in v1.20 and resolved by upgrading to v1.29.26

[xanderflood]

Also, hashicorp/terraform#22992 (comment) this comment contains a workaround for setting traditional credential envars from a token file

[diehlaws]

Hi @steven-cherry, apologies for the long delay in response from our end on this. The kubernetes/cloud-provider-aws repo you mention appears to be using version 1.16.26 of the AWS SDK for Go which was released in Jan 2019 - at the time, EKS did not support web identity token files so functionality for this had not been implemented in that version of the AWS SDK for Go. I see you mentioned you're using v1.28.2 of the SDK, are you building this cloud controller on your end using this version of the SDK? If not, where does this version of the SDK come into play on your end?

I was able to create an EKS cluster that uses a role named go3101NodeInstanceRole as the instance profile for the underlying instances and was able to create a separate role go3101-podRole associated with an OIDC provider in IAM to associate with a Kubernetes pod on this cluster as specified in the YAML file applied to the cluster.

I used an STS get-caller-identity call in the code I pushed to the docker image to be used for this pod, initially I saw that this call returned the go3101NodeInstanceRole I'd set as the EC2 instance profile when I was creating the session used by the STS client with session.New() (which is deprecated according to our docs) and passing in an aws.Config{} with only a Region parameter, however after changing this to session.NewSessionWithOptions() with the same config and the parameter SharedConfigState: session.SharedConfigEnable I saw that the calls were being issued as the go3101-podRole role instead of go3101NodeInstanceRole.

I hope this information helps, but please do feel free to reach out if you have additional questions for us about this.