Terraform looks up wrong AWS Credentials in ECS

[nocode99]

If you look at my issue from Vault, I'm experiencing similar issues with Terraform hashicorp/vault#2979

I'm trying to run terraform on jenkins slave nodes in ECS and I have verified it is trying to get permissions from the host. I use KMS key and kept getting access denied and when I gave my EC2 instance permissions to it as a test, it was able to use the KMS key in the remote state config.

Terraform Version

0.10.5
Providers: 1.4.0, 1.5.0

Affected Resource(s)

Everything :)

Expected Behavior

When using terraform in ECS, it should be looking up the credentials for the ECS Task Role ARN and not the instance profile of the EC2 instance

Actual Behavior

Terraform tries to use the credentials of the EC2 instance profile

Important Factoids

This issue was happening in 0.8.7 and I lazily just set static credentials for the time being and rotated them routinely. Based on the linked Vault issue, this is the way the Go SDK handles environment variables.

References

hashicorp/vault#2979

[myoung34]

@nocode99 related? hashicorp/terraform#8746

[evanstachowiak]

This is also an issue for me on 0.10.7 and 0.11.3.

I've resorted to manually setting the credentials this way:
curl --silent http://169.254.169.254:80/latest/meta-data/iam/security-credentials/$role_name > credentials && export AWS_ACCESS_KEY_ID=$(jq '.AccessKeyId' credentials) && export AWS_SECRET_ACCESS_KEY=$(jq '.SecretAccessKey' credentials) && export AWS_SESSION_TOKEN=$(jq '.Token' credentials) && rm credentials

I've just recently upgraded my ECS agent, perhaps this could be a cause? It is only happening now after the upgrade.

[PurrBiscuit]

I'm seeing the same behaviour with version 1.11.0 of the AWS Terraform provider.

Initializing provider plugins...
- Checking for available provider plugins on https://releases.hashicorp.com...
- Downloading plugin for provider "vault" (1.0.0)...
- Downloading plugin for provider "aws" (1.11.0)...

We have the task running with a Task ARN but it's still falling back to the EC2 IAM Role that's on the instance in the ECS cluster that the task is being run on.

* aws_iam_role.poco: aws_iam_role.poco: Error reading IAM Role poco-us-east-1: AccessDenied: User: arn:aws:sts::111111111111:assumed-role/jenkins-ecs/i-0809111111111 is not authorized to perform: iam:GetRole on resource: role poco-us-east-1

	status code: 403, request id: b9b7f0dd-27ba-11e8-b4d1-ebaba213a394

We're currently using ECS Agent version 1.17.1 in that ECS cluster as well.

[JohannesEbke]

I'm observing a similar thing: When running terraform on an AWS instance, credentials in ~/.aws/config are ignored in favor of the instance profile. This seems to contradict the "credential provider chain" specified in https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html

Unfortunately the workaround of setting the AWS_* environment variables does not work, since we use profiles with assumed roles from ~/.aws/config, and they don't get picked up when the AWS_ credentials are specified.

[paultyng]

Have you tried using the ECS_AWSVPC_BLOCK_IMDS environment variable?

To prevent containers in tasks that use the awsvpc network mode from accessing the credential information supplied to the container instance profile (while still allowing the permissions that are provided by the task role), set the ECS_AWSVPC_BLOCK_IMDS agent configuration variable to true in the agent configuration file and restart the agent.

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html

[paultyng]

It does seem like the provider looks up the value of AWS_CONTAINER_CREDENTIALS_RELATIVE_URI in the code:

https://github.com/terraform-providers/terraform-provider-aws/blob/8427cf84775dae2be92d8da63d037364895c9955/aws/auth_helpers.go#L153-L157

So perhaps its just a precedence problem?

[aeschright]

Hey folks, we're rolling out a fix in the next release that should take care of the credential precedence issues you were seeing. If there's still a problem after that, please open a new bug issue so we can look into the details. Thanks!

[bflad]

Closing as #10379 was merged previously and v2.32.0 has been released. 👍

[ghost]

This has been released in version 2.32.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template for triage. Thanks!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!