Terraform does not use IAM Role for ECS Task as credential provider

[iwat]

Terraform Version

Terraform v0.7.3

Affected Resource(s)

• aws_alb_target_group
• aws_security_group
• a lot

This affects all AWS related command.

Terraform Configuration Files

resource "aws_security_group_rule" "demo_pri_ingress_vpn_service" {
    security_group_id = "${aws_security_group.demo_pri.id}"
    type = "ingress"

    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["${data.terraform_remote_state.infra.vpn-cidr_block}"]
}

Debug Output

https://gist.github.com/iwat/df0b0ebfe2f8db62adfd5953bfd6b92c

Panic Output

None

Expected Behavior

It should work by using IAM Role for ECS Task.
awscli works

Actual Behavior

It was using EC2 Instance Role which does not allow this action.

Error retrieving Target Group: AccessDenied: User: arn:aws:sts::872767853649:assumed-role/myrole/i-0223aeb98c19f2d0d

Steps to Reproduce

• Setup an EC2, do not provide any critical IAM action.
• Setup ECS task, provide required IAM action for testing.
• Try AWSCLI inside the running ECS task, it should work fine.
• Run terraform on AWS ECS Task.

Important Factoids

None

References

• http://docs.aws.amazon.com/de_de/AmazonECS/latest/developerguide/task-iam-roles.html
• https://github.com/hashicorp/terraform/blob/v0.7.3/builtin/providers/aws/auth_helpers.go#L97
• https://github.com/hashicorp/terraform/blob/v0.7.3/builtin/providers/aws/auth_helpers.go#L128
• https://github.com/aws/aws-sdk-go/blob/v1.4.8/aws/defaults/defaults.go#L100

[bacoboy]

A little context to help the enhancement along. I ran into this while trying to run a terraform command from the new AWS CodeBuild service (which is running on an AWS hosted ECS cluster farm it seems).

In the newer versions of the AWS SDK, they've added one more "location" to scan for IAM keys to support IAM roles on docker containers.

The launched docker containers, if they have an IAM role, get passed an extra environment variable like this: AWS_CONTAINER_CREDENTIALS_RELATIVE_URI='/v2/credentials/895e903e-0672-4c41-bdc8-ef0c3b37d178' This is a relative path to the ECS agent running on the EC2 instance which has been NATed via iptables to address http://169.254.170.2/. This means that the container can get its metadata (includes IAM keys) using that relative URL from the ECS agent like this:

$ curl 169.254.170.2$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI
{
    "AccessKeyId": "ACCESS_KEY_ID",
    "Expiration": "EXPIRATION_DATE",
    "RoleArn": "TASK_ROLE_ARN",
    "SecretAccessKey": "SECRET_ACCESS_KEY",
    "Token": "SECURITY_TOKEN_STRING"
}

Since all the SDK's have been updated to scan this location as well as the usual paths, it just seems to work like magic when you run AWS commands.

This sample, and all the gory details are from here

Hopefully this is enough to get somebody going on the terraform enhancement. Any takers?

UPDATE: Of course, now that I just typed all that, it seems that if you are using the aws-sdk-go package, you just need to update the dependency version. These seem to be the minimums

[imyousuf]

Faced the problem from within AWS CodeBuild with v0.8.6 as well.

[s0enke]

Facing the same problem. It seems that the Terraform AWS credential logic is custom and does not use the default
AWS provider chain (

terraform/builtin/providers/aws/auth_helpers.go

Lines 94 to 204 in 351c6be

	// This function is responsible for reading credentials from the
	// environment in the case that they're not explicitly specified
	// in the Terraform configuration.
	func GetCredentials(c *Config) (*awsCredentials.Credentials, error) {
	// build a chain provider, lazy-evaulated by aws-sdk
	providers := []awsCredentials.Provider{
	&awsCredentials.StaticProvider{Value: awsCredentials.Value{
	AccessKeyID: c.AccessKey,
	SecretAccessKey: c.SecretKey,
	SessionToken: c.Token,
	}},
	&awsCredentials.EnvProvider{},
	&awsCredentials.SharedCredentialsProvider{
	Filename: c.CredsFilename,
	Profile: c.Profile,
	},
	}
	// Build isolated HTTP client to avoid issues with globally-shared settings
	client := cleanhttp.DefaultClient()
	// Keep the timeout low as we don't want to wait in non-EC2 environments
	client.Timeout = 100 * time.Millisecond
	cfg := &aws.Config{
	HTTPClient: client,
	}
	usedEndpoint := setOptionalEndpoint(cfg)
	if !c.SkipMetadataApiCheck {
	// Real AWS should reply to a simple metadata request.
	// We check it actually does to ensure something else didn't just
	// happen to be listening on the same IP:Port
	metadataClient := ec2metadata.New(session.New(cfg))
	if metadataClient.Available() {
	providers = append(providers, &ec2rolecreds.EC2RoleProvider{
	Client: metadataClient,
	})
	log.Print("[INFO] AWS EC2 instance detected via default metadata" +
	" API endpoint, EC2RoleProvider added to the auth chain")
	} else {
	if usedEndpoint == "" {
	usedEndpoint = "default location"
	}
	log.Printf("[WARN] Ignoring AWS metadata API endpoint at %s "+
	"as it doesn't return any instance-id", usedEndpoint)
	}
	}
	// This is the "normal" flow (i.e. not assuming a role)
	if c.AssumeRoleARN == "" {
	return awsCredentials.NewChainCredentials(providers), nil
	}
	// Otherwise we need to construct and STS client with the main credentials, and verify
	// that we can assume the defined role.
	log.Printf("[INFO] Attempting to AssumeRole %s (SessionName: %q, ExternalId: %q)",
	c.AssumeRoleARN, c.AssumeRoleSessionName, c.AssumeRoleExternalID)
	creds := awsCredentials.NewChainCredentials(providers)
	cp, err := creds.Get()
	if err != nil {
	if awsErr, ok := err.(awserr.Error); ok && awsErr.Code() == "NoCredentialProviders" {
	return nil, errors.New(`No valid credential sources found for AWS Provider.
	Please see https://terraform.io/docs/providers/aws/index.html for more information on
	providing credentials for the AWS Provider`)
	}
	return nil, fmt.Errorf("Error loading credentials for AWS Provider: %s", err)
	}
	log.Printf("[INFO] AWS Auth provider used: %q", cp.ProviderName)
	awsConfig := &aws.Config{
	Credentials: creds,
	Region: aws.String(c.Region),
	MaxRetries: aws.Int(c.MaxRetries),
	HTTPClient: cleanhttp.DefaultClient(),
	S3ForcePathStyle: aws.Bool(c.S3ForcePathStyle),
	}
	stsclient := sts.New(session.New(awsConfig))
	assumeRoleProvider := &stscreds.AssumeRoleProvider{
	Client: stsclient,
	RoleARN: c.AssumeRoleARN,
	}
	if c.AssumeRoleSessionName != "" {
	assumeRoleProvider.RoleSessionName = c.AssumeRoleSessionName
	}
	if c.AssumeRoleExternalID != "" {
	assumeRoleProvider.ExternalID = aws.String(c.AssumeRoleExternalID)
	}
	providers = []awsCredentials.Provider{assumeRoleProvider}
	assumeRoleCreds := awsCredentials.NewChainCredentials(providers)
	_, err = assumeRoleCreds.Get()
	if err != nil {
	if awsErr, ok := err.(awserr.Error); ok && awsErr.Code() == "NoCredentialProviders" {
	return nil, fmt.Errorf("The role %q cannot be assumed.\n\n"+
	" There are a number of possible causes of this - the most common are:\n"+
	" * The credentials used in order to assume the role are invalid\n"+
	" * The credentials do not have appropriate permission to assume the role\n"+
	" * The role ARN is not valid",
	c.AssumeRoleARN)
	}
	return nil, fmt.Errorf("Error loading credentials for AWS Provider: %s", err)
	}
	return assumeRoleCreds, nil
	}

). This custom chain does not include ECS task yet.

[s0enke]

I blogged about using Terraform within CodeBuild, which includes a workaround for this problem: https://www.ruempler.eu/2017/02/26/continuous-infrastructure-delivery-pipeline-aws-codepipeline-codebuild-terraform/

[danmandle]

This may be a little off-topic, but here's my workaround for use with Jenkins Pipelines (Groovy)

import groovy.json.JsonSlurperClassic

@NonCPS
def jsonParse(def json) {
    new groovy.json.JsonSlurperClassic().parseText(json)
}

node {
  stage('AWS Creds Please'){
    def awsCredsJSON = sh(returnStdout: true, script: "curl 169.254.170.2$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI")
    
    def awsCreds = jsonParse(awsCredsJSON)
    
    env.AWS_ACCESS_KEY_ID = awsCreds.AccessKeyId
    env.AWS_SECRET_ACCESS_KEY = awsCreds.SecretAccessKey
  }
}

[apparentlymart]

Thanks for this feature request @iwat, and thanks to everyone else for the great info that followed.

Terraform is using the official Go SDK for AWS but is customizing the set of valid credential sources. To implement this I expect we would need to upgrade the SDK (assuming we didn't already do that for some other reason) and add one more credential provider to the list in the Terraform AWS provider.

Since this one is gated on the presence of an environment variable it should be safe to add without any unintended consequences for those not using ECS.

The Terraform team doesn't have any immediate plans to work on this but if someone else had the time or motivation we would love to review a PR!

[jekh]

I submitted PR #14199 for this. A couple questions in the PR, but hopefully this will help get terraform working with CodeBuild/ECS slaves.

[jch254]

I can confirm this is now working in CodeBuild WITHOUT the pre_build phase I posted above. Terraform 0.11.7 and Terraform AWS provider 1.14.1. Brilliant work 👍

[vinujan59]

I can confirm that terraform trying to assume the role of EC2's instead of ECS's

Below are some logs
2020/03/03 11:46:52 [INFO] ECS container credentials detected, RemoteCredProvider added to auth chain
2020/03/03 11:46:52 [INFO] AWS EC2 instance detected via default metadata API endpoint, EC2RoleProvider added to the auth chain
2020/03/03 11:46:53 [INFO] AWS Auth provider used: "EC2RoleProvider"

terraform version: 0.12.21

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.