Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Terraform does not use IAM Role for ECS Task as credential provider #8746

iwat opened this issue Sep 9, 2016 · 10 comments

Terraform does not use IAM Role for ECS Task as credential provider #8746

iwat opened this issue Sep 9, 2016 · 10 comments


Copy link

iwat commented Sep 9, 2016

Terraform Version

Terraform v0.7.3

Affected Resource(s)

  • aws_alb_target_group
  • aws_security_group
  • a lot

This affects all AWS related command.

Terraform Configuration Files

resource "aws_security_group_rule" "demo_pri_ingress_vpn_service" {
    security_group_id = "${}"
    type = "ingress"

    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["${data.terraform_remote_state.infra.vpn-cidr_block}"]

Debug Output

Panic Output


Expected Behavior

It should work by using IAM Role for ECS Task.
awscli works

Actual Behavior

It was using EC2 Instance Role which does not allow this action.

Error retrieving Target Group: AccessDenied: User: arn:aws:sts::872767853649:assumed-role/myrole/i-0223aeb98c19f2d0d

Steps to Reproduce

  • Setup an EC2, do not provide any critical IAM action.
  • Setup ECS task, provide required IAM action for testing.
  • Try AWSCLI inside the running ECS task, it should work fine.
  • Run terraform on AWS ECS Task.

Important Factoids



Copy link

bacoboy commented Dec 7, 2016

A little context to help the enhancement along. I ran into this while trying to run a terraform command from the new AWS CodeBuild service (which is running on an AWS hosted ECS cluster farm it seems).

In the newer versions of the AWS SDK, they've added one more "location" to scan for IAM keys to support IAM roles on docker containers.

The launched docker containers, if they have an IAM role, get passed an extra environment variable like this: AWS_CONTAINER_CREDENTIALS_RELATIVE_URI='/v2/credentials/895e903e-0672-4c41-bdc8-ef0c3b37d178' This is a relative path to the ECS agent running on the EC2 instance which has been NATed via iptables to address This means that the container can get its metadata (includes IAM keys) using that relative URL from the ECS agent like this:

    "AccessKeyId": "ACCESS_KEY_ID",
    "Expiration": "EXPIRATION_DATE",
    "RoleArn": "TASK_ROLE_ARN",
    "SecretAccessKey": "SECRET_ACCESS_KEY",

Since all the SDK's have been updated to scan this location as well as the usual paths, it just seems to work like magic when you run AWS commands.

This sample, and all the gory details are from here

Hopefully this is enough to get somebody going on the terraform enhancement. Any takers?

UPDATE: Of course, now that I just typed all that, it seems that if you are using the aws-sdk-go package, you just need to update the dependency version. These seem to be the minimums

Copy link

imyousuf commented Feb 8, 2017

Faced the problem from within AWS CodeBuild with v0.8.6 as well.

Copy link

s0enke commented Feb 20, 2017

Facing the same problem. It seems that the Terraform AWS credential logic is custom and does not use the default
AWS provider chain (

// This function is responsible for reading credentials from the
// environment in the case that they're not explicitly specified
// in the Terraform configuration.
func GetCredentials(c *Config) (*awsCredentials.Credentials, error) {
// build a chain provider, lazy-evaulated by aws-sdk
providers := []awsCredentials.Provider{
&awsCredentials.StaticProvider{Value: awsCredentials.Value{
AccessKeyID: c.AccessKey,
SecretAccessKey: c.SecretKey,
SessionToken: c.Token,
Filename: c.CredsFilename,
Profile: c.Profile,
// Build isolated HTTP client to avoid issues with globally-shared settings
client := cleanhttp.DefaultClient()
// Keep the timeout low as we don't want to wait in non-EC2 environments
client.Timeout = 100 * time.Millisecond
cfg := &aws.Config{
HTTPClient: client,
usedEndpoint := setOptionalEndpoint(cfg)
if !c.SkipMetadataApiCheck {
// Real AWS should reply to a simple metadata request.
// We check it actually does to ensure something else didn't just
// happen to be listening on the same IP:Port
metadataClient := ec2metadata.New(session.New(cfg))
if metadataClient.Available() {
providers = append(providers, &ec2rolecreds.EC2RoleProvider{
Client: metadataClient,
log.Print("[INFO] AWS EC2 instance detected via default metadata" +
" API endpoint, EC2RoleProvider added to the auth chain")
} else {
if usedEndpoint == "" {
usedEndpoint = "default location"
log.Printf("[WARN] Ignoring AWS metadata API endpoint at %s "+
"as it doesn't return any instance-id", usedEndpoint)
// This is the "normal" flow (i.e. not assuming a role)
if c.AssumeRoleARN == "" {
return awsCredentials.NewChainCredentials(providers), nil
// Otherwise we need to construct and STS client with the main credentials, and verify
// that we can assume the defined role.
log.Printf("[INFO] Attempting to AssumeRole %s (SessionName: %q, ExternalId: %q)",
c.AssumeRoleARN, c.AssumeRoleSessionName, c.AssumeRoleExternalID)
creds := awsCredentials.NewChainCredentials(providers)
cp, err := creds.Get()
if err != nil {
if awsErr, ok := err.(awserr.Error); ok && awsErr.Code() == "NoCredentialProviders" {
return nil, errors.New(`No valid credential sources found for AWS Provider.
Please see for more information on
providing credentials for the AWS Provider`)
return nil, fmt.Errorf("Error loading credentials for AWS Provider: %s", err)
log.Printf("[INFO] AWS Auth provider used: %q", cp.ProviderName)
awsConfig := &aws.Config{
Credentials: creds,
Region: aws.String(c.Region),
MaxRetries: aws.Int(c.MaxRetries),
HTTPClient: cleanhttp.DefaultClient(),
S3ForcePathStyle: aws.Bool(c.S3ForcePathStyle),
stsclient := sts.New(session.New(awsConfig))
assumeRoleProvider := &stscreds.AssumeRoleProvider{
Client: stsclient,
RoleARN: c.AssumeRoleARN,
if c.AssumeRoleSessionName != "" {
assumeRoleProvider.RoleSessionName = c.AssumeRoleSessionName
if c.AssumeRoleExternalID != "" {
assumeRoleProvider.ExternalID = aws.String(c.AssumeRoleExternalID)
providers = []awsCredentials.Provider{assumeRoleProvider}
assumeRoleCreds := awsCredentials.NewChainCredentials(providers)
_, err = assumeRoleCreds.Get()
if err != nil {
if awsErr, ok := err.(awserr.Error); ok && awsErr.Code() == "NoCredentialProviders" {
return nil, fmt.Errorf("The role %q cannot be assumed.\n\n"+
" There are a number of possible causes of this - the most common are:\n"+
" * The credentials used in order to assume the role are invalid\n"+
" * The credentials do not have appropriate permission to assume the role\n"+
" * The role ARN is not valid",
return nil, fmt.Errorf("Error loading credentials for AWS Provider: %s", err)
return assumeRoleCreds, nil
). This custom chain does not include ECS task yet.

Copy link

s0enke commented Feb 26, 2017

I blogged about using Terraform within CodeBuild, which includes a workaround for this problem:

Copy link

This may be a little off-topic, but here's my workaround for use with Jenkins Pipelines (Groovy)

import groovy.json.JsonSlurperClassic

def jsonParse(def json) {
    new groovy.json.JsonSlurperClassic().parseText(json)

node {
  stage('AWS Creds Please'){
    def awsCredsJSON = sh(returnStdout: true, script: "curl$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI")
    def awsCreds = jsonParse(awsCredsJSON)
    env.AWS_ACCESS_KEY_ID = awsCreds.AccessKeyId
    env.AWS_SECRET_ACCESS_KEY = awsCreds.SecretAccessKey

Copy link

Thanks for this feature request @iwat, and thanks to everyone else for the great info that followed.

Terraform is using the official Go SDK for AWS but is customizing the set of valid credential sources. To implement this I expect we would need to upgrade the SDK (assuming we didn't already do that for some other reason) and add one more credential provider to the list in the Terraform AWS provider.

Since this one is gated on the presence of an environment variable it should be safe to add without any unintended consequences for those not using ECS.

The Terraform team doesn't have any immediate plans to work on this but if someone else had the time or motivation we would love to review a PR!

Copy link

jekh commented May 5, 2017

I submitted PR #14199 for this. A couple questions in the PR, but hopefully this will help get terraform working with CodeBuild/ECS slaves.

Copy link

jch254 commented Apr 18, 2018

I can confirm this is now working in CodeBuild WITHOUT the pre_build phase I posted above. Terraform 0.11.7 and Terraform AWS provider 1.14.1. Brilliant work 👍

Copy link

I can confirm that terraform trying to assume the role of EC2's instead of ECS's

Below are some logs
2020/03/03 11:46:52 [INFO] ECS container credentials detected, RemoteCredProvider added to auth chain
2020/03/03 11:46:52 [INFO] AWS EC2 instance detected via default metadata API endpoint, EC2RoleProvider added to the auth chain
2020/03/03 11:46:53 [INFO] AWS Auth provider used: "EC2RoleProvider"

terraform version: 0.12.21

Copy link

ghost commented Mar 4, 2020

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@hashicorp hashicorp locked and limited conversation to collaborators Mar 4, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
None yet

No branches or pull requests