AWS CodeBuild Task Role Auth Failure

[ingshtrom]

Terraform Version

0.10.7

Terraform Configuration Files

terraform {
  backend "s3" {
    bucket = "my-bucket"
    key = "main-infrastructure_us-east-2.tfstate"
    region = "us-east-2"
    dynamodb_table = "my-table"
  }
  required_version = "= 0.10.7"
}

provider "aws" {
  region = "us-east-2"
}

Debug Output

• broken build output
• for comparison, here is the successful build output that uses environment variables to work around the issue, but no TF_LOG env var.

Crash Output

n/a

Expected Behavior

Everything should run and the terraform validate should be successful. The Task Role should be grabbed from the AWS_CONTAINER_CREDENTIALS_RELATIVE_URI environment variable, like the documentation says.

Actual Behavior

Terraform fails to setup the s3 backend.

Steps to Reproduce

Please list the full steps required to reproduce the issue, for example:

1. setup codebuild
2. terraform apply

Important Factoids

I am running this in AWS CodeBuild, which means it is using a Task Role since CodeBuild uses ECS under the hood (from my understanding).

Here is my CodeBuild config that produces a failed build, assuming the s3 backend will load from the Task Role environment variable. There is nothing special about it.

Here is my CodeBuild config that produces a successful build using the environment variables workaround. The CodeBuild output is linked in the Debug Output section.

References

Are there any other GitHub issues (open or closed) or Pull Requests that should be linked here? For example:

• Terraform does not use IAM Role for ECS Task as credential provider #8746 => similar ticket from several versions ago, but against the aws provider, not the s3 backend

[poorsquinky]

I'm having the exact same problem with CodeBuild and Terraform 0.10.7, using an S3 back-end.

Error configuring the backend "s3": No valid credential sources found for AWS Provider.

I can provide any other details if they're needed.

[jch254]

I am also facing this issue using Terraform with CodeBuild. I would love to remove my workaround:

phases:
  pre_build:
    commands:
      # Workaround until TF supports creds via Task Roles when running on ECS or CodeBuild
      # See: https://github.com/hashicorp/terraform/issues/8746
      - export AWS_ACCESS_KEY_ID=`curl --silent http://169.254.170.2:80$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI | jq -r '.AccessKeyId'`
      - export AWS_SECRET_ACCESS_KEY=`curl --silent http://169.254.170.2:80$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI | jq -r '.SecretAccessKey'`
      - export AWS_SESSION_TOKEN=`curl --silent http://169.254.170.2:80$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI | jq -r '.Token'`

Any idea when this will be solved for s3 backend?

Thanks

[eredi93]

facing the same issue. i'd love to get a proper fix for this

[barryoneill]

Seeing the same issue on terraform v0.11.1.

[RobbieMcKinstry]

gonna go ahead and report that I also am seeing this issue on v0.11.1

[jch254]

I can confirm this is now working in CodeBuild WITHOUT the pre_build phase I posted above. Terraform 0.11.7 and Terraform AWS provider 1.14.1. Brilliant work 👍

[barryoneill]

Same here, confirmed working with terraform-aws 1.14.1!

[jeancochrane]

@jch254 @barryoneill I'm still seeing this behavior on CodeBuild with Terraform 0.11.7. Have you seen consistently good behavior since April? As I understand it the error occurs in Terraform core while configuring the S3 backend, before the AWS provider gets installed, so I'm not sure why hashicorp/terraform-provider-aws#1425 would be expected to fix it.

[barryoneill]

Yeah, these has been working fine since then. In case it's relevant, in my use case, we ran terraform inside a docker image run from the buildspec, so we still needed to propagate the AWS_CONTAINER_CREDENTIALS_RELATIVE_URI variable to the container.. e.g..

  build:
     commands:
       - |
         docker run -i --rm \
           -e AWS_CONTAINER_CREDENTIALS_RELATIVE_URI=$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI \
           my-image-that-runs-terraform:version arg1 arg2 

[jeancochrane]

@barryoneill I'm also running Terraform inside a container, so that's likely my problem. I'll give your solution a try. Thanks for the tip!

[teamterraform]

Hi all!

The AWS SDK (along with some code shared between the AWS provider and the S3 backend) handles the authentication details here, and we recall that there was a release of it some time ago which added support for automatic authentication in ECS using the AWS_CONTAINER_CREDENTIALS_RELATIVE_URI variable, which is presumably why this is now working for you all.

As @barryoneill noted, it's important to make sure the AWS_CONTAINER_CREDENTIALS_RELATIVE_URI environment variable is set correctly so that the AWS SDK logic can detect that it's running in ECS and find the ECS authentication endpoint.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.