New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Terraform does not use IAM Role for ECS Task as credential provider #8746

Closed
iwat opened this Issue Sep 9, 2016 · 8 comments

Comments

Projects
None yet
10 participants
@iwat

iwat commented Sep 9, 2016

Terraform Version

Terraform v0.7.3

Affected Resource(s)

  • aws_alb_target_group
  • aws_security_group
  • a lot

This affects all AWS related command.

Terraform Configuration Files

resource "aws_security_group_rule" "demo_pri_ingress_vpn_service" {
    security_group_id = "${aws_security_group.demo_pri.id}"
    type = "ingress"

    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["${data.terraform_remote_state.infra.vpn-cidr_block}"]
}

Debug Output

https://gist.github.com/iwat/df0b0ebfe2f8db62adfd5953bfd6b92c

Panic Output

None

Expected Behavior

It should work by using IAM Role for ECS Task.
awscli works

Actual Behavior

It was using EC2 Instance Role which does not allow this action.

Error retrieving Target Group: AccessDenied: User: arn:aws:sts::872767853649:assumed-role/myrole/i-0223aeb98c19f2d0d

Steps to Reproduce

  • Setup an EC2, do not provide any critical IAM action.
  • Setup ECS task, provide required IAM action for testing.
  • Try AWSCLI inside the running ECS task, it should work fine.
  • Run terraform on AWS ECS Task.

Important Factoids

None

References

@bacoboy

This comment has been minimized.

Show comment
Hide comment
@bacoboy

bacoboy Dec 7, 2016

A little context to help the enhancement along. I ran into this while trying to run a terraform command from the new AWS CodeBuild service (which is running on an AWS hosted ECS cluster farm it seems).

In the newer versions of the AWS SDK, they've added one more "location" to scan for IAM keys to support IAM roles on docker containers.

The launched docker containers, if they have an IAM role, get passed an extra environment variable like this: AWS_CONTAINER_CREDENTIALS_RELATIVE_URI='/v2/credentials/895e903e-0672-4c41-bdc8-ef0c3b37d178' This is a relative path to the ECS agent running on the EC2 instance which has been NATed via iptables to address http://169.254.170.2/. This means that the container can get its metadata (includes IAM keys) using that relative URL from the ECS agent like this:

$ curl 169.254.170.2$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI
{
    "AccessKeyId": "ACCESS_KEY_ID",
    "Expiration": "EXPIRATION_DATE",
    "RoleArn": "TASK_ROLE_ARN",
    "SecretAccessKey": "SECRET_ACCESS_KEY",
    "Token": "SECURITY_TOKEN_STRING"
}

Since all the SDK's have been updated to scan this location as well as the usual paths, it just seems to work like magic when you run AWS commands.

This sample, and all the gory details are from here

Hopefully this is enough to get somebody going on the terraform enhancement. Any takers?

UPDATE: Of course, now that I just typed all that, it seems that if you are using the aws-sdk-go package, you just need to update the dependency version. These seem to be the minimums

bacoboy commented Dec 7, 2016

A little context to help the enhancement along. I ran into this while trying to run a terraform command from the new AWS CodeBuild service (which is running on an AWS hosted ECS cluster farm it seems).

In the newer versions of the AWS SDK, they've added one more "location" to scan for IAM keys to support IAM roles on docker containers.

The launched docker containers, if they have an IAM role, get passed an extra environment variable like this: AWS_CONTAINER_CREDENTIALS_RELATIVE_URI='/v2/credentials/895e903e-0672-4c41-bdc8-ef0c3b37d178' This is a relative path to the ECS agent running on the EC2 instance which has been NATed via iptables to address http://169.254.170.2/. This means that the container can get its metadata (includes IAM keys) using that relative URL from the ECS agent like this:

$ curl 169.254.170.2$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI
{
    "AccessKeyId": "ACCESS_KEY_ID",
    "Expiration": "EXPIRATION_DATE",
    "RoleArn": "TASK_ROLE_ARN",
    "SecretAccessKey": "SECRET_ACCESS_KEY",
    "Token": "SECURITY_TOKEN_STRING"
}

Since all the SDK's have been updated to scan this location as well as the usual paths, it just seems to work like magic when you run AWS commands.

This sample, and all the gory details are from here

Hopefully this is enough to get somebody going on the terraform enhancement. Any takers?

UPDATE: Of course, now that I just typed all that, it seems that if you are using the aws-sdk-go package, you just need to update the dependency version. These seem to be the minimums

@imyousuf

This comment has been minimized.

Show comment
Hide comment
@imyousuf

imyousuf Feb 8, 2017

Faced the problem from within AWS CodeBuild with v0.8.6 as well.

imyousuf commented Feb 8, 2017

Faced the problem from within AWS CodeBuild with v0.8.6 as well.

@s0enke

This comment has been minimized.

Show comment
Hide comment
@s0enke

s0enke Feb 20, 2017

Facing the same problem. It seems that the Terraform AWS credential logic is custom and does not use the default
AWS provider chain (

// This function is responsible for reading credentials from the
// environment in the case that they're not explicitly specified
// in the Terraform configuration.
func GetCredentials(c *Config) (*awsCredentials.Credentials, error) {
// build a chain provider, lazy-evaulated by aws-sdk
providers := []awsCredentials.Provider{
&awsCredentials.StaticProvider{Value: awsCredentials.Value{
AccessKeyID: c.AccessKey,
SecretAccessKey: c.SecretKey,
SessionToken: c.Token,
}},
&awsCredentials.EnvProvider{},
&awsCredentials.SharedCredentialsProvider{
Filename: c.CredsFilename,
Profile: c.Profile,
},
}
// Build isolated HTTP client to avoid issues with globally-shared settings
client := cleanhttp.DefaultClient()
// Keep the timeout low as we don't want to wait in non-EC2 environments
client.Timeout = 100 * time.Millisecond
cfg := &aws.Config{
HTTPClient: client,
}
usedEndpoint := setOptionalEndpoint(cfg)
if !c.SkipMetadataApiCheck {
// Real AWS should reply to a simple metadata request.
// We check it actually does to ensure something else didn't just
// happen to be listening on the same IP:Port
metadataClient := ec2metadata.New(session.New(cfg))
if metadataClient.Available() {
providers = append(providers, &ec2rolecreds.EC2RoleProvider{
Client: metadataClient,
})
log.Print("[INFO] AWS EC2 instance detected via default metadata" +
" API endpoint, EC2RoleProvider added to the auth chain")
} else {
if usedEndpoint == "" {
usedEndpoint = "default location"
}
log.Printf("[WARN] Ignoring AWS metadata API endpoint at %s "+
"as it doesn't return any instance-id", usedEndpoint)
}
}
// This is the "normal" flow (i.e. not assuming a role)
if c.AssumeRoleARN == "" {
return awsCredentials.NewChainCredentials(providers), nil
}
// Otherwise we need to construct and STS client with the main credentials, and verify
// that we can assume the defined role.
log.Printf("[INFO] Attempting to AssumeRole %s (SessionName: %q, ExternalId: %q)",
c.AssumeRoleARN, c.AssumeRoleSessionName, c.AssumeRoleExternalID)
creds := awsCredentials.NewChainCredentials(providers)
cp, err := creds.Get()
if err != nil {
if awsErr, ok := err.(awserr.Error); ok && awsErr.Code() == "NoCredentialProviders" {
return nil, errors.New(`No valid credential sources found for AWS Provider.
Please see https://terraform.io/docs/providers/aws/index.html for more information on
providing credentials for the AWS Provider`)
}
return nil, fmt.Errorf("Error loading credentials for AWS Provider: %s", err)
}
log.Printf("[INFO] AWS Auth provider used: %q", cp.ProviderName)
awsConfig := &aws.Config{
Credentials: creds,
Region: aws.String(c.Region),
MaxRetries: aws.Int(c.MaxRetries),
HTTPClient: cleanhttp.DefaultClient(),
S3ForcePathStyle: aws.Bool(c.S3ForcePathStyle),
}
stsclient := sts.New(session.New(awsConfig))
assumeRoleProvider := &stscreds.AssumeRoleProvider{
Client: stsclient,
RoleARN: c.AssumeRoleARN,
}
if c.AssumeRoleSessionName != "" {
assumeRoleProvider.RoleSessionName = c.AssumeRoleSessionName
}
if c.AssumeRoleExternalID != "" {
assumeRoleProvider.ExternalID = aws.String(c.AssumeRoleExternalID)
}
providers = []awsCredentials.Provider{assumeRoleProvider}
assumeRoleCreds := awsCredentials.NewChainCredentials(providers)
_, err = assumeRoleCreds.Get()
if err != nil {
if awsErr, ok := err.(awserr.Error); ok && awsErr.Code() == "NoCredentialProviders" {
return nil, fmt.Errorf("The role %q cannot be assumed.\n\n"+
" There are a number of possible causes of this - the most common are:\n"+
" * The credentials used in order to assume the role are invalid\n"+
" * The credentials do not have appropriate permission to assume the role\n"+
" * The role ARN is not valid",
c.AssumeRoleARN)
}
return nil, fmt.Errorf("Error loading credentials for AWS Provider: %s", err)
}
return assumeRoleCreds, nil
}
). This custom chain does not include ECS task yet.

s0enke commented Feb 20, 2017

Facing the same problem. It seems that the Terraform AWS credential logic is custom and does not use the default
AWS provider chain (

// This function is responsible for reading credentials from the
// environment in the case that they're not explicitly specified
// in the Terraform configuration.
func GetCredentials(c *Config) (*awsCredentials.Credentials, error) {
// build a chain provider, lazy-evaulated by aws-sdk
providers := []awsCredentials.Provider{
&awsCredentials.StaticProvider{Value: awsCredentials.Value{
AccessKeyID: c.AccessKey,
SecretAccessKey: c.SecretKey,
SessionToken: c.Token,
}},
&awsCredentials.EnvProvider{},
&awsCredentials.SharedCredentialsProvider{
Filename: c.CredsFilename,
Profile: c.Profile,
},
}
// Build isolated HTTP client to avoid issues with globally-shared settings
client := cleanhttp.DefaultClient()
// Keep the timeout low as we don't want to wait in non-EC2 environments
client.Timeout = 100 * time.Millisecond
cfg := &aws.Config{
HTTPClient: client,
}
usedEndpoint := setOptionalEndpoint(cfg)
if !c.SkipMetadataApiCheck {
// Real AWS should reply to a simple metadata request.
// We check it actually does to ensure something else didn't just
// happen to be listening on the same IP:Port
metadataClient := ec2metadata.New(session.New(cfg))
if metadataClient.Available() {
providers = append(providers, &ec2rolecreds.EC2RoleProvider{
Client: metadataClient,
})
log.Print("[INFO] AWS EC2 instance detected via default metadata" +
" API endpoint, EC2RoleProvider added to the auth chain")
} else {
if usedEndpoint == "" {
usedEndpoint = "default location"
}
log.Printf("[WARN] Ignoring AWS metadata API endpoint at %s "+
"as it doesn't return any instance-id", usedEndpoint)
}
}
// This is the "normal" flow (i.e. not assuming a role)
if c.AssumeRoleARN == "" {
return awsCredentials.NewChainCredentials(providers), nil
}
// Otherwise we need to construct and STS client with the main credentials, and verify
// that we can assume the defined role.
log.Printf("[INFO] Attempting to AssumeRole %s (SessionName: %q, ExternalId: %q)",
c.AssumeRoleARN, c.AssumeRoleSessionName, c.AssumeRoleExternalID)
creds := awsCredentials.NewChainCredentials(providers)
cp, err := creds.Get()
if err != nil {
if awsErr, ok := err.(awserr.Error); ok && awsErr.Code() == "NoCredentialProviders" {
return nil, errors.New(`No valid credential sources found for AWS Provider.
Please see https://terraform.io/docs/providers/aws/index.html for more information on
providing credentials for the AWS Provider`)
}
return nil, fmt.Errorf("Error loading credentials for AWS Provider: %s", err)
}
log.Printf("[INFO] AWS Auth provider used: %q", cp.ProviderName)
awsConfig := &aws.Config{
Credentials: creds,
Region: aws.String(c.Region),
MaxRetries: aws.Int(c.MaxRetries),
HTTPClient: cleanhttp.DefaultClient(),
S3ForcePathStyle: aws.Bool(c.S3ForcePathStyle),
}
stsclient := sts.New(session.New(awsConfig))
assumeRoleProvider := &stscreds.AssumeRoleProvider{
Client: stsclient,
RoleARN: c.AssumeRoleARN,
}
if c.AssumeRoleSessionName != "" {
assumeRoleProvider.RoleSessionName = c.AssumeRoleSessionName
}
if c.AssumeRoleExternalID != "" {
assumeRoleProvider.ExternalID = aws.String(c.AssumeRoleExternalID)
}
providers = []awsCredentials.Provider{assumeRoleProvider}
assumeRoleCreds := awsCredentials.NewChainCredentials(providers)
_, err = assumeRoleCreds.Get()
if err != nil {
if awsErr, ok := err.(awserr.Error); ok && awsErr.Code() == "NoCredentialProviders" {
return nil, fmt.Errorf("The role %q cannot be assumed.\n\n"+
" There are a number of possible causes of this - the most common are:\n"+
" * The credentials used in order to assume the role are invalid\n"+
" * The credentials do not have appropriate permission to assume the role\n"+
" * The role ARN is not valid",
c.AssumeRoleARN)
}
return nil, fmt.Errorf("Error loading credentials for AWS Provider: %s", err)
}
return assumeRoleCreds, nil
}
). This custom chain does not include ECS task yet.

@s0enke

This comment has been minimized.

Show comment
Hide comment
@s0enke

s0enke Feb 26, 2017

I blogged about using Terraform within CodeBuild, which includes a workaround for this problem: https://www.ruempler.eu/2017/02/26/continuous-infrastructure-delivery-pipeline-aws-codepipeline-codebuild-terraform/

s0enke commented Feb 26, 2017

I blogged about using Terraform within CodeBuild, which includes a workaround for this problem: https://www.ruempler.eu/2017/02/26/continuous-infrastructure-delivery-pipeline-aws-codepipeline-codebuild-terraform/

@danmandle

This comment has been minimized.

Show comment
Hide comment
@danmandle

danmandle Apr 3, 2017

This may be a little off-topic, but here's my workaround for use with Jenkins Pipelines (Groovy)

import groovy.json.JsonSlurperClassic

@NonCPS
def jsonParse(def json) {
    new groovy.json.JsonSlurperClassic().parseText(json)
}

node {
  stage('AWS Creds Please'){
    def awsCredsJSON = sh(returnStdout: true, script: "curl 169.254.170.2$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI")
    
    def awsCreds = jsonParse(awsCredsJSON)
    
    env.AWS_ACCESS_KEY_ID = awsCreds.AccessKeyId
    env.AWS_SECRET_ACCESS_KEY = awsCreds.SecretAccessKey
  }
}

danmandle commented Apr 3, 2017

This may be a little off-topic, but here's my workaround for use with Jenkins Pipelines (Groovy)

import groovy.json.JsonSlurperClassic

@NonCPS
def jsonParse(def json) {
    new groovy.json.JsonSlurperClassic().parseText(json)
}

node {
  stage('AWS Creds Please'){
    def awsCredsJSON = sh(returnStdout: true, script: "curl 169.254.170.2$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI")
    
    def awsCreds = jsonParse(awsCredsJSON)
    
    env.AWS_ACCESS_KEY_ID = awsCreds.AccessKeyId
    env.AWS_SECRET_ACCESS_KEY = awsCreds.SecretAccessKey
  }
}
@apparentlymart

This comment has been minimized.

Show comment
Hide comment
@apparentlymart

apparentlymart Apr 4, 2017

Contributor

Thanks for this feature request @iwat, and thanks to everyone else for the great info that followed.

Terraform is using the official Go SDK for AWS but is customizing the set of valid credential sources. To implement this I expect we would need to upgrade the SDK (assuming we didn't already do that for some other reason) and add one more credential provider to the list in the Terraform AWS provider.

Since this one is gated on the presence of an environment variable it should be safe to add without any unintended consequences for those not using ECS.

The Terraform team doesn't have any immediate plans to work on this but if someone else had the time or motivation we would love to review a PR!

Contributor

apparentlymart commented Apr 4, 2017

Thanks for this feature request @iwat, and thanks to everyone else for the great info that followed.

Terraform is using the official Go SDK for AWS but is customizing the set of valid credential sources. To implement this I expect we would need to upgrade the SDK (assuming we didn't already do that for some other reason) and add one more credential provider to the list in the Terraform AWS provider.

Since this one is gated on the presence of an environment variable it should be safe to add without any unintended consequences for those not using ECS.

The Terraform team doesn't have any immediate plans to work on this but if someone else had the time or motivation we would love to review a PR!

@jekh

This comment has been minimized.

Show comment
Hide comment
@jekh

jekh May 5, 2017

I submitted PR #14199 for this. A couple questions in the PR, but hopefully this will help get terraform working with CodeBuild/ECS slaves.

jekh commented May 5, 2017

I submitted PR #14199 for this. A couple questions in the PR, but hopefully this will help get terraform working with CodeBuild/ECS slaves.

@jch254

This comment has been minimized.

Show comment
Hide comment
@jch254

jch254 Apr 18, 2018

I can confirm this is now working in CodeBuild WITHOUT the pre_build phase I posted above. Terraform 0.11.7 and Terraform AWS provider 1.14.1. Brilliant work 👍

jch254 commented Apr 18, 2018

I can confirm this is now working in CodeBuild WITHOUT the pre_build phase I posted above. Terraform 0.11.7 and Terraform AWS provider 1.14.1. Brilliant work 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment