credential_process using sts get-session-token fails for aws_iam_role and aws_iam_profile with InvalidClientTokenId

[earchibald-lv]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

Terraform v0.12.20
(AWS provider 2.49.0)

Affected Resource(s)

• aws_iam_role
• aws_iam_profile

Terraform Configuration Files

resource "aws_iam_policy" "bootstrap" {
  name        = "ec2_bootstrap_policy"
  path        = "/terraform/"
  description = "Policy to support S3 and EC2 read access for EC2 instances."
  policy      = file("ec2_bootstrap_policy.json")
}

resource "aws_iam_role" "bootstrap" {
  name               = "ec2_bootstrap_role"
  path               = "/terraform/"
  description        = "Role to support S3 and EC2 read access for EC2 instances."
  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Principal": {
        "Service": "ec2.amazonaws.com"
      },
      "Effect": "Allow",
      "Sid": ""
    }
  ]
}
EOF

  tags = {
    Project = "siqna"
  }
}

Debug Output

Panic Output

Expected Behavior

The rest of this module works perfectly, dealing with S3, instances, routes, vpc, all kinds of stuff; the IAM user being used has Administrator access via IAM.

The credential_process invoked takes access and secret keys from various secret backends, then invokes aws sts get-session-token, massages them into the correct format with jq and uses the session credentials.

If I hardcode the same access and secret keys into the AWS credentials file everything works as expected.

Actual Behavior

2020/02/20 13:48:22 [DEBUG] aws_iam_policy.bootstrap: apply errored, but we're indicating that via the Error pointer rather than returning it: Error creating IAM policy ec2_bootstrap_policy: InvalidClientTokenId: The security token included in the request is invalid
        status code: 403, request id: 0156f3b9-c708-4899-98d4-0e4b57e69b85
2020/02/20 13:48:22 [ERROR] <root>: eval: *terraform.EvalApplyPost, err: Error creating IAM Role ec2_bootstrap_role: InvalidClientTokenId: The security token included in the request is invalid
        status code: 403, request id: 35cf094d-89f9-4fed-a11c-abca8d3e6852

Steps to Reproduce

1. terraform apply

Important Factoids

References

• #0000

[earchibald-lv]

Specifically, we get a session token and provide it to credential_process with the excerpt:

aws sts get-session-token | jq '.Credentials + {Version: 1}'

[justinretzolk]

Hi @earchibald-lv 👋 Thank you for taking the time to file this issue. Given that there's been a number of AWS provider releases since you initially filed it, can you confirm whether you're still experiencing this behavior?