Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

InvalidClientTokenId error when creating IAM role with vault credentials and terraform #266

Closed
skeller88 opened this issue Jul 10, 2018 · 25 comments
Closed

InvalidClientTokenId error when creating IAM role with vault credentials and terraform #266

skeller88 opened this issue Jul 10, 2018 · 25 comments

Comments

@skeller88
Copy link

skeller88 commented Jul 10, 2018 •
edited

OS: 10.13.4 (High Sierra)
Terraform version: v0.11.7
provider.aws: v1.26.0

The problem:
When I attempt to create an IAM role via terraform while using vault credentials for a user called "terraform", a 403 error occurs.

Shanes-MacBook-Pro:terraform_test shanekeller$ aws-vault --debug exec terraform -- terraform apply --auto-approve 
2018/07/09 22:43:27 [keyring] Considering backends: [keychain file]
2018/07/09 22:43:27 Loading config file /Users/shanekeller/.aws/config
2018/07/09 22:43:27 Parsing config file /Users/shanekeller/.aws/config
2018/07/09 22:43:27 Looking for sessions for terraform
2018/07/09 22:43:27 Looking up all keys in keyring
2018/07/09 22:43:27 [keyring] Querying keychain for service="aws-vault", keychain="aws-vault.keychain"
2018/07/09 22:43:27 [keyring] Found 3 results
2018/07/09 22:43:27 Session "terraform session (1531204691)" expires in 54m43.768571954s
2018/07/09 22:43:27 [keyring] Querying keychain for service="aws-vault", account="terraform session (1531204691)", keychain="aws-vault.keychain"
2018/07/09 22:43:30 [keyring] Found item "aws-vault session for terraform"
2018/07/09 22:43:30 Using session ********************, expires in 54m40.363844684s
2018/07/09 22:43:30 Setting subprocess env: AWS_DEFAULT_REGION=us-west-2, AWS_REGION=us-west-2
2018/07/09 22:43:30 Setting subprocess env: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY
2018/07/09 22:43:30 Setting subprocess env: AWS_SESSION_TOKEN, AWS_SECURITY_TOKEN
aws_iam_role.iam_for_lambda: Creating...
  arn:                   "" => "<computed>"
  assume_role_policy:    "" => "{\n  \"Version\": \"2012-10-17\",\n  \"Statement\": [\n    {\n      \"Action\": \"sts:AssumeRole\",\n      \"Principal\": {\n        \"Service\": \"lambda.amazonaws.com\"\n      },\n      \"Effect\": \"Allow\",\n      \"Sid\": \"\"\n    }\n  ]\n}\n"
  create_date:           "" => "<computed>"
  force_detach_policies: "" => "false"
  max_session_duration:  "" => "3600"
  name:                  "" => "iam_for_lambda"
  path:                  "" => "/"
  unique_id:             "" => "<computed>"

Error: Error applying plan:

1 error(s) occurred:

* aws_iam_role.iam_for_lambda: 1 error(s) occurred:

* aws_iam_role.iam_for_lambda: Error creating IAM Role iam_for_lambda: InvalidClientTokenId: The security token included in the request is invalid
	status code: 403, request id: 3124a1fa-8404-11e8-b8c9-0f4f651e0f50

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

Here are the terraform debug logs just before the error:

2018-07-09T22:46:53.942-0700 [DEBUG] plugin.terraform-provider-aws_v1.26.0_x4: Accept-Encoding: gzip
2018-07-09T22:46:53.942-0700 [DEBUG] plugin.terraform-provider-aws_v1.26.0_x4: 
2018-07-09T22:46:53.942-0700 [DEBUG] plugin.terraform-provider-aws_v1.26.0_x4: Action=GetRole&RoleName=iam_for_lambda&Version=2010-05-08
2018-07-09T22:46:53.942-0700 [DEBUG] plugin.terraform-provider-aws_v1.26.0_x4: -----------------------------------------------------
2018-07-09T22:46:54.453-0700 [DEBUG] plugin.terraform-provider-aws_v1.26.0_x4: 2018/07/09 22:46:54 [DEBUG] [aws-sdk-go] DEBUG: Response iam/GetRole Details:
2018-07-09T22:46:54.453-0700 [DEBUG] plugin.terraform-provider-aws_v1.26.0_x4: ---[ RESPONSE ]--------------------------------------
2018-07-09T22:46:54.453-0700 [DEBUG] plugin.terraform-provider-aws_v1.26.0_x4: HTTP/1.1 403 Forbidden
2018-07-09T22:46:54.453-0700 [DEBUG] plugin.terraform-provider-aws_v1.26.0_x4: Connection: close
2018-07-09T22:46:54.453-0700 [DEBUG] plugin.terraform-provider-aws_v1.26.0_x4: Content-Length: 305
2018-07-09T22:46:54.453-0700 [DEBUG] plugin.terraform-provider-aws_v1.26.0_x4: Content-Type: text/xml
2018-07-09T22:46:54.453-0700 [DEBUG] plugin.terraform-provider-aws_v1.26.0_x4: Date: Tue, 10 Jul 2018 05:46:53 GMT
2018-07-09T22:46:54.453-0700 [DEBUG] plugin.terraform-provider-aws_v1.26.0_x4: X-Amzn-Requestid: a68ca1e8-8404-11e8-b6a6-e1ba4d916180
2018-07-09T22:46:54.453-0700 [DEBUG] plugin.terraform-provider-aws_v1.26.0_x4: 
2018-07-09T22:46:54.453-0700 [DEBUG] plugin.terraform-provider-aws_v1.26.0_x4: 
2018-07-09T22:46:54.453-0700 [DEBUG] plugin.terraform-provider-aws_v1.26.0_x4: -----------------------------------------------------
2018-07-09T22:46:54.453-0700 [DEBUG] plugin.terraform-provider-aws_v1.26.0_x4: 2018/07/09 22:46:54 [DEBUG] [aws-sdk-go] <ErrorResponse xmlns="https://iam.amazonaws.com/doc/2010-05-08/">

I am able to successfully create the same IAM role via terraform by hardcoding the "terraform" user credentials into the provider definition in the .tf file.

Shanes-MacBook-Pro:terraform_test shanekeller$ aws-vault exec terraform -- terraform apply  --auto-approve

aws_iam_role.iam_for_lambda: Creating...
  arn:                   "" => "<computed>"
  assume_role_policy:    "" => "{\n  \"Version\": \"2012-10-17\",\n  \"Statement\": [\n    {\n      \"Action\": \"sts:AssumeRole\",\n      \"Principal\": {\n        \"Service\": \"lambda.amazonaws.com\"\n      },\n      \"Effect\": \"Allow\",\n      \"Sid\": \"\"\n    }\n  ]\n}\n"
  create_date:           "" => "<computed>"
  force_detach_policies: "" => "false"
  max_session_duration:  "" => "3600"
  name:                  "" => "iam_for_lambda"
  path:                  "" => "/"
  unique_id:             "" => "<computed>"
aws_iam_role.iam_for_lambda: Creation complete after 1s (ID: iam_for_lambda)

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

tf file

provider "aws" {
  region     = "us-west-2"
}

resource "aws_iam_role" "iam_for_lambda" {
  name = "iam_for_lambda"

  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Effect": "Allow",
      "Sid": ""
    }
  ]
}
EOF
}

~/.aws/config file

[profile terraform]
mfa_serial = arn:aws:iam::207283649878:user/terraform
region = us-west-2
source_profile = terraform

What I've tried

Creating other resources with the aws-vault credentials and terraform. I'm able to successfully create s3 buckets and a vpc.

resource "aws_s3_bucket" "terraform_test"{
  bucket = "terraform-test-1983451"
}

resource "aws_vpc" "test" {
  # https://serverfault.com/questions/630022/what-is-the-recommended-cidr-when-creating-vpc-on-aws
  # "...there is no harm in starting with a small prefix such as /16 because you can always create subnets."
  cidr_block = "11.0.0.0/16"

  # enabled by default, but just to be sure
  enable_dns_support   = true
  enable_dns_hostnames = true
}

It seems like the security token generated by aws-vault has different permissions than I'm expecting. But I'm not sure what additional debugging steps to take.

There's another issue that addresses security tokens, but I don't think the solution applies to me because the poster had two roles: #262
@FernandoMiguel
Copy link
Collaborator

Sounds like you are missing MFA

@skeller88
Copy link
Author

Ok I added MFA and a role according to the Usage documentation. Now my .tf file looks like this:

provider "aws" {
  region     = "us-west-2"
  profile = "deployment_user"
  assume_role {
    role_arn     = "arn:aws:iam::[account-number]:role/deployment"
    session_name = "assume_deployment_role"
  }
}

And my ~/.aws/config file:

[profile deployment_user]
mfa_serial = arn:aws:iam::[account-number]:mfa/deployment_user
region = us-west-2
source_profile = deployment_user
role_arn = arn:aws:iam::[account-number]:role/deployment

I added the "deployment_user" access key and secret to the vault.

What policy does the "deployment_user" need to assume the "deployment" Role? I tried this:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::[account-number]:role/deployment"
        }
    ]
}

But get an error:


Shanes-MacBook-Pro:terraform_test shanekeller$ aws-vault exec deployment_user -- terraform apply
Enter token for arn:aws:iam::[account-number]:mfa/deployment_user: [token-entered]

Error: Error refreshing state: 1 error(s) occurred:

* provider.aws: The role "arn:aws:iam::[account-number]:role/deployment" cannot be assumed.

  There are a number of possible causes of this - the most common are:
    * The credentials used in order to assume the role are invalid
    * The credentials do not have appropriate permission to assume the role
    * The role ARN is not valid

I copied role ARN from the "deployment" role via the AWS console, and I just added the AWS secret and key for "deployment_user" to the vault in a fresh bash session. So the user credentials must not have the right permissions. Any ideas?

@FernandoMiguel
Copy link
Collaborator

You seem to be miss understanding how aws-vault works.
Aws-vault assumes a role using an IAM account.
Your terraform then has a provider that runs under that role, or has a provider (or alias) to assume yet another role.
Here's an example
https://github.com/FernandoMiguel/kb/blob/master/PortoLinux/ET-2017Dez/terraform/main.tf

@skeller88
Copy link
Author

skeller88 commented Jul 10, 2018 via email

@frezbo
Copy link
Contributor

frezbo commented Jul 10, 2018

AWS requires to use an MFA to perform IAM operation with an assume-role and you should not specify the profile in the TF file.

@skeller88
Copy link
Author

skeller88 commented Jul 10, 2018 •
edited

I'm using MFA. I tried again without the profile in the TF file

provider "aws" {
  region     = "us-west-2"
  assume_role {
    role_arn     = "arn:aws:iam::[account]:role/deployment"
    session_name = "terraform"
  }
}

and still get the error.

Shanes-MacBook-Pro:terraform_test shanekeller$ aws-vault exec deployment_user -- terraform apply
Enter token for arn:aws:iam::[account]:mfa/deployment_user: 903223
aws-vault: error: Failed to get credentials for deployment_user: AccessDenied: MultiFactorAuthentication failed with invalid MFA one time pass code. 
	status code: 403, request id: afad2a3f-8463-11e8-a9fd-d188f32cdfae
Shanes-MacBook-Pro:terraform_test shanekeller$ aws-vault exec deployment_user -- terraform apply
Enter token for arn:aws:iam::[account]:mfa/deployment_user: 000725

Error: Error refreshing state: 1 error(s) occurred:

* provider.aws: The role "arn:aws:iam::[account]:role/deployment" cannot be assumed.

  There are a number of possible causes of this - the most common are:
    * The credentials used in order to assume the role are invalid
    * The credentials do not have appropriate permission to assume the role
    * The role ARN is not valid

@frezbo
Copy link
Contributor

frezbo commented Jul 10, 2018 •
edited

What does this give:

aws-vault exec deployment_user -- aws sts assume-role --role-arn <arn> --external-id <if there is one> --role-session-name dummy

Make sure to remove the creds when posting 😉

@frezbo
Copy link
Contributor

frezbo commented Jul 10, 2018

@skeller88 I just saw your .aws/config file. Two questions:

  • The source_profile is the same as your current deployment_profile
  • Assuming the file content was wrong, why are you again trying to assume the role in TF?. aws-vault already gives you the assumed creds.

@skeller88
Copy link
Author

Can we DM? I'm on gmail at skeller88[at]gmail[dot]com

@skeller88
Copy link
Author

skeller88 commented Jul 10, 2018 •
edited

What's the difference between "source_profile" and the first line of the ~/.aws/config file that has "profile" in it? I'm referring to

[profile deployment_user]
mfa_serial=arn:aws:iam::207283649878:mfa/deployment_user

Should I be giving the profile name a different name?

I executed the sts command:

Shanes-MacBook-Pro:terraform_test shanekeller$ aws-vault exec deployment_user -- aws sts assume-role --role-arn arn:aws:iam::[account]:role/deployment  --role-session-name dummy

An error occurred (AccessDenied) when calling the AssumeRole operation: User: arn:aws:sts::[account]:assumed-role/deployment/[id] is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::[account]:role/deployment
Shanes-MacBook-Pro:terraform_test shanekeller$ aws-vault exec terraform -- terraform apply
Enter token for arn:aws:iam::[account]:mfa/deployment_user: [mfa-code]

Error: Error refreshing state: 1 error(s) occurred:

* provider.aws: The role "arn:aws:iam::[acccount]:role/deployment" cannot be assumed.

  There are a number of possible causes of this - the most common are:
    * The credentials used in order to assume the role are invalid
    * The credentials do not have appropriate permission to assume the role
    * The role ARN is not valid

Assuming the file content was wrong, why are you again trying to assume the role in TF?. aws-vault already gives you the assumed creds.

I'm copying the format of the tf file in #262. Once the file content is right and I can properly assume the role with aws-vault, I'll want Terraform to assume that role as well.

Here's my understanding of the aws-vault process from the documentation:

  • Create an IAM role with the desired AWS permissions. In my case, that role is "deployment_role". It has access to all the AWS resources I need to manipulate with Terraform.
  • Create an IAM user. In my case, that user is "deployment". Enable MFA for the user.
  • Create a policy that gives the IAM user the permission to assume the "deployment_role". In my case, that policy is "assume_deployment_role":
{
    "Version": "2012-10-17",
    "Statement": {
        "Effect": "Allow",
        "Action": "sts:AssumeRole",
        "Resource": "arn:aws:iam::[account]:role/deployment"
    }
}
  • Add a trust policy to the "deployment" Role that gives the "deployment_user" User permission to assume the role.
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::[account]:user/deployment_user"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "Bool": {
          "aws:MultiFactorAuthPresent": "true"
        }
      }
    }
  ]
}
  • Add the "deployment_user" id and secret to the aws-vault. I did so under the name "deployment_user".
  • Configure my ~/.aws/config profile to use the appropriate mfa role and assume the appropriate role
[profile deployment_user]
mfa_serial=arn:aws:iam::[account]:mfa/deployment_user
region=us-west-2
role_arn=arn:aws:iam::[account]:role/deployment

Do you see anything I'm doing wrong?

@frezbo
Copy link
Contributor

frezbo commented Jul 10, 2018

All your understanding is perfectly right. The source profile is the profile credentials used or assuming the role. Sorry email is real hard to follow up

@skeller88
Copy link
Author

skeller88 commented Jul 10, 2018 via email

@frezbo
Copy link
Contributor

frezbo commented Jul 11, 2018

slack?

@skeller88
Copy link
Author

skeller88 commented Jul 11, 2018 via email

@frezbo
Copy link
Contributor

frezbo commented Jul 11, 2018

I'm available on the k8s and chef slack channels

@0xdabbad00
Copy link

0xdabbad00 commented Jul 11, 2018 •
edited

From the error message

An error occurred (AccessDenied) when calling the AssumeRole operation: User: arn:aws:sts::[account]:assumed-role/deployment/[id] is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::[account]:role/deployment

you already assumed into a role.

Run:

aws-vault exec deployment_user -- aws sts get-caller-identity

@skeller88
Copy link
Author

skeller88 commented Jul 11, 2018 via email

@0xdabbad00
Copy link

Don't worry about the UserId that is its own thing, but I think there may be something to look into with regard to the error message saying your identity was arn:aws:sts::[account]:assumed-role/deployment/[id] but get-caller-identity saying it is arn:aws:iam::[account]:user/deployment_user .

@lox
Copy link
Collaborator

lox commented Jul 12, 2018

I would guess the issue is layering of an STS session and then trying to assume a role with that.

aws-vault is designed with a primary objective of "protect your root credentials". E.g. only disclose short-lived temporary credentials to subprocesses. There are two ways to do this, either creating an STS session and providing those credentials to the subprocess, or assuming a role and providing the session credentials form that to the subprocess.

AWS has complicated rules about chaining temporary credentials together with assuming roles, or assuming roles from other roles.

If you really want, you can disable the session that aws-vault creates with aws-vault exec <profile> --no-session, but I think it waters down a lot of the security that it provides.

In terms of your specific problem, I'd suggest troubleshooting it by getting the simplest config you can working and then adding complexity to it. Start without Terraform, get aws-vault simply assuming a role in your CLI. Prove that you can run something like aws s3 ls or an equivalent simple command.

Note that your config should look something like:

[profile ljd]
region=us-east-1

[profile ljd-admin]
region=us-east-1
source_profile=ljd
mfa_serial=arn:aws:iam::xxx:mfa/lachlan.donald
role_arn=arn:aws:iam::xxx:role/assume-admin-access

The source_profile refers to the profile to look for the credentials in. In this case, credentials are against my IAM user, in myljd profile. I then use an ljd-admin profile to assume a role, via the credentials in the ljd profile.

@skeller88
Copy link
Author

skeller88 commented Jul 13, 2018 •
edited

Here's what I've tried:

Deleted my ~/.aws/config and ~/.aws/credentials files.
Deleted all aws-vault credentials.
Created a new IAM user, test, with no MFA.
Gave the test user permissions for all the AWS services I need.
Successfully executed the following commands from the command line:

aws-vault exec test -- aws sns list-topics
aws-vault exec test -- aws ssm get-parameters --names database_credentials
aws-vault exec test -- aws lambda list-functions
aws-vault exec test -- aws rds describe-instances
aws-vault exec test -- aws s3 ls

Even though I added the "IAMFullAccess" policy to the "test" user permissions, the following command still fails with the same error I saw at the start of this issue:

> aws-vault exec test -- aws iam list-roles 
An error occurred (InvalidClientTokenId) when calling the ListRoles operation: The security token included in the request is invalid

I debugged all of the requests and looked at what's different between the aws iam request and the other requests. The session token is the same. I don't see any noticeable differences other than the fact that the request is denied in the iam case.

I'm able to access iam without aws-vault.

aws configure
AWS Access Key ID [None]: xxx
AWS Secret Access Key [None]: xxx
Default region name [None]: us-west-2
Default output format [None]: 
aws iam list-roles 
{
    "Roles": [
     ...
}

Here's my ~/.aws/config file

[profile deployment]
region=us-west-2
[default]
region = us-west-2

Anything else I could try?

@frezbo
Copy link
Contributor

frezbo commented Jul 13, 2018

yes that is the expected behaviour. AWS does not allow IAM operations with an assumed role unless it's authenticated with an MFA

@skeller88
Copy link
Author

I fixed my issue last week. Thanks everyone for the help. I solved it by authenticating the IAM role with MFA, and taking out the "role_arn" section of the aws provider in my .tf file.

@geoffreywiseman
Copy link

yes that is the expected behaviour. AWS does not allow IAM operations with an assumed role unless it's authenticated with an MFA

Just so I better understand -- aws-vault is always using assumed roles, even if you've just set it up with an access key / secret, the same way that you might have done with aws cli? So with the example with test above, if you go directly through aws cli, there's no assumed role, but if you use aws-vault there is, even if you use the same credentials? I'm hitting this same issue, and just trying to understand what's different between the two paths.

@FernandoMiguel
Copy link
Collaborator

@geoffreywiseman yss, unless you use --no-session

@geoffreywiseman
Copy link

Ok, thanks. I also found this issue, #455, which also set me on a good path (in case anyone else ends up looking at this issue). I added mfa_serial to my .aws/config without doing any work to add roles on to of my IAM user, and that has helped.

This was referenced Nov 30, 2019
Closed
Open
@haydenth haydenth mentioned this issue Jan 6, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@lox @geoffreywiseman @FernandoMiguel @0xdabbad00 @skeller88 @frezbo