terraform commands fail to detect expired AWS security tokens

[ghost]

This issue was originally opened by @scubahub as hashicorp/terraform#18024. It was migrated here as a result of the provider split. The original body of the issue is below.

---

Im using the AWS provider and relying on terraform to access using the values defined in the default ~/.aws/credentials and ~/.aws/config files. The credentials entries include aws_session_token which terraform does seem to pick up. However when that token is expired and I try to terraform plan, terraform tries to connect to AWS 15 times then just hangs.

At that point ctl-c says it's gracefully shutting down, but it never does. Have to ctl-c it a second time.

Terraform Version

Terraform v0.11.7

Terraform Configuration Files

provider "aws" {
  profile = "dz"
}

resource "aws_instance" "example" {
  ami           = "ami-f4533694"
  instance_type = "t2.micro"
}

Debug Output

The following is the result for every attempt to terraform makes to login to aws:

2018-05-10T10:46:37.580-0700 [DEBUG] plugin.terraform-provider-aws_v1.18.0_x4: ---[ REQUEST POST-SIGN ]-----------------------------
2018-05-10T10:46:37.580-0700 [DEBUG] plugin.terraform-provider-aws_v1.18.0_x4: POST / HTTP/1.1
2018-05-10T10:46:37.580-0700 [DEBUG] plugin.terraform-provider-aws_v1.18.0_x4: Host: sts.amazonaws.com
2018-05-10T10:46:37.580-0700 [DEBUG] plugin.terraform-provider-aws_v1.18.0_x4: User-Agent: aws-sdk-go/1.13.42 (go1.9.2; darwin; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.11.7
2018-05-10T10:46:37.580-0700 [DEBUG] plugin.terraform-provider-aws_v1.18.0_x4: Content-Length: 43
2018-05-10T10:46:37.580-0700 [DEBUG] plugin.terraform-provider-aws_v1.18.0_x4: Authorization: AWS4-HMAC-SHA256 Credential=***redacted***
2018-05-10T10:46:37.580-0700 [DEBUG] plugin.terraform-provider-aws_v1.18.0_x4: Content-Type: application/x-www-form-urlencoded; charset=utf-8
2018-05-10T10:46:37.580-0700 [DEBUG] plugin.terraform-provider-aws_v1.18.0_x4: X-Amz-Date: 20180510T174637Z
2018-05-10T10:46:37.581-0700 [DEBUG] plugin.terraform-provider-aws_v1.18.0_x4: X-Amz-Security-Token: *** redacted ***
2018-05-10T10:46:37.581-0700 [DEBUG] plugin.terraform-provider-aws_v1.18.0_x4: Accept-Encoding: gzip
2018-05-10T10:46:37.581-0700 [DEBUG] plugin.terraform-provider-aws_v1.18.0_x4: Action=GetCallerIdentity&Version=2011-06-15
2018-05-10T10:46:37.581-0700 [DEBUG] plugin.terraform-provider-aws_v1.18.0_x4: -----------------------------------------------------
2018-05-10T10:46:37.987-0700 [DEBUG] plugin.terraform-provider-aws_v1.18.0_x4: 2018/05/10 10:46:37 [DEBUG] [aws-sdk-go] DEBUG: Response sts/GetCallerIdentity Details:
2018-05-10T10:46:37.987-0700 [DEBUG] plugin.terraform-provider-aws_v1.18.0_x4: ---[ RESPONSE ]--------------------------------------
2018-05-10T10:46:37.987-0700 [DEBUG] plugin.terraform-provider-aws_v1.18.0_x4: HTTP/1.1 403 Forbidden
2018-05-10T10:46:37.987-0700 [DEBUG] plugin.terraform-provider-aws_v1.18.0_x4: Connection: close
2018-05-10T10:46:37.987-0700 [DEBUG] plugin.terraform-provider-aws_v1.18.0_x4: Content-Length: 297
2018-05-10T10:46:37.987-0700 [DEBUG] plugin.terraform-provider-aws_v1.18.0_x4: Content-Type: text/xml
2018-05-10T10:46:37.987-0700 [DEBUG] plugin.terraform-provider-aws_v1.18.0_x4: Date: Thu, 10 May 2018 17:46:37 GMT
2018-05-10T10:46:37.987-0700 [DEBUG] plugin.terraform-provider-aws_v1.18.0_x4: X-Amzn-Requestid: 16c459b6-547a-11e8-90ea-59844d81b50c
2018-05-10T10:46:37.987-0700 [DEBUG] plugin.terraform-provider-aws_v1.18.0_x4:
2018-05-10T10:46:37.987-0700 [DEBUG] plugin.terraform-provider-aws_v1.18.0_x4:
2018-05-10T10:46:37.987-0700 [DEBUG] plugin.terraform-provider-aws_v1.18.0_x4: -----------------------------------------------------
2018-05-10T10:46:37.987-0700 [DEBUG] plugin.terraform-provider-aws_v1.18.0_x4: 2018/05/10 10:46:37 [DEBUG] [aws-sdk-go] <ErrorResponse xmlns="https://sts.amazonaws.com/doc/2011-06-15/">
2018-05-10T10:46:37.987-0700 [DEBUG] plugin.terraform-provider-aws_v1.18.0_x4:   <Error>
2018-05-10T10:46:37.987-0700 [DEBUG] plugin.terraform-provider-aws_v1.18.0_x4:     <Type>Sender</Type>
2018-05-10T10:46:37.987-0700 [DEBUG] plugin.terraform-provider-aws_v1.18.0_x4:     <Code>ExpiredToken</Code>
2018-05-10T10:46:37.987-0700 [DEBUG] plugin.terraform-provider-aws_v1.18.0_x4:     <Message>The security token included in the request is expired</Message>
2018-05-10T10:46:37.987-0700 [DEBUG] plugin.terraform-provider-aws_v1.18.0_x4:   </Error>
2018-05-10T10:46:37.987-0700 [DEBUG] plugin.terraform-provider-aws_v1.18.0_x4:   <RequestId>16c459b6-547a-11e8-90ea-59844d81b50c</RequestId>
2018-05-10T10:46:37.987-0700 [DEBUG] plugin.terraform-provider-aws_v1.18.0_x4: </ErrorResponse>

Here is the output after ctl-c:

2018/05/10 10:48:08 [WARN] terraform: Stop called, initiating interrupt sequence
2018/05/10 10:48:08 [WARN] terraform: run context exists, stopping
Interrupt received.
Please wait for Terraform to exit or data loss may occur.
Gracefully shutting down...
stopping operation...
2018-05-10T10:48:08.846-0700 [DEBUG] plugin.terraform-provider-aws_v1.18.0_x4: plugin received interrupt signal, ignoring: timestamp=2018-05-10T10:48:08.845-0700 count=1

Expected Behavior

Terraform should exit and notify the user their credentials are expired.

Actual Behavior

Terraform never returns

Steps to Reproduce

terraform init
terraform plan

[skeller88]

I'm experiencing this bug as well.

[skeller88]

@bflad See #4502 which reports the same issue.

[farhank3389]

Any update on this issue?

[skang0601]

This is also affecting me as well whenever I do longer running provisioning such as RDS instances.

Since it takes so long for RDS instances to provision, the token I'm using will time out leaving Terraform in a state of constant waiting. This actually corrupts my state pretty badly as when I exit the command and attempt to rectify, it doesn't seem to sync the state properly.

[bernadinm]

Ideally, this error should be exposed to the user rather than only the debug logs:

The security token included in the request is expired

[k7faq]

In my work environment we authenticate to aws via a Federated connection. I found (the very hard way) that having failed to "login" in this morning terraform plan would 'hang' and I had to use Ctrl + C to stop the action. The only information provided with a TF_LOG="ERROR" set in my MacBookPro environment.

Ideal result would be for terraform plan to halt and report an error message to the user reporting connection failure(s). A simple message of "Connection Failure" would be far more informative than the vague log data. As well, I disagree with forcing the user to depend on having to view a log file to obtain this level of data. This does defeat various levels of automation and allowing 'junior' or 'lesser skilled persons' executing scripts. < My 2 cents ...

Log output:

2019/01/18 10:06:22 [ERROR] root: eval: *terraform.EvalConfigProvider, err: unexpected EOF
2019/01/18 10:06:22 [ERROR] root: eval: *terraform.EvalSequence, err: unexpected EOF
2019/01/18 10:06:22 [ERROR] root: eval: *terraform.EvalOpFilter, err: unexpected EOF
2019/01/18 10:06:22 [ERROR] root: eval: *terraform.EvalSequence, err: unexpected EOF
2019/01/18 10:06:22 [TRACE] [walkRefresh] Exiting eval tree: provider.aws

[kaznak]

I think this could be not a bug because a user can refresh credential while Terraform is retrying.
But this silent retry is still very annoying. Please print some warning messages while retrying for expired credentials.

[k7faq]

From where would I find these warning messages?

[longbowrocks]

Related to issues #1351, #1307, #2068, and PR #6992.

tl;dr If credentials expire and the problem cannot be resolved, terraform should report that there is a problem. Currently credential expiration is opaque to users (it isn't reported at the default logging level).

Currently, when credentials expire, terraform loops until the modification timeout for its active resource expires, then complains about being unable to write its state. This does not clearly convey that there is an issue that needs to be resolved while trying to modify resources, nor does it convey that there was a credential problem earlier on when the persistence of the state file fails. To the user, it appears that the active resource was taking a long time to modify, and as a result credentials expired, resulting in the end of the build at that time. In truth, cause and effect were flipped: the credentials expired, causing the active resource to "take a long time" to modify.

Here are the scenarios when credentials expire midway through apply as I understand them:

• Terraform is running in EC2 and was provided a role to assume for credentials. Terraform assumes the role again to get new credentials.
• Credentials were obtained through the awscli credentials file or environment variables and cannot be refreshed. Terraform default logging level exposes this to the user, and the user restarts the apply with fresh credentials.

The second use case is not addressed. The first use case may be addressed, but I haven't tried it so I can't say.

[devonbleak]

Pretty sure this behavior is coming from the SDK itself and probably related to aws/aws-sdk-go#925

[jamesbibby]

I experienced this as well when creating a large redis cluster. After 1 hour the credentials stopped working and the process hung. I also needed to ctrl+c x 2 to get out and my state was left corrupted as it could not be written back to s3. Can I help out at all or are you sure the behaviour is caused by an underlying sdk issue?

[devonbleak]

I'm 99% sure this is the underlying SDK. Looking at the provider code there's no reference to ExpiredToken errors, and most (all?) of the CRUD functions are just returning whatever error the SDK sends back, which says to me that the SDK just isn't returning when it hits an ExpiredToken error and can't refresh its credentials.

[gkrizek]

Hitting this too. If you are using a system that rotates credentials frequently (like hourly) this gets really annoying.

[jrsmith17]

Why was this closed? This is still an ongoing issue. I'm on what the download page lists as the latest version: 0.12.24.

[dshuvar]

Ideally, this error should be exposed to the user rather than only the debug logs:

The security token included in the request is expired

I have faced with an issue in terraform 0.12.29.
I have received the correct token and can use aws cli but cant forming terraform plan.

[tbugfinder]

I just run into a hung terrform session also.

[tbugfinder]

$ terraform version
Terraform v0.14.7
+ provider registry.terraform.io/hashicorp/aws v3.34.0
+ provider registry.terraform.io/hashicorp/local v2.1.0
+ provider registry.terraform.io/hashicorp/null v3.1.0
+ provider registry.terraform.io/hashicorp/random v3.1.0
+ provider registry.terraform.io/hashicorp/template v2.2.0

and the logs:

021-03-27T09:52:54.191+0100 [INFO]  plugin.terraform-provider-aws_v3.34.0_x5: 2021/03/27 09:52:54 [DEBUG] [aws-sdk-go] DEBUG: Request sts/GetCallerIdentity Details:
---[ REQUEST POST-SIGN ]-----------------------------
POST / HTTP/1.1
Host: sts.amazonaws.com
User-Agent: aws-sdk-go/1.38.0 (go1.16; linux; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.14.7 (+https://www.terraform.io) terraform-provider-aws/dev (+https://registry.terraform.io/providers/hashicorp/aws)
Content-Length: 43
Authorization: AWS4-HMAC-SHA256 Credential=ASIxxxxxxx7U77/20210327/us-east-1/sts/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-date;x-amz-security-token, Signature=72381d4727e16f9xxxxxxxxxxxxxxxxa984ac748
Content-Type: application/x-www-form-urlencoded; charset=utf-8
X-Amz-Date: 20210327T085254Z
X-Amz-Security-Token: FwoGZXIvYXdzEOj//////////wEaDHocqGxxxxxxxxxxxxxxDBsZbDfQOH2OvcDA==
Accept-Encoding: gzip

Action=GetCallerIdentity&Version=2011-06-15
-----------------------------------------------------: timestamp=2021-03-27T09:52:54.190+0100
2021-03-27T09:52:54.893+0100 [INFO]  plugin.terraform-provider-aws_v3.34.0_x5: 2021/03/27 09:52:54 [DEBUG] [aws-sdk-go] DEBUG: Response sts/GetCallerIdentity Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 403 Forbidden
Connection: close
Content-Length: 297
Content-Type: text/xml
Date: Sat, 27 Mar 2021 08:52:54 GMT
X-Amzn-Requestid: 0ee1be08-xxxxxxxxxxxxxxx

[gadelkareem]

Any updates on this?
tf version 0.12.24

I do not face this when I do a terraform plan but when I try to import a resource.

[jacobm3]

Bug still in latest -

$ terraform version
Terraform v1.0.10
on linux_amd64
+ provider registry.terraform.io/hashicorp/aws v3.64.2

[robertlagrant]

I have the same issue. Even refreshing credentials behind the scenes while it's "Still modifying..." doesn't do anything. It just keeps plugging away with the old credentials.

[YakDriver]

This is closed by events. See hashicorp/aws-sdk-go-base#362 for more details.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.