4xx errors being retried

[ghost]

This issue was originally opened by @jschoombee as hashicorp/terraform#24097. It was migrated here as a result of the provider split. The original body of the issue is below.

---

Terraform Version

2020/02/13 12:02:39 [INFO] Terraform version: 0.12.7
2020/02/13 12:02:39 [INFO] Go runtime version: go1.12.9

...

Terraform Configuration Files

...

Debug Output

2020-02-13T11:59:27.839+0200 [DEBUG] plugin.terraform-provider-aws_v2.48.0_x4.exe: 2020/02/13 11:59:27 [DEBUG] [aws-sdk-go] DEBUG: Retrying Request sts/GetCallerIdentity, attempt 8
2020-02-13T11:59:27.845+0200 [DEBUG] plugin.terraform-provider-aws_v2.48.0_x4.exe: 2020/02/13 11:59:27 [DEBUG] [aws-sdk-go] DEBUG: Request sts/GetCallerIdentity Details:
2020-02-13T11:59:27.845+0200 [DEBUG] plugin.terraform-provider-aws_v2.48.0_x4.exe: ---[ REQUEST POST-SIGN ]-----------------------------
2020-02-13T11:59:27.845+0200 [DEBUG] plugin.terraform-provider-aws_v2.48.0_x4.exe: POST / HTTP/1.1
2020-02-13T11:59:27.845+0200 [DEBUG] plugin.terraform-provider-aws_v2.48.0_x4.exe: Host: sts.amazonaws.com
2020-02-13T11:59:27.845+0200 [DEBUG] plugin.terraform-provider-aws_v2.48.0_x4.exe: User-Agent: aws-sdk-go/1.28.9 (go1.13.7; windows; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.12.7 (+https://www.terraform.io)
2020-02-13T11:59:27.845+0200 [DEBUG] plugin.terraform-provider-aws_v2.48.0_x4.exe: Content-Length: 43
2020-02-13T11:59:27.845+0200 [DEBUG] plugin.terraform-provider-aws_v2.48.0_x4.exe: Authorization: AWS4-HMAC-SHA256 Credential=redated/20200213/us-east-1/sts/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-date;x-amz-security-token, Signature=redacted
2020-02-13T11:59:27.845+0200 [DEBUG] plugin.terraform-provider-aws_v2.48.0_x4.exe: Content-Type: application/x-www-form-urlencoded; charset=utf-8
2020-02-13T11:59:27.845+0200 [DEBUG] plugin.terraform-provider-aws_v2.48.0_x4.exe: X-Amz-Date: 20200213T095927Z
2020-02-13T11:59:27.845+0200 [DEBUG] plugin.terraform-provider-aws_v2.48.0_x4.exe: X-Amz-Security-Token: redacted
2020-02-13T11:59:27.845+0200 [DEBUG] plugin.terraform-provider-aws_v2.48.0_x4.exe: Accept-Encoding: gzip
2020-02-13T11:59:27.845+0200 [DEBUG] plugin.terraform-provider-aws_v2.48.0_x4.exe:
2020-02-13T11:59:27.845+0200 [DEBUG] plugin.terraform-provider-aws_v2.48.0_x4.exe: Action=GetCallerIdentity&Version=2011-06-15
2020-02-13T11:59:27.845+0200 [DEBUG] plugin.terraform-provider-aws_v2.48.0_x4.exe: -----------------------------------------------------
2020-02-13T11:59:29.595+0200 [DEBUG] plugin.terraform-provider-aws_v2.48.0_x4.exe: 2020/02/13 11:59:29 [DEBUG] [aws-sdk-go] DEBUG: Response sts/GetCallerIdentity Details:
2020-02-13T11:59:29.595+0200 [DEBUG] plugin.terraform-provider-aws_v2.48.0_x4.exe: ---[ RESPONSE ]--------------------------------------
2020-02-13T11:59:29.595+0200 [DEBUG] plugin.terraform-provider-aws_v2.48.0_x4.exe: HTTP/1.1 403 Forbidden
2020-02-13T11:59:29.595+0200 [DEBUG] plugin.terraform-provider-aws_v2.48.0_x4.exe: Connection: close
2020-02-13T11:59:29.595+0200 [DEBUG] plugin.terraform-provider-aws_v2.48.0_x4.exe: Content-Length: 297
2020-02-13T11:59:29.595+0200 [DEBUG] plugin.terraform-provider-aws_v2.48.0_x4.exe: Content-Type: text/xml
2020-02-13T11:59:29.595+0200 [DEBUG] plugin.terraform-provider-aws_v2.48.0_x4.exe: Date: Thu, 13 Feb 2020 09:59:28 GMT
2020-02-13T11:59:29.595+0200 [DEBUG] plugin.terraform-provider-aws_v2.48.0_x4.exe: X-Amzn-Requestid: a108d2a8-f378-41cf-858a-373fbcd82b30
2020-02-13T11:59:29.595+0200 [DEBUG] plugin.terraform-provider-aws_v2.48.0_x4.exe:
2020-02-13T11:59:29.595+0200 [DEBUG] plugin.terraform-provider-aws_v2.48.0_x4.exe:
2020-02-13T11:59:29.595+0200 [DEBUG] plugin.terraform-provider-aws_v2.48.0_x4.exe: -----------------------------------------------------
2020-02-13T11:59:29.595+0200 [DEBUG] plugin.terraform-provider-aws_v2.48.0_x4.exe: 2020/02/13 11:59:29 [DEBUG] [aws-sdk-go]
2020-02-13T11:59:29.595+0200 [DEBUG] plugin.terraform-provider-aws_v2.48.0_x4.exe:
2020-02-13T11:59:29.595+0200 [DEBUG] plugin.terraform-provider-aws_v2.48.0_x4.exe: Sender
2020-02-13T11:59:29.595+0200 [DEBUG] plugin.terraform-provider-aws_v2.48.0_x4.exe: ExpiredToken
2020-02-13T11:59:29.595+0200 [DEBUG] plugin.terraform-provider-aws_v2.48.0_x4.exe: The security token included in the request is expired
2020-02-13T11:59:29.595+0200 [DEBUG] plugin.terraform-provider-aws_v2.48.0_x4.exe:
2020-02-13T11:59:29.595+0200 [DEBUG] plugin.terraform-provider-aws_v2.48.0_x4.exe: a108d2a8-f378-41cf-858a-373fbcd82b30
2020-02-13T11:59:29.595+0200 [DEBUG] plugin.terraform-provider-aws_v2.48.0_x4.exe:
2020-02-13T11:59:29.595+0200 [DEBUG] plugin.terraform-provider-aws_v2.48.0_x4.exe: 2020/02/13 11:59:29 [DEBUG] [aws-sdk-go] DEBUG: Validate Response sts/GetCallerIdentity failed, attempt 8/25, error ExpiredToken: The security token included in the request is expired
2020-02-13T11:59:29.595+0200 [DEBUG] plugin.terraform-provider-aws_v2.48.0_x4.exe: status code: 403, request id: a108d2a8-f378-41cf-858a-373fbcd82b30

Crash Output

Expected Behavior

Actual Behavior

Steps to Reproduce

terraform apply / plan

Additional Context

References

[jschoombee]

Expected Behavior - not to retry on a 403 or certain 4xx (bad request) errors specifically
Actual Behavior - it is retrying

[ewbankkit]

Related:

• Terraform appears to be hanging while performing 25 attempts due to 403 error (MFA) #9601
• terraform commands fail to detect expired AWS security tokens #4502
• Expired STS token results in terraform to hang #1351

[YakDriver]

This is closed by events. See hashicorp/aws-sdk-go-base#362 for more details.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.