WIP: Automatically retry when encounter connection reset by peer error from aws api

[jiashuChen]

Community Note

• Please vote on this pull request by adding a 👍 reaction to the original pull request comment to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for pull request followers and do not help prioritize the request

Relates #10715

Draft PR, I'm not sure if this is the most correct way to solve this issue, raising as a draft, feedback will be appreciated. The goal of this pull request is to resolve to have a global wrapper function to deal with connection reset by peer error from aws api, which seems to impact multiple aws region and services.

│ Error: error deleting IAM Role (IAM-ROLE-NAME): RequestError: send request failed
│ caused by: Post "https://iam.amazonaws.com/": read tcp IP:PORT->DIFFERENT_IP:PORT: read: connection reset by peer

Furthermore, this helps resolve the issue with resource.Retry being marked deprecated.

https://github.com/hashicorp/terraform-plugin-sdk/blob/main/helper/resource/wait.go#L66-L69

Lastly, this pull request now seems to have more than 100 labels automatically applied to it, which has caused verification to fail. I'm not sure how to resolve it. Some help will be appreciated.

NOTE: although there are many files changes, the core of the change is here in this file:

terraform-provider-aws/aws/internal/tfresource/retry.go

Lines 119 to 129 in 03ea02c

	func RetryOnConnectionResetByPeer(timeout time.Duration, f resource.RetryFunc) error {
	return resource.RetryContext(context.Background(), timeout, func() *resource.RetryError {
	err := f()
	if err != nil && !err.Retryable && tfawserr.ErrMessageContains(err.Err, request.ErrCodeRequestError, "read: connection reset by peer") {
	return resource.RetryableError(err.Err)
	}
	return err
	})
	}

[github-actions]

Thank you for your contribution! 🚀

Please note that typically Go dependency changes are handled in this repository by dependabot or the maintainers. This is to prevent pull request merge conflicts and further delay reviews of contributions. Remove any changes to the go.mod or go.sum files and commit them into this pull request.

Additional details:

• Check open pull requests with the dependencies label to view other dependency updates.
• If this pull request includes an update the AWS Go SDK (or any other dependency) version, only updates submitted via dependabot will be merged. This pull request will need to remove these changes and will need to be rebased after the existing dependency update via dependabot has been merged for this pull request to be reviewed.
• If this pull request is for supporting a new AWS service:
  • Ensure the new AWS service changes are following the Contributing Guide section on new services, in particular that the dependency addition and initial provider support are in a separate pull request from other changes (e.g. new resources). Contributions not following this item will not be reviewed until the changes are split.
  • If this pull request is already a separate pull request from the above item, you can ignore this message.

[jiashuChen]

Main Change

[jiashuChen]

Unit Test

[AdamTylerLynch]

Question for the author: the AWS GoSDK has retry configured by default for all client operations. The retry mechanism will retry 3 times by default and use exponential back off with jitter.

I’m curious as to how this PR will be any different? I would suggest modifying the default policy for the SDK rather than implement additional retry logic, specifically because that logic does not include exponential backoff with jitter.

https://aws.github.io/aws-sdk-go-v2/docs/configuring-sdk/retries-timeouts/

Additionally, TCP reset could be happening for a number of reasons. I use Terraform all the time, running the Acceptance Tests, 20 threads at a time, and I’ve never gotten a TCP reset error.

Have you checked your network route to AWS? A proxy, machine-in-the-middle, or AWS API throttling are all potential sources of TCP reset. I would recommend checking your CloudTrail logs for your account to see if you are being throttled.

[jiashuChen]

Question for the author: the AWS GoSDK has retry configured by default for all client operations. The retry mechanism will retry 3 times by default and use exponential back off with jitter.

I’m curious as to how this PR will be any different? I would suggest modifying the default policy for the SDK rather than implement additional retry logic, specifically because that logic does not include exponential backoff with jitter.

https://aws.github.io/aws-sdk-go-v2/docs/configuring-sdk/retries-timeouts/

Additionally, TCP reset could be happening for a number of reasons. I use Terraform all the time, running the Acceptance Tests, 20 threads at a time, and I’ve never gotten a TCP reset error.

Have you checked your network route to AWS? A proxy, machine-in-the-middle, or AWS API throttling are all potential sources of TCP reset. I would recommend checking your CloudTrail logs for your account to see if you are being throttled.

Thanks. I'll close the current pull requests.

[github-actions]

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.