Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Retry S3 OperationAborted errors #12949

Merged
merged 2 commits into from Nov 19, 2021

Conversation

nijave
Copy link
Contributor

@nijave nijave commented Apr 22, 2020

These crop up when Terraform is performing an operation against a bucket and tries to perform a conflicting operation against the same bucket in parallel. This will allow Terraform to automatically retry instead of having to rerun the Terraform command

Fixes #7628

image
(don't have the error from Terraform handy but here's what the details look like)

Community Note

  • Please vote on this pull request by adding a 👍 reaction to the original pull request comment to help the community and maintainers prioritize this request
  • Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for pull request followers and do not help prioritize the request

Release note for CHANGELOG:

Retry conflicting S3 bucket operations

@nijave nijave requested a review from a team April 22, 2020 15:53
@ghost ghost added needs-triage Waiting for first response or review from a maintainer. size/XS Managed by automation to categorize the size of a PR. provider Pertains to the provider itself, rather than any interaction with AWS. labels Apr 22, 2020
Base automatically changed from master to main January 23, 2021 00:57
@breathingdust breathingdust requested a review from a team as a code owner January 23, 2021 00:57
@breathingdust breathingdust added enhancement Requests to existing resources that expand the functionality or scope. and removed needs-triage Waiting for first response or review from a maintainer. labels Sep 28, 2021
@nijave nijave force-pushed the nv-retry-s3-ops-in-progress-errors branch from dbcc7dd to 480017d Compare October 15, 2021 02:47
@github-actions github-actions bot added size/XL Managed by automation to categorize the size of a PR. and removed size/XS Managed by automation to categorize the size of a PR. labels Oct 15, 2021
@nijave nijave force-pushed the nv-retry-s3-ops-in-progress-errors branch from 480017d to ed93aad Compare October 15, 2021 02:48
@github-actions github-actions bot added size/XS Managed by automation to categorize the size of a PR. and removed provider Pertains to the provider itself, rather than any interaction with AWS. size/XL Managed by automation to categorize the size of a PR. labels Oct 15, 2021
@nijave
Copy link
Contributor Author

nijave commented Oct 15, 2021

I fixed the merge conflict--the changes should still apply but I no longer have a handy way to test this

@zhelding
Copy link
Contributor

Pull request #21306 has significantly refactored the AWS Provider codebase. As a result, most PRs opened prior to the refactor now have merge conflicts that must be resolved before proceeding.

Specifically, PR #21306 relocated the code for all AWS resources and data sources from a single aws directory to a large number of separate directories in internal/service, each corresponding to a particular AWS service. This separation of code has also allowed for us to simplify the names of underlying functions -- while still avoiding namespace collisions.

We recognize that many pull requests have been open for some time without yet being addressed by our maintainers. Therefore, we want to make it clear that resolving these conflicts in no way affects the prioritization of a particular pull request. Once a pull request has been prioritized for review, the necessary changes will be made by a maintainer -- either directly or in collaboration with the pull request author.

For a more complete description of this refactor, including examples of how old filepaths and function names correspond to their new counterparts: please refer to issue #20000.

For a quick guide on how to amend your pull request to resolve the merge conflicts resulting from this refactor and bring it in line with our new code patterns: please refer to our Service Package Refactor Pull Request Guide.

@breathingdust
Copy link
Member

Hi all 👋 Just letting you know that this is issue is featured on this quarters roadmap. If a PR exists to close the issue a maintainer will review and either make changes directly, or work with the original author to get the contribution merged. If you have written a PR to resolve the issue please ensure the "Allow edits from maintainers" box is checked. Thanks for your patience and we are looking forward to getting this merged soon!

@breathingdust breathingdust added this to the Roadmap milestone Nov 10, 2021
@YakDriver YakDriver self-assigned this Nov 19, 2021
@YakDriver
Copy link
Member

@nijave Thanks for your work on this! I do not see a good way of testing this. However, since it is in line with the way we handle this with many other services, I see no problem moving forward. We will have to rely on the community to make that this resolves all the issues in #7628 .

@YakDriver YakDriver added the service/s3 Issues and PRs that pertain to the s3 service. label Nov 19, 2021
@github-actions github-actions bot added client-connections Pertains to the AWS Client and service connections. and removed service/s3 Issues and PRs that pertain to the s3 service. labels Nov 19, 2021
nijave and others added 2 commits November 19, 2021 13:41
These crop up when Terraform is performing an operation against a bucket
and tries to perform a conflicting operation against the same bucket in
parallel. This will allow Terraform to automatically retry instead of
having to rerun the Terraform command
@YakDriver YakDriver force-pushed the nv-retry-s3-ops-in-progress-errors branch from 96d1584 to e8a8982 Compare November 19, 2021 18:41
Copy link
Member

@YakDriver YakDriver left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work! 🎉

Output from acceptance tests:
(Several of the S3 tests are flaky)

--- PASS: TestAccS3Bucket_Tags_ignoreTags (52.21s)
--- PASS: TestAccS3Bucket_Manage_lifecycleBasic (80.69s)
--- PASS: TestAccS3Bucket_Replication_basic (125.98s)
--- PASS: TestAccS3Bucket_Basic_basic (60.95s)
--- PASS: TestAccS3Bucket_Basic_emptyString (59.28s)
--- PASS: TestAccS3Bucket_Basic_forceDestroy (47.66s)
--- PASS: TestAccS3Bucket_Basic_forceDestroyWithEmptyPrefixes (50.35s)
--- PASS: TestAccS3Bucket_Basic_forceDestroyWithObjectLockEnabled (58.71s)
--- PASS: TestAccS3Bucket_Basic_generatedName (78.23s)
--- PASS: TestAccS3Bucket_Basic_keyEnabled (91.82s)
--- PASS: TestAccS3Bucket_Basic_namePrefix (77.49s)
--- PASS: TestAccS3Bucket_Basic_requestPayer (126.90s)
--- PASS: TestAccS3Bucket_Basic_shouldFailNotFound (43.16s)
--- PASS: TestAccS3Bucket_Manage_lifecycleExpireMarkerOnly (214.29s)
--- PASS: TestAccS3Bucket_Manage_lifecycleRuleAbortIncompleteMultipartUploadDaysNoExpiration (121.61s)
--- PASS: TestAccS3Bucket_Manage_lifecycleRuleExpirationEmptyBlock (95.45s)
--- PASS: TestAccS3Bucket_Manage_objectLock (98.43s)
--- PASS: TestAccS3Bucket_Manage_versioning (208.21s)
--- PASS: TestAccS3Bucket_Replication_expectVersioningValidationError (37.20s)
--- PASS: TestAccS3Bucket_Replication_ruleDestinationAccessControlTranslation (140.62s)
--- PASS: TestAccS3Bucket_Replication_ruleDestinationAddAccessControlTranslation (133.89s)
--- PASS: TestAccS3Bucket_Replication_schemaV2 (479.90s)
--- PASS: TestAccS3Bucket_Replication_schemaV2SameRegion (68.91s)
--- PASS: TestAccS3Bucket_Replication_twoDestination (74.42s)
--- PASS: TestAccS3Bucket_Replication_withoutPrefix (73.93s)
--- PASS: TestAccS3Bucket_Replication_withoutStorageClass (86.31s)
--- PASS: TestAccS3Bucket_Security_aclToGrant (140.56s)
--- PASS: TestAccS3Bucket_Security_corsDelete (73.60s)
--- PASS: TestAccS3Bucket_Security_corsEmptyOrigin (61.01s)
--- PASS: TestAccS3Bucket_Security_corsUpdate (156.49s)
--- PASS: TestAccS3Bucket_Security_disableDefaultEncryptionWhenDefaultEncryptionIsEnabled (156.64s)
--- PASS: TestAccS3Bucket_Security_enableDefaultEncryptionWhenAES256IsUsed (90.31s)
--- PASS: TestAccS3Bucket_Security_enableDefaultEncryptionWhenTypical (93.00s)
--- PASS: TestAccS3Bucket_Security_grantToACL (138.78s)
--- PASS: TestAccS3Bucket_Security_logging (74.49s)
--- PASS: TestAccS3Bucket_Security_policy (209.06s)
--- PASS: TestAccS3Bucket_Security_updateACL (139.60s)
--- PASS: TestAccS3Bucket_Security_updateGrant (207.80s)
--- PASS: TestAccS3Bucket_Tags_basic (78.36s)
--- PASS: TestAccS3Bucket_Tags_withNoSystemTags (190.64s)
--- PASS: TestAccS3Bucket_Tags_withSystemTags (247.73s)
--- PASS: TestAccS3Bucket_Web_redirect (207.96s)
--- PASS: TestAccS3Bucket_Web_routingRules (156.11s)
--- PASS: TestAccS3Bucket_Web_simple (208.19s)
--- PASS: TestAccS3BucketPolicy_basic (122.71s)
--- PASS: TestAccS3BucketPolicy_policyUpdate (213.74s)
--- PASS: TestAccS3BucketPublicAccessBlock_basic (115.38s)
--- PASS: TestAccS3BucketPublicAccessBlock_blockPublicACLs (214.57s)
--- PASS: TestAccS3BucketPublicAccessBlock_blockPublicPolicy (179.74s)
--- PASS: TestAccS3BucketPublicAccessBlock_disappears (101.07s)
--- PASS: TestAccS3BucketPublicAccessBlock_Disappears_bucket (66.10s)
--- PASS: TestAccS3BucketPublicAccessBlock_ignorePublicACLs (130.33s)
--- PASS: TestAccS3BucketPublicAccessBlock_restrictPublicBuckets (123.16s)
--- SKIP: TestAccS3Bucket_Basic_acceleration (0.00s)
--- SKIP: TestAccS3Bucket_Replication_multipleDestinationsEmptyFilter (30.46s)
--- SKIP: TestAccS3Bucket_Replication_multipleDestinationsNonEmptyFilter (32.95s)
--- SKIP: TestAccS3Bucket_Replication_RTC_valid (26.40s)

@YakDriver YakDriver merged commit 9e0b7e9 into hashicorp:main Nov 19, 2021
@github-actions
Copy link

github-actions bot commented Dec 1, 2021

This functionality has been released in v3.67.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

@github-actions
Copy link

github-actions bot commented Jun 7, 2022

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Jun 7, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
client-connections Pertains to the AWS Client and service connections. enhancement Requests to existing resources that expand the functionality or scope. service/s3 Issues and PRs that pertain to the s3 service. size/XS Managed by automation to categorize the size of a PR.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Applying aws_s3_bucket_policy and aws_s3_bucket_public_access_block at the same time may cause an error
4 participants