Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

r/aws_sqs_queue: support high-throughput FIFO queues #19639

Merged
merged 18 commits into from
Jun 7, 2021
Merged

r/aws_sqs_queue: support high-throughput FIFO queues #19639

merged 18 commits into from
Jun 7, 2021

Conversation

roberth-k
Copy link
Contributor

@roberth-k roberth-k commented Jun 3, 2021 •
edited by ewbankkit

Community Note

  • Please vote on this pull request by adding a 👍 reaction to the original pull request comment to help the community and maintainers prioritize this request
  • Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for pull request followers and do not help prioritize the request

Closes #19630
Closes #19498
Closes #11848
Closes #14166
Closes #13759
Closes #13980

Adds the deduplication_scope and fifo_throughput_limit, which can be used to enable the high-throughput mode of FIFO queues.

Output from acceptance testing:

$ make testacc TESTARGS='-run=TestAccAWSSQSQueue'
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./aws -v -count 1 -parallel 20 -run=TestAccAWSSQSQueue -timeout 180m
--- PASS: TestAccAWSSQSQueue_ExpectContentBasedDeduplicationError (5.79s)
--- PASS: TestAccAWSSQSQueue_FIFOExpectNameError (5.86s)
--- PASS: TestAccAWSSQSQueuePolicy_disappears_queue (22.37s)
--- PASS: TestAccAWSSQSQueuePolicy_disappears (27.34s)
--- PASS: TestAccAWSSQSQueue_FIFO (27.76s)
--- PASS: TestAccAWSSQSQueue_Name_Generated_FIFOQueue (30.16s)
--- PASS: TestAccAWSSQSQueue_NamePrefix_FIFOQueue (30.22s)
--- PASS: TestAccAWSSQSQueue_FIFOWithContentBasedDeduplication (30.37s)
--- PASS: TestAccAWSSQSQueue_Encryption (30.51s)
--- PASS: TestAccAWSSQSQueue_Name_Generated (30.55s)
--- PASS: TestAccAWSSQSQueue_NamePrefix (30.62s)
--- PASS: TestAccAWSSQSQueue_redrivePolicy (30.78s)
--- PASS: TestAccAWSSQSQueue_queueDeletedRecently (37.34s)
--- PASS: TestAccAWSSQSQueuePolicy_basic (38.52s)
--- PASS: TestAccAWSSQSQueue_FIFOWithHighThroughputMode (42.96s)
--- PASS: TestAccAWSSQSQueue_basic (53.77s)
--- PASS: TestAccAWSSQSQueue_tags (54.01s)
--- PASS: TestAccAWSSQSQueue_policy (75.21s)
--- PASS: TestAccAWSSQSQueue_Policybasic (75.31s)
PASS
ok      github.com/terraform-providers/terraform-provider-aws/aws       76.954s

@roberth-k roberth-k requested a review from a team as a code owner June 3, 2021 08:55
@ghost ghost added size/M Managed by automation to categorize the size of a PR. labels Jun 3, 2021
@github-actions github-actions bot added documentation Introduces or discusses updates to documentation. service/sqs Issues and PRs that pertain to the sqs service. tests PRs: expanded test coverage. Issues: expanded coverage, enhancements to test infrastructure. needs-triage Waiting for first response or review from a maintainer. labels Jun 3, 2021
@roberth-k roberth-k force-pushed the f-aws_sqs_queue-high_throughput branch from 476ade3 to 7eb75ef Compare June 3, 2021 08:57
@roberth-k roberth-k mentioned this pull request Jun 3, 2021
Closed
@ewbankkit ewbankkit removed the needs-triage Waiting for first response or review from a maintainer. label Jun 4, 2021
@ewbankkit ewbankkit self-assigned this Jun 4, 2021
ewbankkit and others added 15 commits June 4, 2021 10:30
@ewbankkit
r/aws_sqs_queue(_policy): Simplify acceptance tests for GovCloud.
d8623c8
@ewbankkit
Add attrmap package.
289a492
@roberth-k @ewbankkit
r/aws_sqs_queue: add deduplication_scope, fifo_throughput_limit attri…
213f692 
…butes
@ewbankkit
Add deduplication_scope and fifo_throughput_limit values to internal …
f8dc547 
…consts.
@ewbankkit
r/aws_sqs_queue: Use paginated ListQueues in test sweeper.
b788701
@ewbankkit
r/aws_sqs_queue: Add internal finder package.
1cfff5e
@ewbankkit
r/aws_sqs_queue: Use internal attrmap package.
9fa6fe2
@ewbankkit
r/aws_sqs_queue: Add argument validation.
cc2a072
@ewbankkit
Add 'attrmap.ApiAttributesToResourceData'.
2eba4dd
@ewbankkit
AttributeMap now includes attribute type.
f839b96
@ewbankkit
r/aws_sqs_queue: All current tests passing in Commercial.
3e09534
@ewbankkit
Simplify 'QueueDeleted' waiter - Now get correct failures in GovCloud.
46f7f48
@ewbankkit
r/aws_sqs_queue_policy: Add 'waiter.QueueAttributesPropagated'.
72a54fd
@ewbankkit
r/aws_sqs_queue: Use 'waiter.QueueAttributesPropagated'.
107cd79
@ewbankkit
r/aws_sqs_queue: Allow 'visibility_timeout_seconds = 0' on creation (h…
e1ef845 
…ashicorp#14166).
@ewbankkit ewbankkit force-pushed the f-aws_sqs_queue-high_throughput branch from 7eb75ef to 31b7d39 Compare June 7, 2021 16:02
@ghost ghost added size/XXL Managed by automation to categorize the size of a PR. and removed size/M Managed by automation to categorize the size of a PR. labels Jun 7, 2021
ewbankkit
ewbankkit approved these changes Jun 7, 2021
View reviewed changes
Copy link
Contributor

@ewbankkit ewbankkit left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 🚀.

Commercial
% make testacc TEST=./aws TESTARGS='-run=TestAccAWSSQSQueue_\|TestAccAWSSQSQueuePolicy_'
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./aws -v -count 1 -parallel 20 -run=TestAccAWSSQSQueue_\|TestAccAWSSQSQueuePolicy_ -timeout 180m
=== RUN   TestAccAWSSQSQueuePolicy_basic
=== PAUSE TestAccAWSSQSQueuePolicy_basic
=== RUN   TestAccAWSSQSQueuePolicy_disappears
=== PAUSE TestAccAWSSQSQueuePolicy_disappears
=== RUN   TestAccAWSSQSQueuePolicy_disappears_queue
=== PAUSE TestAccAWSSQSQueuePolicy_disappears_queue
=== RUN   TestAccAWSSQSQueuePolicy_Update
=== PAUSE TestAccAWSSQSQueuePolicy_Update
=== RUN   TestAccAWSSQSQueue_basic
=== PAUSE TestAccAWSSQSQueue_basic
=== RUN   TestAccAWSSQSQueue_disappears
=== PAUSE TestAccAWSSQSQueue_disappears
=== RUN   TestAccAWSSQSQueue_Name_Generated
=== PAUSE TestAccAWSSQSQueue_Name_Generated
=== RUN   TestAccAWSSQSQueue_Name_Generated_FIFOQueue
=== PAUSE TestAccAWSSQSQueue_Name_Generated_FIFOQueue
=== RUN   TestAccAWSSQSQueue_NamePrefix
=== PAUSE TestAccAWSSQSQueue_NamePrefix
=== RUN   TestAccAWSSQSQueue_NamePrefix_FIFOQueue
=== PAUSE TestAccAWSSQSQueue_NamePrefix_FIFOQueue
=== RUN   TestAccAWSSQSQueue_Tags
=== PAUSE TestAccAWSSQSQueue_Tags
=== RUN   TestAccAWSSQSQueue_Update
=== PAUSE TestAccAWSSQSQueue_Update
=== RUN   TestAccAWSSQSQueue_Policy
=== PAUSE TestAccAWSSQSQueue_Policy
=== RUN   TestAccAWSSQSQueue_RecentlyDeleted
=== PAUSE TestAccAWSSQSQueue_RecentlyDeleted
=== RUN   TestAccAWSSQSQueue_RedrivePolicy
=== PAUSE TestAccAWSSQSQueue_RedrivePolicy
=== RUN   TestAccAWSSQSQueue_FIFOQueue
=== PAUSE TestAccAWSSQSQueue_FIFOQueue
=== RUN   TestAccAWSSQSQueue_FIFOQueue_ExpectNameError
=== PAUSE TestAccAWSSQSQueue_FIFOQueue_ExpectNameError
=== RUN   TestAccAWSSQSQueue_FIFOQueue_ContentBasedDeduplication
=== PAUSE TestAccAWSSQSQueue_FIFOQueue_ContentBasedDeduplication
=== RUN   TestAccAWSSQSQueue_FIFOQueue_HighThroughputMode
=== PAUSE TestAccAWSSQSQueue_FIFOQueue_HighThroughputMode
=== RUN   TestAccAWSSQSQueue_StandardQueue_ExpectContentBasedDeduplicationError
=== PAUSE TestAccAWSSQSQueue_StandardQueue_ExpectContentBasedDeduplicationError
=== RUN   TestAccAWSSQSQueue_Encryption
=== PAUSE TestAccAWSSQSQueue_Encryption
=== RUN   TestAccAWSSQSQueue_ZeroVisibilityTimeoutSeconds
=== PAUSE TestAccAWSSQSQueue_ZeroVisibilityTimeoutSeconds
=== CONT  TestAccAWSSQSQueuePolicy_basic
=== CONT  TestAccAWSSQSQueue_Policy
=== CONT  TestAccAWSSQSQueue_Name_Generated
=== CONT  TestAccAWSSQSQueuePolicy_disappears_queue
=== CONT  TestAccAWSSQSQueuePolicy_Update
=== CONT  TestAccAWSSQSQueue_ZeroVisibilityTimeoutSeconds
=== CONT  TestAccAWSSQSQueue_Encryption
=== CONT  TestAccAWSSQSQueue_disappears
=== CONT  TestAccAWSSQSQueuePolicy_disappears
=== CONT  TestAccAWSSQSQueue_basic
=== CONT  TestAccAWSSQSQueue_StandardQueue_ExpectContentBasedDeduplicationError
=== CONT  TestAccAWSSQSQueue_FIFOQueue_HighThroughputMode
=== CONT  TestAccAWSSQSQueue_FIFOQueue_ContentBasedDeduplication
=== CONT  TestAccAWSSQSQueue_FIFOQueue_ExpectNameError
=== CONT  TestAccAWSSQSQueue_FIFOQueue
=== CONT  TestAccAWSSQSQueue_RedrivePolicy
=== CONT  TestAccAWSSQSQueue_RecentlyDeleted
=== CONT  TestAccAWSSQSQueue_Update
=== CONT  TestAccAWSSQSQueue_Tags
=== CONT  TestAccAWSSQSQueue_NamePrefix_FIFOQueue
=== CONT  TestAccAWSSQSQueue_NamePrefix
--- PASS: TestAccAWSSQSQueue_StandardQueue_ExpectContentBasedDeduplicationError (12.09s)
=== CONT  TestAccAWSSQSQueue_Name_Generated_FIFOQueue
--- PASS: TestAccAWSSQSQueue_FIFOQueue_ExpectNameError (12.43s)
--- PASS: TestAccAWSSQSQueue_disappears (33.29s)
--- PASS: TestAccAWSSQSQueuePolicy_disappears_queue (35.86s)
--- PASS: TestAccAWSSQSQueue_Name_Generated (43.39s)
--- PASS: TestAccAWSSQSQueue_ZeroVisibilityTimeoutSeconds (44.03s)
--- PASS: TestAccAWSSQSQueue_FIFOQueue_ContentBasedDeduplication (44.20s)
--- PASS: TestAccAWSSQSQueue_FIFOQueue (44.79s)
--- PASS: TestAccAWSSQSQueue_NamePrefix_FIFOQueue (46.14s)
--- PASS: TestAccAWSSQSQueue_basic (46.20s)
--- PASS: TestAccAWSSQSQueue_NamePrefix (34.95s)
--- PASS: TestAccAWSSQSQueue_Name_Generated_FIFOQueue (34.71s)
--- PASS: TestAccAWSSQSQueue_RedrivePolicy (49.35s)
--- PASS: TestAccAWSSQSQueue_Encryption (56.53s)
--- PASS: TestAccAWSSQSQueue_FIFOQueue_HighThroughputMode (57.93s)
--- PASS: TestAccAWSSQSQueue_Update (58.33s)
--- PASS: TestAccAWSSQSQueue_Tags (74.18s)
--- PASS: TestAccAWSSQSQueue_Policy (85.11s)
--- PASS: TestAccAWSSQSQueuePolicy_disappears (90.01s)
--- PASS: TestAccAWSSQSQueue_RecentlyDeleted (95.42s)
--- PASS: TestAccAWSSQSQueuePolicy_basic (116.59s)
--- PASS: TestAccAWSSQSQueuePolicy_Update (120.83s)
PASS
ok  	github.com/terraform-providers/terraform-provider-aws/aws	123.887s
GovCloud
% make testacc TEST=./aws TESTARGS='-run=TestAccAWSSQSQueue_\|TestAccAWSSQSQueuePolicy_'
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./aws -v -count 1 -parallel 20 -run=TestAccAWSSQSQueue_\|TestAccAWSSQSQueuePolicy_ -timeout 180m
=== RUN   TestAccAWSSQSQueuePolicy_basic
=== PAUSE TestAccAWSSQSQueuePolicy_basic
=== RUN   TestAccAWSSQSQueuePolicy_disappears
=== PAUSE TestAccAWSSQSQueuePolicy_disappears
=== RUN   TestAccAWSSQSQueuePolicy_disappears_queue
=== PAUSE TestAccAWSSQSQueuePolicy_disappears_queue
=== RUN   TestAccAWSSQSQueuePolicy_Update
=== PAUSE TestAccAWSSQSQueuePolicy_Update
=== RUN   TestAccAWSSQSQueue_basic
=== PAUSE TestAccAWSSQSQueue_basic
=== RUN   TestAccAWSSQSQueue_disappears
=== PAUSE TestAccAWSSQSQueue_disappears
=== RUN   TestAccAWSSQSQueue_Name_Generated
=== PAUSE TestAccAWSSQSQueue_Name_Generated
=== RUN   TestAccAWSSQSQueue_Name_Generated_FIFOQueue
=== PAUSE TestAccAWSSQSQueue_Name_Generated_FIFOQueue
=== RUN   TestAccAWSSQSQueue_NamePrefix
=== PAUSE TestAccAWSSQSQueue_NamePrefix
=== RUN   TestAccAWSSQSQueue_NamePrefix_FIFOQueue
=== PAUSE TestAccAWSSQSQueue_NamePrefix_FIFOQueue
=== RUN   TestAccAWSSQSQueue_Tags
=== PAUSE TestAccAWSSQSQueue_Tags
=== RUN   TestAccAWSSQSQueue_Update
=== PAUSE TestAccAWSSQSQueue_Update
=== RUN   TestAccAWSSQSQueue_Policy
=== PAUSE TestAccAWSSQSQueue_Policy
=== RUN   TestAccAWSSQSQueue_RecentlyDeleted
=== PAUSE TestAccAWSSQSQueue_RecentlyDeleted
=== RUN   TestAccAWSSQSQueue_RedrivePolicy
=== PAUSE TestAccAWSSQSQueue_RedrivePolicy
=== RUN   TestAccAWSSQSQueue_FIFOQueue
=== PAUSE TestAccAWSSQSQueue_FIFOQueue
=== RUN   TestAccAWSSQSQueue_FIFOQueue_ExpectNameError
=== PAUSE TestAccAWSSQSQueue_FIFOQueue_ExpectNameError
=== RUN   TestAccAWSSQSQueue_FIFOQueue_ContentBasedDeduplication
=== PAUSE TestAccAWSSQSQueue_FIFOQueue_ContentBasedDeduplication
=== RUN   TestAccAWSSQSQueue_FIFOQueue_HighThroughputMode
=== PAUSE TestAccAWSSQSQueue_FIFOQueue_HighThroughputMode
=== RUN   TestAccAWSSQSQueue_StandardQueue_ExpectContentBasedDeduplicationError
=== PAUSE TestAccAWSSQSQueue_StandardQueue_ExpectContentBasedDeduplicationError
=== RUN   TestAccAWSSQSQueue_Encryption
=== PAUSE TestAccAWSSQSQueue_Encryption
=== RUN   TestAccAWSSQSQueue_ZeroVisibilityTimeoutSeconds
=== PAUSE TestAccAWSSQSQueue_ZeroVisibilityTimeoutSeconds
=== CONT  TestAccAWSSQSQueuePolicy_basic
=== CONT  TestAccAWSSQSQueue_Policy
=== CONT  TestAccAWSSQSQueue_FIFOQueue_ContentBasedDeduplication
=== CONT  TestAccAWSSQSQueue_FIFOQueue_ExpectNameError
=== CONT  TestAccAWSSQSQueue_StandardQueue_ExpectContentBasedDeduplicationError
=== CONT  TestAccAWSSQSQueue_ZeroVisibilityTimeoutSeconds
=== CONT  TestAccAWSSQSQueue_Name_Generated_FIFOQueue
=== CONT  TestAccAWSSQSQueue_Name_Generated
=== CONT  TestAccAWSSQSQueue_Update
=== CONT  TestAccAWSSQSQueue_Tags
=== CONT  TestAccAWSSQSQueue_NamePrefix_FIFOQueue
=== CONT  TestAccAWSSQSQueue_NamePrefix
=== CONT  TestAccAWSSQSQueue_FIFOQueue_HighThroughputMode
=== CONT  TestAccAWSSQSQueue_Encryption
=== CONT  TestAccAWSSQSQueue_FIFOQueue
=== CONT  TestAccAWSSQSQueue_RedrivePolicy
=== CONT  TestAccAWSSQSQueue_RecentlyDeleted
=== CONT  TestAccAWSSQSQueuePolicy_disappears
=== CONT  TestAccAWSSQSQueuePolicy_disappears_queue
=== CONT  TestAccAWSSQSQueuePolicy_Update
--- PASS: TestAccAWSSQSQueue_StandardQueue_ExpectContentBasedDeduplicationError (15.13s)
=== CONT  TestAccAWSSQSQueue_disappears
--- PASS: TestAccAWSSQSQueue_FIFOQueue_ExpectNameError (15.45s)
=== CONT  TestAccAWSSQSQueue_basic
--- PASS: TestAccAWSSQSQueue_disappears (24.49s)
--- PASS: TestAccAWSSQSQueuePolicy_disappears_queue (39.98s)
--- PASS: TestAccAWSSQSQueue_Name_Generated (43.40s)
--- PASS: TestAccAWSSQSQueue_NamePrefix_FIFOQueue (44.05s)
--- PASS: TestAccAWSSQSQueue_Name_Generated_FIFOQueue (44.35s)
--- PASS: TestAccAWSSQSQueue_FIFOQueue_ContentBasedDeduplication (44.42s)
=== CONT  TestAccAWSSQSQueue_FIFOQueue_HighThroughputMode
    resource_aws_sqs_queue_test.go:540: Step 3/3 error: Check failed: Check 2/4 error: aws_sqs_queue.test: Attribute 'deduplication_scope' expected "messageGroup", got "queue"
=== CONT  TestAccAWSSQSQueue_RedrivePolicy
    testing_new.go:63: Error running post-test destroy, there may be dangling resources: SQS Queue https://sqs.us-gov-west-1.amazonaws.com/123456789012/tf-acc-test-3677550227255856731-2 still exists
--- FAIL: TestAccAWSSQSQueue_RedrivePolicy (47.73s)
--- PASS: TestAccAWSSQSQueue_basic (32.56s)
--- PASS: TestAccAWSSQSQueue_NamePrefix (50.18s)
--- PASS: TestAccAWSSQSQueue_ZeroVisibilityTimeoutSeconds (50.98s)
--- PASS: TestAccAWSSQSQueue_FIFOQueue (51.33s)
=== CONT  TestAccAWSSQSQueue_FIFOQueue_HighThroughputMode
    testing_new.go:63: Error running post-test destroy, there may be dangling resources: SQS Queue https://sqs.us-gov-west-1.amazonaws.com/123456789012/tf-acc-test-271893099192892502.fifo still exists
--- FAIL: TestAccAWSSQSQueue_FIFOQueue_HighThroughputMode (51.75s)
--- PASS: TestAccAWSSQSQueuePolicy_basic (54.87s)
=== CONT  TestAccAWSSQSQueue_Encryption
    testing_new.go:63: Error running post-test destroy, there may be dangling resources: SQS Queue https://sqs.us-gov-west-1.amazonaws.com/123456789012/tf-acc-test-7615187763346557890 still exists
--- FAIL: TestAccAWSSQSQueue_Encryption (56.93s)
--- PASS: TestAccAWSSQSQueuePolicy_Update (63.30s)
--- PASS: TestAccAWSSQSQueue_Update (69.94s)
--- PASS: TestAccAWSSQSQueue_Tags (70.99s)
--- PASS: TestAccAWSSQSQueue_Policy (84.03s)
--- PASS: TestAccAWSSQSQueuePolicy_disappears (91.57s)
--- PASS: TestAccAWSSQSQueue_RecentlyDeleted (101.80s)
FAIL
FAIL	github.com/terraform-providers/terraform-provider-aws/aws	105.600s
FAIL
make: *** [testacc] Error 1

The failures are caused by eventual consistency issues in GovCloud and are not new.

This was referenced Jun 7, 2021
Closed
Closed
Closed
Closed
@ewbankkit
Copy link
Contributor

@roberth-k Thanks for the contribution 🎉 👏.
Everything looks great.
I went ahead and incorporated some other SQS Queue fixes into this PR.

@ewbankkit ewbankkit merged commit 4711040 into hashicorp:main Jun 7, 2021
@github-actions github-actions bot added this to the v3.45.0 milestone Jun 7, 2021
@roberth-k roberth-k deleted the f-aws_sqs_queue-high_throughput branch June 10, 2021 09:51
@github-actions
Copy link

This functionality has been released in v3.45.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

@github-actions
Copy link

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Jul 12, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@ewbankkit ewbankkit ewbankkit approved these changes

Assignees

@ewbankkit ewbankkit

Labels
documentation Introduces or discusses updates to documentation. service/sqs Issues and PRs that pertain to the sqs service. size/XXL Managed by automation to categorize the size of a PR. tests PRs: expanded test coverage. Issues: expanded coverage, enhancements to test infrastructure.
Projects
None yet
Milestone
v3.45.0
3 participants
@roberth-k @ewbankkit @whatsfordinner