[Enhancement]: Check for duplicate keys in IAM policy JSON arguments

[jar-b]

Description

Adds a step to the existing verify.ValidIAMPolicyJSON validator which checks for duplicate keys. This can help prevent situations where key values are overwritten (such as duplicated Condition keys in IAM policy documents), leading to unintended permission definitions.

This particular check will only help when the value is supplied as a heredoc string. Values created from the jsonencode function will already have duplicated keys squashed prior to the provider receiving the argument. #33093 addressed a related issue with Condition keys in the aws_iam_policy_document data source, and this should be the preferred option for generating values for IAM policy JSON arguments, avoiding this class of error altogether.

Relations

Closes #33026
Depends on hashicorp/aws-sdk-go-base#650 (and subsequent release)
Relates hashicorp/terraform#28727

Output from Acceptance Testing

% make testacc PKG=iam TESTS=TestAccIAMPolicy_policyDuplicateKeys
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./internal/service/iam/... -v -count 1 -parallel 20 -run='TestAccIAMPolicy_policyDuplicateKeys'  -timeout 360m
=== RUN   TestAccIAMPolicy_policyDuplicateKeys
=== PAUSE TestAccIAMPolicy_policyDuplicateKeys
=== CONT  TestAccIAMPolicy_policyDuplicateKeys
--- PASS: TestAccIAMPolicy_policyDuplicateKeys (1.90s)
PASS
ok      github.com/hashicorp/terraform-provider-aws/internal/service/iam        5.149s

% make testacc PKG=iam TESTS=TestAccIAMPolicy_
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./internal/service/iam/... -v -count 1 -parallel 20 -run='TestAccIAMPolicy_'  -timeout 360m

--- PASS: TestAccIAMPolicy_policyDuplicateKeys (2.40s)
--- PASS: TestAccIAMPolicy_disappears (19.87s)
--- PASS: TestAccIAMPolicy_path (24.68s)
--- PASS: TestAccIAMPolicy_description (25.09s)
--- PASS: TestAccIAMPolicy_basic (25.14s)
--- PASS: TestAccIAMPolicy_namePrefix (25.34s)
--- PASS: TestAccIAMPolicy_policy (37.10s)
--- PASS: TestAccIAMPolicy_tags (47.87s)
--- PASS: TestAccIAMPolicy_diffs (117.81s)
PASS
ok      github.com/hashicorp/terraform-provider-aws/internal/service/iam        120.914s

% make testacc PKG=iam TESTS=TestAccIAMRole_
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./internal/service/iam/... -v -count 1 -parallel 20 -run='TestAccIAMRole_'  -timeout 360

--- PASS: TestAccIAMRole_badJSON (6.81s)
=== CONT  TestAccIAMRole_policiesForceDetach
--- PASS: TestAccIAMRole_InlinePolicy_empty (53.10s)
=== CONT  TestAccIAMRole_nameGenerated
--- PASS: TestAccIAMRole_basic (60.06s)
=== CONT  TestAccIAMRole_namePrefix
--- PASS: TestAccIAMRole_policiesForceDetach (59.79s)
=== CONT  TestAccIAMRole_description
--- PASS: TestAccIAMRole_ManagedPolicy_outOfBandAdditionIgnored (89.96s)
=== CONT  TestAccIAMRole_disappears
--- PASS: TestAccIAMRole_InlinePolicy_outOfBandAdditionRemovedEmpty (95.21s)
--- PASS: TestAccIAMRole_ManagedPolicy_outOfBandAdditionRemovedEmpty (96.61s)
--- PASS: TestAccIAMRole_InlinePolicy_outOfBandRemovalAddedBack (96.61s)
--- PASS: TestAccIAMRole_ManagedPolicy_outOfBandAdditionRemoved (96.89s)
--- PASS: TestAccIAMRole_InlinePolicy_outOfBandAdditionRemoved (97.86s)
--- PASS: TestAccIAMRole_testNameChange (98.68s)
--- PASS: TestAccIAMRole_ManagedPolicy_outOfBandRemovalAddedBack (98.89s)
--- PASS: TestAccIAMRole_tags (102.81s)
--- PASS: TestAccIAMRole_nameGenerated (54.53s)
--- PASS: TestAccIAMRole_namePrefix (50.86s)
--- PASS: TestAccIAMRole_maxSessionDuration (110.94s)
--- PASS: TestAccIAMRole_InlinePolicy_outOfBandAdditionIgnored (114.27s)
--- PASS: TestAccIAMRole_InlinePolicy_ignoreOrder (117.72s)
--- PASS: TestAccIAMRole_disappears (28.78s)
--- PASS: TestAccIAMRole_ManagedPolicy_basic (122.23s)
--- PASS: TestAccIAMRole_InlinePolicy_basic (122.26s)
--- PASS: TestAccIAMRole_description (74.46s)
--- PASS: TestAccIAMRole_diffsCondition (141.10s)
--- PASS: TestAccIAMRole_permissionsBoundary (144.24s)
--- PASS: TestAccIAMRole_diffs (295.93s)
PASS
ok      github.com/hashicorp/terraform-provider-aws/internal/service/iam        299.073s

% make testacc PKG=iam TESTS=TestAccIAMRolePolicy_
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./internal/service/iam/... -v -count 1 -parallel 20 -run='TestAccIAMRolePolicy_'  -timeout 360m

--- PASS: TestAccIAMRolePolicy_invalidJSON (2.31s)
--- PASS: TestAccIAMRolePolicy_Policy_invalidResource (12.04s)
--- PASS: TestAccIAMRolePolicy_disappears (20.42s)
--- PASS: TestAccIAMRolePolicy_unknownsInPolicy (24.20s)
--- PASS: TestAccIAMRolePolicy_policyOrder (27.64s)
--- PASS: TestAccIAMRolePolicy_generatedName (34.29s)
--- PASS: TestAccIAMRolePolicy_namePrefix (34.78s)
--- PASS: TestAccIAMRolePolicy_basic (34.89s)
PASS
ok      github.com/hashicorp/terraform-provider-aws/internal/service/iam        38.000s

% make testacc PKG=iam TESTS=TestAccIAMGroupPolicy_
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./internal/service/iam/... -v -count 1 -parallel 20 -run='TestAccIAMGroupPolicy_'  -timeout 360m

--- PASS: TestAccIAMGroupPolicy_disappears (17.16s)
--- PASS: TestAccIAMGroupPolicy_unknownsInPolicy (21.94s)
--- PASS: TestAccIAMGroupPolicy_generatedName (30.74s)
--- PASS: TestAccIAMGroupPolicy_namePrefix (31.09s)
--- PASS: TestAccIAMGroupPolicy_basic (33.21s)
PASS
ok      github.com/hashicorp/terraform-provider-aws/internal/service/iam        36.322s

% make testacc PKG=iam TESTS=TestAccIAMUserPolicy_
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./internal/service/iam/... -v -count 1 -parallel 20 -run='TestAccIAMUserPolicy_'  -timeout 360m

--- PASS: TestAccIAMUserPolicy_disappears (18.42s)
--- PASS: TestAccIAMUserPolicy_policyOrder (25.69s)
--- PASS: TestAccIAMUserPolicy_namePrefix (32.95s)
--- PASS: TestAccIAMUserPolicy_generatedName (33.42s)
--- PASS: TestAccIAMUserPolicy_basic (33.82s)
--- PASS: TestAccIAMUserPolicy_multiplePolicies (54.93s)
PASS
ok      github.com/hashicorp/terraform-provider-aws/internal/service/iam        58.040s

[github-actions]

Community Note

Voting for Prioritization

• Please vote on this pull request by adding a 👍 reaction to the original post to help the community and maintainers prioritize this pull request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

For Submitters

• Review the contribution guide relating to the type of change you are making to ensure all of the necessary steps have been taken.
• For new resources and data sources, use skaff to generate scaffolding with comments detailing common expectations.
• Whether or not the branch has been rebased will not impact prioritization, but doing so is always a welcome surprise.

[ewbankkit]

LGTM 🚀.

% make testacc TESTARGS='-run=TestAccIAMPolicy_\|TestAccIAMRole_\|TestAccIAMRolePolicy_\|TestAccIAMGroupPolicy_\|TESTS=TestAccIAMUserPolicy_' PKG=iam ACCTEST_PARALLELISM=2
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./internal/service/iam/... -v -count 1 -parallel 2  -run=TestAccIAMPolicy_\|TestAccIAMRole_\|TestAccIAMRolePolicy_\|TestAccIAMGroupPolicy_\|TESTS=TestAccIAMUserPolicy_ -timeout 360m
=== RUN   TestAccIAMGroupPolicy_basic
=== PAUSE TestAccIAMGroupPolicy_basic
=== RUN   TestAccIAMGroupPolicy_disappears
=== PAUSE TestAccIAMGroupPolicy_disappears
=== RUN   TestAccIAMGroupPolicy_namePrefix
=== PAUSE TestAccIAMGroupPolicy_namePrefix
=== RUN   TestAccIAMGroupPolicy_generatedName
=== PAUSE TestAccIAMGroupPolicy_generatedName
=== RUN   TestAccIAMGroupPolicy_unknownsInPolicy
=== PAUSE TestAccIAMGroupPolicy_unknownsInPolicy
=== RUN   TestAccIAMPolicy_basic
=== PAUSE TestAccIAMPolicy_basic
=== RUN   TestAccIAMPolicy_description
=== PAUSE TestAccIAMPolicy_description
=== RUN   TestAccIAMPolicy_tags
=== PAUSE TestAccIAMPolicy_tags
=== RUN   TestAccIAMPolicy_disappears
=== PAUSE TestAccIAMPolicy_disappears
=== RUN   TestAccIAMPolicy_namePrefix
=== PAUSE TestAccIAMPolicy_namePrefix
=== RUN   TestAccIAMPolicy_path
=== PAUSE TestAccIAMPolicy_path
=== RUN   TestAccIAMPolicy_policy
=== PAUSE TestAccIAMPolicy_policy
=== RUN   TestAccIAMPolicy_diffs
=== PAUSE TestAccIAMPolicy_diffs
=== RUN   TestAccIAMPolicy_policyDuplicateKeys
=== PAUSE TestAccIAMPolicy_policyDuplicateKeys
=== RUN   TestAccIAMRolePolicy_basic
=== PAUSE TestAccIAMRolePolicy_basic
=== RUN   TestAccIAMRolePolicy_disappears
=== PAUSE TestAccIAMRolePolicy_disappears
=== RUN   TestAccIAMRolePolicy_policyOrder
=== PAUSE TestAccIAMRolePolicy_policyOrder
=== RUN   TestAccIAMRolePolicy_namePrefix
=== PAUSE TestAccIAMRolePolicy_namePrefix
=== RUN   TestAccIAMRolePolicy_generatedName
=== PAUSE TestAccIAMRolePolicy_generatedName
=== RUN   TestAccIAMRolePolicy_invalidJSON
=== PAUSE TestAccIAMRolePolicy_invalidJSON
=== RUN   TestAccIAMRolePolicy_Policy_invalidResource
=== PAUSE TestAccIAMRolePolicy_Policy_invalidResource
=== RUN   TestAccIAMRolePolicy_unknownsInPolicy
=== PAUSE TestAccIAMRolePolicy_unknownsInPolicy
=== RUN   TestAccIAMRole_basic
=== PAUSE TestAccIAMRole_basic
=== RUN   TestAccIAMRole_description
=== PAUSE TestAccIAMRole_description
=== RUN   TestAccIAMRole_nameGenerated
=== PAUSE TestAccIAMRole_nameGenerated
=== RUN   TestAccIAMRole_namePrefix
=== PAUSE TestAccIAMRole_namePrefix
=== RUN   TestAccIAMRole_testNameChange
=== PAUSE TestAccIAMRole_testNameChange
=== RUN   TestAccIAMRole_diffs
=== PAUSE TestAccIAMRole_diffs
=== RUN   TestAccIAMRole_diffsCondition
=== PAUSE TestAccIAMRole_diffsCondition
=== RUN   TestAccIAMRole_badJSON
=== PAUSE TestAccIAMRole_badJSON
=== RUN   TestAccIAMRole_disappears
=== PAUSE TestAccIAMRole_disappears
=== RUN   TestAccIAMRole_policiesForceDetach
=== PAUSE TestAccIAMRole_policiesForceDetach
=== RUN   TestAccIAMRole_maxSessionDuration
=== PAUSE TestAccIAMRole_maxSessionDuration
=== RUN   TestAccIAMRole_permissionsBoundary
=== PAUSE TestAccIAMRole_permissionsBoundary
=== RUN   TestAccIAMRole_tags
=== PAUSE TestAccIAMRole_tags
=== RUN   TestAccIAMRole_InlinePolicy_basic
=== PAUSE TestAccIAMRole_InlinePolicy_basic
=== RUN   TestAccIAMRole_InlinePolicy_ignoreOrder
=== PAUSE TestAccIAMRole_InlinePolicy_ignoreOrder
=== RUN   TestAccIAMRole_InlinePolicy_empty
=== PAUSE TestAccIAMRole_InlinePolicy_empty
=== RUN   TestAccIAMRole_ManagedPolicy_basic
=== PAUSE TestAccIAMRole_ManagedPolicy_basic
=== RUN   TestAccIAMRole_ManagedPolicy_outOfBandRemovalAddedBack
=== PAUSE TestAccIAMRole_ManagedPolicy_outOfBandRemovalAddedBack
=== RUN   TestAccIAMRole_InlinePolicy_outOfBandRemovalAddedBack
=== PAUSE TestAccIAMRole_InlinePolicy_outOfBandRemovalAddedBack
=== RUN   TestAccIAMRole_ManagedPolicy_outOfBandAdditionRemoved
=== PAUSE TestAccIAMRole_ManagedPolicy_outOfBandAdditionRemoved
=== RUN   TestAccIAMRole_InlinePolicy_outOfBandAdditionRemoved
=== PAUSE TestAccIAMRole_InlinePolicy_outOfBandAdditionRemoved
=== RUN   TestAccIAMRole_InlinePolicy_outOfBandAdditionIgnored
=== PAUSE TestAccIAMRole_InlinePolicy_outOfBandAdditionIgnored
=== RUN   TestAccIAMRole_ManagedPolicy_outOfBandAdditionIgnored
=== PAUSE TestAccIAMRole_ManagedPolicy_outOfBandAdditionIgnored
=== RUN   TestAccIAMRole_InlinePolicy_outOfBandAdditionRemovedEmpty
=== PAUSE TestAccIAMRole_InlinePolicy_outOfBandAdditionRemovedEmpty
=== RUN   TestAccIAMRole_ManagedPolicy_outOfBandAdditionRemovedEmpty
=== PAUSE TestAccIAMRole_ManagedPolicy_outOfBandAdditionRemovedEmpty
=== CONT  TestAccIAMGroupPolicy_basic
=== CONT  TestAccIAMRole_nameGenerated
--- PASS: TestAccIAMRole_nameGenerated (24.61s)
=== CONT  TestAccIAMRole_InlinePolicy_ignoreOrder
--- PASS: TestAccIAMGroupPolicy_basic (52.67s)
=== CONT  TestAccIAMRole_ManagedPolicy_outOfBandAdditionRemovedEmpty
--- PASS: TestAccIAMRole_InlinePolicy_ignoreOrder (59.25s)
=== CONT  TestAccIAMRole_InlinePolicy_outOfBandAdditionRemovedEmpty
--- PASS: TestAccIAMRole_ManagedPolicy_outOfBandAdditionRemovedEmpty (38.43s)
=== CONT  TestAccIAMRole_ManagedPolicy_outOfBandAdditionIgnored
--- PASS: TestAccIAMRole_InlinePolicy_outOfBandAdditionRemovedEmpty (40.53s)
=== CONT  TestAccIAMRole_InlinePolicy_outOfBandAdditionIgnored
--- PASS: TestAccIAMRole_ManagedPolicy_outOfBandAdditionIgnored (38.25s)
=== CONT  TestAccIAMRole_InlinePolicy_outOfBandAdditionRemoved
--- PASS: TestAccIAMRole_InlinePolicy_outOfBandAdditionRemoved (36.99s)
=== CONT  TestAccIAMRole_ManagedPolicy_outOfBandAdditionRemoved
--- PASS: TestAccIAMRole_InlinePolicy_outOfBandAdditionIgnored (54.63s)
=== CONT  TestAccIAMRole_InlinePolicy_outOfBandRemovalAddedBack
--- PASS: TestAccIAMRole_ManagedPolicy_outOfBandAdditionRemoved (38.33s)
=== CONT  TestAccIAMRole_ManagedPolicy_outOfBandRemovalAddedBack
--- PASS: TestAccIAMRole_InlinePolicy_outOfBandRemovalAddedBack (37.40s)
=== CONT  TestAccIAMRole_ManagedPolicy_basic
--- PASS: TestAccIAMRole_ManagedPolicy_outOfBandRemovalAddedBack (38.84s)
=== CONT  TestAccIAMRole_InlinePolicy_empty
--- PASS: TestAccIAMRole_InlinePolicy_empty (21.40s)
=== CONT  TestAccIAMPolicy_diffs
--- PASS: TestAccIAMRole_ManagedPolicy_basic (60.58s)
=== CONT  TestAccIAMRole_description
--- PASS: TestAccIAMRole_description (57.23s)
=== CONT  TestAccIAMRole_basic
--- PASS: TestAccIAMRole_basic (23.89s)
=== CONT  TestAccIAMRolePolicy_unknownsInPolicy
--- PASS: TestAccIAMRolePolicy_unknownsInPolicy (25.97s)
=== CONT  TestAccIAMRolePolicy_Policy_invalidResource
--- PASS: TestAccIAMRolePolicy_Policy_invalidResource (12.02s)
=== CONT  TestAccIAMRolePolicy_invalidJSON
--- PASS: TestAccIAMRolePolicy_invalidJSON (2.22s)
=== CONT  TestAccIAMRolePolicy_generatedName
--- PASS: TestAccIAMPolicy_diffs (166.40s)
=== CONT  TestAccIAMRolePolicy_namePrefix
--- PASS: TestAccIAMRolePolicy_generatedName (42.99s)
=== CONT  TestAccIAMRolePolicy_policyOrder
--- PASS: TestAccIAMRolePolicy_policyOrder (32.11s)
=== CONT  TestAccIAMRolePolicy_disappears
--- PASS: TestAccIAMRolePolicy_namePrefix (46.62s)
=== CONT  TestAccIAMRolePolicy_basic
--- PASS: TestAccIAMRolePolicy_disappears (25.99s)
=== CONT  TestAccIAMPolicy_policyDuplicateKeys
--- PASS: TestAccIAMPolicy_policyDuplicateKeys (2.81s)
=== CONT  TestAccIAMPolicy_description
--- PASS: TestAccIAMRolePolicy_basic (47.98s)
=== CONT  TestAccIAMPolicy_policy
--- PASS: TestAccIAMPolicy_description (25.66s)
=== CONT  TestAccIAMPolicy_path
--- PASS: TestAccIAMPolicy_path (23.95s)
=== CONT  TestAccIAMPolicy_namePrefix
--- PASS: TestAccIAMPolicy_policy (41.84s)
=== CONT  TestAccIAMPolicy_disappears
--- PASS: TestAccIAMPolicy_namePrefix (23.78s)
=== CONT  TestAccIAMPolicy_tags
--- PASS: TestAccIAMPolicy_disappears (18.24s)
=== CONT  TestAccIAMGroupPolicy_generatedName
--- PASS: TestAccIAMGroupPolicy_generatedName (41.23s)
=== CONT  TestAccIAMPolicy_basic
--- PASS: TestAccIAMPolicy_tags (59.58s)
=== CONT  TestAccIAMGroupPolicy_unknownsInPolicy
--- PASS: TestAccIAMPolicy_basic (24.13s)
=== CONT  TestAccIAMGroupPolicy_namePrefix
--- PASS: TestAccIAMGroupPolicy_unknownsInPolicy (25.71s)
=== CONT  TestAccIAMGroupPolicy_disappears
--- PASS: TestAccIAMGroupPolicy_disappears (20.23s)
=== CONT  TestAccIAMRole_disappears
--- PASS: TestAccIAMGroupPolicy_namePrefix (39.07s)
=== CONT  TestAccIAMRole_InlinePolicy_basic
--- PASS: TestAccIAMRole_disappears (19.21s)
=== CONT  TestAccIAMRole_tags
--- PASS: TestAccIAMRole_tags (39.77s)
=== CONT  TestAccIAMRole_permissionsBoundary
--- PASS: TestAccIAMRole_InlinePolicy_basic (57.17s)
=== CONT  TestAccIAMRole_maxSessionDuration
--- PASS: TestAccIAMRole_maxSessionDuration (46.54s)
=== CONT  TestAccIAMRole_policiesForceDetach
--- PASS: TestAccIAMRole_policiesForceDetach (24.62s)
=== CONT  TestAccIAMRole_diffs
--- PASS: TestAccIAMRole_permissionsBoundary (88.97s)
=== CONT  TestAccIAMRole_badJSON
--- PASS: TestAccIAMRole_badJSON (2.76s)
=== CONT  TestAccIAMRole_diffsCondition
--- PASS: TestAccIAMRole_diffsCondition (92.88s)
=== CONT  TestAccIAMRole_testNameChange
--- PASS: TestAccIAMRole_testNameChange (39.50s)
=== CONT  TestAccIAMRole_namePrefix
--- PASS: TestAccIAMRole_namePrefix (23.01s)
--- PASS: TestAccIAMRole_diffs (311.08s)
PASS
ok  	github.com/hashicorp/terraform-provider-aws/internal/service/iam	1135.256s

[github-actions]

This functionality has been released in v5.19.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions]

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.