Amazon MSK multiple authentication modes and updates to TLS encryption settings

[james-bjss]

Community Note

• Please vote on this pull request by adding a 👍 reaction to the original pull request comment to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for pull request followers and do not help prioritize the request

Closes #20956.
Closes #22839.
Closes #23727.
Closes #21859.

Output from acceptance testing:

 make testacc TESTS=TestAccKafkaCluster_basic PKG=kafka
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./internal/service/kafka/... -v -count 1 -parallel 20 -run='TestAccKafkaCluster_basic' -timeout 180m
=== RUN   TestAccKafkaCluster_basic
=== PAUSE TestAccKafkaCluster_basic
=== CONT  TestAccKafkaCluster_basic
--- PASS: TestAccKafkaCluster_basic (1993.81s)
PASS
ok      github.com/hashicorp/terraform-provider-aws/internal/service/kafka      1993.866s


make testacc TESTS=TestAccKafkaCluster_ClientAuthenticationSASL_scram PKG=kafka
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./internal/service/kafka/... -v -count 1 -parallel 20 -run='TestAccKafkaCluster_ClientAuthenticationSASL_scram' -timeout 180m
=== RUN   TestAccKafkaCluster_ClientAuthenticationSASL_scram
=== PAUSE TestAccKafkaCluster_ClientAuthenticationSASL_scram
=== CONT  TestAccKafkaCluster_ClientAuthenticationSASL_scram
--- PASS: TestAccKafkaCluster_ClientAuthenticationSASL_scram (3611.11s)
PASS
ok      github.com/hashicorp/terraform-provider-aws/internal/service/kafka      3611.202s

[zhelding]

Pull request #21306 has significantly refactored the AWS Provider codebase. As a result, most PRs opened prior to the refactor now have merge conflicts that must be resolved before proceeding.

Specifically, PR #21306 relocated the code for all AWS resources and data sources from a single aws directory to a large number of separate directories in internal/service, each corresponding to a particular AWS service. This separation of code has also allowed for us to simplify the names of underlying functions -- while still avoiding namespace collisions.

We recognize that many pull requests have been open for some time without yet being addressed by our maintainers. Therefore, we want to make it clear that resolving these conflicts in no way affects the prioritization of a particular pull request. Once a pull request has been prioritized for review, the necessary changes will be made by a maintainer -- either directly or in collaboration with the pull request author.

For a more complete description of this refactor, including examples of how old filepaths and function names correspond to their new counterparts: please refer to issue #20000.

For a quick guide on how to amend your pull request to resolve the merge conflicts resulting from this refactor and bring it in line with our new code patterns: please refer to our Service Package Refactor Pull Request Guide.

[ewbankkit]

LGTM 🚀.

% ACCTEST_TIMEOUT=720m make testacc TESTS=TestAccKafkaCluster PKG=kafka ACCTEST_PARALLELISM=3
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./internal/service/kafka/... -v -count 1 -parallel 3 -run='TestAccKafkaCluster'  -timeout 720m
=== RUN   TestAccKafkaClusterDataSource_basic
=== PAUSE TestAccKafkaClusterDataSource_basic
=== RUN   TestAccKafkaCluster_basic
=== PAUSE TestAccKafkaCluster_basic
=== RUN   TestAccKafkaCluster_BrokerNodeGroupInfo_ebsVolumeSize
=== PAUSE TestAccKafkaCluster_BrokerNodeGroupInfo_ebsVolumeSize
=== RUN   TestAccKafkaCluster_BrokerNodeGroupInfo_instanceType
=== PAUSE TestAccKafkaCluster_BrokerNodeGroupInfo_instanceType
=== RUN   TestAccKafkaCluster_BrokerNodeGroupInfo_publicAccessSaslIam
=== PAUSE TestAccKafkaCluster_BrokerNodeGroupInfo_publicAccessSaslIam
=== RUN   TestAccKafkaCluster_ClientAuthenticationSASL_scram
=== PAUSE TestAccKafkaCluster_ClientAuthenticationSASL_scram
=== RUN   TestAccKafkaCluster_ClientAuthenticationSASL_iam
=== PAUSE TestAccKafkaCluster_ClientAuthenticationSASL_iam
=== RUN   TestAccKafkaCluster_ClientAuthenticationTLS_certificateAuthorityARNs
=== PAUSE TestAccKafkaCluster_ClientAuthenticationTLS_certificateAuthorityARNs
=== RUN   TestAccKafkaCluster_ClientAuthenticationTLS_initiallyNoAuthentication
=== PAUSE TestAccKafkaCluster_ClientAuthenticationTLS_initiallyNoAuthentication
=== RUN   TestAccKafkaCluster_Info_revision
=== PAUSE TestAccKafkaCluster_Info_revision
=== RUN   TestAccKafkaCluster_EncryptionInfo_encryptionAtRestKMSKeyARN
=== PAUSE TestAccKafkaCluster_EncryptionInfo_encryptionAtRestKMSKeyARN
=== RUN   TestAccKafkaCluster_EncryptionInfoEncryptionInTransit_clientBroker
=== PAUSE TestAccKafkaCluster_EncryptionInfoEncryptionInTransit_clientBroker
=== RUN   TestAccKafkaCluster_EncryptionInfoEncryptionInTransit_inCluster
=== PAUSE TestAccKafkaCluster_EncryptionInfoEncryptionInTransit_inCluster
=== RUN   TestAccKafkaCluster_enhancedMonitoring
=== PAUSE TestAccKafkaCluster_enhancedMonitoring
=== RUN   TestAccKafkaCluster_numberOfBrokerNodes
=== PAUSE TestAccKafkaCluster_numberOfBrokerNodes
=== RUN   TestAccKafkaCluster_openMonitoring
=== PAUSE TestAccKafkaCluster_openMonitoring
=== RUN   TestAccKafkaCluster_loggingInfo
=== PAUSE TestAccKafkaCluster_loggingInfo
=== RUN   TestAccKafkaCluster_kafkaVersionUpgrade
=== PAUSE TestAccKafkaCluster_kafkaVersionUpgrade
=== RUN   TestAccKafkaCluster_kafkaVersionDowngrade
=== PAUSE TestAccKafkaCluster_kafkaVersionDowngrade
=== RUN   TestAccKafkaCluster_kafkaVersionUpgradeWithInfo
=== PAUSE TestAccKafkaCluster_kafkaVersionUpgradeWithInfo
=== RUN   TestAccKafkaCluster_tags
=== PAUSE TestAccKafkaCluster_tags
=== CONT  TestAccKafkaClusterDataSource_basic
=== CONT  TestAccKafkaCluster_EncryptionInfoEncryptionInTransit_clientBroker
=== CONT  TestAccKafkaCluster_ClientAuthenticationSASL_iam
--- PASS: TestAccKafkaCluster_EncryptionInfoEncryptionInTransit_clientBroker (1941.08s)
=== CONT  TestAccKafkaCluster_EncryptionInfo_encryptionAtRestKMSKeyARN
--- PASS: TestAccKafkaClusterDataSource_basic (1949.57s)
=== CONT  TestAccKafkaCluster_Info_revision
--- PASS: TestAccKafkaCluster_EncryptionInfo_encryptionAtRestKMSKeyARN (1955.82s)
=== CONT  TestAccKafkaCluster_ClientAuthenticationTLS_initiallyNoAuthentication
--- PASS: TestAccKafkaCluster_ClientAuthenticationSASL_iam (3909.38s)
=== CONT  TestAccKafkaCluster_ClientAuthenticationTLS_certificateAuthorityARNs
--- PASS: TestAccKafkaCluster_Info_revision (2688.38s)
=== CONT  TestAccKafkaCluster_loggingInfo
--- PASS: TestAccKafkaCluster_ClientAuthenticationTLS_certificateAuthorityARNs (2288.52s)
=== CONT  TestAccKafkaCluster_tags
--- PASS: TestAccKafkaCluster_loggingInfo (2266.44s)
=== CONT  TestAccKafkaCluster_kafkaVersionUpgradeWithInfo
--- PASS: TestAccKafkaCluster_ClientAuthenticationTLS_initiallyNoAuthentication (4265.93s)
=== CONT  TestAccKafkaCluster_kafkaVersionDowngrade
--- PASS: TestAccKafkaCluster_tags (2148.92s)
=== CONT  TestAccKafkaCluster_kafkaVersionUpgrade
--- PASS: TestAccKafkaCluster_kafkaVersionDowngrade (4154.47s)
=== CONT  TestAccKafkaCluster_BrokerNodeGroupInfo_instanceType
--- PASS: TestAccKafkaCluster_kafkaVersionUpgradeWithInfo (5748.13s)
=== CONT  TestAccKafkaCluster_ClientAuthenticationSASL_scram
--- PASS: TestAccKafkaCluster_kafkaVersionUpgrade (5813.05s)
=== CONT  TestAccKafkaCluster_BrokerNodeGroupInfo_publicAccessSaslIam
--- PASS: TestAccKafkaCluster_ClientAuthenticationSASL_scram (3384.64s)
=== CONT  TestAccKafkaCluster_BrokerNodeGroupInfo_ebsVolumeSize
--- PASS: TestAccKafkaCluster_BrokerNodeGroupInfo_ebsVolumeSize (2466.04s)
=== CONT  TestAccKafkaCluster_numberOfBrokerNodes
--- PASS: TestAccKafkaCluster_BrokerNodeGroupInfo_instanceType (6307.75s)
=== CONT  TestAccKafkaCluster_openMonitoring
--- PASS: TestAccKafkaCluster_BrokerNodeGroupInfo_publicAccessSaslIam (6018.91s)
=== CONT  TestAccKafkaCluster_basic
--- PASS: TestAccKafkaCluster_openMonitoring (1677.99s)
=== CONT  TestAccKafkaCluster_enhancedMonitoring
--- PASS: TestAccKafkaCluster_numberOfBrokerNodes (2644.50s)
=== CONT  TestAccKafkaCluster_EncryptionInfoEncryptionInTransit_inCluster
--- PASS: TestAccKafkaCluster_basic (1729.06s)
--- PASS: TestAccKafkaCluster_enhancedMonitoring (1727.91s)
--- PASS: TestAccKafkaCluster_EncryptionInfoEncryptionInTransit_inCluster (2028.77s)
PASS
ok  	github.com/hashicorp/terraform-provider-aws/internal/service/kafka	22884.818s

[ewbankkit]

@james-bjss @nuttmeister Thanks for the contribution 🎉 👏.

[github-actions]

This functionality has been released in v4.13.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions]

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.