Add support for AWS MSK serverless cluster type #25684

moremagic · 2022-07-05T06:15:18Z

Community Note

Please vote on this pull request by adding a 👍 reaction to the original pull request comment to help the community and maintainers prioritize this request
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for pull request followers and do not help prioritize the request

Closes #22058

Output from acceptance testing:

$ make testacc TESTARGS='-run=TestAccKafkaCluster_' PKG=kafka ACCTEST_PARALLELISM=3

...


$ make testacc \
> PKG=kafka ACCTEST_PARALLELISM=3 \
> TESTARGS='-run=TestAccKafkaCluster_Basic_'
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./internal/service/kafka/... -v -count 1 -parallel 3  -run=TestAccKafkaCluster_Basic_ -timeout 180m
=== RUN   TestAccKafkaCluster_Basic_Provisioned
=== PAUSE TestAccKafkaCluster_Basic_Provisioned
=== RUN   TestAccKafkaCluster_Basic_Serverless
=== PAUSE TestAccKafkaCluster_Basic_Serverless
=== CONT  TestAccKafkaCluster_Basic_Provisioned
=== CONT  TestAccKafkaCluster_Basic_Serverless
--- PASS: TestAccKafkaCluster_Basic_Serverless (401.64s)
--- PASS: TestAccKafkaCluster_Basic_Provisioned (1960.52s)
PASS
ok  	github.com/hashicorp/terraform-provider-aws/internal/service/kafka	1960.622s


$ make testacc PKG=kafka ACCTEST_PARALLELISM=3 TESTARGS='-run=TestAccKafkaCluster_BrokerNodeGroupInfo_' ACCTEST_TIMEOUT=4h
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./internal/service/kafka/... -v -count 1 -parallel 3  -run=TestAccKafkaCluster_BrokerNodeGroupInfo_ -timeout 4h
=== RUN   TestAccKafkaCluster_BrokerNodeGroupInfo_ebsVolumeSize
=== PAUSE TestAccKafkaCluster_BrokerNodeGroupInfo_ebsVolumeSize
=== RUN   TestAccKafkaCluster_BrokerNodeGroupInfo_storageInfo
=== PAUSE TestAccKafkaCluster_BrokerNodeGroupInfo_storageInfo
=== RUN   TestAccKafkaCluster_BrokerNodeGroupInfo_modifyEBSVolumeSizeToStorageInfo
=== PAUSE TestAccKafkaCluster_BrokerNodeGroupInfo_modifyEBSVolumeSizeToStorageInfo
=== RUN   TestAccKafkaCluster_BrokerNodeGroupInfo_instanceType
=== PAUSE TestAccKafkaCluster_BrokerNodeGroupInfo_instanceType
=== RUN   TestAccKafkaCluster_BrokerNodeGroupInfo_publicAccessSASLIAM
=== PAUSE TestAccKafkaCluster_BrokerNodeGroupInfo_publicAccessSASLIAM
=== CONT  TestAccKafkaCluster_BrokerNodeGroupInfo_ebsVolumeSize
=== CONT  TestAccKafkaCluster_BrokerNodeGroupInfo_instanceType
=== CONT  TestAccKafkaCluster_BrokerNodeGroupInfo_publicAccessSASLIAM
--- PASS: TestAccKafkaCluster_BrokerNodeGroupInfo_ebsVolumeSize (2378.66s)
=== CONT  TestAccKafkaCluster_BrokerNodeGroupInfo_modifyEBSVolumeSizeToStorageInfo
--- PASS: TestAccKafkaCluster_BrokerNodeGroupInfo_publicAccessSASLIAM (6151.64s)
=== CONT  TestAccKafkaCluster_BrokerNodeGroupInfo_storageInfo
--- PASS: TestAccKafkaCluster_BrokerNodeGroupInfo_instanceType (6955.93s)
--- PASS: TestAccKafkaCluster_BrokerNodeGroupInfo_modifyEBSVolumeSizeToStorageInfo (4878.60s)
--- PASS: TestAccKafkaCluster_BrokerNodeGroupInfo_storageInfo (2392.74s)
PASS
ok  	github.com/hashicorp/terraform-provider-aws/internal/service/kafka	8544.478s


$ make testacc PKG=kafka ACCTEST_PARALLELISM=3 TESTARGS='-run=TestAccKafkaCluster_ClientAuthentication' ACCTEST_TIMEOUT=4h
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./internal/service/kafka/... -v -count 1 -parallel 3  -run=TestAccKafkaCluster_ClientAuthentication -timeout 4h
=== RUN   TestAccKafkaCluster_ClientAuthenticationSASL_scram
=== PAUSE TestAccKafkaCluster_ClientAuthenticationSASL_scram
=== RUN   TestAccKafkaCluster_ClientAuthenticationSASL_iam
=== PAUSE TestAccKafkaCluster_ClientAuthenticationSASL_iam
=== RUN   TestAccKafkaCluster_ClientAuthenticationTLS_certificateAuthorityARNs
=== PAUSE TestAccKafkaCluster_ClientAuthenticationTLS_certificateAuthorityARNs
=== RUN   TestAccKafkaCluster_ClientAuthenticationTLS_initiallyNoAuthentication
=== PAUSE TestAccKafkaCluster_ClientAuthenticationTLS_initiallyNoAuthentication
=== CONT  TestAccKafkaCluster_ClientAuthenticationSASL_iam
=== CONT  TestAccKafkaCluster_ClientAuthenticationSASL_scram
=== CONT  TestAccKafkaCluster_ClientAuthenticationTLS_certificateAuthorityARNs
--- PASS: TestAccKafkaCluster_ClientAuthenticationTLS_certificateAuthorityARNs (2213.09s)
=== CONT  TestAccKafkaCluster_ClientAuthenticationTLS_initiallyNoAuthentication
--- PASS: TestAccKafkaCluster_ClientAuthenticationSASL_scram (3857.07s)
--- PASS: TestAccKafkaCluster_ClientAuthenticationSASL_iam (3862.73s)
--- PASS: TestAccKafkaCluster_ClientAuthenticationTLS_initiallyNoAuthentication (3636.77s)
PASS
ok  	github.com/hashicorp/terraform-provider-aws/internal/service/kafka	5849.972s


$  make testacc PKG=kafka ACCTEST_PARALLELISM=3 TESTARGS='-run=TestAccKafkaCluster_EncryptionInfo' ACCTEST_TIMEOUT=4h
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./internal/service/kafka/... -v -count 1 -parallel 3  -run=TestAccKafkaCluster_EncryptionInfo -timeout 4h
=== RUN   TestAccKafkaCluster_EncryptionInfo_encryptionAtRestKMSKeyARN
=== PAUSE TestAccKafkaCluster_EncryptionInfo_encryptionAtRestKMSKeyARN
=== RUN   TestAccKafkaCluster_EncryptionInfoEncryptionInTransit_clientBroker
=== PAUSE TestAccKafkaCluster_EncryptionInfoEncryptionInTransit_clientBroker
=== RUN   TestAccKafkaCluster_EncryptionInfoEncryptionInTransit_inCluster
=== PAUSE TestAccKafkaCluster_EncryptionInfoEncryptionInTransit_inCluster
=== CONT  TestAccKafkaCluster_EncryptionInfo_encryptionAtRestKMSKeyARN
=== CONT  TestAccKafkaCluster_EncryptionInfoEncryptionInTransit_inCluster
=== CONT  TestAccKafkaCluster_EncryptionInfoEncryptionInTransit_clientBroker
--- PASS: TestAccKafkaCluster_EncryptionInfo_encryptionAtRestKMSKeyARN (1655.12s)
--- PASS: TestAccKafkaCluster_EncryptionInfoEncryptionInTransit_inCluster (1655.14s)
--- PASS: TestAccKafkaCluster_EncryptionInfoEncryptionInTransit_clientBroker (1659.40s)
PASS
ok  	github.com/hashicorp/terraform-provider-aws/internal/service/kafka	1659.527s


$ make testacc PKG=kafka TESTARGS='-run=TestAccKafkaCluster_Info_revision'
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./internal/service/kafka/... -v -count 1 -parallel 20  -run=TestAccKafkaCluster_Info_revision -timeout 180m
=== RUN   TestAccKafkaCluster_Info_revision
=== PAUSE TestAccKafkaCluster_Info_revision
=== CONT  TestAccKafkaCluster_Info_revision
--- PASS: TestAccKafkaCluster_Info_revision (2339.46s)
PASS
ok  	github.com/hashicorp/terraform-provider-aws/internal/service/kafka	2339.550s


$  make testacc PKG=kafka TESTARGS='-run=TestAccKafkaCluster_tags'
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./internal/service/kafka/... -v -count 1 -parallel 20  -run=TestAccKafkaCluster_tags -timeout 180m
=== RUN   TestAccKafkaCluster_tags
=== PAUSE TestAccKafkaCluster_tags
=== CONT  TestAccKafkaCluster_tags
--- PASS: TestAccKafkaCluster_tags (2183.67s)
PASS
ok  	github.com/hashicorp/terraform-provider-aws/internal/service/kafka	2183.766s

$ make testacc PKG=kafka TESTARGS='-run=TestAccKafkaCluster_enhancedMonitoring'
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./internal/service/kafka/... -v -count 1 -parallel 20  -run=TestAccKafkaCluster_enhancedMonitoring -timeout 180m
=== RUN   TestAccKafkaCluster_enhancedMonitoring
=== PAUSE TestAccKafkaCluster_enhancedMonitoring
=== CONT  TestAccKafkaCluster_enhancedMonitoring
--- PASS: TestAccKafkaCluster_enhancedMonitoring (1880.49s)
PASS
ok  	github.com/hashicorp/terraform-provider-aws/internal/service/kafka	1880.584s


$ make testacc PKG=kafka TESTARGS='-run=TestAccKafkaCluster_openMonitoring'
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./internal/service/kafka/... -v -count 1 -parallel 20  -run=TestAccKafkaCluster_openMonitoring -timeout 180m
=== RUN   TestAccKafkaCluster_openMonitoring
=== PAUSE TestAccKafkaCluster_openMonitoring
=== CONT  TestAccKafkaCluster_openMonitoring
--- PASS: TestAccKafkaCluster_openMonitoring (2904.16s)
PASS
ok  	github.com/hashicorp/terraform-provider-aws/internal/service/kafka	2904.258s

$ make testacc PKG=kafka TESTARGS='-run=TestAccKafkaCluster_numberOfBrokerNodes'
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./internal/service/kafka/... -v -count 1 -parallel 20  -run=TestAccKafkaCluster_numberOfBrokerNodes -timeout 180m
=== RUN   TestAccKafkaCluster_numberOfBrokerNodes
=== PAUSE TestAccKafkaCluster_numberOfBrokerNodes
=== CONT  TestAccKafkaCluster_numberOfBrokerNodes
--- PASS: TestAccKafkaCluster_numberOfBrokerNodes (2231.00s)
PASS
ok  	github.com/hashicorp/terraform-provider-aws/internal/service/kafka	2231.108s


$ make testacc PKG=kafka ACCTEST_PARALLELISM=3 TESTARGS='-run=TestAccKafkaCluster_loggingInfo'
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./internal/service/kafka/... -v -count 1 -parallel 3  -run=TestAccKafkaCluster_loggingInfo -timeout 180m
=== RUN   TestAccKafkaCluster_loggingInfo
=== PAUSE TestAccKafkaCluster_loggingInfo
=== CONT  TestAccKafkaCluster_loggingInfo
--- PASS: TestAccKafkaCluster_loggingInfo (2361.50s)
PASS
ok  	github.com/hashicorp/terraform-provider-aws/internal/service/kafka	2361.599s


make testacc TESTARGS='-run=TestAccKafkaCluster_kafkaVersion' PKG=kafka ACCTEST_PARALLELISM=3 ACCTEST_TIMEOUT=17h
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./internal/service/kafka/... -v -count 1 -parallel 3  -run=TestAccKafkaCluster_kafkaVersion -timeout 17h
=== RUN   TestAccKafkaCluster_kafkaVersionUpgrade
=== PAUSE TestAccKafkaCluster_kafkaVersionUpgrade
=== RUN   TestAccKafkaCluster_kafkaVersionDowngrade
=== PAUSE TestAccKafkaCluster_kafkaVersionDowngrade
=== RUN   TestAccKafkaCluster_kafkaVersionUpgradeWithInfo
=== PAUSE TestAccKafkaCluster_kafkaVersionUpgradeWithInfo
=== CONT  TestAccKafkaCluster_kafkaVersionUpgradeWithInfo
=== CONT  TestAccKafkaCluster_kafkaVersionUpgrade
=== CONT  TestAccKafkaCluster_kafkaVersionDowngrade
--- PASS: TestAccKafkaCluster_kafkaVersionDowngrade (3293.68s)
--- PASS: TestAccKafkaCluster_kafkaVersionUpgradeWithInfo (6184.09s)
--- PASS: TestAccKafkaCluster_kafkaVersionUpgrade (6185.36s)
PASS
ok  	github.com/hashicorp/terraform-provider-aws/internal/service/kafka	6185.560s

github-actions

Welcome @moremagic 👋

It looks like this is your first Pull Request submission to the Terraform AWS Provider! If you haven’t already done so please make sure you have checked out our CONTRIBUTING guide and FAQ to make sure your contribution is adhering to best practice and has all the necessary elements in place for a successful approval.

Also take a look at our FAQ which details how we prioritize Pull Requests for inclusion.

Thanks again, and welcome to the community! 😃

This reverts commit a6a9f50.

This reverts commit b657db4.

This reverts commit 6cbcdef.

This reverts commit df05a43.

This reverts commit 6f86efb.

Acceptance test output: % make testacc TESTARGS='-run=TestAccKafkaServerlessCluster_' PKG=kafka ACCTEST_PARALLELISM=2 ==> Checking that code complies with gofmt requirements... TF_ACC=1 go test ./internal/service/kafka/... -v -count 1 -parallel 2 -run=TestAccKafkaServerlessCluster_ -timeout 180m === RUN TestAccKafkaServerlessCluster_basic === PAUSE TestAccKafkaServerlessCluster_basic === RUN TestAccKafkaServerlessCluster_disappears === PAUSE TestAccKafkaServerlessCluster_disappears === RUN TestAccKafkaServerlessCluster_tags === PAUSE TestAccKafkaServerlessCluster_tags === RUN TestAccKafkaServerlessCluster_securityGroup === PAUSE TestAccKafkaServerlessCluster_securityGroup === CONT TestAccKafkaServerlessCluster_basic === CONT TestAccKafkaServerlessCluster_tags --- PASS: TestAccKafkaServerlessCluster_tags (368.41s) === CONT TestAccKafkaServerlessCluster_disappears --- PASS: TestAccKafkaServerlessCluster_basic (420.83s) === CONT TestAccKafkaServerlessCluster_securityGroup --- PASS: TestAccKafkaServerlessCluster_disappears (256.70s) --- PASS: TestAccKafkaServerlessCluster_securityGroup (367.95s) PASS ok github.com/hashicorp/terraform-provider-aws/internal/service/kafka 792.919s

ewbankkit

LGTM 🚀.

% make testacc TESTARGS='-run=TestAccKafkaServerlessCluster_' PKG=kafka ACCTEST_PARALLELISM=2 
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./internal/service/kafka/... -v -count 1 -parallel 2  -run=TestAccKafkaServerlessCluster_ -timeout 180m
=== RUN   TestAccKafkaServerlessCluster_basic
=== PAUSE TestAccKafkaServerlessCluster_basic
=== RUN   TestAccKafkaServerlessCluster_disappears
=== PAUSE TestAccKafkaServerlessCluster_disappears
=== RUN   TestAccKafkaServerlessCluster_tags
=== PAUSE TestAccKafkaServerlessCluster_tags
=== RUN   TestAccKafkaServerlessCluster_securityGroup
=== PAUSE TestAccKafkaServerlessCluster_securityGroup
=== CONT  TestAccKafkaServerlessCluster_basic
=== CONT  TestAccKafkaServerlessCluster_tags
--- PASS: TestAccKafkaServerlessCluster_tags (368.41s)
=== CONT  TestAccKafkaServerlessCluster_disappears
--- PASS: TestAccKafkaServerlessCluster_basic (420.83s)
=== CONT  TestAccKafkaServerlessCluster_securityGroup
--- PASS: TestAccKafkaServerlessCluster_disappears (256.70s)
--- PASS: TestAccKafkaServerlessCluster_securityGroup (367.95s)
PASS
ok  	github.com/hashicorp/terraform-provider-aws/internal/service/kafka	792.919s
% make testacc TESTARGS='-run=TestAccKafkaCluster_basic\|TestAccKafkaCluster_disappears' PKG=kafka ACCTEST_PARALLELISM=2
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./internal/service/kafka/... -v -count 1 -parallel 2  -run=TestAccKafkaCluster_basic\|TestAccKafkaCluster_disappears -timeout 180m
=== RUN   TestAccKafkaCluster_basic
=== PAUSE TestAccKafkaCluster_basic
=== RUN   TestAccKafkaCluster_disappears
=== PAUSE TestAccKafkaCluster_disappears
=== CONT  TestAccKafkaCluster_basic
=== CONT  TestAccKafkaCluster_disappears
--- PASS: TestAccKafkaCluster_basic (2120.84s)
--- PASS: TestAccKafkaCluster_disappears (2129.36s)
PASS
ok  	github.com/hashicorp/terraform-provider-aws/internal/service/kafka	2133.839s

ewbankkit · 2022-08-17T18:07:39Z

@moremagic Thanks for the contribution 🎉 👏.
As serverless cluster configuration is very different from classic, provisioned cluster configuration, and to match the CloudFormation AWS::MSK::ServerlessCluster resource type, the maintainers decided to implement this as a new resource type, aws_msk_serverless_cluster.

github-actions · 2022-08-19T12:17:04Z

This functionality has been released in v4.27.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

github-actions · 2022-09-19T02:45:34Z

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

github-actions bot reviewed Jul 5, 2022

View reviewed changes

moremagic force-pushed the f-aws_msk_cluster-serverless_cluster_type branch from 828e5a7 to 8e31ea8 Compare July 6, 2022 00:04

github-actions bot added the documentation Introduces or discusses updates to documentation. label Jul 6, 2022

moremagic force-pushed the f-aws_msk_cluster-serverless_cluster_type branch from 9fdb962 to 0eb58f6 Compare July 7, 2022 09:08

justinretzolk added enhancement Requests to existing resources that expand the functionality or scope. and removed needs-triage Waiting for first response or review from a maintainer. labels Jul 7, 2022

moremagic added 5 commits July 12, 2022 12:03

Add MSKServerless Configration

6f86efb

Fixed to use constant value for ClusterType

df05a43

Updated documentation

6cbcdef

Add changelog

b657db4

fix timeout value

a6a9f50

moremagic force-pushed the f-aws_msk_cluster-serverless_cluster_type branch from 45a9bf7 to a6a9f50 Compare July 12, 2022 03:10

moremagic changed the title ~~[WIP] Add support for AWS MSK serverless cluster type~~ Add support for AWS MSK serverless cluster type Jul 12, 2022

ewbankkit added 13 commits August 10, 2022 16:58

Revert "fix timeout value"

98743f1

This reverts commit a6a9f50.

Revert "Add changelog"

12fa0c9

This reverts commit b657db4.

Revert "Updated documentation"

db77c2e

This reverts commit 6cbcdef.

Revert "Fixed to use constant value for ClusterType"

72a7b2f

This reverts commit df05a43.

Revert "Add MSKServerless Configration"

43b296a

This reverts commit 6f86efb.

Merge branch 'main' into HEAD

ac5f072

Add CHANGELOG entry.

5dfbc13

MSK: No need to generate 'ListTags'.

a53ff99

r/aws_msk_cluster: Use 'UpdateTagsWithContext'.

463143a

Merge branch 'main' into HEAD

859784c

Add 'FindClusterV2ByARN' and 'FindServerlessClusterByARN'.

9556447

'statusClusterState' uses 'findClusterV2ByARN'.

83cea95

r/aws_msk_serverless_cluster: Skeleton.

ffe4afe

ewbankkit added 5 commits August 17, 2022 10:43

r/aws_msk_serverless_cluster: Add documentation.

0bd38c8

r/aws_msk_serverless_cluster: Add flex.

6bae639

Add documentation about cluster types (provisioned vs. serverless).

a6e3923

Update CHANGELOG entry.

f32eecb

github-actions bot added generators Relates to code generators. provider Pertains to the provider itself, rather than any interaction with AWS. tags Pertains to resource tagging. labels Aug 17, 2022

ewbankkit added 6 commits August 17, 2022 13:45

Fix typo.

3c8762b

MSK: Sweep both provisioned and serverless clusters.

998a71e

Fix terrafmt error.

610be0c

Fix semgrep 'ci.caps2-in-func-name'.

0cdcb0f

Fix golangi-lint 'unparam'.

735f553

Add 'TestAccKafkaCluster_disappears'.

a1567f7

github-actions bot added the sweeper Pertains to changes to or issues with the sweeper. label Aug 17, 2022

ewbankkit approved these changes Aug 17, 2022

View reviewed changes

ewbankkit added the new-resource Introduces a new resource. label Aug 17, 2022

Fix semgrep 'ci.caps4-in-func-name'.

6691496

ewbankkit merged commit 526591f into hashicorp:main Aug 17, 2022

github-actions bot added this to the v4.27.0 milestone Aug 17, 2022

github-actions bot locked as resolved and limited conversation to collaborators Sep 19, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for AWS MSK serverless cluster type #25684

Add support for AWS MSK serverless cluster type #25684

moremagic commented Jul 5, 2022 •

edited

github-actions bot left a comment

ewbankkit left a comment •

edited

ewbankkit commented Aug 17, 2022

github-actions bot commented Aug 19, 2022

github-actions bot commented Sep 19, 2022

Add support for AWS MSK serverless cluster type #25684

Add support for AWS MSK serverless cluster type #25684

Conversation

moremagic commented Jul 5, 2022 • edited

Community Note

github-actions bot left a comment

Choose a reason for hiding this comment

ewbankkit left a comment • edited

Choose a reason for hiding this comment

ewbankkit commented Aug 17, 2022

github-actions bot commented Aug 19, 2022

github-actions bot commented Sep 19, 2022

moremagic commented Jul 5, 2022 •

edited

ewbankkit left a comment •

edited