chore: update Kafka endpoint settings to include all attributes #20904

bryantbiggs · 2021-09-14T19:06:12Z

Community Note

Please vote on this pull request by adding a 👍 reaction to the original pull request comment to help the community and maintainers prioritize this request
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for pull request followers and do not help prioritize the request

Closes #20138.
Closes #20083.

Output from acceptance testing:

$ make testacc TESTARGS='-run=TestAccAwsDmsEndpoint_Kafka'

==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./aws -v -count 1 -parallel 20 -run=TestAccAwsDmsEndpoint_Kafka -timeout 180m
=== RUN   TestAccAwsDmsEndpoint_Kafka
=== PAUSE TestAccAwsDmsEndpoint_Kafka
=== CONT  TestAccAwsDmsEndpoint_Kafka
    resource_aws_dms_endpoint_test.go:309: Step 1/3 error: Error running apply: exit status 1

        Error: Error creating DMS endpoint: KMSKeyNotAccessibleFault: Key Id is missing

          with aws_dms_endpoint.test,
          on terraform_plugin_test.tf line 4, in resource "aws_dms_endpoint" "test":
           4: resource "aws_dms_endpoint" "test" {

--- FAIL: TestAccAwsDmsEndpoint_Kafka (18.50s)
FAIL
FAIL	github.com/terraform-providers/terraform-provider-aws/aws	20.513s
FAIL
make: *** [testacc] Error 1

⚠️ working on KMSKeyNotAccessibleFault: Key Id is missing

… Sensitive.

…tly unused.

ewbankkit

LGTM 🚀.

Commercial

% make testacc TESTARGS='-run=TestAccAwsDmsEndpoint_'     
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./aws -v -count 1 -parallel 20 -run=TestAccAwsDmsEndpoint_ -timeout 180m
=== RUN   TestAccAwsDmsEndpoint_basic
=== PAUSE TestAccAwsDmsEndpoint_basic
=== RUN   TestAccAwsDmsEndpoint_S3
=== PAUSE TestAccAwsDmsEndpoint_S3
=== RUN   TestAccAwsDmsEndpoint_S3_ExtraConnectionAttributes
=== PAUSE TestAccAwsDmsEndpoint_S3_ExtraConnectionAttributes
=== RUN   TestAccAwsDmsEndpoint_DynamoDb
=== PAUSE TestAccAwsDmsEndpoint_DynamoDb
=== RUN   TestAccAwsDmsEndpoint_Elasticsearch
=== PAUSE TestAccAwsDmsEndpoint_Elasticsearch
=== RUN   TestAccAwsDmsEndpoint_Elasticsearch_ExtraConnectionAttributes
=== PAUSE TestAccAwsDmsEndpoint_Elasticsearch_ExtraConnectionAttributes
=== RUN   TestAccAwsDmsEndpoint_Elasticsearch_ErrorRetryDuration
=== PAUSE TestAccAwsDmsEndpoint_Elasticsearch_ErrorRetryDuration
=== RUN   TestAccAwsDmsEndpoint_Elasticsearch_FullLoadErrorPercentage
=== PAUSE TestAccAwsDmsEndpoint_Elasticsearch_FullLoadErrorPercentage
=== RUN   TestAccAwsDmsEndpoint_Kafka
=== PAUSE TestAccAwsDmsEndpoint_Kafka
=== RUN   TestAccAwsDmsEndpoint_Kinesis
=== PAUSE TestAccAwsDmsEndpoint_Kinesis
=== RUN   TestAccAwsDmsEndpoint_MongoDb
=== PAUSE TestAccAwsDmsEndpoint_MongoDb
=== RUN   TestAccAwsDmsEndpoint_MongoDb_Update
=== PAUSE TestAccAwsDmsEndpoint_MongoDb_Update
=== RUN   TestAccAwsDmsEndpoint_DocDB
=== PAUSE TestAccAwsDmsEndpoint_DocDB
=== RUN   TestAccAwsDmsEndpoint_Db2
=== PAUSE TestAccAwsDmsEndpoint_Db2
=== CONT  TestAccAwsDmsEndpoint_basic
=== CONT  TestAccAwsDmsEndpoint_Kafka
=== CONT  TestAccAwsDmsEndpoint_Elasticsearch_FullLoadErrorPercentage
=== CONT  TestAccAwsDmsEndpoint_Kinesis
=== CONT  TestAccAwsDmsEndpoint_S3
=== CONT  TestAccAwsDmsEndpoint_S3_ExtraConnectionAttributes
=== CONT  TestAccAwsDmsEndpoint_Elasticsearch_ErrorRetryDuration
=== CONT  TestAccAwsDmsEndpoint_Elasticsearch
=== CONT  TestAccAwsDmsEndpoint_Db2
=== CONT  TestAccAwsDmsEndpoint_DocDB
=== CONT  TestAccAwsDmsEndpoint_MongoDb_Update
=== CONT  TestAccAwsDmsEndpoint_MongoDb
=== CONT  TestAccAwsDmsEndpoint_Elasticsearch_ExtraConnectionAttributes
=== CONT  TestAccAwsDmsEndpoint_DynamoDb
--- PASS: TestAccAwsDmsEndpoint_MongoDb (56.06s)
--- PASS: TestAccAwsDmsEndpoint_Elasticsearch_FullLoadErrorPercentage (64.05s)
--- PASS: TestAccAwsDmsEndpoint_Elasticsearch_ExtraConnectionAttributes (64.23s)
--- PASS: TestAccAwsDmsEndpoint_Elasticsearch_ErrorRetryDuration (64.23s)
--- PASS: TestAccAwsDmsEndpoint_Kafka (73.13s)
--- PASS: TestAccAwsDmsEndpoint_DocDB (73.78s)
--- PASS: TestAccAwsDmsEndpoint_basic (73.95s)
--- PASS: TestAccAwsDmsEndpoint_Db2 (74.56s)
--- PASS: TestAccAwsDmsEndpoint_S3_ExtraConnectionAttributes (77.48s)
--- PASS: TestAccAwsDmsEndpoint_Elasticsearch (77.81s)
--- PASS: TestAccAwsDmsEndpoint_S3 (83.34s)
--- PASS: TestAccAwsDmsEndpoint_DynamoDb (83.80s)
--- PASS: TestAccAwsDmsEndpoint_MongoDb_Update (87.16s)
--- PASS: TestAccAwsDmsEndpoint_Kinesis (105.96s)
PASS
ok  	github.com/terraform-providers/terraform-provider-aws/aws	110.726s

GovCloud

% make testacc TESTARGS='-run=TestAccAwsDmsEndpoint_'
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./aws -v -count 1 -parallel 20 -run=TestAccAwsDmsEndpoint_ -timeout 180m
=== RUN   TestAccAwsDmsEndpoint_basic
=== PAUSE TestAccAwsDmsEndpoint_basic
=== RUN   TestAccAwsDmsEndpoint_S3
=== PAUSE TestAccAwsDmsEndpoint_S3
=== RUN   TestAccAwsDmsEndpoint_S3_ExtraConnectionAttributes
=== PAUSE TestAccAwsDmsEndpoint_S3_ExtraConnectionAttributes
=== RUN   TestAccAwsDmsEndpoint_DynamoDb
=== PAUSE TestAccAwsDmsEndpoint_DynamoDb
=== RUN   TestAccAwsDmsEndpoint_Elasticsearch
=== PAUSE TestAccAwsDmsEndpoint_Elasticsearch
=== RUN   TestAccAwsDmsEndpoint_Elasticsearch_ExtraConnectionAttributes
=== PAUSE TestAccAwsDmsEndpoint_Elasticsearch_ExtraConnectionAttributes
=== RUN   TestAccAwsDmsEndpoint_Elasticsearch_ErrorRetryDuration
=== PAUSE TestAccAwsDmsEndpoint_Elasticsearch_ErrorRetryDuration
=== RUN   TestAccAwsDmsEndpoint_Elasticsearch_FullLoadErrorPercentage
=== PAUSE TestAccAwsDmsEndpoint_Elasticsearch_FullLoadErrorPercentage
=== RUN   TestAccAwsDmsEndpoint_Kafka
=== PAUSE TestAccAwsDmsEndpoint_Kafka
=== RUN   TestAccAwsDmsEndpoint_Kinesis
=== PAUSE TestAccAwsDmsEndpoint_Kinesis
=== RUN   TestAccAwsDmsEndpoint_MongoDb
=== PAUSE TestAccAwsDmsEndpoint_MongoDb
=== RUN   TestAccAwsDmsEndpoint_MongoDb_Update
=== PAUSE TestAccAwsDmsEndpoint_MongoDb_Update
=== RUN   TestAccAwsDmsEndpoint_DocDB
=== PAUSE TestAccAwsDmsEndpoint_DocDB
=== RUN   TestAccAwsDmsEndpoint_Db2
=== PAUSE TestAccAwsDmsEndpoint_Db2
=== CONT  TestAccAwsDmsEndpoint_basic
=== CONT  TestAccAwsDmsEndpoint_DocDB
=== CONT  TestAccAwsDmsEndpoint_S3_ExtraConnectionAttributes
=== CONT  TestAccAwsDmsEndpoint_Elasticsearch
=== CONT  TestAccAwsDmsEndpoint_Elasticsearch_FullLoadErrorPercentage
=== CONT  TestAccAwsDmsEndpoint_Elasticsearch_ErrorRetryDuration
=== CONT  TestAccAwsDmsEndpoint_Elasticsearch_ExtraConnectionAttributes
=== CONT  TestAccAwsDmsEndpoint_Kinesis
=== CONT  TestAccAwsDmsEndpoint_MongoDb
=== CONT  TestAccAwsDmsEndpoint_Kafka
=== CONT  TestAccAwsDmsEndpoint_DynamoDb
=== CONT  TestAccAwsDmsEndpoint_S3
=== CONT  TestAccAwsDmsEndpoint_Db2
=== CONT  TestAccAwsDmsEndpoint_MongoDb_Update
--- PASS: TestAccAwsDmsEndpoint_Elasticsearch_ErrorRetryDuration (160.59s)
--- PASS: TestAccAwsDmsEndpoint_Elasticsearch_ExtraConnectionAttributes (162.16s)
--- PASS: TestAccAwsDmsEndpoint_MongoDb (162.45s)
--- PASS: TestAccAwsDmsEndpoint_Elasticsearch_FullLoadErrorPercentage (162.83s)
--- PASS: TestAccAwsDmsEndpoint_Elasticsearch (163.26s)
--- PASS: TestAccAwsDmsEndpoint_S3_ExtraConnectionAttributes (163.37s)
--- PASS: TestAccAwsDmsEndpoint_basic (170.87s)
--- PASS: TestAccAwsDmsEndpoint_Kafka (171.15s)
--- PASS: TestAccAwsDmsEndpoint_Db2 (171.43s)
--- PASS: TestAccAwsDmsEndpoint_DocDB (171.44s)
--- PASS: TestAccAwsDmsEndpoint_DynamoDb (180.72s)
--- PASS: TestAccAwsDmsEndpoint_S3 (181.96s)
--- PASS: TestAccAwsDmsEndpoint_MongoDb_Update (185.04s)
--- PASS: TestAccAwsDmsEndpoint_Kinesis (203.18s)
PASS
ok  	github.com/terraform-providers/terraform-provider-aws/aws	207.418s

ewbankkit · 2021-10-07T22:00:37Z

@bryantbiggs Thanks for the contribution 🎉 👏.

The "KMSKeyNotAccessibleFault: Key Id is missing" error was fixed by not sending the various Kafka SSL attributes if they are empty.

github-actions · 2021-10-08T00:58:40Z

This functionality has been released in v3.62.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

kduvekot-wehkamp-nl · 2021-10-08T05:01:39Z

@bryantbiggs @ewbankkit
Are these changes only available with a kafka output source? Or also for the Kinesis output source?

#18750 was intended for the Kinesis output source, so just doing a double check if I can now also use this for the Kinesis output.

edit: based on this page : https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/dms_endpoint#kinesis_settings
its not added for Kinesis output

ewbankkit · 2021-10-08T11:01:25Z

@kduvekot-wehkamp-nl My mistake 😄. Yes, this applies to the Kafka output source only.
I have re-opened your PR and we will get it reviewed.

bryantbiggs · 2021-10-08T11:23:37Z

thank you @ewbankkit !

bryantbiggs · 2021-10-08T16:43:18Z

hmm, @ewbankkit I think something in this ☝🏽 changeset introduced a bug - I'm now seeing errors like:

│ Error: 1 error occurred:
│ 	* s3_settings must be set when engine_name = "s3"
│ 
│   with module.dms_aurora_postgresql_aurora_mysql.aws_dms_endpoint.this["s3-source"],
│   on ../../main.tf line 141, in resource "aws_dms_endpoint" "this":
│  141: resource "aws_dms_endpoint" "this" {

this example worked all the way up to 3.61.0, and now throws this error with 3.62.0

here is where s3_settings is defined https://github.com/clowdhaus/terraform-aws-dms/blob/ec9134ce18a04a393274ab7c745ec90f1a45c158/examples/complete/main.tf#L370 and here is where its referenced in the resource https://github.com/clowdhaus/terraform-aws-dms/blob/ec9134ce18a04a393274ab7c745ec90f1a45c158/main.tf#L203

github-actions · 2022-06-16T02:35:02Z

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

breathingdust added enhancement Requests to existing resources that expand the functionality or scope. and removed needs-triage Waiting for first response or review from a maintainer. labels Sep 16, 2021

This was referenced Oct 7, 2021

Include additional kinesis-settings in dms endpoint #18750

Closed

Add support for additional kinesis_settings to dms_endpoint #20084

Merged

bryantbiggs and others added 9 commits October 7, 2021 13:41

chore: update Kafka endpoint settings to include all attributes

b8226c5

chore: update website documentation

0545074

Add CHANGELOG entry.

e869bb5

r/aws_dms_endpoint: Alphabetic order for structure fields.

4797dda

Add and use 'internal/service/dms/finder'.

0fa9ac9

'dmsEndpointDestroy' -> 'testAccCheckAWSDmsEndpointDestroy'.

b19cb21

'checkDmsEndpointExists' -> 'testAccCheckAWSDmsEndpointExists'.

35336c9

Add 'expandDmsKafkaSettings'.

84cc5bb

Add 'internal/service/dms/waiter'.

461b157

ewbankkit marked this pull request as ready for review October 7, 2021 21:30

ewbankkit added 2 commits October 7, 2021 17:45

r/aws_dms_endpoint: 'sasl_password' and 'ssl_client_key_password' are…

b81a9ed

… Sensitive.

Comment out 'testAccCheckResourceAttrHostnameWithPort' as it's curren…

883516f

…tly unused.

ewbankkit approved these changes Oct 7, 2021

View reviewed changes

ewbankkit merged commit 557b3c7 into hashicorp:main Oct 7, 2021

github-actions bot added this to the v3.62.0 milestone Oct 7, 2021

bryantbiggs deleted the chore/update-dms-endpoint-settings branch October 8, 2021 11:23

bryantbiggs mentioned this pull request Oct 8, 2021

feat: add new kafka_settings attributes for DMS endpoint, and add cdc_start_position to DMS replication task terraform-aws-modules/terraform-aws-dms#3

Merged

github-actions bot locked as resolved and limited conversation to collaborators Jun 16, 2022

justinretzolk added the partner Contribution from a partner. label May 16, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore: update Kafka endpoint settings to include all attributes #20904

chore: update Kafka endpoint settings to include all attributes #20904

bryantbiggs commented Sep 14, 2021 •

edited by ewbankkit

ewbankkit left a comment

ewbankkit commented Oct 7, 2021

github-actions bot commented Oct 8, 2021

kduvekot-wehkamp-nl commented Oct 8, 2021 •

edited

ewbankkit commented Oct 8, 2021

bryantbiggs commented Oct 8, 2021

bryantbiggs commented Oct 8, 2021 •

edited

github-actions bot commented Jun 16, 2022

chore: update Kafka endpoint settings to include all attributes #20904

chore: update Kafka endpoint settings to include all attributes #20904

Conversation

bryantbiggs commented Sep 14, 2021 • edited by ewbankkit

Community Note

ewbankkit left a comment

Choose a reason for hiding this comment

Commercial

GovCloud

ewbankkit commented Oct 7, 2021

github-actions bot commented Oct 8, 2021

kduvekot-wehkamp-nl commented Oct 8, 2021 • edited

ewbankkit commented Oct 8, 2021

bryantbiggs commented Oct 8, 2021

bryantbiggs commented Oct 8, 2021 • edited

github-actions bot commented Jun 16, 2022

bryantbiggs commented Sep 14, 2021 •

edited by ewbankkit

kduvekot-wehkamp-nl commented Oct 8, 2021 •

edited

bryantbiggs commented Oct 8, 2021 •

edited