Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

r/aws_kinesis_firehose_delivery_stream: Set 'kinesis_source_configuration' in Read function #13536

Merged
merged 4 commits into from Jun 4, 2020
Merged

r/aws_kinesis_firehose_delivery_stream: Set 'kinesis_source_configuration' in Read function #13536

merged 4 commits into from Jun 4, 2020

Conversation

ewbankkit
Copy link
Contributor

@ewbankkit ewbankkit commented May 28, 2020 •
edited

Community Note

  • Please vote on this pull request by adding a 👍 reaction to the original pull request comment to help the community and maintainers prioritize this request
  • Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for pull request followers and do not help prioritize the request

Closes #13530.

Also:

  • Refactored to use SDK-defined constant strings
  • Added a _disappears acceptance test

Release note for CHANGELOG:

resource/aws_kinesis_firehose_delivery_stream: `kinesis_source_configuration` is now correctly set during import

Output from acceptance testing:

$ make testacc TEST=./aws/ TESTARGS='-run=TestAccAWSKinesisFirehoseDeliveryStream_'
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./aws/ -v -count 1 -parallel 20 -run=TestAccAWSKinesisFirehoseDeliveryStream_ -timeout 120m
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_basic
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_basic
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_disappears
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_disappears
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_s3basic
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_s3basic
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_s3basicWithSSE
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_s3basicWithSSE
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_s3basicWithTags
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_s3basicWithTags
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_s3KinesisStreamSource
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_s3KinesisStreamSource
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_s3WithCloudwatchLogging
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_s3WithCloudwatchLogging
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_s3ConfigUpdates
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_s3ConfigUpdates
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3basic
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3basic
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_Enabled
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_Enabled
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_ExternalUpdate
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_ExternalUpdate
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_Deserializer_Update
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_Deserializer_Update
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_HiveJsonSerDe_Empty
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_HiveJsonSerDe_Empty
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_OpenXJsonSerDe_Empty
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_OpenXJsonSerDe_Empty
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_OrcSerDe_Empty
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_OrcSerDe_Empty
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_ParquetSerDe_Empty
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_ParquetSerDe_Empty
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_Serializer_Update
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_Serializer_Update
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_ErrorOutputPrefix
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_ErrorOutputPrefix
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_ProcessingConfiguration_Empty
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_ProcessingConfiguration_Empty
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3KmsKeyArn
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3KmsKeyArn
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3Updates
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3Updates
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_KinesisStreamSource
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_KinesisStreamSource
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_RedshiftConfigUpdates
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_RedshiftConfigUpdates
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_SplunkConfigUpdates
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_SplunkConfigUpdates
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchConfigUpdates
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchConfigUpdates
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_missingProcessingConfiguration
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_missingProcessingConfiguration
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_basic
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_OrcSerDe_Empty
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_OpenXJsonSerDe_Empty
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_missingProcessingConfiguration
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_HiveJsonSerDe_Empty
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchConfigUpdates
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_Deserializer_Update
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_SplunkConfigUpdates
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_ExternalUpdate
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_RedshiftConfigUpdates
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_Enabled
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_KinesisStreamSource
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3Updates
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3basic
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3KmsKeyArn
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_s3ConfigUpdates
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_ProcessingConfiguration_Empty
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_ErrorOutputPrefix
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_Serializer_Update
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_ParquetSerDe_Empty
--- FAIL: TestAccAWSKinesisFirehoseDeliveryStream_RedshiftConfigUpdates (32.85s)
    testing.go:684: Step 0 error: errors during apply:
        
        Error: InvalidVPCNetworkStateFault: Cannot create a publicly accessible cluster because customer VPC vpc-b6fbb6cf has no internet gateway attached.
        	status code: 400, request id: 160e2744-9762-4284-be7c-caa991e3df3d
        
          on /tmp/tf-test787880588/main.tf line 74:
          (source code not available)
        
        
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_s3WithCloudwatchLogging
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_KinesisStreamSource (117.35s)
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_s3KinesisStreamSource
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_ProcessingConfiguration_Empty (135.94s)
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_s3basicWithTags
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_Serializer_Update (141.89s)
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_s3basicWithSSE
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_HiveJsonSerDe_Empty (143.15s)
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_s3basic
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_missingProcessingConfiguration (145.42s)
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_disappears
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_ParquetSerDe_Empty (145.72s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_basic (146.48s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_Deserializer_Update (152.85s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3KmsKeyArn (154.51s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_OrcSerDe_Empty (155.72s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3basic (161.03s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_ErrorOutputPrefix (165.37s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_ExternalUpdate (167.92s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_Enabled (169.97s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_s3WithCloudwatchLogging (139.57s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_OpenXJsonSerDe_Empty (173.16s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_SplunkConfigUpdates (191.12s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3Updates (205.08s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_s3KinesisStreamSource (111.79s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_s3basic (99.85s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_s3ConfigUpdates (247.44s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_disappears (123.66s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_s3basicWithTags (183.28s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_s3basicWithSSE (288.73s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchConfigUpdates (891.78s)
FAIL
FAIL	github.com/terraform-providers/terraform-provider-aws/aws	891.871s
FAIL
GNUmakefile:26: recipe for target 'testacc' failed
make: *** [testacc] Error 1

The failure is not related to this change - My default VPC has no attached IGW.
I'll open a tech-debt issue to add an acceptance test PreCheck for that - #13537.

@ewbankkit ewbankkit requested a review from a team May 28, 2020 14:48
@ghost ghost added size/L Managed by automation to categorize the size of a PR. service/firehose Issues and PRs that pertain to the firehose service. tests PRs: expanded test coverage. Issues: expanded coverage, enhancements to test infrastructure. needs-triage Waiting for first response or review from a maintainer. labels May 28, 2020
@ewbankkit ewbankkit mentioned this pull request May 28, 2020
Closed
@ewbankkit
Copy link
Contributor Author

ewbankkit commented May 28, 2020 •
edited

I'll fix the Redshift test case to make it independent of the default VPC.

Changed Redshift cluster to run in-VPC.
The corresponding test now passes:

$ make testacc TEST=./aws/ TESTARGS='-run=TestAccAWSKinesisFirehoseDeliveryStream_RedshiftConfigUpdates'
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./aws/ -v -count 1 -parallel 20 -run=TestAccAWSKinesisFirehoseDeliveryStream_RedshiftConfigUpdates -timeout 120m
=== RUN   TestAccAWSKinesisFirehoseDeliveryStream_RedshiftConfigUpdates
=== PAUSE TestAccAWSKinesisFirehoseDeliveryStream_RedshiftConfigUpdates
=== CONT  TestAccAWSKinesisFirehoseDeliveryStream_RedshiftConfigUpdates
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_RedshiftConfigUpdates (503.48s)
PASS
ok  	github.com/terraform-providers/terraform-provider-aws/aws	503.517s

@ghost ghost added size/XL Managed by automation to categorize the size of a PR. and removed size/L Managed by automation to categorize the size of a PR. labels May 29, 2020
@romankydybets
Copy link

Could anyone take a look of this PR ?

@romankydybets
Copy link

@ewbankkit who could help with this approve to merge?

@anGie44 anGie44 removed the needs-triage Waiting for first response or review from a maintainer. label Jun 4, 2020
anGie44
anGie44 approved these changes Jun 4, 2020
View reviewed changes
Copy link
Contributor

@anGie44 anGie44 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this fix @ewbankkit ! Validations, setting of attribute, and tests looks great 👍

Output of acceptance tests:

--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_s3KinesisStreamSource (85.94s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_OpenXJsonSerDe_Empty (97.74s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_s3WithCloudwatchLogging (97.89s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_OrcSerDe_Empty (98.15s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_s3basic (98.85s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_ErrorOutputPrefix (102.76s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_disappears (107.86s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_ParquetSerDe_Empty (108.90s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3KmsKeyArn (118.96s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_ExternalUpdate (125.20s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_ProcessingConfiguration_Empty (127.74s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_basic (130.87s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3basic (135.62s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_Enabled (136.33s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_Serializer_Update (138.41s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_HiveJsonSerDe_Empty (141.10s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_s3basicWithTags (144.11s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_DataFormatConversionConfiguration_Deserializer_Update (145.91s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3_KinesisStreamSource (84.20s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_s3ConfigUpdates (190.31s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_missingProcessingConfiguration (99.62s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_s3basicWithSSE (202.28s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ExtendedS3Updates (130.34s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_SplunkConfigUpdates (123.12s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_RedshiftConfigUpdates (475.88s)
--- PASS: TestAccAWSKinesisFirehoseDeliveryStream_ElasticsearchConfigUpdates (854.05s)

@anGie44 anGie44 merged commit 69de6dd into hashicorp:master Jun 4, 2020
@anGie44 anGie44 added this to the v2.65.0 milestone Jun 4, 2020
anGie44 added a commit that referenced this pull request Jun 4, 2020
@anGie44
Update CHANGELOG for #13536
13b7a83
anGie44 added a commit that referenced this pull request Jun 4, 2020
@anGie44
Update CHANGELOG for #13536
d7e87ee 
Add GH link to PR
@ewbankkit ewbankkit deleted the b-kinesis-firehose-import-kinesis_source_configuration branch June 4, 2020 18:41
@ghost
Copy link

ghost commented Jun 5, 2020

This has been released in version 2.65.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template for triage. Thanks!

@ghost
Copy link

ghost commented Jul 5, 2020

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!

@hashicorp hashicorp locked and limited conversation to collaborators Jul 5, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@anGie44 anGie44 anGie44 approved these changes

Assignees
No one assigned
Labels
service/firehose Issues and PRs that pertain to the firehose service. size/XL Managed by automation to categorize the size of a PR. tests PRs: expanded test coverage. Issues: expanded coverage, enhancements to test infrastructure.
Projects
None yet
Milestone
v2.65.0
Development

Successfully merging this pull request may close these issues.

kinesis-firehose recreates during import
3 participants
@ewbankkit @romankydybets @anGie44