Datastream Stream resource #6806

melinath · 2022-11-09T23:46:18Z

Initial implementation of datastream stream resource. Resolved hashicorp/terraform-provider-google#10810

This does not cover all the fields in this very large resource. In particular I'm excluding CMEK fields and oracle & postgres (because those require specific handwritten tests) and oracle & postgres support (because that requires separate, non-automated testing.)

If this PR is for Terraform, I acknowledge that I have:

Searched through the issue tracker for an open issue that this either resolves or contributes to, commented on it to claim it, and written "fixes {url}" or "part of {url}" in this PR description. If there were no relevant open issues, I opened one and commented that I would like to work on it (not necessary for very small changes).
Generated Terraform, and ran make test and make lint to ensure it passes unit and linter tests.
Ensured that all new fields I added that can be set by a user appear in at least one example (for generated resources) or third_party test (for handwritten resources or update tests).
Ran relevant acceptance tests (If the acceptance tests do not yet pass or you are unable to run them, please let your reviewer know).
Read the Release Notes Guide before writing my release note below.

Release Note Template for Downstream PRs (will be copied)

`google_datastream_stream`

modular-magician · 2022-11-10T00:17:18Z

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 9 files changed, 5319 insertions(+), 6 deletions(-))
Terraform Beta: Diff ( 9 files changed, 5319 insertions(+), 6 deletions(-))
TF Validator: Diff ( 3 files changed, 1643 insertions(+), 3 deletions(-))
TF OiCS: Diff ( 8 files changed, 508 insertions(+))

modular-magician · 2022-11-10T00:56:09Z

Tests analytics

Total tests: 2237
Passed tests 1993
Skipped tests: 241
Failed tests: 3

Action taken

Triggering VCR tests in RECORDING mode for the tests that failed during VCR. Click here to see the failed tests

TestAccDatastreamStream_datastreamStreamFullExample|TestAccDatastreamStream_datastreamStreamBasicExample|TestAccComputeForwardingRule_update

modular-magician · 2022-11-10T00:57:29Z

Tests passed during RECORDING mode:
TestAccComputeForwardingRule_update[Debug log]

Tests failed during RECORDING mode:
TestAccDatastreamStream_datastreamStreamFullExample[Error message] [Debug log]
TestAccDatastreamStream_datastreamStreamBasicExample[Error message] [Debug log]

Please fix these to complete your PR
View the build log or the debug log for each test

mmv1/products/datastream/api.yaml

modular-magician · 2022-11-10T17:43:11Z

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 9 files changed, 7205 insertions(+), 6 deletions(-))
Terraform Beta: Diff ( 9 files changed, 7205 insertions(+), 6 deletions(-))
TF Validator: Diff ( 3 files changed, 2330 insertions(+), 3 deletions(-))
TF OiCS: Diff ( 8 files changed, 508 insertions(+))

modular-magician · 2022-11-10T18:23:35Z

Tests analytics

Total tests: 2237
Passed tests 1992
Skipped tests: 241
Failed tests: 4

Action taken

Triggering VCR tests in RECORDING mode for the tests that failed during VCR. Click here to see the failed tests

TestAccFirebaserulesRelease_BasicRelease|TestAccDatastreamStream_datastreamStreamFullExample|TestAccDatastreamStream_datastreamStreamBasicExample|TestAccComputeForwardingRule_update

modular-magician · 2022-11-10T18:25:00Z

Tests passed during RECORDING mode:
TestAccFirebaserulesRelease_BasicRelease[Debug log]
TestAccComputeForwardingRule_update[Debug log]

Tests failed during RECORDING mode:
TestAccDatastreamStream_datastreamStreamFullExample[Error message] [Debug log]
TestAccDatastreamStream_datastreamStreamBasicExample[Error message] [Debug log]

Please fix these to complete your PR
View the build log or the debug log for each test

mmv1/products/datastream/api.yaml

modular-magician · 2022-11-14T17:16:26Z

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 9 files changed, 7254 insertions(+), 6 deletions(-))
Terraform Beta: Diff ( 9 files changed, 7254 insertions(+), 6 deletions(-))
TF Validator: Diff ( 3 files changed, 2340 insertions(+), 3 deletions(-))
TF OiCS: Diff ( 8 files changed, 509 insertions(+))

modular-magician · 2022-11-14T18:03:28Z

Tests analytics

Total tests: 2253
Passed tests 2010
Skipped tests: 241
Failed tests: 2

Action taken

Triggering VCR tests in RECORDING mode for the tests that failed during VCR. Click here to see the failed tests

TestAccDatastreamStream_datastreamStreamFullExample|TestAccDatastreamStream_datastreamStreamBasicExample

modular-magician · 2022-11-14T18:03:41Z

Tests failed during RECORDING mode:
TestAccDatastreamStream_datastreamStreamFullExample[Error message] [Debug log]
TestAccDatastreamStream_datastreamStreamBasicExample[Error message] [Debug log]

Please fix these to complete your PR
View the build log or the debug log for each test

modular-magician · 2022-11-22T23:40:02Z

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 9 files changed, 3732 insertions(+), 6 deletions(-))
Terraform Beta: Diff ( 9 files changed, 3732 insertions(+), 6 deletions(-))
TF Validator: Diff ( 3 files changed, 1041 insertions(+), 3 deletions(-))
TF OiCS: Diff ( 8 files changed, 509 insertions(+))

modular-magician · 2022-11-23T00:29:55Z

Tests analytics

Total tests: 2283
Passed tests 2034
Skipped tests: 244
Failed tests: 5

Action taken

Triggering VCR tests in RECORDING mode for the tests that failed during VCR. Click here to see the failed tests

TestAccLoggingBucketConfigProject_cmekSettings|TestAccDatastreamStream_datastreamStreamFullExample|TestAccDatastreamStream_datastreamStreamBasicExample|TestAccFirebaserulesRelease_BasicRelease|TestAccComputeForwardingRule_update

modular-magician · 2022-11-23T00:31:11Z

Tests passed during RECORDING mode:
TestAccLoggingBucketConfigProject_cmekSettings[Debug log]
TestAccFirebaserulesRelease_BasicRelease[Debug log]
TestAccComputeForwardingRule_update[Debug log]

Tests failed during RECORDING mode:
TestAccDatastreamStream_datastreamStreamFullExample[Error message] [Debug log]
TestAccDatastreamStream_datastreamStreamBasicExample[Error message] [Debug log]

Please fix these to complete your PR
View the build log or the debug log for each test

modular-magician · 2022-11-29T19:40:48Z

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 11 files changed, 4086 insertions(+), 7 deletions(-))
Terraform Beta: Diff ( 11 files changed, 4086 insertions(+), 7 deletions(-))
TF Validator: Diff ( 4 files changed, 1052 insertions(+), 4 deletions(-))
TF OiCS: Diff ( 8 files changed, 509 insertions(+))

modular-magician · 2022-11-29T20:19:14Z

Tests analytics

Total tests: 2305
Passed tests 2053
Skipped tests: 244
Failed tests: 8

Action taken

Triggering VCR tests in RECORDING mode for the tests that failed during VCR. Click here to see the failed tests

TestAccFirebaserulesRelease_BasicRelease|TestAccDatastreamStream_update|TestAccDatastreamStream_datastreamStreamFullExample|TestAccDatastreamStream_datastreamStreamBasicExample|TestAccCloudFunctions2Function_fullUpdate|TestAccCloudfunctions2function_cloudfunctions2BasicAuditlogsExample|TestAccCloudfunctions2function_cloudfunctions2BasicGcsExample|TestAccLoggingBucketConfigProject_cmekSettings

…desired_state

modular-magician · 2022-12-20T17:53:44Z

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 12 files changed, 4142 insertions(+), 7 deletions(-))
Terraform Beta: Diff ( 12 files changed, 4142 insertions(+), 7 deletions(-))
TF Validator: Diff ( 4 files changed, 1052 insertions(+), 4 deletions(-))
TF OiCS: Diff ( 9 files changed, 513 insertions(+))

modular-magician · 2022-12-20T18:24:00Z

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 12 files changed, 4142 insertions(+), 7 deletions(-))
Terraform Beta: Diff ( 12 files changed, 4142 insertions(+), 7 deletions(-))
TF Validator: Diff ( 4 files changed, 1052 insertions(+), 4 deletions(-))
TF OiCS: Diff ( 9 files changed, 513 insertions(+))

modular-magician · 2022-12-20T19:11:28Z

Tests analytics

Total tests: 2359
Passed tests 2107
Skipped tests: 247
Failed tests: 5

Action taken

Triggering VCR tests in RECORDING mode for the tests that failed during VCR. Click here to see the failed tests

TestAccDatastreamStream_update|TestAccDatastreamStream_datastreamStreamFullExample|TestAccDatastreamStream_datastreamStreamBasicExample|TestAccContainerCluster_withInvalidGatewayApiConfigChannel|TestAccLoggingBucketConfigProject_cmekSettings

modular-magician · 2022-12-20T19:42:57Z

Tests passed during RECORDING mode:
TestAccDatastreamStream_datastreamStreamFullExample[Debug log]
TestAccDatastreamStream_datastreamStreamBasicExample[Debug log]
TestAccContainerCluster_withInvalidGatewayApiConfigChannel[Debug log]
TestAccLoggingBucketConfigProject_cmekSettings[Debug log]

Tests failed during RECORDING mode:
TestAccDatastreamStream_update[Error message] [Debug log]

Please fix these to complete your PR
View the build log or the debug log for each test

juancalvof · 2023-01-02T09:51:04Z

Looking forward to this merge. Thanks for the effort! :)

melinath · 2023-01-03T17:45:47Z

/gcbrun

modular-magician · 2023-01-03T18:02:03Z

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 12 files changed, 4142 insertions(+), 7 deletions(-))
Terraform Beta: Diff ( 12 files changed, 4142 insertions(+), 7 deletions(-))
TF Validator: Diff ( 4 files changed, 1052 insertions(+), 4 deletions(-))
TF OiCS: Diff ( 9 files changed, 513 insertions(+))

modular-magician · 2023-01-03T18:54:44Z

Tests analytics

Total tests: 2385
Passed tests 2132
Skipped tests: 248
Failed tests: 5

Action taken

Triggering VCR tests in RECORDING mode for the tests that failed during VCR. Click here to see the failed tests

TestAccLoggingBucketConfigProject_cmekSettings|TestAccDatastreamStream_update|TestAccDatastreamStream_datastreamStreamFullExample|TestAccDatastreamStream_datastreamStreamBasicExample|TestAccContainerCluster_withInvalidGatewayApiConfigChannel

modular-magician · 2023-01-03T19:14:38Z

Tests passed during RECORDING mode:
TestAccLoggingBucketConfigProject_cmekSettings[Debug log]
TestAccDatastreamStream_datastreamStreamFullExample[Debug log]
TestAccDatastreamStream_datastreamStreamBasicExample[Debug log]
TestAccContainerCluster_withInvalidGatewayApiConfigChannel[Debug log]

Tests failed during RECORDING mode:
TestAccDatastreamStream_update[Error message] [Debug log]

Please fix these to complete your PR
View the build log or the debug log for each test

modular-magician · 2023-01-04T17:41:15Z

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 12 files changed, 4155 insertions(+), 7 deletions(-))
Terraform Beta: Diff ( 12 files changed, 4155 insertions(+), 7 deletions(-))
TF Validator: Diff ( 4 files changed, 1052 insertions(+), 4 deletions(-))
TF OiCS: Diff ( 9 files changed, 513 insertions(+))

modular-magician · 2023-01-04T18:28:21Z

Tests analytics

Total tests: 2389
Passed tests 2135
Skipped tests: 248
Failed tests: 6

Action taken

Triggering VCR tests in RECORDING mode for the tests that failed during VCR. Click here to see the failed tests

TestAccFirebaserulesRelease_BasicRelease|TestAccDatastreamStream_datastreamStreamFullExample|TestAccDatastreamStream_datastreamStreamBasicExample|TestAccDatastreamStream_update|TestAccContainerCluster_withInvalidGatewayApiConfigChannel|TestAccComputeForwardingRule_update

modular-magician · 2023-01-04T18:52:01Z

Tests passed during RECORDING mode:
TestAccFirebaserulesRelease_BasicRelease[Debug log]
TestAccDatastreamStream_datastreamStreamFullExample[Debug log]
TestAccDatastreamStream_datastreamStreamBasicExample[Debug log]
TestAccDatastreamStream_update[Debug log]
TestAccContainerCluster_withInvalidGatewayApiConfigChannel[Debug log]
TestAccComputeForwardingRule_update[Debug log]

All tests passed
View the build log or the debug log for each test

melinath · 2023-01-04T22:04:16Z

@shuyama1 This is now ready for review!

* Added initial datastream Stream structure * Added basic test for datastream * Removed oracle support since it's hard to test * Added some additional fields that could not previously be handled * Added full example for datastream stream with MySQL -> GCS * Removed additional references to oracle on datastream stream * cleaned up api.yaml and terraform.yaml * Revert "Removed additional references to oracle on datastream stream" This reverts commit 6ce0d70. * Revert "Removed oracle support since it's hard to test" This reverts commit 732cf30. * Added oracleSourceConfig.dropLargeObjects and oracleSourceConfig.streamLargeObjects * Require a value inside jsonFileFormat * Require a value inside bigqueryDestinationConfig * Corrected type of fileRotationMb * Added state field and added default_from_api to a few fields * Fixed typo * Corrected name of default_value property * Removed support for oracle and postgres from Datastream Stream This simplifies the PR; also they are not automatically testable * Converted status field to be computed We will likely want to use either default_from_api or use a desired_state field because this field is only partially specified * Added desired_state field * removed input: true from stream * ignore read on stream connection profiles * Added update test * Added unit test for customizediff * Added unit tests for customize diff * Added external providers * gofmt * Corrected deletion_protection usage * Updates due to code review * Cleaned up test issues * Corrections to tests * Fixed test issues * Added gcs file format to fix tests * Added projectNumberDiffSuppress * Added default_from_api for fileRotationMb * Corrected labels usage in tests * Corrected stream update verb * Cleaned up references to google_storage_bucket.bucket.name * Source / destination profile are not updatable * Marked column length fields as output-only * Made update test use checks instead of importstateverify for state / desired_state * gofmt * Add state to update mask if desired_state is changed

melinath force-pushed the b-233804820-stream branch from 1c9f83a to f76f865 Compare November 10, 2022 00:02

iperetz-goo reviewed Nov 10, 2022

View reviewed changes

mmv1/products/datastream/api.yaml Outdated Show resolved Hide resolved

mmv1/products/datastream/api.yaml Outdated Show resolved Hide resolved

mmv1/products/datastream/api.yaml Outdated Show resolved Hide resolved

mmv1/products/datastream/api.yaml Show resolved Hide resolved

iperetz-goo suggested changes Nov 10, 2022

View reviewed changes

mmv1/products/datastream/api.yaml Show resolved Hide resolved

mmv1/products/datastream/api.yaml Show resolved Hide resolved

melinath requested a review from iperetz-goo November 10, 2022 17:28

iperetz-goo suggested changes Nov 10, 2022

View reviewed changes

mmv1/products/datastream/api.yaml Outdated Show resolved Hide resolved

iperetz-goo suggested changes Nov 14, 2022

View reviewed changes

mmv1/products/datastream/api.yaml Outdated Show resolved Hide resolved

melinath force-pushed the b-233804820-stream branch from fda902f to b4e4be7 Compare November 22, 2022 23:24

melinath force-pushed the b-233804820-stream branch from b4e4be7 to 03e204f Compare November 29, 2022 19:25

melinath requested review from a team, c2thorn, shuyama1 and iperetz-goo and removed request for a team and c2thorn November 29, 2022 19:28

Made update test use checks instead of importstateverify for state / …

95e9175

…desired_state

gofmt

75ae319

Add state to update mask if desired_state is changed

b2719d8

shuyama1 approved these changes Jan 4, 2023

View reviewed changes

melinath merged commit ba93120 into main Jan 4, 2023

melinath deleted the b-233804820-stream branch January 4, 2023 22:36

iperetz-goo mentioned this pull request Jan 12, 2023

Finish missing sources from Datastream Stream #7115

Closed

5 tasks

This was referenced Jan 26, 2023

Finish datastream #7174

Closed

Finish missing sources from Datastream Stream #7175

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Datastream Stream resource #6806

Datastream Stream resource #6806

melinath commented Nov 9, 2022 •

edited

modular-magician commented Nov 10, 2022

modular-magician commented Nov 10, 2022

modular-magician commented Nov 10, 2022

modular-magician commented Nov 10, 2022

modular-magician commented Nov 10, 2022

modular-magician commented Nov 10, 2022

modular-magician commented Nov 14, 2022

modular-magician commented Nov 14, 2022

modular-magician commented Nov 14, 2022

modular-magician commented Nov 22, 2022

modular-magician commented Nov 23, 2022

modular-magician commented Nov 23, 2022

modular-magician commented Nov 29, 2022

modular-magician commented Nov 29, 2022

modular-magician commented Dec 20, 2022

modular-magician commented Dec 20, 2022

modular-magician commented Dec 20, 2022

modular-magician commented Dec 20, 2022

juancalvof commented Jan 2, 2023

melinath commented Jan 3, 2023

modular-magician commented Jan 3, 2023

modular-magician commented Jan 3, 2023

modular-magician commented Jan 3, 2023

modular-magician commented Jan 4, 2023

modular-magician commented Jan 4, 2023

modular-magician commented Jan 4, 2023

melinath commented Jan 4, 2023

Datastream Stream resource #6806

Datastream Stream resource #6806

Conversation

melinath commented Nov 9, 2022 • edited

modular-magician commented Nov 10, 2022

Diff report

modular-magician commented Nov 10, 2022

Tests analytics

Action taken

modular-magician commented Nov 10, 2022

modular-magician commented Nov 10, 2022

Diff report

modular-magician commented Nov 10, 2022

Tests analytics

Action taken

modular-magician commented Nov 10, 2022

modular-magician commented Nov 14, 2022

Diff report

modular-magician commented Nov 14, 2022

Tests analytics

Action taken

modular-magician commented Nov 14, 2022

modular-magician commented Nov 22, 2022

Diff report

modular-magician commented Nov 23, 2022

Tests analytics

Action taken

modular-magician commented Nov 23, 2022

modular-magician commented Nov 29, 2022

Diff report

modular-magician commented Nov 29, 2022

Tests analytics

Action taken

modular-magician commented Dec 20, 2022

Diff report

modular-magician commented Dec 20, 2022

Diff report

modular-magician commented Dec 20, 2022

Tests analytics

Action taken

modular-magician commented Dec 20, 2022

juancalvof commented Jan 2, 2023

melinath commented Jan 3, 2023

modular-magician commented Jan 3, 2023

Diff report

modular-magician commented Jan 3, 2023

Tests analytics

Action taken

modular-magician commented Jan 3, 2023

modular-magician commented Jan 4, 2023

Diff report

modular-magician commented Jan 4, 2023

Tests analytics

Action taken

modular-magician commented Jan 4, 2023

melinath commented Jan 4, 2023

melinath commented Nov 9, 2022 •

edited