Added customizeDiff for a couple of subfields of params field in google_bigquery_data_transfer_config

[AlfatahB]

Added customizeDiff for data_path_template and destination_table_name_template subfields of params field in google_bigquery_data_transfer_config
fixes hashicorp/terraform-provider-google#6992

If this PR is for Terraform, I acknowledge that I have:

• Searched through the issue tracker for an open issue that this either resolves or contributes to, commented on it to claim it, and written "fixes {url}" or "part of {url}" in this PR description. If there were no relevant open issues, I opened one and commented that I would like to work on it (not necessary for very small changes).
• Generated Terraform, and ran make test and make lint to ensure it passes unit and linter tests.
• Ensured that all new fields I added that can be set by a user appear in at least one example (for generated resources) or third_party test (for handwritten resources or update tests).
• Ran relevant acceptance tests (If the acceptance tests do not yet pass or you are unable to run them, please let your reviewer know).
• Read the Release Notes Guide before writing my release note below.

Release Note Template for Downstream PRs (will be copied)

bigquerydatatransfer: recreate `google_bigquery_data_transfer_config` for Cloud Storage transfers when immutable params `data_path_template` and `destination_table_name_template` are changed

[modular-magician]

Hello! I am a robot who works on Magic Modules PRs.

I've detected that you're a community contributor. @melinath, a repository maintainer, has been assigned to assist you and help review your changes.

❓ First time contributing? Click here for more details

---

Your assigned reviewer will help review your code by:

• Ensuring it's backwards compatible, covers common error cases, etc.
• Summarizing the change into a user-facing changelog note.
• Passes tests, either our "VCR" suite, a set of presubmit tests, or with manual test runs.

You can help make sure that review is quick by running local tests and ensuring they're passing in between each push you make to your PR's branch. Also, try to leave a comment with each push you make, as pushes generally don't generate emails.

If your reviewer doesn't get back to you within a week after your most recent change, please feel free to leave a comment on the issue asking them to take a look! In the absence of a dedicated review dashboard most maintainers manage their pending reviews through email, and those will sometimes get lost in their inbox.

---

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 2 files changed, 93 insertions(+), 1 deletion(-))
Terraform Beta: Diff ( 2 files changed, 93 insertions(+), 1 deletion(-))
TF Validator: Diff ( 3 files changed, 28 insertions(+), 3 deletions(-))

[modular-magician]

Tests analytics

Total tests: 2216
Passed tests 1976
Skipped tests: 239
Failed tests: 1

Action taken

Triggering VCR tests in RECORDING mode for the tests that failed during VCR. Click here to see the failed tests

TestAccComputeForwardingRule_update

[modular-magician]

Tests passed during RECORDING mode:
TestAccComputeForwardingRule_update[Debug log]

All tests passed
View the build log or the debug log for each test

[AlfatahB]

I've tried to modify the value of the destination_table_name_template field. I was getting the error stating Immutable parameter destination_table_name_template with value string_value: \t \"the-table\"\n cannot be changed to string_value: \t \"the-table-new\"\n as shown in the attached screenshot. This was the same error I was getting while changing the value of data_path_template. Hence, I used customizeDiff to call diff.ForceNew("params") if either of them have changed in value.

[melinath]

assigning to @roaks3 to do a preliminary review

[roaks3]

This PR looks good to me, but two comments:

• I noticed that this looks to be the same implementation that was previously reverted. I see the request/response that seems to confirm that destination_table_name_template is immutable, but I'm wondering why we believed it was not when we reverted. Is there any way we can confirm that this won't re-trigger a case where we need to revert?
• Since we are implementing a particular ForceNew behavior here, I think it would be best to include a simple unit test like this: https://github.com/GoogleCloudPlatform/magic-modules/blob/main/mmv1/third_party/terraform/tests/resource_spanner_database_test.go.erb#L375-L450. That way we can confirm these two fields trigger a ForceNew, and the other fields within params do not. However, since the previous behavior here was a failure if you tried to update, I don't think it's strictly necessary.

[melinath]

I'm wondering why we believed it was not when we reverted.

@c2thorn may have context related to this.

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 2 files changed, 177 insertions(+), 1 deletion(-))
Terraform Beta: Diff ( 2 files changed, 177 insertions(+), 1 deletion(-))
TF Validator: Diff ( 3 files changed, 32 insertions(+), 3 deletions(-))

[modular-magician]

Tests analytics

Total tests: 2235
Passed tests 1992
Skipped tests: 241
Failed tests: 2

Action taken

Triggering VCR tests in RECORDING mode for the tests that failed during VCR. Click here to see the failed tests

TestAccFirebaserulesRelease_BasicRelease|TestAccComputeForwardingRule_update

[modular-magician]

Tests passed during RECORDING mode:
TestAccFirebaserulesRelease_BasicRelease[Debug log]
TestAccComputeForwardingRule_update[Debug log]

All tests passed
View the build log or the debug log for each test

[AlfatahB]

• Since we are implementing a particular ForceNew behavior here, I think it would be best to include a simple unit test like this: https://github.com/GoogleCloudPlatform/magic-modules/blob/main/mmv1/third_party/terraform/tests/resource_spanner_database_test.go.erb#L375-L450. That way we can confirm these two fields trigger a ForceNew, and the other fields within params do not. However, since the previous behavior here was a failure if you tried to update, I don't think it's strictly necessary.

I have added the unit tests for the particular ForceNew behaviour.

[c2thorn]

I'm wondering why we believed it was not when we reverted.

@c2thorn may have context related to this.

I'm still able to update this field successfully via the provider. I believe this field may have conditional mutability. I've attached my logs at b/251556517

[AlfatahB]

Thanks @c2thorn for the inputs. I've found out that if data_source_id is set to "google_cloud_storage" than data_path_template and destination_table_name_template are immutable fields. So, I've added a condition in customizeDiff to check if the value of data_source_id is set to "google_cloud_storage" then we have to call diff.ForceNew("params").

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 2 files changed, 201 insertions(+), 1 deletion(-))
Terraform Beta: Diff ( 2 files changed, 201 insertions(+), 1 deletion(-))
TF Validator: Diff ( 3 files changed, 36 insertions(+), 3 deletions(-))

[modular-magician]

Tests analytics

Total tests: 2249
Passed tests 2007
Skipped tests: 241
Failed tests: 1

Action taken

Triggering VCR tests in RECORDING mode for the tests that failed during VCR. Click here to see the failed tests

TestAccFirebaserulesRelease_BasicRelease

[modular-magician]

Tests passed during RECORDING mode:
TestAccFirebaserulesRelease_BasicRelease[Debug log]

All tests passed
View the build log or the debug log for each test

[roaks3]

For reference, I did find this line in the docs that appears to confirm the limitation with Cloud Storage (at least for destination_table):

Unlike individual loads of data from Cloud Storage to BigQuery, for ongoing transfers you need to create the destination table and its schema in advance of setting up the transfer. BigQuery cannot create the table as part of the recurring data transfer process.

I think this is definitely service-specific logic that we prefer not to re-implement in the provider, but I'm unsure of a better way to solve the problem. For that reason, I believe this custom ForceNew logic is our best option, and moving forward we would need to keep an eye out for other data sources that need similar adjustments.

Adding Stephen for follow-up review.

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 2 files changed, 221 insertions(+), 1 deletion(-))
Terraform Beta: Diff ( 2 files changed, 221 insertions(+), 1 deletion(-))
TF Validator: Diff ( 3 files changed, 37 insertions(+), 3 deletions(-))

[modular-magician]

Tests analytics

Total tests: 2277
Passed tests 2023
Skipped tests: 244
Failed tests: 10

Action taken

Triggering VCR tests in RECORDING mode for the tests that failed during VCR. Click here to see the failed tests

TestAccLoggingBucketConfigProject_cmekSettings|TestAccComputeGlobalForwardingRule_globalForwardingRuleInternalExample|TestAccComputeGlobalForwardingRule_externalHttpLbMigBackendCustomHeaderExample|TestAccComputeGlobalForwardingRule_externalTcpProxyLbMigBackendExample|TestAccComputeForwardingRule_forwardingRuleHttpLbExample|TestAccComputeForwardingRule_internalHttpLbWithMigBackendExample|TestAccComputeForwardingRule_update|TestAccComputeForwardingRule_internalTcpUdpLbWithMigBackendExample|TestAccComputeForwardingRule_forwardingRuleRegionalHttpXlbExample|TestAccComputeGlobalForwardingRule_internalLoadBalancing

[modular-magician]

Tests passed during RECORDING mode:
TestAccLoggingBucketConfigProject_cmekSettings[Debug log]
TestAccComputeGlobalForwardingRule_globalForwardingRuleInternalExample[Debug log]
TestAccComputeGlobalForwardingRule_externalHttpLbMigBackendCustomHeaderExample[Debug log]
TestAccComputeGlobalForwardingRule_externalTcpProxyLbMigBackendExample[Debug log]
TestAccComputeForwardingRule_forwardingRuleHttpLbExample[Debug log]
TestAccComputeForwardingRule_internalHttpLbWithMigBackendExample[Debug log]
TestAccComputeForwardingRule_update[Debug log]
TestAccComputeForwardingRule_internalTcpUdpLbWithMigBackendExample[Debug log]
TestAccComputeForwardingRule_forwardingRuleRegionalHttpXlbExample[Debug log]
TestAccComputeGlobalForwardingRule_internalLoadBalancing[Debug log]

All tests passed
View the build log or the debug log for each test

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 2 files changed, 240 insertions(+), 1 deletion(-))
Terraform Beta: Diff ( 2 files changed, 240 insertions(+), 1 deletion(-))
TF Validator: Diff ( 3 files changed, 37 insertions(+), 3 deletions(-))

[modular-magician]

Tests analytics

Total tests: 2302
Passed tests 2047
Skipped tests: 244
Failed tests: 11

Action taken

Triggering VCR tests in RECORDING mode for the tests that failed during VCR. Click here to see the failed tests

TestAccDataFusionInstance_dataFusionInstanceZoneExample|TestAccDataFusionInstance_dataFusionInstanceEventExample|TestAccDataFusionInstance_dataFusionInstanceFullExample|TestAccLoggingBucketConfigProject_cmekSettings|TestAccVertexAIIndex_updated|TestAccVertexAIIndex_vertexAiIndexStreamingExample|TestAccFirebaserulesRelease_BasicRelease|TestAccVertexAIIndex_vertexAiIndexExample|TestAccDataFusionInstanceIamPolicyGenerated|TestAccDataFusionInstanceIamMemberGenerated|TestAccDataFusionInstanceIamBindingGenerated

[modular-magician]

Tests passed during RECORDING mode:
TestAccDataFusionInstance_dataFusionInstanceZoneExample[Debug log]
TestAccDataFusionInstance_dataFusionInstanceEventExample[Debug log]
TestAccDataFusionInstance_dataFusionInstanceFullExample[Debug log]
TestAccLoggingBucketConfigProject_cmekSettings[Debug log]
TestAccVertexAIIndex_updated[Debug log]
TestAccVertexAIIndex_vertexAiIndexStreamingExample[Debug log]
TestAccFirebaserulesRelease_BasicRelease[Debug log]
TestAccDataFusionInstanceIamPolicyGenerated[Debug log]
TestAccDataFusionInstanceIamBindingGenerated[Debug log]

Tests failed during RECORDING mode:
TestAccVertexAIIndex_vertexAiIndexExample[Error message] [Debug log]
TestAccDataFusionInstanceIamMemberGenerated[Error message] [Debug log]

Please fix these to complete your PR
View the build log or the debug log for each test

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 2 files changed, 240 insertions(+), 1 deletion(-))
Terraform Beta: Diff ( 2 files changed, 240 insertions(+), 1 deletion(-))
TF Validator: Diff ( 3 files changed, 37 insertions(+), 3 deletions(-))

[modular-magician]

Tests analytics

Total tests: 2302
Passed tests 2054
Skipped tests: 244
Failed tests: 4

Action taken

Triggering VCR tests in RECORDING mode for the tests that failed during VCR. Click here to see the failed tests

TestAccLoggingBucketConfigProject_cmekSettings|TestAccFirebaserulesRelease_BasicRelease|TestAccVertexAIIndex_vertexAiIndexExample|TestAccDataFusionInstanceIamMemberGenerated

[modular-magician]

Tests passed during RECORDING mode:
TestAccLoggingBucketConfigProject_cmekSettings[Debug log]
TestAccFirebaserulesRelease_BasicRelease[Debug log]
TestAccVertexAIIndex_vertexAiIndexExample[Debug log]
TestAccDataFusionInstanceIamMemberGenerated[Debug log]

All tests passed
View the build log or the debug log for each test