Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(bigquery): add option to use Non-incremental materialized views #8725

Conversation

sachinpro
Copy link
Contributor

feat(bigquery): add option to use Non-incremental materialized views

This adds an option use allow_non_incremental_definition to create a non-incremental materialized view in bigquery, and a basic test for this.

If this PR is for Terraform, I acknowledge that I have:

  • Searched through the issue tracker for an open issue that this either resolves or contributes to, commented on it to claim it, and written "fixes {url}" or "part of {url}" in this PR description. If there were no relevant open issues, I opened one and commented that I would like to work on it (not necessary for very small changes).
  • Ensured that all new fields I added that can be set by a user appear in at least one example (for generated resources) or third_party test (for handwritten resources or update tests).
  • Generated Terraform providers, and ran make test and make lint in the generated providers to ensure it passes unit and linter tests.
  • Ran relevant acceptance tests using my own Google Cloud project and credentials (If the acceptance tests do not yet pass or you are unable to run them, please let your reviewer know).
  • Read Write release notes before writing my release note below.

Release Note Template for Downstream PRs (will be copied)

bigquery: added `allow_non_incremental_definition` to `google_bigquery_table` resource

@modular-magician
Copy link
Collaborator

Hello! I am a robot. It looks like you are a: Community Contributor Googler Core Contributor. Tests will run automatically.

@shuyama1, a repository maintainer, has been assigned to review your changes. If you have not received review feedback within 2 business days, please leave a comment on this PR asking them to take a look.

You can help make sure that review is quick by doing a self-review and by running impacted tests locally.

@modular-magician
Copy link
Collaborator

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 2 files changed, 142 insertions(+))
Terraform Beta: Diff ( 2 files changed, 142 insertions(+))

@modular-magician
Copy link
Collaborator

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 2 files changed, 126 insertions(+))
Terraform Beta: Diff ( 2 files changed, 126 insertions(+))

@sachinpro sachinpro force-pushed the bigquery-materialized-view-non-incremental branch from 71e6162 to a398412 Compare August 21, 2023 22:09
@modular-magician
Copy link
Collaborator

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 3 files changed, 129 insertions(+))
Terraform Beta: Diff ( 3 files changed, 129 insertions(+))

@modular-magician
Copy link
Collaborator

Tests analytics

Total tests: 2984
Passed tests 2686
Skipped tests: 295
Affected tests: 3

Action taken

Found 3 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests
TestAccBigQueryTable_MaterializedView_NonIncremental_basic|TestAccBigQueryTable_MaterializedView_DailyTimePartioning_Update|TestAccBigQueryTable_MaterializedView_DailyTimePartioning_Basic

Get to know how VCR tests work

@modular-magician
Copy link
Collaborator

$\textcolor{green}{\textsf{Tests passed during RECORDING mode:}}$
TestAccBigQueryTable_MaterializedView_DailyTimePartioning_Update[Debug log]
TestAccBigQueryTable_MaterializedView_DailyTimePartioning_Basic[Debug log]

Rerun these tests in REPLAYING mode to catch issues

$\textcolor{green}{\textsf{No issues found for passed tests after REPLAYING rerun.}}$


$\textcolor{red}{\textsf{Tests failed during RECORDING mode:}}$
TestAccBigQueryTable_MaterializedView_NonIncremental_basic[Error message] [Debug log]

$\textcolor{red}{\textsf{Please fix these to complete your PR.}}$
View the build log or the debug log for each test

Copy link
Member

@shuyama1 shuyama1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @sachinpro, thanks for the contribution! Looks like the test failed due to

vcr_utils.go:152: Step 1/3 error: After applying this test step, the plan was not empty.
        stdout:
        
        
        Terraform used the selected providers to generate the following execution
        plan. Resource actions are indicated with the following symbols:
          ~ update in-place
        
        Terraform will perform the following actions:
        
          # google_bigquery_table.mv_test will be updated in-place
          ~ resource "google_bigquery_table" "mv_test" {
                id                  = "projects/ci-test-project-188019/datasets/tf_test_l6k10l1tyh/tables/tf_test_susyysrtg2"
                # (16 unchanged attributes hidden)
        
              ~ materialized_view {
                  ~ allow_non_incremental_definition = false -> true
                    # (3 unchanged attributes hidden)
                }
        
                # (1 unchanged block hidden)
            }
        
        Plan: 0 to add, 1 to change, 0 to destroy.

It mean there's a drift detected between config and TF state after terraform apply, please let me know if you have questions debugging it.

Plus, is the field updatable? If so, it'd be nice to add/modify an update test. See details at https://googlecloudplatform.github.io/magic-modules/develop/test/#add-an-update-test

@shuyama1
Copy link
Member

shuyama1 commented Sep 7, 2023

/gcbrun

@modular-magician
Copy link
Collaborator

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 3 files changed, 129 insertions(+))
Terraform Beta: Diff ( 3 files changed, 129 insertions(+))

@modular-magician
Copy link
Collaborator

Tests analytics

Total tests: 3018
Passed tests 2718
Skipped tests: 297
Affected tests: 3

Action taken

Found 3 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests
TestAccBigQueryTable_MaterializedView_NonIncremental_basic|TestAccBigQueryTable_MaterializedView_DailyTimePartioning_Update|TestAccBigQueryTable_MaterializedView_DailyTimePartioning_Basic

Get to know how VCR tests work

@modular-magician
Copy link
Collaborator

$\textcolor{green}{\textsf{Tests passed during RECORDING mode:}}$
TestAccBigQueryTable_MaterializedView_NonIncremental_basic[Debug log]
TestAccBigQueryTable_MaterializedView_DailyTimePartioning_Update[Debug log]
TestAccBigQueryTable_MaterializedView_DailyTimePartioning_Basic[Debug log]

Rerun these tests in REPLAYING mode to catch issues

$\textcolor{green}{\textsf{No issues found for passed tests after REPLAYING rerun.}}$


$\textcolor{green}{\textsf{All tests passed!}}$
View the build log or the debug log for each test

@shuyama1
Copy link
Member

shuyama1 commented Sep 8, 2023

/gcbrun

@modular-magician
Copy link
Collaborator

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 3 files changed, 129 insertions(+))
Terraform Beta: Diff ( 3 files changed, 129 insertions(+))

@modular-magician
Copy link
Collaborator

Tests analytics

Total tests: 3021
Passed tests 2724
Skipped tests: 297
Affected tests: 0

$\textcolor{green}{\textsf{All tests passed in REPLAYING mode.}}$
View the build log

Copy link
Member

@shuyama1 shuyama1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks at the API issue is resolved. Does this field support in-place update? If so, can you add an update step to test it (https://googlecloudplatform.github.io/magic-modules/develop/test/#add-an-update-test). Otherwise, please add forcenew: true to the field. Thanks!

@sachinpro
Copy link
Contributor Author

Looks at the API issue is resolved. Does this field support in-place update? If so, can you add an update step to test it (https://googlecloudplatform.github.io/magic-modules/develop/test/#add-an-update-test). Otherwise, please add forcenew: true to the field. Thanks!

Added ForceNew in the commit.

@modular-magician
Copy link
Collaborator

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 3 files changed, 130 insertions(+))
Terraform Beta: Diff ( 3 files changed, 130 insertions(+))

@modular-magician
Copy link
Collaborator

Tests analytics

Total tests: 3028
Passed tests 2731
Skipped tests: 297
Affected tests: 0

$\textcolor{green}{\textsf{All tests passed in REPLAYING mode.}}$
View the build log

Copy link
Member

@shuyama1 shuyama1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

@shuyama1 shuyama1 merged commit 37206b2 into GoogleCloudPlatform:main Sep 12, 2023
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
3 participants