autogeneration of tests for singular datasources #14872

luckyswaminathan · 2025-08-15T21:43:42Z

hashicorp/terraform-provider-google#12595 part of this initiative

Release Note Template for Downstream PRs (will be copied)

autogeneration of tests enabled

modular-magician · 2025-08-15T21:55:21Z

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 3 files changed, 142 insertions(+), 12 deletions(-))
google-beta provider: Diff ( 3 files changed, 142 insertions(+), 12 deletions(-))

modular-magician · 2025-08-15T21:57:07Z

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 3 files changed, 142 insertions(+), 12 deletions(-))
google-beta provider: Diff ( 3 files changed, 142 insertions(+), 12 deletions(-))

modular-magician · 2025-08-15T22:02:00Z

Tests analytics

Total tests: 123
Passed tests: 101
Skipped tests: 20
Affected tests: 2

Click here to see the affected service packages

iap

redis

Action taken

Found 2 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests

TestAccDataSourceIapClient_basic

TestAccDataSourceRedisCluster_basic

Get to know how VCR tests work

modular-magician · 2025-08-15T22:03:12Z

Tests analytics

Total tests: 123
Passed tests: 101
Skipped tests: 20
Affected tests: 2

Click here to see the affected service packages

iap

redis

Action taken

Found 2 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests

TestAccDataSourceIapClient_basic

TestAccDataSourceRedisCluster_basic

Get to know how VCR tests work

modular-magician · 2025-08-15T22:36:09Z

🟢 Tests passed during RECORDING mode:
TestAccDataSourceRedisCluster_basic [Debug log]

🟢 No issues found for passed tests after REPLAYING rerun.

🔴 Tests failed during RECORDING mode:
TestAccDataSourceIapClient_basic [Error message] [Debug log]

🔴 Errors occurred during RECORDING mode. Please fix them to complete your PR.

View the build log or the debug log for each test

modular-magician · 2025-08-15T22:39:57Z

🟢 Tests passed during RECORDING mode:
TestAccDataSourceRedisCluster_basic [Debug log]

🟢 No issues found for passed tests after REPLAYING rerun.

🔴 Tests failed during RECORDING mode:
TestAccDataSourceIapClient_basic [Error message] [Debug log]

🔴 Errors occurred during RECORDING mode. Please fix them to complete your PR.

View the build log or the debug log for each test

modular-magician · 2025-08-18T16:50:18Z

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 2 files changed, 77 insertions(+), 12 deletions(-))
google-beta provider: Diff ( 2 files changed, 77 insertions(+), 12 deletions(-))

modular-magician · 2025-08-18T16:53:48Z

Tests analytics

Total tests: 35
Passed tests: 35
Skipped tests: 0
Affected tests: 0

Click here to see the affected service packages

redis

🟢 All tests passed!

View the build log

github-actions · 2025-08-18T17:24:29Z

Hello! I am a robot. Tests will require approval from a repository maintainer to run.

Googlers: For automatic test runs see go/terraform-auto-test-runs.

@melinath, a repository maintainer, has been assigned to review your changes. If you have not received review feedback within 2 business days, please leave a comment on this PR asking them to take a look.

You can help make sure that review is quick by doing a self-review and by running impacted tests locally.

melinath

Rather than converting a new datasource to have generated tests, could you take a resource that already has a generated datasource and convert its tests?

Ideally, it would be nice to rename the handwritten test file to have the same name as the generated file in a separate PR, then remove the handwritten test file in this PR. That would make the diff easier to review.

melinath · 2025-08-18T17:33:43Z

mmv1/products/redis/Cluster.yaml

 autogen_async: true
+datasource:
+  generate: true
+  generate_tests: true


Given there are only four datasources generated at the moment, could we actually just convert all four in this PR (with simplified diffs) and make that the default behavior, instead of having a generate_tests flag that needs to be set?

so the problem with this is that there are generatable datasources (ie storagecontrol) that don't have generatable tests, as they only have handrolled tests, so no real way of generating from that template

yeah, we probably want to have an exclude_test attribute that you can set to not generate, similar to the exclude_docs & exclude_test for examples, which defaults to false.

The alternative would be to make generate_test default to true (advantage being that we wouldn't end up with a double-negative) but we generally try to avoid that at the moment (I forget why - maybe hard to tell the difference between "unset" and "set to false"?)

Basically, the default behavior should be "generate tests" since that's what we expect people to do in most cases.

luckyswaminathan · 2025-08-18T19:28:57Z

Rather than converting a new datasource to have generated tests, could you take a resource that already has a generated datasource and convert its tests?

Ideally, it would be nice to rename the handwritten test file to have the same name as the generated file in a separate PR, then remove the handwritten test file in this PR. That would make the diff easier to review.

I can for certain datasources yeah, I think it might best to keep the flag(so datasources with handrolled resource tests are compatible), but still replacing tests for compatible datasources such as the redis/iapclient ones done in this PR, let me know if you think this isn't the right way

modular-magician · 2025-08-18T19:38:16Z

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 3 files changed, 69 insertions(+), 86 deletions(-))
google-beta provider: Diff ( 3 files changed, 69 insertions(+), 86 deletions(-))

modular-magician · 2025-08-18T19:44:22Z

Tests analytics

Total tests: 121
Passed tests: 100
Skipped tests: 20
Affected tests: 1

Click here to see the affected service packages

iap

redis

Action taken

Found 1 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests

TestAccDataSourceIapClient_basic

Get to know how VCR tests work

modular-magician · 2025-08-18T19:46:41Z

🟢 Tests passed during RECORDING mode:
TestAccDataSourceIapClient_basic [Debug log]

🟢 No issues found for passed tests after REPLAYING rerun.

🟢 All tests passed!

View the build log or the debug log for each test

melinath · 2025-08-18T20:14:42Z

mmv1/provider/terraform.go

+	if err := os.MkdirAll(targetFolder, os.ModePerm); err != nil {
+		log.Println(fmt.Errorf("error creating parent directory %v: %v", targetFolder, err))
+	}
+	targetFilePath := path.Join(targetFolder, fmt.Sprintf("data_source_%s_test.go", t.ResourceGoFilename(object)))


👍 doing this for now so the diff is clearer seems reasonable - we should switch to _generated_test.go in the near future, but that can be in a separate PR.

mmv1/templates/terraform/examples/base_configs/datasource_test_file.go.tmpl

melinath · 2025-08-18T20:21:35Z

mmv1/templates/terraform/examples/base_configs/datasource_test_file.go.tmpl

+		{{ $fieldName }} = {{ $e.ResourceType $.Res.TerraformName }}.{{ $e.PrimaryResourceId }}.{{ $fieldName }}
+		{{- end }}
+		}`,
+		testAcc{{ $e.TestSlug $.Res.ProductMetadata.Name $.Res.Name }}(context),


We should probably inline the configuration (similar to what IAM tests do) rather than using string formatting - string formatting like this makes it harder to read the final result (both for humans and for the missing test detector).

modular-magician · 2025-08-19T17:12:57Z

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 3 files changed, 134 insertions(+), 62 deletions(-))
google-beta provider: Diff ( 3 files changed, 134 insertions(+), 62 deletions(-))

modular-magician · 2025-08-19T17:19:21Z

Tests analytics

Total tests: 121
Passed tests: 99
Skipped tests: 20
Affected tests: 2

Click here to see the affected service packages

iap

redis

Action taken

Found 2 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests

TestAccDataSourceIapClient_basic

TestAccDataSourceRedisCluster_basic

Get to know how VCR tests work

melinath · 2025-08-27T15:49:34Z

I think this is probably working but it's hard to tell because the test runs were interleaved. Rerunning.

/gcbrun

mmv1/products/alloydb/Cluster.yaml

modular-magician · 2025-08-28T17:46:58Z

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 1 file changed, 24 insertions(+), 19 deletions(-))
google-beta provider: Diff ( 1 file changed, 24 insertions(+), 19 deletions(-))

modular-magician · 2025-08-28T17:52:36Z

Tests analytics

Total tests: 103
Passed tests: 79
Skipped tests: 24
Affected tests: 0

Click here to see the affected service packages

iap

🟢 All tests passed!

View the build log

melinath

LGTM, diff looks correct

…#14872) Co-authored-by: Stephen Lewis (Burrows) <stephen.r.burrows@gmail.com>

remake pr

55429c2

luckyswaminathan changed the title ~~remake pr~~ autogeneration of tests for singular datasources Aug 15, 2025

linting:

08c9da3

modular-magician added service/iap service/redis-cluster labels Aug 15, 2025

it works

ae0ddba

luckyswaminathan marked this pull request as ready for review August 18, 2025 17:23

github-actions bot requested a review from melinath August 18, 2025 17:24

melinath requested changes Aug 18, 2025

View reviewed changes

correct config generation and expressivity

4d817cb

github-actions bot requested a review from melinath August 18, 2025 19:25

melinath requested changes Aug 18, 2025

View reviewed changes

luckyswaminathan added 2 commits August 19, 2025 09:59

slightly altered config logic

5a9231a

remove output.log

c87cc66

github-actions bot requested a review from melinath August 19, 2025 17:00

modular-magician added awaiting-approval Pull requests that need reviewer's approval to run presubmit tests and removed awaiting-approval Pull requests that need reviewer's approval to run presubmit tests labels Aug 27, 2025

luckyswaminathan and others added 3 commits August 27, 2025 10:09

cluster bk

351ba46

Merge branch 'GoogleCloudPlatform:main' into main

0ebe5e5

fix cloudrun test issue

19553c8

modular-magician added the awaiting-approval Pull requests that need reviewer's approval to run presubmit tests label Aug 28, 2025

merge main

8ec557a

modular-magician removed the awaiting-approval Pull requests that need reviewer's approval to run presubmit tests label Aug 28, 2025

import path fix

1a2cdbf

modular-magician added awaiting-approval Pull requests that need reviewer's approval to run presubmit tests and removed awaiting-approval Pull requests that need reviewer's approval to run presubmit tests labels Aug 28, 2025

This comment was marked as outdated.

Sign in to view

melinath requested changes Aug 28, 2025

View reviewed changes

mmv1/products/alloydb/Cluster.yaml Outdated Show resolved Hide resolved

Update mmv1/products/alloydb/Cluster.yaml

2c344b5

modular-magician added the awaiting-approval Pull requests that need reviewer's approval to run presubmit tests label Aug 28, 2025

github-actions bot requested a review from melinath August 28, 2025 17:37

modular-magician removed the awaiting-approval Pull requests that need reviewer's approval to run presubmit tests label Aug 28, 2025

melinath approved these changes Aug 28, 2025

View reviewed changes

melinath added this pull request to the merge queue Aug 28, 2025

Merged via the queue into GoogleCloudPlatform:main with commit d6166b5 Aug 28, 2025
27 of 28 checks passed

NandiniAgrawal15 pushed a commit to NandiniAgrawal15/magic-modules that referenced this pull request Sep 4, 2025

autogeneration of tests for singular datasources (GoogleCloudPlatform…

49ebc63

…#14872) Co-authored-by: Stephen Lewis (Burrows) <stephen.r.burrows@gmail.com>

BBBmau pushed a commit to BBBmau/magic-modules that referenced this pull request Sep 26, 2025

autogeneration of tests for singular datasources (GoogleCloudPlatform…

3ecc2ce

…#14872) Co-authored-by: Stephen Lewis (Burrows) <stephen.r.burrows@gmail.com>

jkrish-c pushed a commit to jkrish-c/magic-modules that referenced this pull request Oct 14, 2025

autogeneration of tests for singular datasources (GoogleCloudPlatform…

c168eb8

…#14872) Co-authored-by: Stephen Lewis (Burrows) <stephen.r.burrows@gmail.com>

autogeneration of tests for singular datasources #14872

autogeneration of tests for singular datasources #14872

Uh oh!

Conversation

luckyswaminathan commented Aug 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

modular-magician commented Aug 15, 2025

Diff report

Uh oh!

modular-magician commented Aug 15, 2025

Diff report

Uh oh!

modular-magician commented Aug 15, 2025

Tests analytics

Action taken

Uh oh!

modular-magician commented Aug 15, 2025

Tests analytics

Action taken

Uh oh!

modular-magician commented Aug 15, 2025

Uh oh!

modular-magician commented Aug 15, 2025

Uh oh!

modular-magician commented Aug 18, 2025

Diff report

Uh oh!

modular-magician commented Aug 18, 2025

Tests analytics

Uh oh!

github-actions bot commented Aug 18, 2025

Uh oh!

melinath left a comment

Choose a reason for hiding this comment

Uh oh!

melinath Aug 18, 2025

Choose a reason for hiding this comment

Uh oh!

luckyswaminathan Aug 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

melinath Aug 18, 2025

Choose a reason for hiding this comment

Uh oh!

melinath Aug 18, 2025

Choose a reason for hiding this comment

Uh oh!

luckyswaminathan commented Aug 18, 2025

Uh oh!

modular-magician commented Aug 18, 2025

Diff report

Uh oh!

modular-magician commented Aug 18, 2025

Tests analytics

Action taken

Uh oh!

modular-magician commented Aug 18, 2025

Uh oh!

melinath Aug 18, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

melinath Aug 18, 2025

Choose a reason for hiding this comment

Uh oh!

modular-magician commented Aug 19, 2025

Diff report

Uh oh!

modular-magician commented Aug 19, 2025

Tests analytics

Action taken

Uh oh!

melinath commented Aug 27, 2025

Uh oh!

This comment was marked as outdated.

Uh oh!

modular-magician commented Aug 28, 2025

Diff report

luckyswaminathan commented Aug 15, 2025 •

edited

Loading

luckyswaminathan Aug 18, 2025 •

edited

Loading