Sync main feature branch 6.0.0 - 7/16

.changelog/11179.txt

@@ -0,0 +1,3 @@
+```release-note:none
+
+```

.teamcity/CONTRIBUTION_GUIDE.md

@@ -78,3 +78,76 @@ Note: this is likely to go out of date, so this description is kept at a high le
    # Makefile is essential for testing your code changes!
 # There are other misc files here, e.g. .gitignore
 ```
+
+## Feature branch testing
+
+If you want to test a feature branch on a schedule ahead of a release you can update the TeamCity configuration to include a new project specifically for testing that feature branch.
+
+### Testing major releases
+
+#### Adding a testing project for a major release
+
+First, make sure that the feature branch `FEATURE-BRANCH-major-release-X.0.0` is created in the downstream TPG and TPGB repositories, where X is the major version.
+
+See this PR as an example of adding a major release testing project: https://github.com/SarahFrench/magic-modules/pull/9/files
+
+That PR creates a new file at `.teamcity/components/projects/feature_branches/FEATURE-BRANCH-major-release-X.0.0.kt` (replacing `X` with the version number). This file defines a new project that will contain all the builds run against the feature branch. See [FEATURE-BRANCH-major-release-6.0.0.kt](https://github.com/GoogleCloudPlatform/magic-modules/blob/main/mmv1/third_party/terraform/.teamcity/components/projects/feature_branches/FEATURE-BRANCH-major-release-6.0.0.kt) as an example.
+
+This file must:
+
+* [Define a function that returns an instance of Project](https://github.com/GoogleCloudPlatform/magic-modules/blob/30ab2a2eea61cc34f439ddfe7cf840abf746ab1f/mmv1/third_party/terraform/.teamcity/components/projects/feature_branches/FEATURE-BRANCH-major-release-6.0.0.kt#L50)
+* [The project should include sub projects for Google and Google Beta, and inside each use reusable code to provision nightly testing projects](https://github.com/GoogleCloudPlatform/magic-modules/blob/main/mmv1/third_party/terraform/.teamcity/components/projects/feature_branches/FEATURE-BRANCH-major-release-6.0.0.kt#L59-L97)
+    * Note: Including the Google and Google Beta projects is done to avoid two projects with the name "Nightly Tests" existing side-by-side. Names have to be unique within the scope of a containing project.
+* For VCS roots, [create 2 new roots that default to using the feature branch from the TPG and TPGB repos](https://github.com/GoogleCloudPlatform/magic-modules/blob/main/mmv1/third_party/terraform/.teamcity/components/projects/feature_branches/FEATURE-BRANCH-major-release-6.0.0.kt#L22-L38).
+   * By using a new VCS root, instead of importing [existing roots](https://github.com/GoogleCloudPlatform/magic-modules/blob/main/mmv1/third_party/terraform/.teamcity/components/vcs_roots/vcs_roots.kt#L14-L32), you can ensure that the new project will default to using the feature branch even when builds are triggered manually.
+   * Be aware of how ["logical branch names"](https://www.jetbrains.com/help/teamcity/working-with-feature-branches.html#Logical+Branch+Name) may interact with the branch filter on the cron trigger. Always manually test your configuration is able to launch builds using that trigger!
+* For both nightly test projects you need to pass in a argument describing the CRON trigger that will be used to trigger builds.
+    * When testing major release feature branches we tend to:
+        * Trigger tests on the GA version of the [feature branch on Thursdays](https://github.com/GoogleCloudPlatform/magic-modules/blob/30ab2a2eea61cc34f439ddfe7cf840abf746ab1f/mmv1/third_party/terraform/.teamcity/components/projects/feature_branches/FEATURE-BRANCH-major-release-6.0.0.kt#L72)
+        * Trigger tests on the Beta version of the [feature branch on Fridays](https://github.com/GoogleCloudPlatform/magic-modules/blob/30ab2a2eea61cc34f439ddfe7cf840abf746ab1f/mmv1/third_party/terraform/.teamcity/components/projects/feature_branches/FEATURE-BRANCH-major-release-6.0.0.kt#L92)
+        * The non-feature branch projects will need to be updated to run on all days except these!
+    * You'll also need to [pass the feature branch name into the CRON trigger config class](https://github.com/GoogleCloudPlatform/magic-modules/blob/2778e6b73d802c6709d10d56fc3b8a3891168e6e/mmv1/third_party/terraform/.teamcity/components/projects/feature_branches/FEATURE-BRANCH-major-release-6.0.0.kt#L71). Note that the string needs to start with `refs/heads/`.
+* Don't forget to update the files that define the long-lived nightly test projects, making their CRON schedules the opposite of what's described for the feature branch testing projects:
+
+
+```diff
+// .teamcity/components/projects/google_ga_subproject.kt
+
+- subProject(nightlyTests(gaId, ProviderNameGa, HashiCorpVCSRootGa, gaConfig))
++ subProject(nightlyTests(gaId, ProviderNameGa, HashiCorpVCSRootGa, gaConfig, NightlyTriggerConfiguration(daysOfWeek="1-4,6-7"))) // All nights except Thursday (5) for GA; feature branch testing happens on Thursdays and TeamCity numbers days Sun=1...Sat=7
+
+// .teamcity/components/projects/google_beta_subproject.kt
+
+- subProject(nightlyTests(betaId, ProviderNameBeta, HashiCorpVCSRootBeta, betaConfig))
++ subProject(nightlyTests(betaId, ProviderNameBeta, HashiCorpVCSRootBeta, betaConfig, NightlyTriggerConfiguration(daysOfWeek="1-5,7"))) // All nights except Friday (6) for Beta; feature branch testing happens on Fridays and TeamCity numbers days Sun=1...Sat=7
+```
+
+Finally, you need to register the new feature branch testing project in [the root "Google Cloud" project](https://hashicorp.teamcity.com/project/TerraformProviders_GoogleCloud). Update [root_project.kt](https://github.com/GoogleCloudPlatform/magic-modules/blob/main/mmv1/third_party/terraform/.teamcity/components/projects/root_project.kt) to import code from your feature branch specific file, and [register a new sub project in the root project](https://github.com/GoogleCloudPlatform/magic-modules/blob/fd1a2272507d09214cf225b2ac05dfb363d3fb98/mmv1/third_party/terraform/.teamcity/components/projects/root_project.kt#L67-L68):
+
+```java
+Project {
+    description = "Contains all testing projects for the GA and Beta versions of the Google provider."
+
+    ...
+
+    // Feature branch testing
+    subProject(featureBranchMajorRelease600_Project(allConfig)) // FEATURE-BRANCH-major-release-6.0.0
+
+}
+```
+
+Don't forget to check that the code builds by running `make test` in the `.teamcity` folder while you're making these changes.
+
+To test your changes before merging a PR in magic-modules to update the TeamCity configuration, you can create a new TeamCity project that pulls its config from [the `auto-pr-N` branch for your PR in the modular-magician/terraform-provider-google repo](https://github.com/GoogleCloudPlatform/magic-modules/pull/11104#issuecomment-2206785710). See [USE_CONFIG_WITH_TEAMCITY.md](./USE_CONFIG_WITH_TEAMCITY.md) for details on creating new projects.
+
+#### Removing a test project, after a major release
+
+Once a major release is out you can safely delete the new project in TeamCity, and return the cron schedules to normal (i.e. every day of the week testing main branch).
+
+Here is PR illustrating those changes: https://github.com/SarahFrench/magic-modules/pull/8
+
+### Other feature branches
+
+You can do the above for feature branches that aren't major releases! However depending on the feature it may make sense to limit what builds run in that project; major releases need us to test all services, but is that also true for your feature branch?
+
+An example you can look at is this [PR](https://github.com/GoogleCloudPlatform/magic-modules/pull/10088) that creates a project within TeamCity for testing provider-defined functions. The project only ran builds for running acceptance tests for provider-defined functions and we did not need to be mindful of cron schedules as the provider-defined function acceptance tests did not have the ability to conflict with nightly tests.

.teamcity/components/builds/build_triggers.kt

@@ -18,10 +18,20 @@ import jetbrains.buildServer.configs.kotlin.triggers.schedule
 class NightlyTriggerConfiguration(
     val branch: String = DefaultBranchName,
     val nightlyTestsEnabled: Boolean = true,
-    val startHour: Int = DefaultStartHour,
-    val daysOfWeek: String = DefaultDaysOfWeek,
+    var startHour: Int = DefaultStartHour,
+    var daysOfWeek: String = DefaultDaysOfWeek,
     val daysOfMonth: String = DefaultDaysOfMonth
-)
+){
+    fun clone(): NightlyTriggerConfiguration{
+        return NightlyTriggerConfiguration(
+            this.branch,
+            this.nightlyTestsEnabled,
+            this.startHour,
+            this.daysOfWeek,
+            this.daysOfMonth
+        )
+    }
+}
 
 fun Triggers.runNightly(config: NightlyTriggerConfiguration) {
 

.teamcity/components/inputs/services_beta.kt

@@ -661,6 +661,11 @@ var ServicesListBeta = mapOf(
         "displayName" to "Serviceusage",
         "path" to "./google-beta/services/serviceusage"
     ),
+    "siteverification" to mapOf(
+        "name" to "siteverification",
+        "displayName" to "Siteverification",
+        "path" to "./google-beta/services/siteverification"
+    ),
     "sourcerepo" to mapOf(
         "name" to "sourcerepo",
         "displayName" to "Sourcerepo",

.teamcity/components/inputs/services_ga.kt

@@ -656,6 +656,11 @@ var ServicesListGa = mapOf(
         "displayName" to "Serviceusage",
         "path" to "./google/services/serviceusage"
     ),
+    "siteverification" to mapOf(
+        "name" to "siteverification",
+        "displayName" to "Siteverification",
+        "path" to "./google/services/siteverification"
+    ),
     "sourcerepo" to mapOf(
         "name" to "sourcerepo",
         "displayName" to "Sourcerepo",

.teamcity/components/projects/feature_branches/FEATURE-BRANCH-major-release-6.0.0.kt

@@ -0,0 +1,105 @@
+/*
+ * Copyright (c) HashiCorp, Inc.
+ * SPDX-License-Identifier: MPL-2.0
+ */
+
+// This file is controlled by MMv1, any changes made here will be overwritten
+
+package projects.feature_branches
+
+import ProviderNameBeta
+import ProviderNameGa
+import builds.*
+import jetbrains.buildServer.configs.kotlin.Project
+import jetbrains.buildServer.configs.kotlin.vcs.GitVcsRoot
+import projects.reused.nightlyTests
+import replaceCharsId
+
+const val branchName = "FEATURE-BRANCH-major-release-6.0.0"
+
+// VCS Roots specifically for pulling code from the feature branches in the downstream repos
+
+object HashicorpVCSRootGa_featureBranchMajorRelease600: GitVcsRoot({
+    name = "VCS root for the hashicorp/terraform-provider-${ProviderNameGa} repo @ refs/heads/${branchName}"
+    url = "https://github.com/hashicorp/terraform-provider-${ProviderNameGa}"
+    branch = "refs/heads/${branchName}"
+    branchSpec = """
+        +:(refs/heads/*)
+        -:refs/pulls/*
+    """.trimIndent()
+})
+
+object HashicorpVCSRootBeta_featureBranchMajorRelease600: GitVcsRoot({
+    name = "VCS root for the hashicorp/terraform-provider-${ProviderNameBeta} repo @ refs/heads/${branchName}"
+    url = "https://github.com/hashicorp/terraform-provider-${ProviderNameBeta}"
+    branch = "refs/heads/${branchName}"
+    branchSpec = """
+        +:(refs/heads/*)
+        -:refs/pulls/*
+    """.trimIndent()
+})
+
+fun featureBranchMajorRelease600_Project(allConfig: AllContextParameters): Project {
+
+    val projectId = replaceCharsId(branchName)
+    val gaProjectId = replaceCharsId(projectId + "_GA")
+    val betaProjectId= replaceCharsId(projectId + "_BETA")
+
+    // Get config for using the GA and Beta identities
+    val gaConfig = getGaAcceptanceTestConfig(allConfig)
+    val betaConfig = getBetaAcceptanceTestConfig(allConfig)
+
+    return Project{
+        id(projectId)
+        name = "6.0.0 Major Release Testing"
+        description = "Subproject for testing feature branch $branchName"
+
+        // Register feature branch-specific VCS roots in the project
+        vcsRoot(HashicorpVCSRootGa_featureBranchMajorRelease600)
+        vcsRoot(HashicorpVCSRootBeta_featureBranchMajorRelease600)
+
+        // Nested Nightly Test project that uses hashicorp/terraform-provider-google
+        subProject(
+            Project{
+                id(gaProjectId)
+                name = "Google"
+                subProject(
+                    nightlyTests(
+                        gaProjectId,
+                        ProviderNameGa,
+                        HashicorpVCSRootGa_featureBranchMajorRelease600,
+                        gaConfig,
+                        NightlyTriggerConfiguration(
+                            branch = "refs/heads/${branchName}", // Make triggered builds use the feature branch
+                            daysOfWeek = "5"     // Thursday for GA, TeamCity numbers days Sun=1...Sat=7
+                        ), 
+                    )
+                )
+            }
+        )
+
+        // Nested Nightly Test project that uses hashicorp/terraform-provider-google-beta
+        subProject(
+            Project {
+                id(betaProjectId)
+                name = "Google Beta"
+                subProject(
+                    nightlyTests(
+                        betaProjectId,
+                        ProviderNameBeta,
+                        HashicorpVCSRootBeta_featureBranchMajorRelease600,
+                        betaConfig,
+                        NightlyTriggerConfiguration(
+                            branch = "refs/heads/${branchName}", // Make triggered builds use the feature branch
+                            daysOfWeek="6"       // Friday for Beta, TeamCity numbers days Sun=1...Sat=7
+                        ),
+                    )
+                )
+            }
+        )
+
+        params {
+            readOnlySettings()
+        }
+    }
+}

.teamcity/components/projects/feature_branches/README.md

@@ -1,5 +1,6 @@
 ## Feature Branch Folder
 
-This folder is meant for testing of new features in teamcity.
+This folder is meant for testing of new features in TeamCity.
+
+See the [Contribution Guide](../../../CONTRIBUTION_GUIDE.md) for information about feature branch testing.
 
-For example, this [PR](https://github.com/GoogleCloudPlatform/magic-modules/pull/10088) creates a project within teamcity for testing provider functions.

.teamcity/components/projects/google_beta_subproject.kt

@@ -8,10 +8,7 @@
 package projects
 
 import ProviderNameBeta
-import builds.AllContextParameters
-import builds.getBetaAcceptanceTestConfig
-import builds.getVcrAcceptanceTestConfig
-import builds.readOnlySettings
+import builds.*
 import jetbrains.buildServer.configs.kotlin.Project
 import projects.reused.mmUpstream
 import projects.reused.nightlyTests
@@ -23,7 +20,7 @@ import vcs_roots.ModularMagicianVCSRootBeta
 // googleSubProjectBeta returns a subproject that is used for testing terraform-provider-google-beta (Beta)
 fun googleSubProjectBeta(allConfig: AllContextParameters): Project {
 
-    var betaId = replaceCharsId("GOOGLE_BETA")
+    val betaId = replaceCharsId("GOOGLE_BETA")
 
     // Get config for using the Beta and VCR identities
     val betaConfig = getBetaAcceptanceTestConfig(allConfig)
@@ -35,10 +32,10 @@ fun googleSubProjectBeta(allConfig: AllContextParameters): Project {
         description = "Subproject containing builds for testing the Beta version of the Google provider"
 
         // Nightly Test project that uses hashicorp/terraform-provider-google-beta
-        subProject(nightlyTests(betaId, ProviderNameBeta, HashiCorpVCSRootBeta, betaConfig))
+        subProject(nightlyTests(betaId, ProviderNameBeta, HashiCorpVCSRootBeta, betaConfig, NightlyTriggerConfiguration(daysOfWeek="1-5,7"))) // All nights except Friday (6) for Beta; feature branch testing happens on Fridays and TeamCity numbers days Sun=1...Sat=7
 
         // MM Upstream project that uses modular-magician/terraform-provider-google-beta
-        subProject(mmUpstream(betaId, ProviderNameBeta, ModularMagicianVCSRootBeta, HashiCorpVCSRootBeta, vcrConfig))
+        subProject(mmUpstream(betaId, ProviderNameBeta, ModularMagicianVCSRootBeta, HashiCorpVCSRootBeta, vcrConfig, NightlyTriggerConfiguration()))
 
         // VCR recording project that allows VCR recordings to be made using hashicorp/terraform-provider-google-beta OR modular-magician/terraform-provider-google-beta
         // This is only present for the Beta provider, as only TPGB VCR recordings are used.

.teamcity/components/projects/google_ga_subproject.kt

@@ -8,10 +8,7 @@
 package projects
 
 import ProviderNameGa
-import builds.AllContextParameters
-import builds.getGaAcceptanceTestConfig
-import builds.getVcrAcceptanceTestConfig
-import builds.readOnlySettings
+import builds.*
 import jetbrains.buildServer.configs.kotlin.Project
 import projects.reused.mmUpstream
 import projects.reused.nightlyTests
@@ -22,7 +19,7 @@ import vcs_roots.ModularMagicianVCSRootGa
 // googleSubProjectGa returns a subproject that is used for testing terraform-provider-google (GA)
 fun googleSubProjectGa(allConfig: AllContextParameters): Project {
 
-    var gaId = replaceCharsId("GOOGLE")
+    val gaId = replaceCharsId("GOOGLE")
 
     // Get config for using the GA and VCR identities
     val gaConfig = getGaAcceptanceTestConfig(allConfig)
@@ -34,10 +31,10 @@ fun googleSubProjectGa(allConfig: AllContextParameters): Project {
         description = "Subproject containing builds for testing the GA version of the Google provider"
 
         // Nightly Test project that uses hashicorp/terraform-provider-google
-        subProject(nightlyTests(gaId, ProviderNameGa, HashiCorpVCSRootGa, gaConfig))
+        subProject(nightlyTests(gaId, ProviderNameGa, HashiCorpVCSRootGa, gaConfig, NightlyTriggerConfiguration(daysOfWeek="1-4,6-7"))) // All nights except Thursday (5) for GA; feature branch testing happens on Thursdays and TeamCity numbers days Sun=1...Sat=7
 
         // MM Upstream project that uses modular-magician/terraform-provider-google
-        subProject(mmUpstream(gaId, ProviderNameGa, ModularMagicianVCSRootGa, HashiCorpVCSRootGa, vcrConfig))
+        subProject(mmUpstream(gaId, ProviderNameGa, ModularMagicianVCSRootGa, HashiCorpVCSRootGa, vcrConfig, NightlyTriggerConfiguration()))
 
         params {
             readOnlySettings()

.teamcity/components/projects/project_sweeper_project.kt

@@ -23,7 +23,7 @@ fun projectSweeperSubProject(allConfig: AllContextParameters): Project {
 
     val projectId = replaceCharsId("PROJECT_SWEEPER")
 
-    // Get config for using the GA identity (arbitrary choice as sweeper isn't confined by GA/Beta etc)
+    // Get config for using the GA identity (arbitrary choice as sweeper isn't confined by GA/Beta etc.)
     val gaConfig = getGaAcceptanceTestConfig(allConfig)
 
     // List of ALL shared resources; avoid clashing with any other running build

.teamcity/components/projects/reused/mm_upstream.kt

@@ -25,7 +25,7 @@ import jetbrains.buildServer.configs.kotlin.Project
 import jetbrains.buildServer.configs.kotlin.vcs.GitVcsRoot
 import replaceCharsId
 
-fun mmUpstream(parentProject: String, providerName: String, vcsRoot: GitVcsRoot, cronSweeperVcsRoot: GitVcsRoot, config: AccTestConfiguration): Project {
+fun mmUpstream(parentProject: String, providerName: String, vcsRoot: GitVcsRoot, cronSweeperVcsRoot: GitVcsRoot, config: AccTestConfiguration, cron: NightlyTriggerConfiguration): Project {
 
     // Create unique ID for the dynamically-created project
     var projectId = "${parentProject}_${MMUpstreamProjectId}"
@@ -45,11 +45,13 @@ fun mmUpstream(parentProject: String, providerName: String, vcsRoot: GitVcsRoot,
         ProviderNameBeta -> sweepersList = SweepersListBeta
         else -> throw Exception("Provider name not supplied when generating a nightly test subproject")
     }
+
+    // This build is for manually-initiated runs of sweepers, to test changes to sweepers from the upstream repo
     val serviceSweeperManualConfig = BuildConfigurationForServiceSweeper(providerName, ServiceSweeperManualName, sweepersList, projectId, vcsRoot, sharedResources, config)
 
+    // This build runs on a schedule to do actual sweeping of the VCR project, using the downstream repo's code
     val serviceSweeperCronConfig = BuildConfigurationForServiceSweeper(providerName, ServiceSweeperCronName, sweepersList, projectId, cronSweeperVcsRoot, sharedResources, config)
-    val trigger  = NightlyTriggerConfiguration(startHour=12)
-    serviceSweeperCronConfig.addTrigger(trigger) // Only the sweeper is on a schedule in this project
+    serviceSweeperCronConfig.addTrigger(cron)
 
     return Project {
         id(projectId)