Skip to content

Commit

Permalink
Update PERFORMING_TASKS_IN_TEAMCITY.md (#10700)
Browse files Browse the repository at this point in the history
  • Loading branch information
melinath committed May 15, 2024
1 parent 4ee0d37 commit d676d82
Showing 1 changed file with 11 additions and 9 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -42,8 +42,9 @@ Google Cloud/
A CRON trigger causes all acceptance tests for the GA and Beta providers to run overnight, Mon-Sun. These tests use the `main` branches of hashicorp/terraform-provider-google(-beta) repos. The tests interact fully with Google APIs and are used to identify any breaking changes introduced by either a recent PR or a change in the API itself.

You can find the builds for nightly tests at:
* Google > Nightly Tests
* Google Beta > Nightly Tests

* [Google > Nightly Tests](https://hashicorp.teamcity.com/project/TerraformProviders_GoogleCloud_GOOGLE_NIGHTLYTESTS?mode=builds#all-projects)
* [Google Beta > Nightly Tests](https://hashicorp.teamcity.com/project/TerraformProviders_GoogleCloud_GOOGLE_BETA_NIGHTLYTESTS#all-projects)

These projects contain a build configuration per service package.

Expand All @@ -60,7 +61,7 @@ To view all the failed tests for a given commit:

### Ad-hoc testing of branches in the downstream repositories

In preparation for a release you may need to run tests on a release branch present in the downstream hashicorp/terraform-provider-google(-beta) repos. To do this you should navigate to `Google > Nightly Tests` or `Google Beta > Nightly Tests` and run a Custom Build. The resulting build will still use the terraform-provider-google(-beta) repo and the appropriate nightly testing GCP project, but you can change which branch is used and can limit which tests are run. See the official TeamCity documentation for [information on how to run a Custom Build](https://www.jetbrains.com/help/teamcity/running-custom-build.html).
In preparation for a release you may need to run tests on a release branch present in the downstream hashicorp/terraform-provider-google(-beta) repos. To do this you should navigate to [`Google > Nightly Tests`](https://hashicorp.teamcity.com/project/TerraformProviders_GoogleCloud_GOOGLE_NIGHTLYTESTS?mode=builds#all-projects) or [`Google Beta > Nightly Tests`](https://hashicorp.teamcity.com/project/TerraformProviders_GoogleCloud_GOOGLE_BETA_NIGHTLYTESTS#all-projects) and run a Custom Build. The resulting build will still use the terraform-provider-google(-beta) repo and the appropriate nightly testing GCP project, but you can change which branch is used and can limit which tests are run. See the official TeamCity documentation for [information on how to run a Custom Build](https://www.jetbrains.com/help/teamcity/running-custom-build.html).

To use a release branch, or any other branch that isn't main, use [the `Build branch` dropdown menu](https://www.jetbrains.com/help/teamcity/running-custom-build.html#Build+Branch) in the `Run Custom Build` modal. Changing branches is no longer controlled by a parameter.

Expand All @@ -70,8 +71,8 @@ To use a release branch, or any other branch that isn't main, use [the `Build br
When reviewing a PR you may need to run acceptance tests using the code shown in the Diff Report, present in branches in the modular-magician/terraform-provider-google(-beta) repositories.

To do this you should navigate to either of these projects and run a custom build:
* `Google > Upstream MM Testing`
* `Google Beta > Upstream MM Testing`
* [`Google > Upstream MM Testing`](https://hashicorp.teamcity.com/project/TerraformProviders_GoogleCloud_GOOGLE_MMUPSTREAMTESTS#all-projects)
* [`Google Beta > Upstream MM Testing`](https://hashicorp.teamcity.com/project/TerraformProviders_GoogleCloud_GOOGLE_BETA_MMUPSTREAMTESTS#all-projects)

Builds in these projects will test the code present in the Modular Magician's forks and will run tests against the VCR testing project in GCP.

Expand All @@ -81,7 +82,8 @@ See the section above about how to run a Custom Build.
### Triggering VCR tests to record new cassettes

Sometimes VCR cassettes need to be re-recorded by manual intervention, for example if a VCR test is failing across all PRs due to a bad cassette that isn't being replaced. Our VCR tests on PRs only use the Beta provider, so the only place to record VCR cassettes in TeamCity is:
* `Google Beta > VCR Recording`.

* [`Google Beta > VCR Recording`](https://hashicorp.teamcity.com/project/TerraformProviders_GoogleCloud_GOOGLE_BETA_VCRRECORDING#all-projects)

There are two build configurations in this project: one for using hashicorp/terraform-provider-google-beta to run the tests, and the other for using modular-magician/terraform-provider-google-beta. Make sure to use the correct one for your use case.

Expand All @@ -102,14 +104,14 @@ In REPLAYING mode the build will download VCR cassettes from a GCS Bucket and ru

### Sweeping the Nightly Test Projects

The Service Sweeper builds in `Google > Nightly Tests` and `Google Beta > Nightly Tests` run every night via CRON. They are designed to not run until there are no builds testing any services in the GA or Beta nightly test GCP projects. No acceptance testing builds will start until the sweeper stops.
The Service Sweeper builds in [`Google > Nightly Tests`](https://hashicorp.teamcity.com/project/TerraformProviders_GoogleCloud_GOOGLE_NIGHTLYTESTS?mode=builds#all-projects) and [`Google Beta > Nightly Tests`](https://hashicorp.teamcity.com/project/TerraformProviders_GoogleCloud_GOOGLE_BETA_NIGHTLYTESTS#all-projects) run every night via CRON. They are designed to not run until there are no builds testing any services in the GA or Beta nightly test GCP projects. No acceptance testing builds will start until the sweeper stops.

### Sweeping the VCR Project

The Service Sweeper builds in `Google > Upstream MM Testing` and `Google Beta > Upstream MM Testing` run every night via CRON. They are redundant as both sweep the VCR project, but I've left them both in. They are designed to not run until there are no builds testing any services in the VCR test GCP project. No acceptance testing builds will start until the sweeper stops.
The Service Sweeper builds in [`Google > Upstream MM Testing`](https://hashicorp.teamcity.com/project/TerraformProviders_GoogleCloud_GOOGLE_MMUPSTREAMTESTS#all-projects) and [`Google Beta > Upstream MM Testing`](https://hashicorp.teamcity.com/project/TerraformProviders_GoogleCloud_GOOGLE_BETA_MMUPSTREAMTESTS#all-projects) run every night via CRON. They are redundant as both sweep the VCR project, but I've left them both in. They are designed to not run until there are no builds testing any services in the VCR test GCP project. No acceptance testing builds will start until the sweeper stops.

### Sweeping `google_project` Resources

When testing the GA and Beta providers we can run tests in parallel because those tests use separate GCP projects. This creates a boundary between the two test suites and ensures they don't clash. However if an acceptance test provisions `google_project` resources in the process then there is no longer a clear GA/Beta boundary based on which host project is in use. This makes sweeping up these resources tough, as there's potential to disrupt any other running build.

The `google_project` resource can be swept up safely if there are no other ongoing builds testing the GA/Beta Google providers. The `Project Sweeper` project contains a special build configuration for this sweeper that locks access to the GA/Beta/VCR GCP projects while it runs. This means the buid must wait for all other builds to stop before it starts, and while it is running no other Google-related builds can leave the queue and start running.
The `google_project` resource can be swept up safely if there are no other ongoing builds testing the GA/Beta Google providers. The `Project Sweeper` project contains a special build configuration for this sweeper that locks access to the GA/Beta/VCR GCP projects while it runs. This means the buid must wait for all other builds to stop before it starts, and while it is running no other Google-related builds can leave the queue and start running.

0 comments on commit d676d82

Please sign in to comment.