chore: run spanner integration tests against a real test instance (#604) by IAM-marco · Pull Request #629 · zitadel/nextgen

IAM-marco · 2026-07-24T15:58:58Z

Summary

The Cloud Spanner emulator only supports one transaction at a time, so concurrent spanner_integration tests are flaky (Transaction ... aborted due to another transaction getting priority. The emulator only supports one transaction at a time). This lets the integration suites run against a real, long-lived Spanner test instance, giving each run its own isolated database. Fixes #604.

New internal/storage/database/dialect/spanner/testdb helper: Provision creates a uniquely named database on an existing instance (via ZITADEL_TEST_SPANNER_INSTANCE=projects/<p>/instances/<i>, ADC auth) and drops it on teardown. Database ids are itest_<run-id>_<random>, always including a random suffix so the two test binaries in one CI run never collide.
Emulator path refactored to reuse the same CreateDatabase helper (emulator kept as the local default).
Both suites (repository + API integration) wired for the new mode. Precedence: ZITADEL_TEST_SPANNER_INSTANCE > ZITADEL_TEST_SPANNER_URL > emulator container > Postgres.
CI authenticates to Google Cloud with Workload Identity Federation on trusted runs and targets the shared instance; fork PRs fall back to the emulator. The auth + spanner steps run last in the job so their (currently expected) failure does not skip the unrelated build/journey/e2e steps.
CONTRIBUTING.md documents the new env var and precedence.

Important

The Authenticate to Google Cloud (Spanner test instance) step is expected to FAIL on this PR and will keep failing on trusted runs until the GCP infrastructure and GCP_* secrets are provisioned (see prerequisite below). This is intentional: a red check is a visible signal that the real test instance is not yet configured, rather than silently masking it. It runs last so every unrelated step still executes and validates.

Validation

gofmt -l on all touched files: clean.
go vet -tags spanner_integration ./internal/storage/database/dialect/spanner/... ./internal/storage/database/repository/ ./internal/storage/database/dbtest/ ./internal/api/integration_test/: clean.
go vet -tags postgres_integration ./internal/storage/database/repository/ (stub path): clean.
ci.yml validated as well-formed YAML.
Not run: the integration suite itself was not executed locally (no Docker for the emulator path / no GCP credentials for the instance path). Relying on CI's emulator run for now.

Release notes / changeset

No changeset required — no shipped behavior changed (test-only Go code under the spanner_integration build tag, CI workflow, and docs).

Notes

CI auth choice: Workload Identity Federation (keyless) was chosen. The considered alternative was a service-account JSON key stored as a GitHub secret: simpler to set up, but a long-lived credential to manage and rotate. WIF avoids the standing secret.

One-time infra prerequisite (not in this PR). Before trusted CI actually targets the real instance, ops must provision:

A Google Cloud test project with the Spanner API enabled.
One small Spanner instance (minimal processing units / regional config).
A service account with Spanner Database Admin (create/drop databases) + Spanner Database User (read/write).
A Workload Identity Federation pool + provider bound to zitadel/nextgen; grant the SA roles/iam.workloadIdentityUser.
GitHub Actions config: secrets GCP_WORKLOAD_IDENTITY_PROVIDER and GCP_SERVICE_ACCOUNT, and variable SPANNER_TEST_INSTANCE = projects/<project>/instances/<instance>.

Until these exist, the auth step fails by design on trusted runs (see the note above). Because it runs last and the trailing artifact-upload steps use always()/failure(), no unrelated step is skipped or fails on its account.

The Spanner emulator only supports one transaction at a time, so concurrent integration tests are flaky. Add a testdb helper that creates a uniquely named database on an existing instance (dropped on teardown) so the suites can target a real, long-lived test instance with isolated per-run databases. Refactor the emulator path to reuse the shared CreateDatabase helper. Refs #604 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Wire ZITADEL_TEST_SPANNER_INSTANCE into both the repository and API integration suites. When set, they provision a fresh database on the shared instance and drop it after the run. Precedence: instance > URL > emulator container > Postgres. The postgres-only stub keeps that build compiling since the testdb package is spanner_integration-only. Refs #604 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Authenticate to Google Cloud with Workload Identity Federation (keyless) on trusted runs and point the spanner integration step at the shared instance via ZITADEL_TEST_SPANNER_INSTANCE. Fork PRs (no secrets) and unconfigured setups resolve the env empty and fall back to the emulator. Refs #604 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Document ZITADEL_TEST_SPANNER_INSTANCE, ADC authentication, and the override precedence in the Go database integration tests section. Refs #604 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

vercel · 2026-07-24T15:59:01Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
nextgen	Ready	Preview	Jul 31, 2026 9:26am
nextgen-docs	Ready	Preview	Jul 31, 2026 9:26am
nextgen-mock-zitadel	Ready	Preview	Jul 31, 2026 9:26am

github-actions · 2026-07-24T16:04:04Z

⚠️ No Changeset found

Latest commit: 999f1fd

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

Copilot

Pull request overview

Enables Spanner integration tests to run reliably under concurrency by optionally provisioning a fresh per-run database on a shared real Spanner test instance (instead of relying solely on the single-transaction emulator), with CI and docs updates to support the new mode.

Changes:

Added a spanner_integration-tagged testdb helper that provisions and later drops a uniquely named database on a configured Spanner instance.
Refactored the emulator setup to reuse the same database-create helper and wired repository integration tests to prefer the shared-instance mode when configured.
Updated CI to authenticate to GCP via Workload Identity Federation on trusted runs and set the instance env var; documented the new env var and precedence in CONTRIBUTING.md.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
internal/storage/database/repository/repository_test.go	Adds shared-instance precedence before URL/container/emulator selection for repository integration tests.
internal/storage/database/repository/repository_spanner_stub_test.go	Adds stubbed shared-instance hooks for non-spanner builds.
internal/storage/database/repository/repository_spanner_setup_test.go	Implements shared-instance provisioning + migration for `spanner_integration` tests.
internal/storage/database/dialect/spanner/testdb/testdb.go	New helper to create/drop per-run databases on a real Spanner instance (ADC/WIF auth).
internal/storage/database/dialect/spanner/embedded/start.go	Reuses `testdb.CreateDatabase` for emulator database creation.
internal/storage/database/dbtest/spanner.go	Updates Spanner test connector selection precedence to include shared-instance provisioning.
CONTRIBUTING.md	Documents `ZITADEL_TEST_SPANNER_INSTANCE` and its precedence over URL/emulator.
.github/workflows/ci.yml	Adds OIDC permissions, optional GCP auth step, and passes the instance env var into the spanner test step.

Move the Spanner auth and integration steps to the end of the job. Until the GCP infrastructure and GCP_* secrets are provisioned, the auth step fails on trusted runs by design (a visible signal that the real test instance is not configured yet); running it last keeps that expected failure from skipping the unrelated build, journey, and e2e steps. The trailing artifact-upload steps use always()/failure() and are unaffected. Refs #604 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…test-instance # Conflicts: # .github/workflows/ci.yml # internal/storage/database/dbtest/spanner.go # internal/storage/database/repository/repository_spanner_setup_test.go # internal/storage/database/repository/repository_spanner_stub_test.go # internal/storage/database/repository/repository_test.go # internal/storage/v2/testdb/spanner.go Co-authored-by: Silvan <adlerhurst@users.noreply.github.com>

Skip GCP auth unless the WIF secrets and SPANNER_TEST_INSTANCE var are set, and run Spanner integration tests with -parallel 1 -p 1 when no shared instance is configured so the emulator stays reliable for OSS and unconfigured CI. Co-authored-by: Silvan <adlerhurst@users.noreply.github.com>

Drop job-level GCP secret env. Always attempt Spanner WIF auth on trusted full runs with continue-on-error; set ZITADEL_TEST_SPANNER_INSTANCE only when auth succeeds, otherwise leave it empty so server:test-spanner uses the serial emulator path. Co-authored-by: Silvan <adlerhurst@users.noreply.github.com>

Keep a single server:test-spanner Moon task for local use. Split the CI Spanner suite into two mutually exclusive steps so the job summary shows whether the emulator or the real test instance ran. Co-authored-by: Silvan <adlerhurst@users.noreply.github.com>

Goose migrations are embedded via //go:embed sql/*.sql. Without tracking *.sql in serverInputs, Moon could cache-hit test/build after schema-only changes while Go would rebuild if invoked. Co-authored-by: Silvan <adlerhurst@users.noreply.github.com>

Inline the serial-emulator Moon script, unexport testdb provisioning helpers, keep uniqueness ahead of any ID clamp, and gate WIF auth on the public SPANNER_TEST_INSTANCE var so misconfigured infra hard-fails instead of silently falling back to the emulator. Co-authored-by: Silvan <adlerhurst@users.noreply.github.com>

## Summary  - Enables some integration tests to run when spanner. - Use `require.IsType` instead of type-case + `require.True`. ## Validation - `go test ./...` - `go test -tags postgres_integration` - `go test -tags spanner_integration` ## Release notes / changeset - No changeset required ## Notes - blocked by #629 --------- Co-authored-by: adlerhurst <27845747+adlerhurst@users.noreply.github.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Silvan <adlerhurst@users.noreply.github.com>

IAM-marco and others added 4 commits July 24, 2026 17:57

docs(spanner): document the shared test instance env var

823119e

Document ZITADEL_TEST_SPANNER_INSTANCE, ADC authentication, and the override precedence in the Go database integration tests section. Refs #604 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Copilot AI review requested due to automatic review settings July 24, 2026 15:59

vercel Bot deployed to Preview – nextgen-mock-zitadel July 24, 2026 15:59 View deployment

vercel Bot deployed to Preview – nextgen-docs July 24, 2026 15:59 View deployment

zitadel-kanban Bot added this to Engineering Kanban Jul 24, 2026

zitadel-kanban Bot moved this to Inbox in Engineering Kanban Jul 24, 2026

Copilot started reviewing on behalf of IAM-marco July 24, 2026 16:03 View session

Copilot AI reviewed Jul 24, 2026

View reviewed changes

Comment thread .github/workflows/ci.yml Outdated

IAM-marco changed the title ~~test(spanner): run integration tests against a real test instance (#604)~~ test: run integration tests against a real test instance (#604) Jul 24, 2026

IAM-marco changed the title ~~test: run integration tests against a real test instance (#604)~~ feat: run integration tests against a real test instance (#604) Jul 24, 2026

IAM-marco changed the title ~~feat: run integration tests against a real test instance (#604)~~ feat: run spanner integration tests against a real test instance (#604) Jul 24, 2026

vercel Bot deployed to Preview – nextgen July 24, 2026 16:10 View deployment

vercel Bot deployed to Preview – nextgen-mock-zitadel July 24, 2026 16:10 View deployment

vercel Bot deployed to Preview – nextgen-docs July 24, 2026 16:11 View deployment

IAM-marco changed the title ~~feat: run spanner integration tests against a real test instance (#604)~~ feat: run spanner integration tests against a real test instance (#604) Jul 24, 2026

elinashoko requested a review from adlerhurst July 29, 2026 09:28

adlerhurst mentioned this pull request Jul 31, 2026

test: enabled more integration tests for spanner #670

Merged

vercel Bot deployed to Preview – nextgen-mock-zitadel July 31, 2026 04:38 View deployment

vercel Bot deployed to Preview – nextgen July 31, 2026 04:38 View deployment

vercel Bot deployed to Preview – nextgen-docs July 31, 2026 04:39 View deployment

adlerhurst reviewed Jul 31, 2026

View reviewed changes

Comment thread .github/workflows/ci.yml Outdated

vercel Bot deployed to Preview – nextgen July 31, 2026 05:12 View deployment

vercel Bot deployed to Preview – nextgen-mock-zitadel July 31, 2026 05:20 View deployment

vercel Bot deployed to Preview – nextgen-docs July 31, 2026 05:21 View deployment

vercel Bot deployed to Preview – nextgen July 31, 2026 05:31 View deployment

vercel Bot deployed to Preview – nextgen-mock-zitadel July 31, 2026 05:31 View deployment

vercel Bot deployed to Preview – nextgen-docs July 31, 2026 05:31 View deployment

vercel Bot deployed to Preview – nextgen-mock-zitadel July 31, 2026 05:45 View deployment

vercel Bot deployed to Preview – nextgen July 31, 2026 05:45 View deployment

vercel Bot deployed to Preview – nextgen-docs July 31, 2026 05:46 View deployment

vercel Bot temporarily deployed to Preview – nextgen-mock-zitadel July 31, 2026 05:59 Inactive

vercel Bot temporarily deployed to Preview – nextgen-docs July 31, 2026 05:59 Inactive

vercel Bot deployed to Preview – nextgen July 31, 2026 05:59 View deployment

vercel Bot deployed to Preview – nextgen July 31, 2026 06:20 View deployment

vercel Bot deployed to Preview – nextgen-mock-zitadel July 31, 2026 06:20 View deployment

vercel Bot deployed to Preview – nextgen-docs July 31, 2026 06:20 View deployment

adlerhurst changed the title ~~feat: run spanner integration tests against a real test instance (#604)~~ chore: run spanner integration tests against a real test instance (#604) Jul 31, 2026

Merge branch 'main' into 604-spanner-emulator-test-instance

999f1fd

adlerhurst enabled auto-merge (squash) July 31, 2026 09:25

vercel Bot deployed to Preview – nextgen July 31, 2026 09:26 View deployment

vercel Bot deployed to Preview – nextgen-mock-zitadel July 31, 2026 09:26 View deployment

vercel Bot deployed to Preview – nextgen-docs July 31, 2026 09:26 View deployment

adlerhurst merged commit 5e2f1b7 into main Jul 31, 2026
14 checks passed

adlerhurst deleted the 604-spanner-emulator-test-instance branch July 31, 2026 09:32

github-project-automation Bot moved this from Inbox to Done in Engineering Kanban Jul 31, 2026

This was referenced Aug 7, 2026

Make Spanner abort-retry provable: preserve the ABORTED status, bound it, replay-safe closures #788

Open

fix(session): keep Spanner ABORTED retryable through exchange conflicts #795

Open

Navigation Menu

Uh oh!

chore: run spanner integration tests against a real test instance (#604) - #629

chore: run spanner integration tests against a real test instance (#604) #629
adlerhurst merged 13 commits into
mainfrom
604-spanner-emulator-test-instance

IAM-marco commented Jul 24, 2026 •

edited

Loading

Uh oh!

vercel Bot commented Jul 24, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 24, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

IAM-marco commented Jul 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Validation

Release notes / changeset

Notes

Uh oh!

vercel Bot commented Jul 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ No Changeset found

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

IAM-marco commented Jul 24, 2026 •

edited

Loading

vercel Bot commented Jul 24, 2026 •

edited

Loading

github-actions Bot commented Jul 24, 2026 •

edited

Loading