Repo Radius: externalize control-plane and Terraform state (rad startup / rad shutdown)

[sylvainsf]

Repo Radius: externalize control-plane and Terraform state (rad startup / rad shutdown)

Description

Implements Investment 2 of the Repo Radius feature spec — externalization of the Radius data
store, scoped to the state-storage aspects only (control-plane PostgreSQL state and Terraform
recipe state). It adds two kind-agnostic commands, rad startup and rad shutdown, that back up
and restore all durable Radius state across an ephemeral control plane, plus the Helm and
Terraform-state plumbing they depend on.

Design note: eng/design-notes/2026-06-repo-radius-state-storage.md.

There is no dedicated workspace kind. The commands operate on the current workspace's
Kubernetes context like any other command and do not create or delete clusters or install Radius —
cluster lifecycle is the caller's responsibility.

What's included

The change is organized as three logical commits:

1. PostgreSQL enablement (closes the chart gaps for database.enabled=true)

   • UCP, Applications RP, and Dynamic RP configmaps/deployments now switch to the postgresql
     provider and inject POSTGRES_PASSWORD when database.enabled=true (previously hardcoded to
     apiserver).
   • Adds an init-db ConfigMap (option 3 from FEATURE: Postgres DB initialization #8398) that creates the per-RP databases, users, and
     tables before the servers start.
   • Fixes the POSTGRES_DB secret value and pins the postgres image to 16-alpine.
   • Fixes the databaseProvider URL env-var substitution in factory.go.
   • Adds helm-unittest coverage and a unit test for the env-var helper.

2. Terraform recipe state backup/restore (pkg/cli/tfstate)

   • Terraform recipes store state in Kubernetes Secrets (tfstate=true), not in PostgreSQL, so
     that state is lost on teardown. This package exports and restores those Secrets, including the
     chunked tfstate-{workspace}-{suffix}-{index} Secrets for large state.

3. rad startup / rad shutdown + end-to-end lifecycle test

   • pkg/cli/pgbackup: control-plane PostgreSQL backup/restore via kubectl exec pg_dump/psql.
   • pkg/cli/gitstate: persists the state directory to a radius-state git orphan branch in an
     isolated worktree; the backup push fails loudly when a remote is configured (a failed push
     would otherwise be silent data loss) and tolerates the no-remote local/test case.
   • rad shutdown backs up both stores then commits and pushes; rad startup waits for the
     database then restores both stores.

Testing

Unit tests cover the chart rendering, the Terraform-state round-trip (fake Kubernetes client), the
git orphan-branch worktree behaviour (real temporary repos), and the command runners
(hand-written fakes). All pass, along with the existing 85 Helm chart tests.

End-to-end test dependency (please read)

The end-to-end lifecycle test lives at test/functional-portable/statestore and exercises the
full path that this work exists to protect: install → deploy a Terraform-backed resource →
rad shutdown → tear down → reinstall → rad startup → deploy an update to the same resource.
The update is the path that fails when Terraform state is lost.

This test depends on the separate Repo Radius workflow code (in flight) that creates the
ephemeral cluster, installs Radius, and runs the deploy. Because rad startup / rad shutdown
are intentionally kind-agnostic and do not manage cluster lifecycle, the test needs that workflow
to stand up the environment. Until it lands, the test drives the install/uninstall itself and is
gated behind the RADIUS_STATE_E2E environment variable, so it does not run in the normal
functional suite. Once the shared cluster-create + deploy workflow is merged, the test's
install/uninstall helpers should be re-pointed at that code rather than duplicating it.

Related

• Feature spec: Repo Radius (Add Repo Radius feature specification #12078)
• Relates to External data store for Radius #8096 (External data store for Radius), FEATURE: Postgres DB initialization #8398 (Postgres DB initialization)
• Supersedes the unmerged prototype feat(github-workspace): add GitHub Actions workspace kind with git-backed state persistence #11457

Type of change

This pull request adds new features (state externalization commands) for Radius.

[github-actions]

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Scanned Files

None

[Copilot]

Pull request overview

Adds Repo Radius “state externalization” support by enabling PostgreSQL-backed control-plane state and introducing rad shutdown / rad startup to back up/restore both PostgreSQL databases and Terraform recipe state (Kubernetes Secrets) via a git orphan branch.

Changes:

• Enable database.enabled=true end-to-end in the Helm chart (PostgreSQL provider wiring, init-db ConfigMap, image pinning, chart tests).
• Add Terraform-state Secret backup/restore (pkg/cli/tfstate) and PostgreSQL dump/restore (pkg/cli/pgbackup), persisted through a git orphan-branch worktree (pkg/cli/gitstate).
• Add rad startup / rad shutdown commands plus unit tests and an opt-in destructive functional lifecycle test.

Reviewed changes

Copilot reviewed 27 out of 27 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
test/functional-portable/statestore/noncloud/statestore_lifecycle_test.go	Adds opt-in destructive E2E lifecycle test for shutdown/startup preserving Terraform state across reinstall.
pkg/components/database/databaseprovider/storageprovider_test.go	Adds unit tests for ${VAR} env substitution in PostgreSQL connection URLs.
pkg/components/database/databaseprovider/factory.go	Implements ${VAR} expansion for PostgreSQL URL config via environment variables.
pkg/cli/tfstate/tfstate.go	New package to back up/restore Terraform Kubernetes-backend state Secrets to/from the state directory.
pkg/cli/tfstate/tfstate_test.go	Unit tests for tfstate backup/restore (fake clientset).
pkg/cli/pgbackup/pgbackup.go	New helpers to pg_dump / psql the control-plane DBs via kubectl exec, plus readiness wait.
pkg/cli/gitstate/gitstate.go	New orphan-branch worktree manager for persisting state outside the main working tree.
pkg/cli/gitstate/gitstate_test.go	Unit tests for orphan-branch worktree and push behavior using real temp git repos.
pkg/cli/cmd/startup/stateclient.go	Startup command wrapper interface over pg/tf restore operations (testable).
pkg/cli/cmd/startup/startup.go	Implements rad startup to open worktree, wait for DB, restore DB + Terraform state.
pkg/cli/cmd/startup/startup_test.go	Unit tests ensuring restore order and failure short-circuiting for rad startup.
pkg/cli/cmd/shutdown/stateclient.go	Shutdown command wrapper interface over pg/tf backup operations (testable).
pkg/cli/cmd/shutdown/shutdown.go	Implements rad shutdown to back up DB + Terraform state then commit/push to orphan branch.
pkg/cli/cmd/shutdown/shutdown_test.go	Unit tests ensuring both stores are backed up and commit/push happens (or fails correctly).
eng/design-notes/2026-06-repo-radius-state-storage.md	Adds technical design note for Repo Radius state storage approach and decisions.
deploy/Chart/values.yaml	Pins PostgreSQL image tag default to 16-alpine.
deploy/Chart/tests/database_test.yaml	Adds helm-unittest coverage for database-enabled/disabled rendering and wiring.
deploy/Chart/templates/ucp/deployment.yaml	Injects POSTGRES_PASSWORD env var into UCP when database.enabled=true.
deploy/Chart/templates/ucp/configmaps.yaml	Switches UCP database provider to PostgreSQL when enabled (URL uses ${POSTGRES_PASSWORD}).
deploy/Chart/templates/rp/deployment.yaml	Injects POSTGRES_PASSWORD into applications-rp when enabled.
deploy/Chart/templates/rp/configmaps.yaml	Switches applications-rp database provider to PostgreSQL when enabled.
deploy/Chart/templates/dynamic-rp/deployment.yaml	Injects POSTGRES_PASSWORD into dynamic-rp when enabled.
deploy/Chart/templates/dynamic-rp/configmaps.yaml	Switches dynamic-rp database provider to PostgreSQL when enabled.
deploy/Chart/templates/database/statefulset.yaml	Mounts init-db scripts into the Postgres StatefulSet.
deploy/Chart/templates/database/configmaps.yaml	Fixes POSTGRES_DB secret value to literal "radius".
deploy/Chart/templates/database/configmap-initdb.yaml	Adds init-db script ConfigMap to create per-RP DBs/users and the resources table.
cmd/rad/cmd/root.go	Wires rad startup and rad shutdown into the CLI root command.

[github-actions]

Unit Tests

    2 files  ± 0    450 suites  +8   7m 39s ⏱️ +10s
5 586 tests +44  5 584 ✅ +44  2 💤 ±0  0 ❌ ±0 
6 783 runs  +61  6 781 ✅ +61  2 💤 ±0  0 ❌ ±0 

Results for commit 5e2ee54. ± Comparison against base commit 385f38e.

♻️ This comment has been updated with latest results.

[codecov]

Codecov Report

❌ Patch coverage is 56.03774% with 233 lines in your changes missing coverage. Please review.
✅ Project coverage is 52.84%. Comparing base (385f38e) to head (5e2ee54).
⚠️ Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
pkg/cli/pgbackup/pgbackup.go	6.00%	94 Missing ⚠️
pkg/cli/tfstate/tfstate.go	66.29%	18 Missing and 12 partials ⚠️
pkg/cli/gitstate/gitstate.go	70.10%	16 Missing and 13 partials ⚠️
pkg/cli/cmd/startup/startup.go	72.50%	15 Missing and 7 partials ⚠️
pkg/cli/controlplane/controlplane.go	67.74%	13 Missing and 7 partials ⚠️
pkg/cli/cmd/shutdown/shutdown.go	72.58%	13 Missing and 4 partials ⚠️
pkg/cli/cmd/startup/stateclient.go	15.38%	11 Missing ⚠️
pkg/cli/cmd/shutdown/stateclient.go	22.22%	7 Missing ⚠️
...kg/components/database/databaseprovider/factory.go	78.57%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #12214      +/-   ##
==========================================
+ Coverage   52.81%   52.84%   +0.03%     
==========================================
  Files         743      751       +8     
  Lines       47788    48313     +525     
==========================================
+ Hits        25238    25532     +294     
- Misses      20197    20385     +188     
- Partials     2353     2396      +43     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[brooke-hamilton]

Review: externalize control-plane & Terraform state (rad startup / rad shutdown)

I reviewed the code statically, ran the unit tests, reproduced the failures against a real Postgres container, and ran a full local end-to-end install on kind with database.enabled=true. The new core Go packages (gitstate, tfstate, controlplane) are well-structured and the factory.go env-var fix is correct — but the database.enabled=true startup path does not work out of the box. There are two independent blocking bugs, both reproduced end-to-end, plus some non-blocking notes and test-coverage gaps.

The two blocking items also have inline comments anchored to the exact lines (deploy/Chart/values.yaml and deploy/Chart/templates/database/configmap-initdb.yaml).

---

🔴 Blocking

1. database.tag: 16-alpine points at a mirror image that does not exist

deploy/Chart/values.yaml:200-201 changes tag: latest → 16-alpine, resolving via the radius.image helper to ghcr.io/radius-project/mirror/postgres:16-alpine, which is not in the mirror. database-0 sits in ImagePullBackOff and the control plane never starts.

EXISTS     ghcr.io/radius-project/mirror/postgres:latest      <- pre-PR value
NOT FOUND  ghcr.io/radius-project/mirror/postgres:16-alpine   <- this PR's value
NOT FOUND  ghcr.io/radius-project/mirror/postgres:16
NOT FOUND  ghcr.io/radius-project/mirror/postgres:16.4-alpine
EXISTS     docker.io/library/postgres:16-alpine               <- real upstream

The upstream tag exists on Docker Hub, but the radius mirror was never populated with it. This breaks database.enabled=true in CI and every default-registry environment. I could only get past it with --set database.image=docker.io/library/postgres.

Fix (pick one): populate the mirror with postgres:16-alpine before merge, revert to a tag that exists in the mirror, or point image/tag at an upstream-resolvable reference. Either way, add CI coverage for database.enabled=true so this can't regress silently.

2. permission denied for table resources — init-db never grants the per-RP users

deploy/Chart/templates/database/configmap-initdb.yaml:33-48 creates the resources table while connected as $POSTGRES_USER (superuser radius), but each RP connects as its own per-RP user (ucp, applications_rp, dynamic_rp) and is never granted privileges. After overriding the image so Postgres started, init-db ran cleanly and created the per-RP DBs/tables, then UCP crashed on startup:

Service api terminated with error: ERROR: permission denied for table resources (SQLSTATE 42501)

followed by a nil-pointer panic on the failed-startup shutdown path. applications-rp/dynamic-rp hit the same wall as soon as they take traffic.

Fix: in the table-creation loop, grant the per-RP user (precedent already exists in build/scripts/start-radius.sh:247-248):

GRANT ALL PRIVILEGES ON TABLE resources TO "$RESOURCE_PROVIDER";
GRANT ALL PRIVILEGES ON ALL SEQUENCES IN SCHEMA public TO "$RESOURCE_PROVIDER";

or create the table while connected as the per-RP user instead of the superuser.

---

🟡 Non-blocking

• expandEnvURL silently expands unset vars to empty string (pkg/components/database/databaseprovider/factory.go:88-98). os.Getenv returns "" for a missing var, so a typo'd/unset variable yields a malformed connection string and a confusing downstream error rather than a clear "required env var X not set". Consider os.LookupEnv + fail-fast.
• init-db schema duplicates deploy/init-db/db.sql.txt. The resources DDL now lives in two places; they will drift. Source it from one place.
• replicasOf coerces 0 → 1. An explicit replicas: 0 (intentional scale-to-zero) is silently overridden to 1. Distinguish "unset" from "explicitly zero".
• Postgres password is regenerated on every chart apply, which will break an already-initialized data directory whose users were created with the previous password. Persist/Secret-pin it.
• rad startup/rad shutdown require a git repo but the error path when none is present could be friendlier.
• startup.Run defers a second ScaleUp that can double-scale on the success path. Worth a second look.

---

🧪 Test coverage

Core packages are well covered (gitstate ~80%, tfstate ~79%, controlplane ~78%). Gaps:

• pkg/cli/pgbackup has 0% coverage — no test file at all. HasBackup is pure file logic and trivially unit-testable; please add pgbackup_test.go.
• Validate() is 0% in both pkg/cli/cmd/startup and pkg/cli/cmd/shutdown. 51 other cmd packages exercise Validate() via the shared radcli.SharedCommandValidation harness — please follow that pattern. (Run() is otherwise well covered via fakes.)

---

How I validated

Static review of all 30 changed files → go build/go test on changed packages (pass) → standalone postgres:16-alpine container to reproduce the GRANT failure → full kind install with custom-built images and database.enabled=true to confirm both blockers end-to-end. Both reproduce from a clean install. Happy to re-validate once they're addressed.

[sylvainsf]

Thanks for the thorough end-to-end review — both blocking bugs are fixed in 9da4f58, and I've addressed most of the non-blocking notes. Summary:

🔴 Blocking — fixed

• Postgres image: now docker.io/library/postgres:16-alpine (pullable; mirror only has :latest).
• permission denied for table resources: init-db now grants the per-RP users on the table + sequences (matches start-radius.sh).

🟡 Non-blocking — fixed

• expandEnvURL fail-fast: now uses os.LookupEnv and returns an error listing any unset variables instead of silently producing a malformed connection string.
• replicasOf coercing 0 → 1: now preserves an explicit replicas: 0; only an unset count defaults to 1.
• startup.Run double-ScaleUp: the deferred scale-up is guarded by a scaledBackUp flag that the success path sets, so it never double-scales — it only runs as a safety net when a restore step returns early. Left as-is (correct), happy to refactor for clarity if you'd prefer.

🧪 Test coverage — fixed

• Added pkg/cli/pgbackup/pgbackup_test.go (covers HasBackup).
• Added Validate() + command-shape tests for rad startup and rad shutdown via the shared radcli harness.
• Added helm-unittest guards asserting the database image is the pullable reference and that the init-db script grants the per-RP users — so both blocking regressions are now caught in CI without a cluster.

Deferred (with rationale) — happy to do these if you'd like them in this PR

• CI job for database.enabled=true: the chart-render guards above catch the two regressions you hit cheaply. A full install-and-pull functional job is a larger CI/infra addition I'd prefer to scope separately (it also overlaps the gated statestore e2e lifecycle test). Tracking as a follow-up.
• init-db schema duplicates deploy/init-db/db.sql.txt: real drift risk, but db.sql.txt lives outside the chart directory so Helm .Files.Get can't source it. De-duplicating cleanly needs a small build step to vendor the SQL into the chart — proposing a follow-up rather than a risky restructure here.
• Postgres password regenerated on every apply (randAlphaNum 16): this is pre-existing (unchanged by this PR) and the proper fix is the lookup-or-generate Secret pattern, which is a separate hardening change. Filing a follow-up.

Ready for re-validation whenever you have a chance.

[sylvainsf]

Filed the deferred follow-ups as tracking issues:

• Add CI coverage for database.enabled=true install #12227 — Add CI coverage for database.enabled=true install
• De-duplicate the resources table schema between Helm init-db and start-radius.sh #12228 — De-duplicate the resources table schema between Helm init-db and start-radius.sh
• Postgres password is regenerated on every chart apply, breaking existing installs #12229 — Postgres password regenerated on every chart apply (pre-existing; breaks existing installs)

[github-actions]

Functional Tests - corerp-cloud

27 tests  ±0   27 ✅ ±0   36m 6s ⏱️ - 1m 43s
 2 suites ±0    0 💤 ±0 
 1 files   ±0    0 ❌ ±0 

Results for commit 5e2ee54. ± Comparison against base commit 385f38e.

♻️ This comment has been updated with latest results.

[radius-functional-tests]

Radius functional test overview

🔍 Go to test action run

Click here to see the test run details

Name	Value
Repository	radius-project/radius
Commit ref	5e2ee54
Unique ID	func602f934113
Image tag	pr-func602f934113

• KinD: v0.29.0
• Dapr: 1.14.4
• Azure KeyVault CSI driver: 1.4.2
• Azure Workload identity webhook: 1.3.0
• Bicep recipe location ghcr.io/radius-project/dev/test/testrecipes/test-bicep-recipes/<name>:pr-func602f934113
• Terraform recipe location http://tf-module-server.radius-test-tf-module-server.svc.cluster.local/<name>.zip (in cluster)
• applications-rp test image location: ghcr.io/radius-project/dev/applications-rp:pr-func602f934113
• dynamic-rp test image location: ghcr.io/radius-project/dev/dynamic-rp:pr-func602f934113
• controller test image location: ghcr.io/radius-project/dev/controller:pr-func602f934113
• ucp test image location: ghcr.io/radius-project/dev/ucpd:pr-func602f934113
• deployment-engine test image location: ghcr.io/radius-project/deployment-engine:latest

Test Status

⌛ Building Radius and pushing container images for functional tests...
✅ Container images build succeeded
⌛ Publishing Bicep Recipes for functional tests...
✅ Recipe publishing succeeded
⌛ Starting ucp-cloud functional tests...
⌛ Starting corerp-cloud functional tests...
✅ ucp-cloud functional tests succeeded
✅ corerp-cloud functional tests succeeded

[github-actions]

Functional Tests - upgrade-noncloud

3 tests  ±0   1 ✅  - 2   7m 9s ⏱️ + 3m 45s
1 suites ±0   0 💤 ±0 
1 files   ±0   2 ❌ +2 

For more details on these failures, see this check.

Results for commit 5e2ee54. ± Comparison against base commit 385f38e.