Terraform

Terraform (GCP)

English | 日本語

Deploy maestro on Google Cloud with the modules under terraform/: WebUI/API (app), dedicated MCP Gateway, self-hosted worker (poller Worker Pool + session Job), and a Global HTTPS edge (LB, IAP, Cloud Armor, Certificate Manager).

In-repo reference (inputs, security table, out-of-scope list): terraform/README.md.

What you get

                    ┌──────────────────────────────────────────┐
  browsers ────────►│ Global HTTPS LB                          │
                    │  app.example.com  ─IAP──► Cloud Run app  │
  GitHub webhook ──►│  POST /webhooks/… ─open─► (manual = 1)   │
  Anthropic MCP ───►│  mcp.example.com ─Armor─► Cloud Run mcp  │
                    └──────────────────────────────────────────┘

  Anthropic work queue ◄──outbound── Cloud Run Worker Pool (poller)
                                         │ gcloud run jobs execute
                                         ▼
                                   Cloud Run Job (session × N)

Module	Role
modules/app	Cloud Run service (start.sh), Litestream GCS bucket, app secrets/SA
modules/mcp-gateway	Dedicated Cloud Run (start-mcp.sh), MCP-only secrets/SA
modules/worker	Worker Pool poller + session Job, outputs bucket, worker secrets/SAs
modules/edge	Global IP, certs, serverless NEGs, IAP/open backends, Cloud Armor, URL map
examples/complete	Wires all four modules together

Prerequisites (outside Terraform)

1. GCP project with billing and APIs: run, secretmanager, compute, certificatemanager, iap, storage, iam.
2. Existing VPC + subnet + Cloud NAT with a static egress IP (GitHub App / Anthropic / registries). Modules only attach Direct VPC egress (ALL_TRAFFIC).
3. Images the project can pull:
   • maestro app/MCP image from this repo (make docker-build / make docker-push)
   • self-hosted worker image (ops-owned; must provide ant beta:worker and /spawn.sh)
4. Anthropic Console self-hosted environment — keep env_… and environment key sk-ant-oat01-….
5. IAP OAuth brand + client created outside Terraform. Pass client id/secret into modules/edge.
6. DNS control of app_hostname and mcp_hostname (Certificate Manager CNAMEs, then LB A records).

Also complete the shared maestro setup from Installation: GitHub App + Anthropic API key.

Quick start

git clone https://github.com/ca-srg/maestro.git
cd maestro/terraform/examples/complete
cp terraform.tfvars.example terraform.tfvars
# edit terraform.tfvars

terraform init
terraform plan
terraform apply

Minimal terraform.tfvars fields (see the example file for the full set):

Variable	Purpose
project_id, region, name_prefix	Placement / naming
app_hostname, mcp_hostname	Public FQDNs
app_image	maestro image with tag (same image for app + MCP; different command)
worker_image, worker_version	Self-hosted worker repo + tag
network, subnetwork	Existing VPC
anthropic_environment_id	Self-hosted env id on the app only
iap_oauth2_client_id / secret, iap_members	Browser access via IAP

Provider pin: Terraform >= 1.5.0, hashicorp/google and google-beta >= 7.24.0 (Worker Pools need google-beta).

Inject secrets after apply

Secret containers are empty. Add versions before services stay healthy. Exact IDs are in module outputs (app_secret_ids, mcp_secret_ids, worker_secret_ids).

# App
echo -n "$ANTHROPIC_API_KEY" | gcloud secrets versions add maestro-app-anthropic-api-key --data-file=-
echo -n "$GITHUB_APP_ID"     | gcloud secrets versions add maestro-app-github-app-id --data-file=-
echo -n "$GITHUB_WEBHOOK_SECRET" | gcloud secrets versions add maestro-app-github-webhook-secret --data-file=-
gcloud secrets versions add maestro-app-github-app-private-key --data-file=./github-app.pem
# OIDC (when enable_oidc=true on the app module)
echo -n "$OIDC_ISSUER_URL"     | gcloud secrets versions add maestro-app-oidc-issuer-url --data-file=-
echo -n "$OIDC_CLIENT_ID"      | gcloud secrets versions add maestro-app-oidc-client-id --data-file=-
echo -n "$OIDC_CLIENT_SECRET"  | gcloud secrets versions add maestro-app-oidc-client-secret --data-file=-
echo -n "$OIDC_REDIRECT_URI"   | gcloud secrets versions add maestro-app-oidc-redirect-uri --data-file=-
echo -n "$OIDC_ADMIN_EMAILS"   | gcloud secrets versions add maestro-app-oidc-admin-emails --data-file=-
echo -n "$SESSION_SECRET"      | gcloud secrets versions add maestro-app-session-secret --data-file=-

# MCP gateway (separate SA — never share app secrets here)
echo -n "$MCP_GATEWAY_TOKEN" | gcloud secrets versions add maestro-mcp-mcp-gateway-token --data-file=-
echo -n "$FIGMA_API_KEY"     | gcloud secrets versions add maestro-mcp-figma-api-key --data-file=-

# Worker (environment KEY must never reach the app)
echo -n "$ANTHROPIC_ENVIRONMENT_KEY" | gcloud secrets versions add maestro-anthropic-environment-key --data-file=-
echo -n "$ANTHROPIC_ENVIRONMENT_ID"  | gcloud secrets versions add maestro-anthropic-environment-id --data-file=-
gcloud secrets versions add maestro-worker-github-app-private-key --data-file=./github-app.pem
echo -n "$GITHUB_APP_ID"       | gcloud secrets versions add maestro-session-github-app-id --data-file=-
echo -n "$MCP_GATEWAY_TOKEN"   | gcloud secrets versions add maestro-session-mcp-gateway-token --data-file=-

Prefix names follow name_prefix (default maestro). Prefer the Terraform outputs over hard-coded IDs.

DNS cutover (Certificate Manager)

1. terraform apply → read dns_authorization_records
2. Create the two PER_PROJECT_RECORD CNAMEs at your DNS provider — do not move apex/app A records yet
3. Wait until both managed certificates are ACTIVE
   (gcloud certificate-manager certificates list --location=global)
4. Point app_hostname and mcp_hostname A records at lb_ip_address
5. Keep any previous origin until health checks pass

App single-writer / Litestream revision updates

The app service uses manual scaling with exactly one instance. Litestream forbids concurrent writers. Every revision-generating change (image, env, secret ref, CPU/memory, VPC, …) needs three applies:

# 1) drain
manual_instance_count = 0
# terraform apply  → wait until the instance and Litestream stop

# 2) change the template while still at 0
image = "…:new"
# terraform apply

# 3) restore
manual_instance_count = 1
# terraform apply

Do not use traffic tags, revision-level minimums, or a normal rolling revision update while Litestream is the SQLite path.

Security boundaries (summary)

Surface	Control
Browser → app	URL map → IAP → Cloud Run (INTERNAL_LOAD_BALANCER, default_uri_disabled) → app OIDC
GitHub review webhook	Exact POST path → open backend → Cloud Armor → app X-Hub-Signature-256
Anthropic → MCP	Host + /servers/<name>/mcp + method + source 160.79.104.0/21 (Armor) → Bearer MCP_GATEWAY_TOKEN
App SA vs MCP SA	Split services so metadata credentials cannot cross secret boundaries
Worker session SA	Blast radius of agent-generated code; session secrets + outputs bucket only
Environment key	Worker only. App gets ANTHROPIC_ENVIRONMENT_ID as plain env

roles/run.invoker for allUsers on app/MCP is intentional with LB-only ingress, URL-map routing, Cloud Armor on open backends, and app-level Bearer/HMAC checks. Details: terraform/README.md.

Out of scope

• VPC / Cloud Router / Cloud NAT creation
• DNS record resources (outputs give values)
• IAP OAuth brand / consent screen
• Artifact Registry repos and image builds
• Worker image source / patches (ops-owned)
• Writing secret values (containers only)

After deploy

1. Open https://<app_hostname> through IAP
2. Register MCP servers in the WebUI with public URLs on mcp_hostname (see MCP-Servers)
3. Confirm self-hosted: app has environment id only; workers hold the key (Installation → Self-hosted)
4. Run a small issue end-to-end

Next

• Installation — GitHub App and shared secrets
• Architecture — runtime components
• Configuration — env and WebUI knobs
• Troubleshooting — when something fails
• Full module variable tables: terraform/README.md