Bug Description
Stage 1 enables 22 APIs on each tenant main project (fast/stages-aw/1-resman/branch-tenants.tf, services lists at lines 183+ and 308+); neither sqladmin.googleapis.com nor appengine.googleapis.com is among them. A sweep of all 56 blueprints at 3728fc9 shows the PostgreSQL and App Engine blueprints are the only two that deploy into an existing Stage-1 tenant project, hard-require an API absent from that list, and contain no google_project_service resource to enable it. Specifically: postgresql's only service-related resource is google_project_service_identity (main.tf:24-27), which references the Cloud SQL service agent but does not enable the API — so google_sql_database_instance fails with 403 SERVICE_DISABLED, and the KMS grant to the (never actually provisioned) service agent misbehaves alongside it; the app-engine blueprint is a bare google_app_engine_application via modules/app-engine, with no enablement in either blueprint or module. This is an outlier pattern: the majority of blueprints enable their required services explicitly via google_project_service (e.g. cloud-run, cloud-functions, bigquery, datastore, firestore, gke, secret-manager, document-ai, cloud-ids, and ~20 more), others need only APIs Stage 1 already enables (compute, cloudkms, storage, pubsub, bigquery, container, …), and the rest create their own projects with their own service lists (vertex-mlops, gitlab). Both failures and the manual workaround are live-verified.
Environment and Deployment Context
- Stellar Engine Version/Commit:
main at commit 3728fc98
- Deployment Type:
- FAST Stage (if applicable):
- Affected Component:
fast/stages-aw/1-resman/branch-tenants.tf (tenant services lists, lines 183+/308+ — zero occurrences of either API); blueprints/il5/postgresql/main.tf (google_project_service count: 0; google_project_service_identity at line 24); blueprints/fedramp-high/app-engine/main.tf + modules/app-engine/main.tf (google_project_service count: 0)
- Terraform Version: 1.15.7
- GCP Provider Version:
hashicorp/google 6.50.0 (postgresql lock file; the failure is API-enablement and provider-version-independent)
Steps to Reproduce
- Deploy Stages 0–1 with a fresh tenant (services list as shipped, unmodified).
- Deploy
blueprints/il5/postgresql against the tenant's main project. The apply fails on google_sql_database_instance with 403 SERVICE_DISABLED (log below).
- Run
gcloud services enable sqladmin.googleapis.com --project=<tenant-main-project> and re-apply — it now succeeds. The manual enable is the only intervention required, confirming root cause.
- Repeat with
blueprints/fedramp-high/app-engine: google_app_engine_application fails identically until gcloud services enable appengine.googleapis.com is run; after enablement the apply succeeds (module.app-engine.google_app_engine_application.app confirmed in state).
- Code confirmation:
grep -c "sqladmin\|appengine" fast/stages-aw/1-resman/branch-tenants.tf → 0; grep -c 'resource "google_project_service"' blueprints/il5/postgresql/main.tf blueprints/fedramp-high/app-engine/main.tf modules/app-engine/main.tf → 0 in each.
Expected Behavior
Blueprints shipped for tenant projects can apply against a fresh tenant: either Stage 1 enables the APIs the shipped workload blueprints require (as it already does for e.g. cloudkms.googleapis.com), or these two blueprints enable their own APIs via google_project_service, as the majority of sibling blueprints already do.
Actual Behavior
Every fresh-tenant deployment of either blueprint hard-fails on apply and cannot self-heal; the operator must discover and run a manual gcloud services enable that no README documents.
Relevant Logs and Errors
Captured from a live failed apply (postgresql, TF_LOG debug; project number redacted):
google_project_service_identity.cloudsql_sa: Refreshing state...
[id=projects/<tenant-main-project>/services/sqladmin.googleapis.com]
...
Error: Error, failed to create instance report: googleapi: Error 403:
Cloud SQL Admin API has not been used in project <PROJECT_NUMBER> before or it
is disabled. Enable it by visiting
https://console.developers.google.com/apis/api/sqladmin.googleapis.com/overview?project=<PROJECT_NUMBER>
then retry. ...
"reason": "SERVICE_DISABLED"
"service": "sqladmin.googleapis.com"
Note the misleading shape: google_project_service_identity.cloudsql_sa refreshes without error while the instance create fails — nothing points to a missing enablement in the framework itself as the root cause.
Additional Context
Blocks primary workload deliverables (Cloud SQL, App Engine) on every fresh tenant. blueprints/fedramp-high/postgresql is a symlink to blueprints/il5/postgresql, so FRH and IL5 consume identical code and are equally affected. Proposed fix (either or both): add sqladmin.googleapis.com and appengine.googleapis.com to Stage 1's tenant services lists, and/or add google_project_service resources to the two blueprints, matching the majority-blueprint pattern. Note that enabling the API does not by itself make impersonated deploys work — see the companion documentation issue on deployer permissions and user_project_override: {companion-issue link}. Related (not duplicate): #47 proposes centralizing bootstrap-level prerequisite/API documentation; this issue is a code gap at the tenant/blueprint level.
Bug Description
Stage 1 enables 22 APIs on each tenant main project (fast/stages-aw/1-resman/branch-tenants.tf, services lists at lines 183+ and 308+); neither sqladmin.googleapis.com nor appengine.googleapis.com is among them. A sweep of all 56 blueprints at 3728fc9 shows the PostgreSQL and App Engine blueprints are the only two that deploy into an existing Stage-1 tenant project, hard-require an API absent from that list, and contain no google_project_service resource to enable it. Specifically: postgresql's only service-related resource is google_project_service_identity (main.tf:24-27), which references the Cloud SQL service agent but does not enable the API — so google_sql_database_instance fails with 403 SERVICE_DISABLED, and the KMS grant to the (never actually provisioned) service agent misbehaves alongside it; the app-engine blueprint is a bare google_app_engine_application via modules/app-engine, with no enablement in either blueprint or module. This is an outlier pattern: the majority of blueprints enable their required services explicitly via google_project_service (e.g. cloud-run, cloud-functions, bigquery, datastore, firestore, gke, secret-manager, document-ai, cloud-ids, and ~20 more), others need only APIs Stage 1 already enables (compute, cloudkms, storage, pubsub, bigquery, container, …), and the rest create their own projects with their own service lists (vertex-mlops, gitlab). Both failures and the manual workaround are live-verified.
Environment and Deployment Context
mainat commit3728fc98fast/stages-aw/1-resman/branch-tenants.tf(tenant services lists, lines 183+/308+ — zero occurrences of either API);blueprints/il5/postgresql/main.tf(google_project_servicecount: 0;google_project_service_identityat line 24);blueprints/fedramp-high/app-engine/main.tf+modules/app-engine/main.tf(google_project_servicecount: 0)hashicorp/google6.50.0 (postgresql lock file; the failure is API-enablement and provider-version-independent)Steps to Reproduce
blueprints/il5/postgresqlagainst the tenant's main project. The apply fails ongoogle_sql_database_instancewith 403SERVICE_DISABLED(log below).gcloud services enable sqladmin.googleapis.com --project=<tenant-main-project>and re-apply — it now succeeds. The manual enable is the only intervention required, confirming root cause.blueprints/fedramp-high/app-engine:google_app_engine_applicationfails identically untilgcloud services enable appengine.googleapis.comis run; after enablement the apply succeeds (module.app-engine.google_app_engine_application.appconfirmed in state).grep -c "sqladmin\|appengine" fast/stages-aw/1-resman/branch-tenants.tf→ 0;grep -c 'resource "google_project_service"' blueprints/il5/postgresql/main.tf blueprints/fedramp-high/app-engine/main.tf modules/app-engine/main.tf→ 0 in each.Expected Behavior
Blueprints shipped for tenant projects can apply against a fresh tenant: either Stage 1 enables the APIs the shipped workload blueprints require (as it already does for e.g.
cloudkms.googleapis.com), or these two blueprints enable their own APIs viagoogle_project_service, as the majority of sibling blueprints already do.Actual Behavior
Every fresh-tenant deployment of either blueprint hard-fails on apply and cannot self-heal; the operator must discover and run a manual
gcloud services enablethat no README documents.Relevant Logs and Errors
Captured from a live failed apply (postgresql,
TF_LOGdebug; project number redacted):Note the misleading shape:
google_project_service_identity.cloudsql_sarefreshes without error while the instance create fails — nothing points to a missing enablement in the framework itself as the root cause.Additional Context
Blocks primary workload deliverables (Cloud SQL, App Engine) on every fresh tenant.
blueprints/fedramp-high/postgresqlis a symlink toblueprints/il5/postgresql, so FRH and IL5 consume identical code and are equally affected. Proposed fix (either or both): addsqladmin.googleapis.comandappengine.googleapis.comto Stage 1's tenant services lists, and/or addgoogle_project_serviceresources to the two blueprints, matching the majority-blueprint pattern. Note that enabling the API does not by itself make impersonated deploys work — see the companion documentation issue on deployer permissions anduser_project_override: {companion-issue link}. Related (not duplicate): #47 proposes centralizing bootstrap-level prerequisite/API documentation; this issue is a code gap at the tenant/blueprint level.