Skip to content

[Bug] postgresql and app-engine blueprints require APIs the framework never enables (sqladmin.googleapis.com, appengine.googleapis.com) — fresh-tenant applies fail until the operator enables them manually #104

Description

@JohnHales

Bug Description

Stage 1 enables 22 APIs on each tenant main project (fast/stages-aw/1-resman/branch-tenants.tf, services lists at lines 183+ and 308+); neither sqladmin.googleapis.com nor appengine.googleapis.com is among them. A sweep of all 56 blueprints at 3728fc9 shows the PostgreSQL and App Engine blueprints are the only two that deploy into an existing Stage-1 tenant project, hard-require an API absent from that list, and contain no google_project_service resource to enable it. Specifically: postgresql's only service-related resource is google_project_service_identity (main.tf:24-27), which references the Cloud SQL service agent but does not enable the API — so google_sql_database_instance fails with 403 SERVICE_DISABLED, and the KMS grant to the (never actually provisioned) service agent misbehaves alongside it; the app-engine blueprint is a bare google_app_engine_application via modules/app-engine, with no enablement in either blueprint or module. This is an outlier pattern: the majority of blueprints enable their required services explicitly via google_project_service (e.g. cloud-run, cloud-functions, bigquery, datastore, firestore, gke, secret-manager, document-ai, cloud-ids, and ~20 more), others need only APIs Stage 1 already enables (compute, cloudkms, storage, pubsub, bigquery, container, …), and the rest create their own projects with their own service lists (vertex-mlops, gitlab). Both failures and the manual workaround are live-verified.

Environment and Deployment Context

  • Stellar Engine Version/Commit: main at commit 3728fc98
  • Deployment Type:
    • US Region Restricted (e.g., Access Policy constraint)
    • FedRAMP Medium
    • FedRAMP High
    • DoD IL4
    • DoD IL5
    • Stand-alone / Custom
  • FAST Stage (if applicable):
    • Stage 0 (Bootstrap)
    • Stage 1 (Resource Management)
    • Stage 2 (Network Creation)
    • Stage 3 (Security and Audit)
  • Affected Component: fast/stages-aw/1-resman/branch-tenants.tf (tenant services lists, lines 183+/308+ — zero occurrences of either API); blueprints/il5/postgresql/main.tf (google_project_service count: 0; google_project_service_identity at line 24); blueprints/fedramp-high/app-engine/main.tf + modules/app-engine/main.tf (google_project_service count: 0)
  • Terraform Version: 1.15.7
  • GCP Provider Version: hashicorp/google 6.50.0 (postgresql lock file; the failure is API-enablement and provider-version-independent)

Steps to Reproduce

  1. Deploy Stages 0–1 with a fresh tenant (services list as shipped, unmodified).
  2. Deploy blueprints/il5/postgresql against the tenant's main project. The apply fails on google_sql_database_instance with 403 SERVICE_DISABLED (log below).
  3. Run gcloud services enable sqladmin.googleapis.com --project=<tenant-main-project> and re-apply — it now succeeds. The manual enable is the only intervention required, confirming root cause.
  4. Repeat with blueprints/fedramp-high/app-engine: google_app_engine_application fails identically until gcloud services enable appengine.googleapis.com is run; after enablement the apply succeeds (module.app-engine.google_app_engine_application.app confirmed in state).
  5. Code confirmation: grep -c "sqladmin\|appengine" fast/stages-aw/1-resman/branch-tenants.tf → 0; grep -c 'resource "google_project_service"' blueprints/il5/postgresql/main.tf blueprints/fedramp-high/app-engine/main.tf modules/app-engine/main.tf → 0 in each.

Expected Behavior

Blueprints shipped for tenant projects can apply against a fresh tenant: either Stage 1 enables the APIs the shipped workload blueprints require (as it already does for e.g. cloudkms.googleapis.com), or these two blueprints enable their own APIs via google_project_service, as the majority of sibling blueprints already do.

Actual Behavior

Every fresh-tenant deployment of either blueprint hard-fails on apply and cannot self-heal; the operator must discover and run a manual gcloud services enable that no README documents.

Relevant Logs and Errors

Captured from a live failed apply (postgresql, TF_LOG debug; project number redacted):

google_project_service_identity.cloudsql_sa: Refreshing state...
  [id=projects/<tenant-main-project>/services/sqladmin.googleapis.com]
...
Error: Error, failed to create instance report: googleapi: Error 403:
Cloud SQL Admin API has not been used in project <PROJECT_NUMBER> before or it
is disabled. Enable it by visiting
https://console.developers.google.com/apis/api/sqladmin.googleapis.com/overview?project=<PROJECT_NUMBER>
then retry. ...
    "reason": "SERVICE_DISABLED"
    "service": "sqladmin.googleapis.com"

Note the misleading shape: google_project_service_identity.cloudsql_sa refreshes without error while the instance create fails — nothing points to a missing enablement in the framework itself as the root cause.

Additional Context

Blocks primary workload deliverables (Cloud SQL, App Engine) on every fresh tenant. blueprints/fedramp-high/postgresql is a symlink to blueprints/il5/postgresql, so FRH and IL5 consume identical code and are equally affected. Proposed fix (either or both): add sqladmin.googleapis.com and appengine.googleapis.com to Stage 1's tenant services lists, and/or add google_project_service resources to the two blueprints, matching the majority-blueprint pattern. Note that enabling the API does not by itself make impersonated deploys work — see the companion documentation issue on deployer permissions and user_project_override: {companion-issue link}. Related (not duplicate): #47 proposes centralizing bootstrap-level prerequisite/API documentation; this issue is a code gap at the tenant/blueprint level.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Level of Effort - LowQuick, well-defined tasks with no unknowns; takes a few hours up to one day to completePriority - MediumStandard features and non-blocking bugs; important for the current milestone but not urgentbugSomething isn't working

    Type

    Fields

    No fields configured for Bug.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions