ci(preview): on-demand previews from any branch, teardown, ephemeral backend - #352
Merged
Merged
Conversation
…backend BDMS-1173. Previews were PR-only and always pointed at the staging API, so there was no way to demo a branch that has no PR yet, and no way to review a frontend change that depends on unreleased OcotilloAPI work. Split the build/deploy and teardown mechanics into two reusable workflows and give them three callers: the existing PR trigger, a new workflow_dispatch that takes any branch, and a teardown workflow with manual, branch-delete, and nightly-sweep entry points. Sweeping keys off preview-* Cloud Run labels; preview-expires=0 exempts PR previews, whose lifetime the PR itself owns. The ephemeral backend is the Cypress job's docker-compose stack re-expressed as one multi-container Cloud Run service: the same OcotilloAPI image and the same postgis image, with the compose "db" host becoming 127.0.0.1 because sidecars share a network namespace. Cloud Run has no exec, so the seed step Cypress runs as a follow-up command becomes a command override that migrates, seeds, then execs uvicorn. Postgres lives on a tmpfs, so the service is pinned to a single always-warm instance and the data is disposable by construction. Ephemeral backends default to AUTHENTIK_DISABLE_AUTHENTICATION=1 because transfers.seed creates no permission rows and an enforcing backend would lock every user out. That makes them public sandboxes; docs say so in as many words. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…cript
Rendering the Cloud Run template locally showed `if [ "$RUN_SEED" = "true" ]`
coming out as `if [ "true" = "true" ]`. envsubst substitutes `$VAR` as readily
as `${VAR}`, so putting RUN_SEED on the allowlist to fill the env entry also
rewrote the identically-named reference inside the container startup script.
Behaviour happened to be correct either way, but the script read as if it
consulted the environment at runtime when it no longer did.
Prefix every render-time placeholder with TPL_ so the two namespaces cannot
overlap, and say why in the template header.
Also quote the Secret Manager reads, silence the deliberate SC2016 on the
envsubst allowlist, and correct the docs: this repo's default branch is
staging, not production, which is where the dispatch and schedule triggers
have to land before they register.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Preview DeploymentPreview URL: https://preview-ci-bdms-1173-on-demand-preview-deploymen-auejgdbofq-uc.a.run.app |
jirhiker
marked this pull request as ready for review
August 21, 2026 20:38
The authentik PATCH answers with the full provider object, client_secret included, and curl was writing that response to stdout. Every preview deploy has therefore been publishing the OAuth2 provider's client_secret into a run log readable by anyone with read access to the repo. The provider is a public client, but client_credentials is among its enabled grant types, so the value is usable rather than merely cosmetic. Discard the response body on both the add and the remove path. The GET earlier in each step already writes to a temp file rather than stdout, so this was the only leak. Predates this branch -- the same curl is in the CD_preview.yml being replaced here -- so the secret is in the logs of past preview runs too and should be rotated independently of this fix. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
jirhiker
added
the
preview-backend
Preview DeploymentPreview URL: https://preview-ci-bdms-1173-on-demand-preview-deploymen-auejgdbofq-uc.a.run.app |
Empty commit. CD_preview.yml listens on synchronize, not labeled, so the preview-backend label added to this PR needs a push to take effect.
…o start A failed "gcloud run services replace" says only that the revision never became ready on PORT=8000. Which container died, and why, is in Cloud Logging, and nothing in the job went and got it -- the existing log dump lives in the readiness-probe step, which never runs when the deploy itself fails. Pull the revision's logs and condition list on failure. Both containers share the revision log stream, tagged by container name, so this distinguishes a postgis sidecar that never opened 5432 from an API container that crashed in alembic or the seed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The ephemeral backend got as far as migrating and seeding, then uvicorn died
importing main:app:
RuntimeError: PYGEOAPI_POSTGRES_PASSWORD must be set to generate the
pygeoapi configuration.
create_api_app() calls mount_pygeoapi() unconditionally, so every boot needs
these settings. docker-compose.yml supplies the whole PYGEOAPI_POSTGRES_* block
and this template dropped it when porting the environment across.
Only the password strictly matters -- host, port, db and user each fall back to
their POSTGRES_* equivalent, whereas the password is checked for None with no
fallback. Set all five regardless, so the template reads as a peer of the
compose file rather than something that happens to work.
Also drop MOUNT_PYGEOAPI_IN_API. It is copied from OcotilloAPI's app.yaml and
reads like an opt-out, but no Python in that repo reads it; leaving it in would
imply the mount is switchable when it is not.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two cost fixes now that the ephemeral path is proven working. The API container was provisioned 2 vCPU and 2Gi. The expensive part of startup -- migrate plus seed -- measured in seconds on the first successful run, so the headroom was buying nothing. Drop it to 1 vCPU and 1536Mi, taking the service from 3 vCPU / 4Gi to 2 vCPU / 3.5Gi across both containers. The sizing matters more than it looks: the database is the instance, so the service cannot scale to zero and bills continuously from the moment it is deployed until teardown. Second, deploying a preview with backend=staging now deletes any ephemeral API left over from an earlier ephemeral deploy of the same branch. Dropping the preview-backend label and pushing used to redeploy the frontend against staging and silently strand the old always-warm API, with nothing reclaiming it until the PR closed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Member
Author
|
Closing briefly to exercise the teardown workflow (BDMS-1173 acceptance criterion). Reopening immediately after. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes BDMS-1173.
Why
Previews were PR-only and always pointed at the staging API. That leaves two gaps: there is no way to demo a branch that has no PR yet, and no way to review a frontend change that depends on unreleased OcotilloAPI work.
What
Split the build/deploy and teardown mechanics into two reusable workflows, then give them three callers.
.github/workflows/_preview_deploy.yml.github/workflows/_preview_teardown.yml.github/workflows/CD_preview.yml.github/workflows/CD_preview_ondemand.ymlworkflow_dispatchfor any branch.github/workflows/CD_preview_teardown.yml.github/preview/api-service.tmpl.yamldocs/preview-deployments.mdAgainst the acceptance criteria:
gh workflow run CD_preview_ondemand.yml --ref my-branch-f backend=ephemeralThe backend deployment option
Selected per deploy, defaulting to today's behaviour:
backend=staging(default) —ocotillo-api-staging.newmexicowaterdata.org, unchanged.backend=ephemeral— a throwaway API + database for this branch alone.backend_refpicks which OcotilloAPI branch to build, so a frontend branch can preview against unreleased API work.The ephemeral path is the Cypress job's docker-compose stack re-expressed as one multi-container Cloud Run service. Same
docker/app/Dockerfile, samepostgis/postgis:17-3.5, deployed as sidecars rather than compose services. Sidecars share a network namespace, so compose'sdbhost simply becomes127.0.0.1:5432— no OcotilloAPI changes required.Three deltas from compose:
exec, sodocker compose exec app python -m transfers.seedbecomes a containercommandoverride: wait for postgres,alembic upgrade head, seed, thenexec uvicorn.minScale: 1, maxScale: 1with CPU always allocated. Data is disposable by construction and re-seeds on restart.TPL_-prefixed soenvsubstcannot freeze a runtime variable — see the second commit, which fixes exactly that bug.Please look closely at
Ephemeral backends default to
AUTHENTIK_DISABLE_AUTHENTICATION=1, and the API is publicly reachable. This is not a shortcut:transfers.seedcreates no users and no permission rows, so an enforcing backend locks every user out of its own preview.-f backend_auth=enabledopts out. The tradeoff is called out in the docs, in every job summary, and is worth a second opinion.Other judgement calls:
ttl_hours: 0, exempting them from the nightly sweep, because the PR itself owns their lifetime. Only on-demand previews expire on a TTL.refs/pull/N/mergeexplicitly. That preserves whatactions/checkoutdid implicitly before the refactor, but it is now load-bearing rather than incidental.minScale: 1means an ephemeral preview bills continuously until torn down.Testing
Exercised end to end against real infrastructure on this PR, not just linted.
pull_requestruns workflows from the merge ref, so every run below used this branch's own workflow definitions.Static
actionlint1.7.12 withshellcheck0.10.0 — clean on all five workflows, including the reusable-workflow wiring and the sweep'sfromJSONmatrix.envsubstexactly as the workflow does. This caught theRUN_SEEDcollision fixed in the second commit.Staging-backend preview — deployed, served HTTP 200, PR comment posted, Cloud Run labels applied, TTL sentinel resolved to
never (torn down when the PR closes), authentik origin registered.Ephemeral backend — took three runs to get right, and the failures were the useful part:
gcloud run services replaceaccepted the multi-container spec first time, but the revision never became ready and the job surfaced nothing but the bare gcloud error. Added afailure()-gated log dump; the existing one was stranded in the readiness-probe step, which never runs when the deploy itself fails.Seed complete: 5 contacts, 5 locations, 5 things, 5 sensors, 5 samples, 10 observations, then uvicorn died onPYGEOAPI_POSTGRES_PASSWORD must be set. This template had dropped compose'sPYGEOAPI_POSTGRES_*block./healthreturns{"status":"ok","db":"ok"}, and/contact,/thing,/locationall serve seeded rows. The frontend bundle has the ephemeral API URL baked in, not staging.Teardown — closed this PR to fire it, then reopened.
Removed 1 preview origin(s) from authentik.The reopen redeployed cleanly with a fresh seed — the new rows carry a later
created_at, which is the disposable database behaving as designed.Not yet exercised:
workflow_dispatchand the nightly sweep, because both only register once the file reaches the default branch (staging). The dispatch path calls the same_preview_deploy.ymlthat the six runs above went through; the sweep's discovery query is the untested part.A note on cost
An ephemeral backend cannot scale to zero — the instance is the database — so it bills continuously from deploy until teardown. After the sizing commit that is 2 vCPU / 3.5 GiB, roughly $3–4 a day. This PR currently has one running. Dropping the
preview-backendlabel and pushing now reaps it, which is the orphan gap the last commit closes.🤖 Generated with Claude Code