fix(deploy): repair L1 role provisioning + plumb AUTO_MIGRATE into the container#466
Merged
Conversation
…e container
Two real prod-deployment bugs the release-gate diagnosis uncovered. Both only
bite L1 role-separated stacks (operators who set POSTGRES_APP_PASSWORD, which
install.sh does); legacy single-role deployments were unaffected, which is why
these went unnoticed until the gate booted the production compose from scratch.
(A) scripts/postgres-init.sh never created `trustedoss_app`. The role-creation
used psql `:'app_user'` / `:'app_password'` INSIDE a `DO $$ … $$`
dollar-quoted block, where psql performs NO variable substitution — so the
literal `:'app_user'` reached the server: `syntax error at or near ":"`.
With `set -e` + `ON_ERROR_STOP=1` the init script aborted, the role was never
created, and every L1 backend then failed password auth as `trustedoss_app`.
Fix: move interpolation out of the dollar-quote into a plain
`SELECT format('CREATE ROLE %I WITH LOGIN INHERIT PASSWORD %L',
:'app_user', :'app_password') WHERE NOT EXISTS (…) \gexec`. psql now SQL-quotes
each value (M4 double-quote defence intact), `\gexec` runs the generated DDL
without re-substituting, and `WHERE NOT EXISTS` keeps it idempotent.
Verified on postgres:17.2-alpine with a hostile password
(`p@ss:with'quote\and\special`): reproduced the syntax-error/role-absent
failure before; after, the role is created, SCRAM auth with the special-char
password succeeds, a wrong password is rejected, no extra roles appear, and
three replays stay at count 1.
(B) AUTO_MIGRATE was never plumbed into the container. install.sh writes
`AUTO_MIGRATE=false` to .env on L1 stacks (the runtime app role has no DDL
grant; migrations run once as the owner), but the compose file never
referenced `${AUTO_MIGRATE}`, so the value stayed visible only to compose
interpolation and never reached the backend entrypoint — which defaulted back
to true and attempted DDL as trustedoss_app. Add
`AUTO_MIGRATE: ${AUTO_MIGRATE:-true}` to the shared x-backend-env anchor
(backend/worker/beat share it and the same entrypoint). Default `true`
matches .env.example and the single-role path, so existing deployments are
unchanged.
Not a bug (left as-is): DATABASE_URL_OWNER is deliberately NOT propagated into
runtime containers (security — a runtime RCE must not hold the owner DSN); the
owner migration runs via install.sh's explicit `-e DATABASE_URL=$DATABASE_URL_OWNER`.
haksungjang
added a commit
that referenced
this pull request
Jul 5, 2026
The L1 role-separated first-boot path had NO automated coverage — CI test(backend) uses a postgres service container (the init script is never mounted) and the release-gate boots in dev single-role mode (POSTGRES_APP_PASSWORD unset → the script skips). That blind spot is exactly how the #466 `DO $$ … :'app_user' … $$` bug (trustedoss_app never created → every L1 install.sh deploy fails backend auth) reached a release. Add a `postgres-init-l1` job to ci.yml that boots a real postgres:17.2-alpine on a fresh volume with scripts/postgres-init.sh mounted at the operator's /docker-entrypoint-initdb.d path (pure docker run, no compose) and asserts: - L1 mode (hostile password with @ : ' \) — init clean (no syntax error), role trustedoss_app present (LOGIN INHERIT), real SCRAM TCP login with the special password succeeds, a wrong password is rejected, and no unexpected roles appear; - legacy single-role (password unset) — script skips cleanly, no role, no error; - idempotency — replaying the script keeps the role count at 1. scripts/postgres-init.sh is not in ci.yml's paths-ignore, so any edit to the script forces this gate. Verified locally with real Docker (3 consecutive green runs) plus a negative test: a copy reintroducing the #466 dollar-quote bug fails the gate three independent ways (serving timeout, syntax-error grep, role census). actionlint + shellcheck clean. Only ci.yml changed.
haksungjang
added a commit
that referenced
this pull request
Jul 5, 2026
…bug (#468) * test(ci): add L1 role-separated install-uat + fix V1 nested-DSN prod bug Covers the deployment layer the release-gate never exercises (it boots dev single-role mode). New `install-uat-l1` job boots the PRODUCTION compose in L1 role-separated mode end-to-end under docker-compose V1 1.29.2 — staged (postgres+redis+backend → owner `alembic upgrade head` → worker/beat/frontend) — and asserts the full L1 contract: - trustedoss_app role created from a special-character POSTGRES_APP_PASSWORD (exercises postgres-init.sh's M4 psql --variable quoting live); - AUTO_MIGRATE=false actually reaches the container (printenv + entrypoint log "migrations skipped", NOT "auto-migration enabled") — the #466 (B1) fix; - the backend runs on the restricted trustedoss_app DSN (pg_stat_activity), and DDL as that role is denied (privilege boundary); - owner migration brings the schema to HEAD; /health + /health/ready 200; /auth/login + authed /v1/projects smoke green. Fixes a real prod bug found while building it: the L1 runtime DSN used a NESTED default `${DATABASE_URL_APP:-${DATABASE_URL}}` that docker-compose V1 1.29.2 (the project standard) cannot expand — it leaves a stray trailing `}`, so every L1 deployment on V1 got `.../trustedoss}` and could not connect. The single-brace form `${DATABASE_URL_APP:-$DATABASE_URL}` resolves cleanly under both V1 and V2; semantics are unchanged (L1 → app DSN in the container, owner creds never leak; single-role still resolves via DATABASE_URL). This is the base-compose root fix; release-gate #465 only overrode the keys in its gate overlay. Verified with 1.29.2 `config` + a full L1 boot of the published images. The L1 job intentionally does NOT drive install.sh yet — see the PR for two further install.sh L1 bugs (owner-password mismatch; up-d deadlock under AUTO_MIGRATE=false) escalated for a separate fix. * test(ci): point install-uat published-image tags at 0.13.0 (was stale 2.0.0) The workflow_dispatch `pull_image_tag` default and the published-image-pull fallback were pinned to `2.0.0`, a tag that was never published, so a manual dispatch of install-uat pulled nonexistent images and failed on both published-image-pull and the new install-uat-l1 job. Point all three tag references (dispatch default, published-image-pull fallback, install-uat-l1 fallback) at 0.13.0, the current release. Nightly schedule (empty input) now also exercises the current release instead of a stale/absent tag. * test(ci): retry the install-uat-l1 liveness probe (cold-runner uvicorn bind race) The liveness /health assertion did a single exec-curl immediately after Stage 1 `up -d`, racing uvicorn's bind to :8000. It passed on a warm local Docker but failed in CI (connection refused) on a cold runner. Poll for up to 60s like the readiness step already does. * test(ci): fix published-image-pull grep (trusca- image-name prefix) The config-resolution assertion grepped for ghcr.io/trustedoss/(backend| backend-worker|frontend): but the images carry the trusca- prefix (trusca-backend, trusca-backend-worker, trusca-frontend), so the grep never matched and the job failed. It was masked before by the stale 2.0.0 tag pulling nonexistent images first. Match the real image names.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two real prod-deployment bugs the release-gate diagnosis uncovered. Both bite
L1 role-separated stacks (operators who set
POSTGRES_APP_PASSWORD— whichinstall.shdoes); legacy single-role deployments were unaffected, which is whythey went unnoticed until the gate booted the production compose from scratch.
(A)
scripts/postgres-init.shnever createdtrustedoss_app— criticalThe role-creation referenced psql
:'app_user'/:'app_password'inside aDO $$ … $$dollar-quoted block, where psql performs no variablesubstitution — so the literal
:'app_user'reached the server:syntax error at or near ":". Withset -e+ON_ERROR_STOP=1the init scriptaborted, the role was never created, and every L1 backend then failed password
auth as
trustedoss_app→ boot failure.Fix — move interpolation out of the dollar-quote into a plain
SELECT format('CREATE ROLE %I WITH LOGIN INHERIT PASSWORD %L', :'app_user', :'app_password') WHERE NOT EXISTS (…) \gexec.psql SQL-quotes each value (the M4 double-quote defence stays intact),
\gexecruns the generated DDL without re-substituting,
WHERE NOT EXISTSkeeps itidempotent.
Verified on
postgres:17.2-alpinewith a hostile password(
p@ss:with'quote\and\special): reproduced the syntax-error / role-absent failurebefore; after, the role is created, SCRAM auth with the special-char password
succeeds, a wrong password is rejected, no extra roles appear, and three replays
stay at count 1.
(B)
AUTO_MIGRATEnever plumbed into the container — mediuminstall.shwritesAUTO_MIGRATE=falseto.envon L1 stacks (the runtime approle has no DDL grant; migrations run once as the owner), but the compose file
never referenced
${AUTO_MIGRATE}, so the value stayed visible only to composeinterpolation and never reached the backend entrypoint — which defaulted back to
trueand attempted DDL astrustedoss_app. AddAUTO_MIGRATE: ${AUTO_MIGRATE:-true}to the sharedx-backend-envanchor(backend / worker / beat share it and the same entrypoint). Default
truematches
.env.exampleand the single-role path, so existing deployments areunchanged.
Not a bug (left as-is)
DATABASE_URL_OWNERis deliberately not propagated into runtime containers(security — a runtime RCE must not hold the owner DSN); the owner migration runs
via
install.sh's explicit-e DATABASE_URL=$DATABASE_URL_OWNER. The third-partyreport flagged this, but it is intentional.