CI: migration smoke-test against ephemeral prod-schema Postgres - #72
Conversation
) The provision_workspace_membership migration (PR #60) passed CI + 5-agent review yet failed at prod apply with 42P17 (INSERT trigger WHEN cannot reference OLD). DDL-only errors are invisible to Deno type-check + offline pytest; only a real Postgres catches them. Add a CI net before prod. Changes: - .github/workflows/ci.yml: new migration-smoke job with a postgres:17 service container (major pinned to prod's verified 17.4). Fails loud with the exact `supabase db dump` command when supabase/schema.snapshot.sql is absent, loads the snapshot, then applies the newest migration under supabase/migrations/ with psql -v ON_ERROR_STOP=1; non-zero exit fails the job. - docs/operations.md §1: snapshot refresh procedure, auth prerequisite, the GoTrue limitation, the version-pin rationale, and the newest-only scope decision. - README.md: repo-layout pointer (marked NOT YET COMMITTED) + Develop & deploy note. schema.snapshot.sql is intentionally NOT created — it needs an interactive `supabase db dump` as renaud@bluegreen.ai against shared prod. The job is expected RED until the human commits it; a silent pass is the failure mode this issue removes. Fixes #65 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
⚡ Self-Fix Report (Aggressive)Status: COMPLETE Fixes Applied (0 total)
The consolidated review found 0 CRITICAL/HIGH/MEDIUM issues. Code-review checked all 8 scope focus areas (fail-loud behavior, Tests Added(none — PR touches 0 source/test files, CI config + docs only) Skipped (2)
Suggested Follow-up Issues(none) ValidationN/A — no changes made. Existing CI on this PR: Self-fix by Archon · aggressive mode · no changes needed |
The job as first written could not have worked. Each fault below was found by running it locally against a real prod dump, not by review: - postgres:17 has neither the cluster roles the snapshot assigns ownership to (supabase_admin, supabase_auth_admin, anon, authenticated, service_role, dashboard_user) nor the extensions its types come from. It died on line 18 with `role "supabase_admin" does not exist`, and later on `type "public.geometry" does not exist`. Use public.ecr.aws/supabase/postgres at prod's exact version (17.4.1.052) — the same image `supabase db dump` runs — where postgis, vector and pg_net all resolve to prod's exact versions. - Connect as supabase_admin. In that image `postgres` is restricted and the snapshot's first statements fail with `permission denied for schema auth`. - Drop the image's own auth schema first. It ships a GoTrue-initialised `auth` older than prod's, and the snapshot uses CREATE TABLE IF NOT EXISTS — so it silently skipped auth.users, then failed on `column "is_sso_user" ... does not exist`. - Install prod's public-schema extensions before loading (tests/fixtures/ ci_extensions.sql). The dump carries the contents of auth/public but not the CREATE EXTENSION statements behind their types. - Apply what the branch ADDS, not "the newest migration". The snapshot is a dump of prod, so it already contains every migration prod has applied — replaying the newest one fails `relation ... already exists` on every single run. Diff against the base branch instead, which is why checkout needs fetch-depth: 0. - Dump auth,public,private, not auth,public: objects in public reference prod's private schema, and the load aborts on `schema "private" does not exist`. Verified locally on 2026-07-24 against a real dump: extensions + snapshot load clean (62 public tables, 23 auth tables, 8 halcrm tables, 23 triggers); a branch adding no migration passes; and a trigger re-introducing the pre-fix `WHEN (OLD...)` form fails with `INSERT trigger's WHEN condition cannot reference OLD values` — the exact error that reached prod through PR #60. The snapshot itself is still not committed, now for a new reason: a prod dump of `public` carries a hardcoded service_role key out of edifice's edifice_trigger_map_generation. Tracked separately in edifice; #65 stays open. Refs #65 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Unblocks the last acceptance criterion of #65. The snapshot could not be committed until today: a dump of `public` carried a hardcoded service_role key out of edifice's edifice_trigger_map_generation(). That is now fixed on the edifice side (key moved to Vault, then replaced by a dedicated narrow-purpose secret, and both compromised keys rotated out), so a plain dump is finally clean. `supabase db dump --schema auth,public,private` against zgkvbjqlvebttbnkklpo, 2026-07-24. Audited before committing, not assumed: - no sb_secret_/sb_publishable_/eyJhbGci/SCRAM-SHA/PASSWORD literal, no private key block; the only long quoted strings are identifiers (`edifice_map_trigger_secret` is the NAME of a Vault entry, not its value); - edifice_trigger_map_generation() reads `vault.decrypted_secrets` — no credential in the body; - 0 COPY/INSERT statements: schema only, no data. Load verified against public.ecr.aws/supabase/postgres:17.4.1.052 — extensions + snapshot apply clean (62 public tables, 23 auth, 8 halcrm, 23 user triggers). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Why
The
provision_workspace_membershiptrigger migration (PR #60) passed the 5-agent review and CI, then failed at prod apply with42P17: INSERT trigger's WHEN condition cannot reference OLD values. Nothing in this repo's CI (Deno type-check + offline pytest) exercises a real Postgres, so DDL-only failures are invisible until a human runsapply_migrationagainst the shared prod project. This PR adds that net, implementing the issue's recommended design (committed schema snapshot + ephemeral Postgres in CI).Status: complete
Job written, snapshot committed, verified end-to-end. All four acceptance criteria met.
The first version of this job could not have worked
Every fault below was found by actually running the job locally against a real dump — none was visible to review. This is worth recording, because the failures came one at a time, each hidden behind the previous one:
postgres:17has none of Supabase's cluster rolesrole "supabase_admin" does not existat line 18type "public.geometry" does not existat line 227postgrespermission denied for schema authat line 35authschema; the snapshot usesCREATE TABLE IF NOT EXISTS, so it skippedauth.userscolumn "is_sso_user" of relation "auth.users" does not existat line 1866--schema auth,publicomits a schemapublicdepends onschema "private" does not existat line 4852relation "halcrm_workspaces_company_id_key" already existsThat last one is a design fault, not a detail. The snapshot is a dump of prod, so it already contains every migration prod has applied — including the newest one on
main. Replaying it fails on every single run, forever, for reasons unrelated to any defect. The reasoning in the original PR body ("the snapshot already contains the cumulative effect, so replaying the full ledger would conflict — therefore apply only the newest") was right about the premise and wrong about the conclusion: the newest migration is in the snapshot too.What is genuinely unapplied is what the branch adds on top of its base. The job now diffs
origin/<base>...HEADoversupabase/migrations/*.sql, which is whyactions/checkoutneedsfetch-depth: 0. A branch adding no migration has nothing to smoke-test and passes.Changes
.github/workflows/ci.yml—migration-smokejob:public.ecr.aws/supabase/postgres:17.4.1.052(prod's exact version, the same imagesupabase db dumpuses), connect assupabase_admin, drop the image's ownauthschema, install prod extensions, load the snapshot, apply the branch's added migrations.ON_ERROR_STOP=1throughout. Fails loud with the exactsupabase db dumpcommand if the snapshot is absent — never skips, never passes silently.tests/fixtures/ci_extensions.sql— postgis / postgis_topology / vector / pg_net, the extensions prod installs intopublic. The dump carries the contents of the dumped schemas but not theCREATE EXTENSIONstatements behind their types.docs/operations.md§1 — refresh procedure, the Docker prerequisite, and the four load-bearing decisions above with the symptom each one prevents.README.md— repo-layout pointer.Verified locally, 2026-07-24
Against a real
supabase db dump --schema auth,public,privateofzgkvbjqlvebttbnkklpo:halcrm_*tables, 23 user triggers;AFTER INSERT ... FOR EACH ROW WHEN (OLD.company_id IS DISTINCT FROM NEW.company_id), fails withINSERT trigger's WHEN condition cannot reference OLD valuesand a non-zero exit. That is the exact error that reached prod through PR feat(db): auto-provision workspace_members on user invite #60 — this job would have blocked it.The snapshot took a detour worth recording
For most of this work the snapshot could not be committed at all — and not for the reason the issue anticipated. The dump ran fine. The problem was its contents: a dump of
publiccarried a hardcodedservice_rolekey, inlined inedifice'sedifice_trigger_map_generation()since 2026-05-29. Committing it would have published an RLS-bypassing credential into this repo's history.That was fixed on the edifice side first (key moved into Vault, then replaced by a dedicated narrow-purpose secret, both compromised keys rotated out and deleted). Only then was a clean dump possible. The smoke-test net is what surfaced it — a fair result for a job whose whole purpose is to look at prod's real schema before it bites.
The committed snapshot was audited before commit, not assumed clean: no
sb_secret_/sb_publishable_/eyJhbGci/SCRAM-SHA/PASSWORDliteral and no private-key block; the only long quoted strings are identifiers (edifice_map_trigger_secretis the name of a Vault entry, not its value);edifice_trigger_map_generation()readsvault.decrypted_secretsrather than carrying a credential; and 0COPY/INSERTstatements — schema only, no data.Acceptance criteria
supabase/schema.snapshot.sqlcommitted (schemasauth+public+private, schema-only) with a documented refresh procedure.migration-smokeloads the snapshot into an ephemeral Postgres pinned to prod's version and applies the migrations under review, failing the build on any SQL error.WHEN (OLD…)form turns the job red.README/docs/operations.mdnote pointing to the snapshot refresh step.4 of 4.
Fixes #65