Skip to content

CI: migration smoke-test against ephemeral prod-schema Postgres - #72

Merged
BluegReeno merged 4 commits into
mainfrom
archon/task-fix-issue-65
Jul 24, 2026
Merged

CI: migration smoke-test against ephemeral prod-schema Postgres#72
BluegReeno merged 4 commits into
mainfrom
archon/task-fix-issue-65

Conversation

@BluegReeno

@BluegReeno BluegReeno commented Jul 24, 2026

Copy link
Copy Markdown
Owner

Why

The provision_workspace_membership trigger migration (PR #60) passed the 5-agent review and CI, then failed at prod apply with 42P17: INSERT trigger's WHEN condition cannot reference OLD values. Nothing in this repo's CI (Deno type-check + offline pytest) exercises a real Postgres, so DDL-only failures are invisible until a human runs apply_migration against the shared prod project. This PR adds that net, implementing the issue's recommended design (committed schema snapshot + ephemeral Postgres in CI).

Status: complete

Job written, snapshot committed, verified end-to-end. All four acceptance criteria met.

The first version of this job could not have worked

Every fault below was found by actually running the job locally against a real dump — none was visible to review. This is worth recording, because the failures came one at a time, each hidden behind the previous one:

Fault Symptom
postgres:17 has none of Supabase's cluster roles role "supabase_admin" does not exist at line 18
…nor the extensions the dumped types come from type "public.geometry" does not exist at line 227
Connecting as postgres permission denied for schema auth at line 35
The Supabase image ships an older auth schema; the snapshot uses CREATE TABLE IF NOT EXISTS, so it skipped auth.users column "is_sso_user" of relation "auth.users" does not exist at line 1866
--schema auth,public omits a schema public depends on schema "private" does not exist at line 4852
"Apply the newest migration" relation "halcrm_workspaces_company_id_key" already exists

That last one is a design fault, not a detail. The snapshot is a dump of prod, so it already contains every migration prod has applied — including the newest one on main. Replaying it fails on every single run, forever, for reasons unrelated to any defect. The reasoning in the original PR body ("the snapshot already contains the cumulative effect, so replaying the full ledger would conflict — therefore apply only the newest") was right about the premise and wrong about the conclusion: the newest migration is in the snapshot too.

What is genuinely unapplied is what the branch adds on top of its base. The job now diffs origin/<base>...HEAD over supabase/migrations/*.sql, which is why actions/checkout needs fetch-depth: 0. A branch adding no migration has nothing to smoke-test and passes.

Changes

  • .github/workflows/ci.ymlmigration-smoke job: public.ecr.aws/supabase/postgres:17.4.1.052 (prod's exact version, the same image supabase db dump uses), connect as supabase_admin, drop the image's own auth schema, install prod extensions, load the snapshot, apply the branch's added migrations. ON_ERROR_STOP=1 throughout. Fails loud with the exact supabase db dump command if the snapshot is absent — never skips, never passes silently.
  • tests/fixtures/ci_extensions.sql — postgis / postgis_topology / vector / pg_net, the extensions prod installs into public. The dump carries the contents of the dumped schemas but not the CREATE EXTENSION statements behind their types.
  • docs/operations.md §1 — refresh procedure, the Docker prerequisite, and the four load-bearing decisions above with the symptom each one prevents.
  • README.md — repo-layout pointer.

Verified locally, 2026-07-24

Against a real supabase db dump --schema auth,public,private of zgkvbjqlvebttbnkklpo:

  • extensions + snapshot load clean — 62 public tables, 23 auth tables, 8 halcrm_* tables, 23 user triggers;
  • a branch adding no migration → nothing to apply → green;
  • negative test: a trigger re-introducing the pre-fix form, AFTER INSERT ... FOR EACH ROW WHEN (OLD.company_id IS DISTINCT FROM NEW.company_id), fails with INSERT trigger's WHEN condition cannot reference OLD values and a non-zero exit. That is the exact error that reached prod through PR feat(db): auto-provision workspace_members on user invite #60 — this job would have blocked it.

The snapshot took a detour worth recording

For most of this work the snapshot could not be committed at all — and not for the reason the issue anticipated. The dump ran fine. The problem was its contents: a dump of public carried a hardcoded service_role key, inlined in edifice's edifice_trigger_map_generation() since 2026-05-29. Committing it would have published an RLS-bypassing credential into this repo's history.

That was fixed on the edifice side first (key moved into Vault, then replaced by a dedicated narrow-purpose secret, both compromised keys rotated out and deleted). Only then was a clean dump possible. The smoke-test net is what surfaced it — a fair result for a job whose whole purpose is to look at prod's real schema before it bites.

The committed snapshot was audited before commit, not assumed clean: no sb_secret_ / sb_publishable_ / eyJhbGci / SCRAM-SHA / PASSWORD literal and no private-key block; the only long quoted strings are identifiers (edifice_map_trigger_secret is the name of a Vault entry, not its value); edifice_trigger_map_generation() reads vault.decrypted_secrets rather than carrying a credential; and 0 COPY/INSERT statements — schema only, no data.

Acceptance criteria

  • supabase/schema.snapshot.sql committed (schemas auth + public + private, schema-only) with a documented refresh procedure.
  • CI job migration-smoke loads the snapshot into an ephemeral Postgres pinned to prod's version and applies the migrations under review, failing the build on any SQL error.
  • Verified negative — the pre-fix WHEN (OLD…) form turns the job red.
  • README / docs/operations.md note pointing to the snapshot refresh step.

4 of 4.

Fixes #65

)

The provision_workspace_membership migration (PR #60) passed CI + 5-agent
review yet failed at prod apply with 42P17 (INSERT trigger WHEN cannot
reference OLD). DDL-only errors are invisible to Deno type-check + offline
pytest; only a real Postgres catches them. Add a CI net before prod.

Changes:
- .github/workflows/ci.yml: new migration-smoke job with a postgres:17
  service container (major pinned to prod's verified 17.4). Fails loud with
  the exact `supabase db dump` command when supabase/schema.snapshot.sql is
  absent, loads the snapshot, then applies the newest migration under
  supabase/migrations/ with psql -v ON_ERROR_STOP=1; non-zero exit fails the job.
- docs/operations.md §1: snapshot refresh procedure, auth prerequisite, the
  GoTrue limitation, the version-pin rationale, and the newest-only scope decision.
- README.md: repo-layout pointer (marked NOT YET COMMITTED) + Develop & deploy note.

schema.snapshot.sql is intentionally NOT created — it needs an interactive
`supabase db dump` as renaud@bluegreen.ai against shared prod. The job is
expected RED until the human commits it; a silent pass is the failure mode
this issue removes.

Fixes #65

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@BluegReeno

Copy link
Copy Markdown
Owner Author

⚡ Self-Fix Report (Aggressive)

Status: COMPLETE
Pushed: N/A — no code/doc changes required
Philosophy: Fix everything unless clearly a new concern


Fixes Applied (0 total)

Severity Count
🔴 CRITICAL 0
🟠 HIGH 0
🟡 MEDIUM 0
🟢 LOW 0

The consolidated review found 0 CRITICAL/HIGH/MEDIUM issues. Code-review checked all 8 scope focus areas (fail-loud behavior, ON_ERROR_STOP propagation, migration-selection safety, version-pin honesty, tradeoff documentation, no prod contact, no new abstractions, honest completion claims) and found zero findings.


Tests Added

(none — PR touches 0 source/test files, CI config + docs only)


Skipped (2)

Finding Reason
README schema.snapshot.sql "NOT YET COMMITTED" annotation will go stale later Explicitly marked "no action needed" by docs-impact-agent — self-corrects when the separate, out-of-scope human supabase db dump step lands in its own future commit.
docs/features/ops-1-purge-ci.md / github-portfolio-cleanup.md describe pre-PR CI job set Explicitly marked "no action" — archival design notes by repo convention, not living references.

Suggested Follow-up Issues

(none)


Validation

N/A — no changes made. Existing CI on this PR: Deno type-check ✅, Python tests (offline) ✅ (157 passed), Migration smoke-test ❌ (expected/intentional — blocked on the separate human supabase db dump step, not a defect).


Self-fix by Archon · aggressive mode · no changes needed

BluegReeno and others added 2 commits July 24, 2026 17:11
The job as first written could not have worked. Each fault below was found by
running it locally against a real prod dump, not by review:

- postgres:17 has neither the cluster roles the snapshot assigns ownership to
  (supabase_admin, supabase_auth_admin, anon, authenticated, service_role,
  dashboard_user) nor the extensions its types come from. It died on line 18 with
  `role "supabase_admin" does not exist`, and later on `type "public.geometry"
  does not exist`. Use public.ecr.aws/supabase/postgres at prod's exact version
  (17.4.1.052) — the same image `supabase db dump` runs — where postgis, vector
  and pg_net all resolve to prod's exact versions.
- Connect as supabase_admin. In that image `postgres` is restricted and the
  snapshot's first statements fail with `permission denied for schema auth`.
- Drop the image's own auth schema first. It ships a GoTrue-initialised `auth`
  older than prod's, and the snapshot uses CREATE TABLE IF NOT EXISTS — so it
  silently skipped auth.users, then failed on `column "is_sso_user" ... does not
  exist`.
- Install prod's public-schema extensions before loading (tests/fixtures/
  ci_extensions.sql). The dump carries the contents of auth/public but not the
  CREATE EXTENSION statements behind their types.
- Apply what the branch ADDS, not "the newest migration". The snapshot is a dump
  of prod, so it already contains every migration prod has applied — replaying
  the newest one fails `relation ... already exists` on every single run. Diff
  against the base branch instead, which is why checkout needs fetch-depth: 0.
- Dump auth,public,private, not auth,public: objects in public reference prod's
  private schema, and the load aborts on `schema "private" does not exist`.

Verified locally on 2026-07-24 against a real dump: extensions + snapshot load
clean (62 public tables, 23 auth tables, 8 halcrm tables, 23 triggers); a branch
adding no migration passes; and a trigger re-introducing the pre-fix
`WHEN (OLD...)` form fails with `INSERT trigger's WHEN condition cannot reference
OLD values` — the exact error that reached prod through PR #60.

The snapshot itself is still not committed, now for a new reason: a prod dump of
`public` carries a hardcoded service_role key out of edifice's
edifice_trigger_map_generation. Tracked separately in edifice; #65 stays open.

Refs #65

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Unblocks the last acceptance criterion of #65. The snapshot could not be committed
until today: a dump of `public` carried a hardcoded service_role key out of
edifice's edifice_trigger_map_generation(). That is now fixed on the edifice side
(key moved to Vault, then replaced by a dedicated narrow-purpose secret, and both
compromised keys rotated out), so a plain dump is finally clean.

`supabase db dump --schema auth,public,private` against zgkvbjqlvebttbnkklpo,
2026-07-24. Audited before committing, not assumed:

- no sb_secret_/sb_publishable_/eyJhbGci/SCRAM-SHA/PASSWORD literal, no private key
  block; the only long quoted strings are identifiers (`edifice_map_trigger_secret`
  is the NAME of a Vault entry, not its value);
- edifice_trigger_map_generation() reads `vault.decrypted_secrets` — no credential
  in the body;
- 0 COPY/INSERT statements: schema only, no data.

Load verified against public.ecr.aws/supabase/postgres:17.4.1.052 — extensions +
snapshot apply clean (62 public tables, 23 auth, 8 halcrm, 23 user triggers).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@BluegReeno
BluegReeno marked this pull request as ready for review July 24, 2026 16:21
@BluegReeno
BluegReeno merged commit 41279ae into main Jul 24, 2026
3 checks passed
@BluegReeno
BluegReeno deleted the archon/task-fix-issue-65 branch July 24, 2026 19:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CI — migration smoke-test against an ephemeral DB with the prod schema (catch 42P17-class errors before apply)

1 participant