fix(db): migration 026 reintroduced the #750 root-org crashloop - #831
Merged
Conversation
#750: migration 015 has legitimate branches that leave ROOT_ORG_ID absent (adopting a pre-existing platform org under a different id, a slug conflict, or no user yet to own it) precisely because repointing/ reparenting an existing root onto ROOT_ORG_ID is a deliberate data migration the code refuses to do unattended. That defect — inferring root-org presence instead of verifying it, then FK-violating — was already fixed and tested on master (c472efa, #751): 015 now re-reads the row before referencing it and 022 resolves the actual root org rather than assuming the constant. Migration 026 (d44ee55, merged 2026-08-26, one day before this fix) reintroduced the exact same defect class one step later. It asserted ROOT_ORG_ID must exist because "this tree owns 015, so there is no cross-service ordering hazard to guard against here" — true, but it conflated "015 has run" with "015 successfully created the row", which is exactly the distinction #750 exists to make. Production is currently in the state where 015 legitimately left ROOT_ORG_ID absent (the platform org adopted from the 2026-07-29 rebuild lives under a different id, and the repoint-or-reparent decision #750 asks for has never been made — confirmed via the issue thread, no such migration exists on master). So 026's unconditional throw is live: it never gets marked applied, so it retries and throws again on every single boot — the same crashloop shape as the original #750 incident, one migration downstream of the fix. THE FIX: narrow the throw to only fire when it is actually warranted — there are org-less `apps` rows that would need backfilling to a still-absent root org (i.e. an unverified reference really would FK-violate). When there is nothing to backfill, skip with a clear log line instead of asserting a precondition the code cannot make true on its own; retry on a later boot once the root org exists. This exactly mirrors the fix already proven and merged for this migration's own sibling (applications-service's 011_apps_organization_id_not_null.ts, merged the same day, d44ee55) — that tree's throw was narrowed for this same reason, but the twin migration in backend/src was missed. WHY AN EDIT TO 026 IN PLACE, NOT A NEW MIGRATION: knex only marks a migration applied after `up()` returns without throwing. On any environment where ROOT_ORG_ID is absent — which includes prod today — 026 currently throws every time it runs and is therefore NEVER recorded in `knex_migrations`. Editing it in place is correct and will re-run with the fix on the next boot; a new migration is unnecessary here because this one never completed. (This is different from 015, which already required no further schema-level fix and none is made here — 015/022 are untouched, only their downstream consumer is corrected.) Safe to run against a DB where 026 already applied successfully elsewhere (dev/CI, where a fresh schema lets 015 create the root org outright): the `root` lookup finds the row and the migration proceeds exactly as before, with SET DEFAULT / SET NOT NULL applied idempotently (Postgres no-ops both on an already-conforming column). Idempotent: re-running is a no-op once either (a) the root org exists — SET DEFAULT/SET NOT NULL on an already-conforming column raises nothing, and the backfill UPDATE's WHERE clause matches zero rows once already applied — or (b) it still doesn't and there is still nothing to backfill, which repeats the same skip-and-log every time with no side effects. The still-open half of #750 — whether to repoint or reparent the existing platform org onto ROOT_ORG_ID — remains a deliberate data decision this migration does not make, consistent with 015's own adopt-branch, which already refuses to repoint unattended for the same reason (rows already reference the existing id). Tests: extended backend/tests/rootOrgAbsentGuards.test.ts (the existing #750 regression suite) with three cases for migration 026 — skip without throwing when root is absent and nothing needs backfilling (the crash this commit fixes), throw with a clear message when root is absent AND rows genuinely need it, and the unchanged happy path when the root org exists. VERIFIED: ran the full 015/022/026 stub-knex test suite plus the project's real jest setup (global beforeAll/afterAll against a live Postgres, migration chain executed end to end) — 10/10 pass, including the 3 new cases. tsc --noEmit on the changed files shows no new errors (the two pre-existing @fuzefront/custom-hostname-client errors are unrelated and present on unmodified master too). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013tMciHkPE8To7V67CsgKKc
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Works #750. The literal defect the issue describes — migration 015 inferring root-org presence from a constraint conflict, then FK-violating — was already fixed and tested on master (
c472efa6, #751): 015 now re-reads the row before referencing it, and 022 resolves the actual root org instead of assuming the hard-coded constant.While investigating, I found the same defect class reintroduced one migration downstream:
026_apps_organization_id_not_null.ts(mergedd44ee551, 2026-08-26 — one day before this PR) unconditionally throws whenROOT_ORG_IDis absent, on the premise that "this tree owns 015, so there is no cross-service ordering hazard to guard against." That's true but irrelevant — it conflates "015 has run" with "015 successfully created the row," exactly the distinction #750 established. Production is confirmed (via the #750 issue thread) to still be in the state where 015 legitimately leftROOT_ORG_IDabsent (an adopted platform org from the 2026-07-29 rebuild lives under a different id, and the repoint-or-reparent decision has never been made). So 026's throw is live: it never gets marked applied inknex_migrations, so it retries and throws on every boot — the #750 crashloop, one migration later.The fix mirrors the pattern already proven and merged the same day for this migration's own sibling (applications-service's
011_apps_organization_id_not_null.ts) but never ported to this tree: only throw when there's actually something that would FK-violate (org-lessappsrows needing backfill to a still-absent root org); otherwise skip with a clear log line and retry on a later boot.Why an edit to 026 in place, not a new migration
Knex only marks a migration applied after
up()returns without throwing. On any environment whereROOT_ORG_IDis absent (prod, currently), 026 throws every time and is therefore never recorded as applied — editing it in place is correct and will take effect on the next boot. A new migration isn't needed here because 026 never completed. (015/022 are untouched — they already have the fix from #751.)Idempotency
SET DEFAULT/SET NOT NULL(Postgres no-ops both on an already-conforming column).The still-open half of #750 — whether to repoint or reparent the existing platform org onto
ROOT_ORG_ID— remains a deliberate data decision this migration does not make, same as 015's own adopt-branch (never repoints unattended).Test plan
backend/tests/rootOrgAbsentGuards.test.ts(the existing Backend CrashLoopBackOff in prod: migration 015 infers "root org present" from a slug conflict, then FK-violates #750 regression suite) with 3 cases for migration 026: skip-without-throw (the crash this fixes), throw-when-genuinely-needed, and the unchanged happy path.tsc --noEmiton changed files — no new errors (two pre-existing, unrelated@fuzefront/custom-hostname-clientresolution errors present on unmodified master too).🤖 Generated with Claude Code
https://claude.ai/code/session_013tMciHkPE8To7V67CsgKKc
Generated by Claude Code