Backend CrashLoopBackOff in prod: migration 015 infers "root org present" from a slug conflict, then FK-violates

[izzywdev]

Symptom

fuzefront-backend cannot start in prod. Measured 2026-08-20 01:27 UTC via FuzeInfra cluster-query (run 32320250962), pod fuzefront-backend-cd94867b4-qdwhn, 27 restarts:

❌ Failed to start server: error: INSERT INTO organization_memberships
       (id, user_id, organization_id, role, status, joined_at, permissions, metadata)
     SELECT gen_random_uuid(), u.id, $1, 'member', ...
   violates foreign key constraint "organization_memberships_organization_id_foreign"

  code:       '23503'
  detail:     Key (organization_id)=(00000000-0000-0000-0000-000000000010)
              is not present in table "organizations".
  constraint: organization_memberships_organization_id_foreign

Argo shows fuzefront Synced / Degraded, sync PHASE: Failed — "one or more synchronization tasks completed unsuccessfully (retried 5 times)".

This is not an outage. The previous ReplicaSet's pods (fuzefront-backend-578fff4d66, 8d old) are still Running and serving. The rollout is stuck, not the site. Also failing to roll out: fuzefront-security (CrashLoopBackOff, 27 restarts), fuzefront-applications (Error, 28 restarts), fuzefront-config-service (CreateContainerConfigError — one instance 8h old, predating tonight's deploy).

Root cause — an untargeted ON CONFLICT DO NOTHING

backend/src/migrations/015_seed_root_platform_organization.ts:

const result = await knex.raw(
  `INSERT INTO organizations
     (id, name, slug, parent_id, owner_id, type, ...)
   VALUES (?, 'FuzeFront', ?, NULL, ?, 'platform', ...)
   ON CONFLICT DO NOTHING`,                       // ← no target: pkey OR slug
  [ROOT_ORG_ID, ROOT_ORG_SLUG, owner.id, ...]
)

if (result.rowCount > 0) { /* created */ }
else console.log('[015] root platform organization already present — nothing to do')

await knex.raw(
  `INSERT INTO organization_memberships (... organization_id ...)
   VALUES (gen_random_uuid(), ?, ?, 'owner', ...)`,
  [owner.id, ROOT_ORG_ID]                          // ← assumes ROOT_ORG_ID exists
)

The migration's own comment states the intent plainly: "ON CONFLICT DO NOTHING with no target covers both the pkey and the slug unique index." That is exactly the problem.

• pkey conflict → a row with ROOT_ORG_ID exists → membership insert is fine.
• slug conflict → a different organization already holds slug fuzefront → no row with ROOT_ORG_ID is created → the membership insert FK-violates.

The else branch then logs “already present — nothing to do”, which is false in the slug case. Presence was inferred from a conflict that a different row caused. rowCount === 0 means "I inserted nothing", not "the row I wanted is there."

The earlier adopt-an-existing-platform-org branch does not cover this, because it only looks for type = 'platform'. A row with slug fuzefront and some other type slips past it and lands in exactly this path.

Why it surfaced now

Nothing had rolled out for days — the old pods are 8d and 41h old — because the Argo sync was failing before it got this far (the consumer-registration-seed PostSync hook could not pull its image; fixed in #749, merged 22:44, deployed as e410199c). With that unblocked, new pods were finally created at ~23:09 and are now failing on this older, previously-masked defect.

To be explicit about attribution: #749 did not cause this. It is chart-only — the seed Job template, one values key, a helm-validate step — and touches no deployment, no migration and no application source. The crashlooping images are built from 1107482f, whose application source is identical to dc373bf9. The 8h-old config-service error predates the merge by ~5.5 hours.

What is inferred vs. measured

• Measured: the error, the constraint, the missing organization_id, the pod states, the Argo sync phase, and the migration source above.
• Inferred: that the conflict is on slug rather than pkey, i.e. that an organization with slug fuzefront exists under a different id and a non-platform type. That follows from the error (a pkey conflict would mean the row exists and no FK violation could occur), but I have no database access from here and have not confirmed the row directly. Worth confirming before fixing.

Suggested fix (not implemented — wants a decision)

Make the migration assert its postcondition instead of assuming it:

1. After the insert, re-read the row by ROOT_ORG_ID. If it is absent, do not proceed to the membership insert.
2. Handle the slug collision explicitly — either adopt the slug-holder as root (consistent with the existing adopt-a-platform-org branch, which already refuses to repoint ids because "rows already reference its id"), or fail with a message that names the colliding row.
3. Fix the misleading log line: distinguish "already present" from "insert skipped for some other conflict".

This is the same shape as two other defects found tonight — the seed Job that exit 0'd without writing a secret, and gate-secret-scan reporting green from a crashed uploader. A step that cannot tell "I did nothing because it was already done" from "I did nothing because something else was in the way" will eventually report success for the wrong reason.

I have not changed anything: prod is GitOps, and a migration that mutates organization/membership rows is not something to push unreviewed at 01:30. Happy to implement the fix above on request.

[izzywdev]

Fix opened as #751. One correction to the root cause I wrote above, and one consequence I had not connected.

The failing statement is migration 022's backfill, not 015's owner-membership insert. I quoted the error correctly (SELECT gen_random_uuid(), u.id, $1, 'member', ...) and then attributed it to 015's VALUES ... 'owner' insert, which is a different statement. That distinction matters, because it identifies which path 015 actually took: 015's own membership insert would have raised the same 23503 first if it had been reached. It was not reached, so 015 returned early — and the only branch that returns without erroring, given migration 014 seeds the registrar, is the adopt-a-pre-existing-platform-org branch.

So the mechanism is not the slug conflict I hypothesised. This database has a type='platform' organization under some other id — the one the old ensureRootPortal() created — and 00000000-0000-0000-0000-000000000010 has never existed here. Migration 022 (FF-EPIC-17-S2) is simply the first statement to hard-require it, which is also why this surfaced now and not months ago. The slug conflict remains a real second way to reach the same state; both are fixed the same way.

The consequence I had not connected: Argo runs PostSync only after the Sync phase succeeds. With Sync failing, the consumer-registration-seed PostSync hook from #749 never fires, so Secret/fuzefront-registration is never written. FuzeInfra's publish-sealed-handoff run 32322759156 (01:55 UTC, after the #749 merge and after Argo reached e410199c) reports FAIL: source fuzefront/fuzefront-registration key 'token' not readable for all eight consumer namespaces. That also retracts an inference I drew earlier: I read the seed Job's absence as evidence it had run and been garbage-collected on success. It is better explained by the Job never having been created at all.

#751 does not decide the data question. Whether to repoint this database's platform org onto ROOT_ORG_ID or reparent it under a newly created root is a deliberate data migration and your call — I have not touched prod data. What #751 does is stop both services crash-looping and make them say plainly what is wrong, so the rollout completes and the registration funnel opens.

master is deploy-on-push and these migrations run on deploy, so #751 wants a deploy window. I left the auto-merge label off for that reason.

---

Generated by Claude Code

[claude]

Opened #831.

The literal defect described here — 015 inferring root-org presence from a conflict, then FK-violating — was already fixed on master by #751 (c472efa6) and is covered by rootOrgAbsentGuards.test.ts. That part is done.

While verifying, I found the same defect class reintroduced one migration downstream: 026_apps_organization_id_not_null.ts (d44ee551, merged 2026-08-26) unconditionally throws when ROOT_ORG_ID is absent. Production is confirmed still in that state (this thread — the repoint/reparent decision hasnt been made), so 026 never gets marked applied and retries+throws on every boot: the #750 crashloop, one step later. #831 narrows that throw to only fire when it is actually warranted, mirroring the fix already merged the same day for its sibling (applications-services 011).

#831 does not decide the repoint-or-reparent question — that remains open and is, per your comment above, your call.

---

Generated by Claude Code