[P1] One PR: account deletion fails closed — mandatory Cloud Tasks dispatch, job-scoped payloads, one execution owner

[Git-on-my-level]

Decision

Ship one backend PR that closes the failure class behind the 2026-07-14 production incident: the safe account-deletion path (durable Cloud Tasks dispatch, built in #8405/#9117) is optional, and the unsafe path (in-process execution + a five-minute reconciler in every shared FastAPI service) is the silent default. Production ran the unsafe default because nothing forced the configuration to exist.

Three commits, one reviewable unit:

1. Fail closed on configuration — production cannot boot or deploy with inline deletion dispatch.
2. Job-scoped task payloads — the wiper can no longer be handed an arbitrary UID.
3. One execution owner — shared services stop executing wipes; the reconciler becomes dispatch-only.

No IaC module, no datastore registry, no client changes, no macOS local-erase work, no canaries. Every guard added here runs in a lane that already executes automatically (CI unit suite + the existing runtime-env deploy contract), per the automatic-or-dead rule.

Production evidence (2026-07-14)

Accepted DELETE /v1/users/delete-account requests were followed by repeated background failures in conversation_recordings; five users produced 28 retries in one day across backend, backend-listen, backend-sync-backfill, and backend-integration. The recordings-bucket deadlock fix (#9632, merged 2026-07-13) was not on the serving image, and no environment had ACCOUNT_DELETION_DISPATCH_MODE=cloud_tasks — so every shared service fell back to in-process execution plus its own periodic reconciler.

Immediate operational step (not PR content, do now): deploy the current backend image so #9632 is serving, and drain/verify the stuck pending deletions. Record the deploy run and outcome in a comment here.

Grounding (verified in current code)

• backend/utils/cloud_tasks.py:77 — is_account_deletion_dispatch_enabled() returns os.getenv('ACCOUNT_DELETION_DISPATCH_MODE', 'inline') == 'cloud_tasks'. The default is the unsafe path.
• backend/utils/cloud_tasks.py:179 — enqueue_account_deletion_wipe(uid) enqueues {'uid': uid}. The OIDC handler (backend/routers/users.py, /v1/users/account-deletion-wipes/run) reads payload['uid'] and acts on it: any enqueue-side bug hands the wiper an arbitrary account.
• Same file, enqueue_listen_finalization_job — the repo's own correct pattern: an opaque {'job_id', 'dispatch_generation'} payload with the Firestore job as canonical, deliberately carrying no uid. Deletion must match this.
• backend/main.py:216–247 — startup_deletion_wipe_reconcile + _periodic_deletion_wipe_reconcile (300s) start in every service that shares main.py, and reconcile_pending_deletion_wipes (backend/services/users/account_deletion.py:332) executes wipes in-process. This is the source of the multi-service retry storm.
• backend/database/users.py — the canonical job state already lives in the top-level account_deletions/{uid} collection (mark_user_deletion_wipe_intent/started/running/failed/completed).
• The repo's existing deploy contract — backend/deploy/runtime_env.yaml + backend/scripts/validate-backend-runtime-env.py + pre-deploy-check.sh — already enforces multi-var enablement contracts (memory-maintenance does this today). That is the enforcement seam; do not build new provisioning infrastructure.

Commit 1 — fail closed on configuration

• In production profiles, ACCOUNT_DELETION_DISPATCH_MODE=cloud_tasks with a nonempty ACCOUNT_DELETION_TASKS_QUEUE and ACCOUNT_DELETION_HANDLER_URL is mandatory. Add the rule to validate-backend-runtime-env.py and the prod entries to backend/deploy/runtime_env.yaml (config-as-code, one time — after this PR there is nothing for an operator to remember).
• Add a startup guard: a production process whose deletion dispatch resolves to inline refuses to start with an actionable error. Silent fallback is deleted.
• inline remains valid only for dev/test profiles as the deterministic local dispatcher, calling the same worker function — no deletion-specific mode flag beyond the existing env var, no new env vars.

Commit 2 — job-scoped task payload

• start_account_deletion writes a random wipe_job_id onto the account_deletions/{uid} document in the same transaction that records the deletion intent.
• enqueue_account_deletion_wipe sends {'job_id': wipe_job_id} only — mirror the enqueue_listen_finalization_job pattern, including its rationale comment.
• The OIDC handler resolves the job by querying account_deletions for that wipe_job_id, derives the UID only from the matched document ID, and drops (200-acks with a structured log) any payload that resolves to zero or multiple documents, or a document already completed. It must never accept a UID from the payload.
• Compatibility: for one release, the handler also accepts a legacy {'uid': ...} payload only if a matching account_deletions/{uid} document exists in a nonterminal state (this is stricter than today). New enqueues always send job_id. Remove the legacy branch in a follow-up once the queue's max retry window has passed; reference this issue in the TODO.

Commit 3 — one execution owner

• reconcile_pending_deletion_wipes becomes dispatch-only: for a stale nonterminal job it re-enqueues a Cloud Task (job-scoped payload) and never calls the wipe function in-process. The handler's existing run-lock and claim logic make duplicate tasks harmless.
• The in-process execution branch used by the reconciler and the inline fallback is removed from production paths entirely; only the dev/test dispatcher may invoke the worker directly.
• Result: exactly one execution owner (the OIDC handler), Cloud Tasks owns retries/backoff, and shared services carry only a cheap idempotent re-dispatch loop instead of four competing executors.

Required tests (hermetic, discovered by backend/test.sh)

#	Case	Asserts
1	Prod env profile without cloud_tasks deletion config	validate-backend-runtime-env.py fails with the new rule; passing profile passes
2	Prod boot with inline dispatch resolved	startup guard raises; dev/test profile boots
3	Enqueue path	payload is exactly {'job_id': ...}; no uid present
4	Handler with unknown/ambiguous/completed job_id	dropped with structured log, zero mutations
5	Handler with legacy uid payload and no nonterminal job doc	dropped, zero mutations
6	Crash injected between job persist and enqueue	reconciler re-enqueues; handler converges to completed; wipe side effects execute once (run-lock/claim)
7	Reconciler under cloud_tasks mode	performs enqueues only; never invokes the wipe function in-process

Tests 4–6 are the wrong-account and durability regression tests for this incident class; they must drive the production handler/reconciler through injected fakes, not reimplementations. Register the deletion workflow in backend/testing/workflow_contracts.json if not already present (checkpoint/resume + retry/idempotency class).

Rollout note (the one piece of sequencing that matters)

Deploy order: this PR's image first, prod env additions in the same deploy. In-flight legacy {'uid'} tasks remain valid through the compatibility branch. Before removing the legacy branch (follow-up), confirm the queue has drained past its max-attempts window. The dispatch-only reconciler preserves recovery for any job stranded by the crash-between-persist-and-enqueue window, so no pre-deploy drain is required beyond the #9632 remediation above.

Acceptance criteria

• Tests 1–7 pass in CI; tests 4 and 6 fail against pre-fix code.
• grep proves no production code path reads a deletion UID from a task payload or executes a wipe outside the OIDC handler (dev/test dispatcher excepted).
• backend/scripts/pre-deploy-check.sh passes with the new rule; a prod profile stripped of the deletion vars fails it.
• Real-path evidence on dev: request deletion for a seeded test account → observe durable job doc, Cloud Task with job-only payload, handler execution, completed marker; record IDs/log lines in the PR.
• make preflight and backend/test.sh pass.
• AGENTS.md backend service map updated (deletion lane is mandatory in prod; reconciler is dispatch-only) in the same PR.
• PR body records root cause, recurrence evidence (Bug: Account deletion does not purge data from all storage backends (GDPR risk) #5088 → delete-account does not cancel active Stripe subscription #6750 → Account deletion leaves verified Twilio caller IDs orphaned #7640 → Make account deletion cleanup durable and observable #8405/Make account deletion cleanup durable #9117 → fix(backend): unconfigured recordings bucket deadlocks account deletion (6 users stuck on prod) #9632 → this incident), the durable guard (env-contract rule + startup guard + tests), and deferred work.

Deliberately cut from the original version, and why

• IaC provisioning module / queue auto-creation / canaries — new automation authority with no owner; the existing runtime-env deploy contract is the living check, and it already runs on every deploy. Queue/IAM exist; drift now fails the deploy instead of silently degrading.
• Universal datastore registry (store → inventory/delete/verify) with static enforcement — the right long-term idea and the true fix for the Bug: Account deletion does not purge data from all storage backends (GDPR risk) #5088/delete-account does not cancel active Stripe subscription #6750/Account deletion leaves verified Twilio caller IDs orphaned #7640/fix(backend): unconfigured recordings bucket deadlocks account deletion (6 users stuck on prod) #9632 recurrence, but epic-sized and independent of this incident's root cause. File separately if pursued; it must arrive as a ratchet check in the CI manifest, not a convention.
• macOS eraseLocalAccountData + client copy changes — desktop/app scope with its own review surface; separate small issues.
• Auth-last ordering, generation locks, byte-for-byte adversarial suites — the current wipe already orders auth deletion after data purge via background_wipe_user_data; formalizing generations without the registry adds state without closing a demonstrated gap.

Any revived item must meet the same bar: enforced in an existing CI/deploy lane, or assume it is a dead check.

Related

• fix(backend): unconfigured recordings bucket deadlocks account deletion (6 users stuck on prod) #9632 — recordings-bucket deadlock (merged; deploy + drain is the immediate action above).
• Make account deletion cleanup durable and observable #8405 / Make account deletion cleanup durable #9117 — built the durable seam this PR makes mandatory.
• Bug: Account deletion does not purge data from all storage backends (GDPR risk) #5088, delete-account does not cancel active Stripe subscription #6750, Account deletion leaves verified Twilio caller IDs orphaned #7640 — prior store-coverage fixes; the registry idea that would close their class is explicitly deferred above.