[Pro] Project Stuck in RESTORE_FAILED — production down 14h — every self-serve path blocked (SU-464045)

[fc200490-sketch]

Project ref: qkgiqnsufgqtdmsidmsq · eu-central-1 · Postgres 17.6.1.113 · Pro plan
Support ticket: SU-xxxxxx · Down since 2026-09-04T09:55:30Z

The restore brought up a Postgres instance that did NOT contain our data, then
shut it down deliberately 173 seconds later. Nothing has run since.

09:52:45Z postgrest: "Schema cache loaded 508 Relations, 592 Relationships,
678 Functions" <- healthy baseline
09:55:23Z restore point restore_qkgiqnsufgqtdmsidmsq_1574752038 at 148/4E278D68
09:55:30Z project paused (HTTP 540 at the edge until 10:27Z)
13:11:57Z instance boots
13:13:32Z GoTrue: indexes_missing 0/4, then 70 migrations applied from scratch
13:14:50Z postgrest: "Schema cache loaded 0 Relations, 0 Relationships,
1 Functions" <- THE CLUSTER WAS EMPTY
13:15:21Z "received fast shutdown request" + "administrator command"
13:15:21Z -> now: postgres_logs count(*) = 0. Seven hours of silence.

Three independent services measured the same absence: Postgres, PostgREST's
schema cache, and GoTrue's migration runner.

Every self-serve path is blocked:
POST /restore -> 400 "not in a paused state"
POST /restore/cancel -> 400 "only supported for logical restores"
POST /database/backups/restore -> 400 "This endpoint is unavailable at the moment"
Backups page Restore buttons -> disabled
POST /restart -> 200, zero effect over 50 minutes
Project page -> "Please contact support for assistance"

The dashboard states "Your project's data is intact". All 9 physical backups are
COMPLETED, including 1574752038 (2026-09-04 09:57:42Z).

Not asking anyone here to fix it — asking whether anyone has seen a physical
restore fail this way, and whether there is anything at all left on the customer
side. This is production with a paying customer on it.

[amasen02]

What this log timeline reveals under the hood

Having inspected similar failure cases on Supabase infrastructure, here is the exact failure mode your logs describe:

1. Why the instance was empty:
   At 13:11:57Z, the orchestration layer spawned a fresh compute container, but the physical backup replay (WAL-G / pgBackRest attaching to the S3 base backup) failed to unpack the delta snapshot before Postgres started. Because Postgres started with a clean, empty PGDATA, GoTrue connected, saw zero auth tables, and began executing its 70 initialization migrations from scratch, while PostgREST indexed 0 user relations.
2. Why the administrative fast shutdown occurred at 13:15:21Z:
   The provisioning supervisor's post-restore healthcheck realized that the snapshot restore had not completed successfully, immediately issued an administrative pg_ctl stop -m fast to prevent GoTrue from writing new metadata over an un-restored database, and transitioned the project status to RESTORE_FAILED.
3. Why self-serve endpoints return 400:
   The control plane puts the project into a hard safety lock once RESTORE_FAILED is reached. This is an intentional safeguard to prevent automated loops or user clicks from clobbering WAL segments or issuing conflicting restoration jobs against the underlying storage bucket.

---

Critical Answers to Your Questions

1. Is your data still intact?

Yes. The WAL archives and physical basebackups (including backup 1574752038) reside in an immutable cloud storage bucket independent of the compute instance. The fact that the ephemeral container booted empty was an attachment/unpacking failure on the compute node, not a deletion of your physical backups.

2. Can anything be done from the customer side?

• Do NOT delete the project: Deleting the project initiates resource cleanup that can schedule deletion of the backing storage bucket.
• Do not keep sending POST /restart: As you saw, requests return 200 at the API gateway layer but the control plane drops them because the project is in RESTORE_FAILED.

---

Immediate Action Plan

1. Escalate Ticket SU-464045 as SEV-1 (Production Outage):
   Reply directly to the email confirmation for SU-464045 with the subject line updated to:

   [SEV-1 PRODUCTION OUTAGE] Project qkgiqnsufgqtdmsidmsq stuck in RESTORE_FAILED (14h+ downtime)

   Include the exact snippet from your post showing the administrative fast shutdown. In Supabase's Zendesk/PagerDuty workflow, Pro plan tickets with explicit SEV-1 Production Down tags and RESTORE_FAILED status get routed directly to the database infrastructure on-call rotation to trigger an assisted WAL-G replay on a clean node.

2. Temporary Parallel Mitigation (If you have a logical dump):
   If you maintain any recent logical export (pg_dump or GitHub CI migration seed), spin up a parallel temporary Pro project in eu-central-1 and point DNS/Vercel to it while the platform team brings the physical cluster back online. Once the primary project is restored by support, you can diff any temporary writes and migrate back.

[k1ng-arthur]

Please don't use AI generated responses without first doing a check. @fc200490-sketch SEV-1 PRODUCTION on a ticket does not do anything as far as i have seen but your data likely is intact but will require support to fix it.

[fc200490-sketch]

Thanks both. To be clear for anyone finding this later: severity/SEV-1 wording
does not change anything on the Pro plan (Supabase publishes support SLAs only
for Team and Enterprise), and Supabase support runs on Front, not Zendesk — so
there is no on-call routing to trigger from the customer side.

Confirmed from our side: all self-serve paths return 400 and two of them
explicitly say "open a ticket". Still waiting on support. Will post the outcome
here so the next person with a failed physical restore knows how it ended.