Production Supabase down and nobody responding

[samueller]

Anybody know what to do? Supabase is having major issues with our database and nobody is responding to us. Our entire site has been down and we cannot access our data.

The disk size was downgraded from 100GB after updating the Postgres version for some reason. When we did a REINDEX DATABASE postgres; the severely limited disk space caused the disk to run out, and the entire database to be wiped. Then we tried a PITR and it said the restore failed. The project has been inaccessible.

[samueller]

CRITICAL UPDATE: Escalated to Hacker News

We are still experiencing a total production outage with severe data loss and have received zero response on our critical support ticket (#SU-342355).

To summarize the current state: after the Postgres upgrade caused an unexpected disk downgrade, a standard REINDEX wiped the DB due to the disk running out of space, and Supabase's automated PITR restore is actively failing.

We have escalated this issue to Hacker News in hopes of getting an infrastructure engineer's attention before our underlying EBS volume is permanently overwritten:
https://news.ycombinator.com/item?id=47394240

If any maintainers or Supabase team members see this, please flag ticket #SU-342355 to the on-call infra team immediately. We need an engineer to manually mount the volume to extract the WAL or raw data pages before the data is gone forever.

[samueller]

Update: Database Recovered

Supabase support successfully restored the database and our site is back online.

For anyone finding this in the future: standard PITR failed, and it required manual intervention from the support team to recover the data after the out-of-space event.

Note for the Supabase team: Our data is safe, but our dashboard is still completely broken/inaccessible following the recovery.