Skip to content

Backups

github-actions[bot] edited this page Nov 22, 2026 · 4 revisions

Backups and restoring

Spotcraft keeps two kinds of backups (ADR 0030):

  • Hetzner's daily backups of the server. These are 7 images of the whole disk, on by default in kodentuzo/hetzner (apps_server_backups, +20% of the server's price).

  • The agent's nightly backups, to a bucket. Every night at 03:00 UTC, each app gets a folder <name>/<time>/ with:

    • database.dump, if the app has a database: a dump of its schema in apps;
    • app.json, its settings and env;
    • secrets.json, its database password and deploy hook hash.

    After the apps come the platform's own databases, in folders whose names start with _, which app names can't:

    • _auth/<time>/auth.dump, the user pool's auth schema (ADR 0035);
    • _umami/<time>/umami.dump, analytics' whole umami database (ADR 0038).

    With the R2 bucket from kodentuzo/hetzner's cloudflare/, they can't be deleted or overwritten for 29 days, not even with the agent's key, and R2 deletes them after 30 (backup_retention_days). With another S3-compatible store, such as Hetzner Object Storage (https://fsn1.your-objectstorage.com), set a lifecycle rule yourself, or they are kept forever.

The dashboard's app list shows each app's backup state. It turns red when the last backup failed or none has worked for 26 hours. On the app's page, Back up now makes one at once.

Set up the bucket

  1. Enable R2 on the Cloudflare account (R2 in the Cloudflare dashboard). It asks for a payment method, even within the free tier, which includes 10 GB.

  2. Make the bucket. In kodentuzo/hetzner's cloudflare/terraform.tfvars, set backup_bucket, e.g. spotcraft-backups, and backup_jurisdiction = "eu" to keep the backups in the EU. Apply it with the Tofu workflow. Its token needs Workers R2 Storage: Edit on the account. Deploy reads the endpoint and the bucket from the root's backup_settings output.

    For a bucket you made by hand, import it first, with its jurisdiction at the end:

    scripts/tofu.sh cloudflare import 'cloudflare_r2_bucket.backups[0]' '<account id>/<bucket>/eu'
  3. Make the agent's key. In R2 → Manage API tokens → Create API token:

    • Object Read & Write, for the backup bucket only;
    • no expiry, or a date you'll remember.

    Add its Access Key ID and Secret Access Key as the secrets BACKUP_ACCESS_KEY_ID and BACKUP_SECRET_ACCESS_KEY.

  4. Deploy Spotcraft. The agent logs nightly backups at 03:00 UTC.

  5. In the dashboard, choose Back up now on each app and check that the folder appears in the bucket.

The bucket holds every app's env and database password. Keep it private. The key belongs only on the server. It can write new objects, but R2's bucket lock keeps it from deleting or overwriting a backup before it expires.

Restore one app's data

For example, after a bad migration. On the app's Backups page in the dashboard, pick a backup under Restore and confirm. The agent then:

  1. backs up the data as it is now, so you can go back to it;
  2. puts the app's schema back as it was at that backup, as the app's own role, so a dump can do no more than the app could;
  3. deploys the same version again, which runs its migrations on the restored data.

People who deleted their data since that backup stay deleted. The user pool logs the id of every deleted user for as long as backups are kept (ADR 0043), and the restore removes their rows again.

While it restores, the app may answer with errors for a few seconds. If the restore fails, the app's data is as it was and nothing is deployed; the dashboard shows pg_restore's error. A restore of a dump that has rows for users deleted before the log existed fails that way: restore the newest backup instead, or delete those rows by hand.

Restore after losing the server

  1. Make a new server, from a Hetzner backup image or from scratch (kodentuzo/hetzner), and deploy Spotcraft as in Getting started.
  2. With a Hetzner image, everything is back as it was at that night's image, and you're done. Otherwise, go on.
  3. For each app, open its latest folder in R2:
    1. In the dashboard, add the app again with the same name and settings, from app.json. Copy every env variable.
    2. Tick Database if it had one, and deploy. That creates an empty schema.
    3. Restore database.dump into it with the commands above, then redeploy.

The app's role gets a new password. The app gets it as DATABASE_URL when it deploys, so secrets.json is only needed if you want the old one.

Clone this wiki locally