Repository navigation
Backups
Spotcraft keeps two kinds of backups (ADR 0030):
-
Hetzner's daily backups of the server. These are 7 images of the whole disk, on by default in kodentuzo/hetzner (
apps_server_backups, +20% of the server's price). -
The agent's nightly backups, to a bucket. Every night at 03:00 UTC, each app gets a folder
<name>/<time>/with:-
database.dump, if the app has a database: a dump of its schema inapps; -
app.json, its settings and env; -
secrets.json, its database password and deploy hook hash.
After the apps come the platform's own databases, in folders whose names start with
_, which app names can't:-
_auth/<time>/auth.dump, the user pool'sauthschema (ADR 0035); -
_umami/<time>/umami.dump, analytics' wholeumamidatabase (ADR 0038).
With the R2 bucket from kodentuzo/hetzner's
cloudflare/, they can't be deleted or overwritten for 29 days, not even with the agent's key, and R2 deletes them after 30 (backup_retention_days). With another S3-compatible store, such as Hetzner Object Storage (https://fsn1.your-objectstorage.com), set a lifecycle rule yourself, or they are kept forever. -
The dashboard's app list shows each app's backup state. It turns red when the last backup failed or none has worked for 26 hours. On the app's page, Back up now makes one at once.
-
Enable R2 on the Cloudflare account (R2 in the Cloudflare dashboard). It asks for a payment method, even within the free tier, which includes 10 GB.
-
Make the bucket. In kodentuzo/hetzner's
cloudflare/terraform.tfvars, setbackup_bucket, e.g.spotcraft-backups, andbackup_jurisdiction = "eu"to keep the backups in the EU. Apply it with the Tofu workflow. Its token needs Workers R2 Storage: Edit on the account. Deploy reads the endpoint and the bucket from the root'sbackup_settingsoutput.For a bucket you made by hand, import it first, with its jurisdiction at the end:
scripts/tofu.sh cloudflare import 'cloudflare_r2_bucket.backups[0]' '<account id>/<bucket>/eu'
-
Make the agent's key. In R2 → Manage API tokens → Create API token:
- Object Read & Write, for the backup bucket only;
- no expiry, or a date you'll remember.
Add its Access Key ID and Secret Access Key as the secrets
BACKUP_ACCESS_KEY_IDandBACKUP_SECRET_ACCESS_KEY. -
Deploy Spotcraft. The agent logs
nightly backups at 03:00 UTC. -
In the dashboard, choose Back up now on each app and check that the folder appears in the bucket.
The bucket holds every app's env and database password. Keep it private. The key belongs only on the server. It can write new objects, but R2's bucket lock keeps it from deleting or overwriting a backup before it expires.
For example, after a bad migration. On the app's Backups page in the dashboard, pick a backup under Restore and confirm. The agent then:
- backs up the data as it is now, so you can go back to it;
- puts the app's schema back as it was at that backup, as the app's own role, so a dump can do no more than the app could;
- deploys the same version again, which runs its migrations on the restored data.
People who deleted their data since that backup stay deleted. The user pool logs the id of every deleted user for as long as backups are kept (ADR 0043), and the restore removes their rows again.
While it restores, the app may answer with errors for a few seconds. If the restore fails,
the app's data is as it was and nothing is deployed; the dashboard shows pg_restore's error.
A restore of a dump that has rows for users deleted before the log existed fails that way:
restore the newest backup instead, or delete those rows by hand.
- Make a new server, from a Hetzner backup image or from scratch (kodentuzo/hetzner), and deploy Spotcraft as in Getting started.
- With a Hetzner image, everything is back as it was at that night's image, and you're done. Otherwise, go on.
- For each app, open its latest folder in R2:
- In the dashboard, add the app again with the same name and settings, from
app.json. Copy every env variable. - Tick Database if it had one, and deploy. That creates an empty schema.
- Restore
database.dumpinto it with the commands above, then redeploy.
- In the dashboard, add the app again with the same name and settings, from
The app's role gets a new password. The app gets it as DATABASE_URL when it deploys, so
secrets.json is only needed if you want the old one.
Published from docs/. Edit the files there; changes here are overwritten.