Backup and Restore

Forge takes scheduled snapshots of itself out of the box. What it does not do is restore them for you, push them anywhere off the host, or capture the secrets you need to have a machine to restore into. This page covers what a snapshot contains, what it deliberately leaves out, and what recovery actually looks like — so that the first time you read the restore procedure is not the day you need it.

Canonical runbook: docs/backup-restore.md in forge-deploy, which carries the exact commands for a rehearsal restore and a production-down restore. This page is the orientation and the decisions; that document is the procedure.

What a snapshot is

The forge-backup sidecar writes one timestamped folder per run under ./backups/ in the deploy tree, containing three things:

Artifact	What it is
forge.dump	A pg_dump of the Forge database in custom (compressed) format, restorable with pg_restore
minio/<bucket>/…	A full mirror of every object-storage bucket — attachments, uploads, generated documents
manifest.json	What the snapshot came from: source database and object-storage endpoints, and the artifact filenames

Each snapshot is an independent point-in-time copy, not an incremental chain, so any one folder is restorable on its own and losing an older one costs you nothing but that point in time. A retention sweep at the end of each run deletes folders older than the configured window.

The sidecar is a separate container on purpose: backups must keep running when the API is down, which is the scenario you most need them for. It also means a failed backup does not crash anything — the failure is logged and the next scheduled run tries again. A backup that has been quietly failing for weeks looks exactly like one that has been working, so put forge-backup's logs in whatever you watch; see Operations and Troubleshooting.

What controls it

Three variables in .env, documented in .env.example under "Backups":

• BACKUP_SCHEDULE — a cron expression, evaluated in UTC, not your local time zone.
• BACKUP_RETENTION_DAYS — how many days of snapshot folders to keep.
• BACKUP_RUN_ON_START — take a snapshot immediately when the container starts. Off in steady state; worth flipping on temporarily around a risky change.

Changes take effect when the sidecar restarts (docker compose up -d forge-backup). To take a snapshot right now without touching the schedule, run the same script the deploy tool runs: docker compose exec forge-backup /usr/local/bin/backup.sh.

Snapshot size is dominated by attachments rather than rows, so size your ./backups/ disk against retention × snapshot size and revisit it once real documents are flowing.

What a snapshot does not contain

Two categories, both needed to bring an install back, and neither is in any snapshot:

1. Your .env — the database and object-storage passwords, JWT_KEY, deployment target, image pins. Without the passwords a restore has nothing to restore into; without the same JWT_KEY every existing session is invalidated on the way back up.
2. Your TLS certificates and reverse-proxy configuration — certs, vhosts, tunnel config. Recoverable by re-issuing and reapplying, but only if you know what they were.

Copy both somewhere else, once, and again whenever they change. They change rarely, which is exactly why they are easy to forget. Hardening a Production Install ends on the same instruction for the same reason: the secrets you set during hardening are not protected by the thing that protects your data.

The reverse is worth stating too. Integration credentials are in the snapshot: they are encrypted rows in the database (see Configuration and Integrations), and the Data Protection keys that unseal them are persisted in the database as well — so a restored database comes back with its integrations intact. The flip side of that convenience is that a snapshot is exactly as sensitive as the database itself. The sidecar does not encrypt what it writes. Put ./backups/ on encrypted storage, or encrypt on the way to wherever you copy it, and restrict access to the directory accordingly.

There is no off-site copy

./backups/ is a bind mount in the deploy tree — the same disk as the database it is protecting. It defends you against a bad migration, a bad import and a deleted record. It does not defend you against losing the host.

Nothing in the stack pushes snapshots anywhere. Point rsync, rclone or restic at that directory on your own schedule, to a destination that is not that machine. (Be aware that forge-deploy's older DEPLOY.md describes an off-site rclone push and a backup-now script; neither exists — the sidecar writes locally and the script is backup.sh. Where a doc and the code disagree, the code wins.) Admin-configurable S3 and SFTP destinations are named in the repo as a later addition; until they ship, off-site is your job.

Restoring

There is no forge-deploy --restore. Restore is a manual procedure, and the shape of it is short enough to hold in your head:

1. Stop the API first. Restoring underneath a running API corrupts state.
2. pg_restore --clean --if-exists --no-owner the snapshot's forge.dump into the database.
3. Mirror each bucket back from the snapshot's minio/ directory into object storage.
4. Start the API and check /api/v1/health before letting anyone back in.

The exact commands, including credentials handling and the spot-check queries, are in the forge-deploy runbook — follow it there rather than a copy that can drift from the one your own tree ships.

That runbook opens with the rule this whole page exists to serve: you don't trust a backup you haven't restored. It carries a rehearsal procedure that restores a snapshot into throwaway containers on different ports, side by side with the live stack, so you can prove the snapshot is recoverable and learn the steps under no time pressure. Do that once before you depend on backups for anything that matters, and again after any change to the stack's shape.

One reassurance about the worst case: forge-deploy --fresh-start removes containers, volumes and generated configuration, and leaves ./backups/ alone. Wipe-and-restore is a real recovery path, not a data-loss event.

How backups interlock with upgrades

Two connections worth knowing before your first upgrade, both covered in Upgrades and Rollback:

• The schema reconcile takes a fresh snapshot before it touches anything, using this same sidecar, and aborts the whole deploy if that backup fails — nothing is applied and the app is not swapped. It fails closed on purpose. The consequence for anyone who prunes containers aggressively: no forge-backup, no upgrade.
• That pre-reconcile snapshot is the artifact you restore if you need to back out a schema-carrying release, because the reconcile runs before the image swap and schema changes are forward-only.

There is also a version coupling that only bites during a Postgres major upgrade: the sidecar image carries the pg_dump client, and pg_dump refuses to dump a server newer than itself. A Postgres server major that moves without the backup image moving with it breaks the nightly backup and every upgrade, since the upgrade path depends on that backup succeeding. Treat the two as one change.

Where the snapshots live on a split install

The backup sidecar follows the database. On a split topology, boxes that do not run Postgres scope out forge-backup as well, so snapshots exist only in the deploy tree of the box that owns the database — that is the tree whose ./backups/ your off-host copy must read, and that box's .env is the one holding the credentials a restore needs. See Installation § Split topologies.

Disaster-recovery checklist

Everything else is re-fetchable. The deploy tree comes back from GitHub; the images come back from the registry; the stack rebuilds from setup.sh. So the whole of your disaster-recovery obligation is four lines:

1. Hold snapshots off-host, on your own schedule.
2. Hold .env off-host, refreshed whenever a credential changes.
3. Hold TLS certificates and proxy configuration off-host.
4. Rehearse a restore from what you hold, on a machine that is not the production one, before you need it.

Backups are not an export

A snapshot is a recovery artifact in Postgres's own dump format, aimed at putting this install back the way it was. It is not the answer to "can I get my data out of Forge and into something else." That has a separate and better answer — an admin-triggered database dump in an open, documented archive format, plus per-domain CSV and spreadsheet exports — covered in Data Ownership and Export. Neither replaces the other: the export does not carry object storage or a schedule, and the snapshot is not a format another system will read.