Upgrades and Rollback

Upgrading Forge is one gated procedure run by the forge-deploy CLI, and it has one failure mode worth understanding before you need it: the database schema moves forward before the application does, and it does not move back. This page covers what an upgrade does, where the state lives, the schema reconcile and the flag that gates it, what rollback can and cannot undo, and the one upgrade that is not automated at all.

Canonical detail lives in forge-deploy's docs/DEPLOY.md § Upgrading and rolling back and in the umbrella README. First install is Installation; getting data back is Backup and Restore.

Use the CLI for every version change. A bare docker compose pull / up skips the backup, the schema reconcile, the health gate and the audit log, and it composes a different file list than the CLI does — which on a public-facing box can quietly drop your port bindings. Where you genuinely need compose, go through forge-deploy compose …, which keeps the scope. See Operations and Troubleshooting.

What an upgrade does, in order

For each component this box runs, the CLI: verifies the tag actually exists in GHCR → rewrites the pin in .env → pulls the image → (API only) takes a fresh backup and reconciles the schema → recreates the container → waits for it to report healthy → and on failure re-pins the previous tag and recreates. An API box whose database lives elsewhere additionally checks that the remote database and object storage are reachable before anything is touched.

Two ordering rules are safety properties rather than conveniences. The set deploys schema, then API, then UI, and a failed tier stops the ones behind it — that is what stops a shop ending up with a new UI in front of the API it was never built for. And the schema reconcile runs before the container swap, so a halt or a failure there reverts the pin and returns without ever recreating anything: the running app and its database are untouched.

Because the health gate rolls itself back, a failed upgrade normally needs no operator action. The API's health check is composite — it warms Postgres, Hangfire, object storage and SignalR before reporting 200 — so first boot on a populated database is legitimately slow, and the gate's timeout is tunable in .env for large databases and slow hosts. The UI is the one tier that can move with no downtime: where there is an edge nginx upstream to flip, it deploys by cutover onto a standby container rather than by replacement, and a standby that never answers is thrown away while the live one keeps serving.

Where the state lives

Three places, and they are not interchangeable:

What	Where	Why it matters
The pins	SERVER_IMAGE_TAG, UI_IMAGE_TAG, SCHEMA_IMAGE_TAG in .env	Rewritten on every deploy. This is configuration, and a hand-edit can diverge from what runs
Per-component current / prior / deployedAt	A deploy-state JSON on the host	What --status and --rollback read, deliberately in preference to .env
A deploy event log	A host log file, tailed by --logs	Every attempt with its outcome — the timestamps you need to find the right pre-reconcile backup

forge-deploy --status prints the recorded deploy alongside the tag the container is actually running, and flags the two differing — the signature of a change made outside the deploy path.

There is no docker-compose.pin.yml. If you have read that anywhere, including in an older revision of this wiki, it does not exist: the pins are the three .env variables above.

Which tags deploy

Only immutable tags: a semver release (X.Y.Z, optionally with a pre-release suffix) or a build tag of the form main-<7 hex>. latest is refused outright, both at deploy time and at start-up, because a floating tag turns a restart into an unplanned upgrade.

forge-deploy --update moves the box to the newest release published for every component it runs — an intersection, so a half-published release is never offered. --update --check reports and exits non-zero when behind without changing anything, which is what cron or monitoring should call. --list shows recent versions paired with their build shas.

The schema reconcile

This is the part to read twice.

forge-db owns the schema and forge-api ships no migrations. The API's SchemaBootstrapper provisions a fresh database once and is a no-op on a populated one — it does not reconcile drift. On an install that already has data, the forge-db reconcile is therefore the only thing that carries the schema forward to a new release's expectations. See Architecture § Schema ownership for why the split exists.

Two facts about how it is switched on produce false confidence, and both belong on your pre-upgrade checklist:

• ENABLE_SCHEMA_RECONCILE ships false. A release deploy sets it to true for you and pins SCHEMA_IMAGE_TAG in lockstep with the release — that covers a semver tag, --update, the guided console and the setup wizard's version convergence. A build-tag (main-<sha>) deploy does not. If you deploy build tags, set the flag yourself, or the new image starts against a database that lacks the relations it expects.
• On a box whose database is elsewhere, the reconcile skips and reports success. In a split topology the Postgres container is on the DB box, not the API box's compose network, so the API deploy warns and proceeds rather than failing. That is deliberate — but it means a split install will upgrade its API against an unmigrated database unless someone runs forge-deploy --reconcile <tag> on the DB box. That subcommand exists for exactly this reason, and it is the whole schema-migration story for a split install. A DB-only box declares that it runs no versioned component, so it never deploys anything else.

When it does run, the reconcile: takes a fresh backup through the forge-backup sidecar and aborts the entire deploy if that fails; pulls the pinned forge-db image; and applies in three phases — pre-migrate scripts (renames, and anything else a state diff cannot express), the schema plan, then data backfills and reference seeds.

Additive changes apply automatically. A plan containing destructive changes halts the deploy: no destructive DDL runs, the app stays on its old image, and the changes are enumerated for you to disposition one by one. Re-run with --allow-destructive once you have decided. An install that records a support contact in .env is marked as customer-operated and the bypass is withheld rather than merely discouraged — the operator standing in front of the box is often not the person who can price a DROP.

Two sharp edges the tool will tell you about, worth knowing in advance:

• Pre-migrate scripts commit before the plan is computed. If a destructive halt reports that they already ran, the database has moved ahead of the still-running application, and stopping there is not a safe resting place — roll forward once the items are dispositioned, or restore the pre-reconcile backup.
• The pre-reconcile backup is taken by executing inside the backup sidecar. If you prune containers aggressively and forge-backup is not running, an upgrade of a populated install fails closed. That is the correct behaviour, and it is a surprising way to discover the sidecar was gone.

Rollback, and its one real caveat

Most of the time you will not need this section: when a new image fails its health gate the CLI re-pins the previous tag and recreates the container by itself.

Manual rollback is for the case where the new version comes up healthy and you then decide to go back. Here the ordering above turns around and bites: the reconcile ran before the swap, and schema changes are forward-only, so after a successful upgrade the database is at or ahead of the release you want to return to. forge-deploy --rollback re-pins the previously deployed application tag — it does not move SCHEMA_IMAGE_TAG, and it cannot un-apply a schema change. It is the correct and complete answer only for a release that carried no schema change.

Otherwise the database has to come back too, and the only route to that is the backup:

1. Stop the API.
2. Find the pg_dump the reconcile wrote immediately before it applied — the newest snapshot older than the upgrade; --logs timestamps the run.
3. Restore that dump into the database container (Backup and Restore has the procedure).
4. Re-pin SERVER_IMAGE_TAG, UI_IMAGE_TAG and SCHEMA_IMAGE_TAG together. Leaving the schema tag forward means the next deploy cheerfully re-applies exactly what you just backed out.
5. Recreate the API.

A release whose reconcile drops or rewrites anything is required to say so in its changelog entry. Read the changelog before upgrading, not after.

Postgres major versions

Application upgrades are gated and automatic. A Postgres major version bump is neither, and it arrives through an ordinary refresh of the deploy tree rather than through the gated deploy path — so it is worth knowing the shape before it lands.

A new Postgres major refuses to start against the previous major's data directory. It fails loudly rather than corrupting anything, but the stack is down until you migrate. The shape of the move is: dump with the old server still running, set a copy of the old volume aside, start the new server clean so it initialises, restore, then verify. Keeping that copy is what makes the step reversible.

Two pieces of advice generalise well beyond this one upgrade. Run the whole procedure through forge-deploy compose … rather than bare docker compose, so the port bindings in your override file survive the recreates — and do not park a version hold in docker-compose.override.yml, which setup.sh owns and rewrites. And verify by exercising the application — a login, one job, one search — not by watching a container go green: a healthy service proves the process started, not that your data came through.

The exact commands live in forge-deploy's Postgres upgrade runbook. They are deliberately not repeated here, because the version numbers are most of the content and they will rot.

Three ways to run the same upgrade

There is one mechanism and three front doors. All of them go through the gated path above; none of them can skip a gate.

• The guided console — forge-deploy with no arguments. It reads the machine, reports in plain language whether a newer release exists, and offers a short numbered menu. This is the customer-facing surface; every flag on this page is operator tooling underneath it.
• The upgrade wrapper — forge-upgrade.sh, or npx @armoryworks/forge-deploy. It refreshes the deploy tree (compose files and scripts, preserving .env, overrides and volumes) and then runs the same gated deploy. A wrapper, not a second mechanism.
• In the app — Admin → Updates. The executor is a small host service installed once from the deploy tree, because the thing performing an upgrade cannot live inside the container the upgrade replaces. It runs the same CLI. The command line stays available either way, and it is the recovery path — if the app will not start, that is where you go.

Before you upgrade

A short list, in order. Most of it is one-time.

1. Confirm you have a restorable backup, not just a backup — Backup and Restore.
2. Read the release changelog for a destructive-schema note.
3. On a split install, plan to run --reconcile on the DB box.
4. On a build-tag box, set ENABLE_SCHEMA_RECONCILE yourself.
5. Check that forge-backup is running, or the deploy will stop at the pre-reconcile backup.
6. Afterwards, log in and exercise one real workflow. Then check --status for tag drift.

Integration credentials, capability selections and label overrides are rows in your database, not files, so they survive an image upgrade untouched — see Configuration and Integrations and Capability Gating. What does not survive a host loss is .env and your TLS material; keep both off the box, per Hardening a Production Install.