Recovery

This repo ships three doctors and one demolition tool, and the difference between them is what they are allowed to change. forge-preflight reads and reports. doctor.sh reads and reports. forge-deploy --recover fixes what it safely can and resumes an unfinished install without touching your data. forge-deploy --fresh-start deletes everything on the box and starts over. Knowing which one you are running is most of the skill.

The hub's Operations and Troubleshooting covers the failures an operator sees from the application's side. This page covers the deploy tooling's own diagnosis paths.

forge-preflight — the read-only pre-deploy doctor

Run it before a change. It validates the things that have actually bitten deployments, fails loud with the exact fix, and changes nothing. --quiet prints only problems and the verdict; the exit code is 1 for NO-GO and 0 for GO, with or without warnings.

What it checks, and why each check exists:

Check	The incident behind it
The deploy tree is a git checkout of this repo, with the expected origin	The scripts never landed on the server, or you are standing in the wrong repo
git is usable as this user and .git is writable	"dubious ownership" blocking a pull, usually after one stray sudo
No CRLF in any *.sh	bad interpreter: …^M on Linux after a checkout on Windows
Docker reachable as this user, and the Compose v2 plugin present	The two prerequisites everything else assumes
.env exists and declares a hosting mode	A tree that was cloned but never set up
Every component this box runs is pinned to an immutable tag	A floating latest crash-looped the API. Only this box's components gate — a UI box must not fail because the API it does not run is unpinned
The running compose project's overlay set matches the hosting mode	A cohost box brought up without its overlay, which drifts on the next recreate
Deploy state exists	Written on the first pin; its absence just means "never deployed"

Two limits worth knowing. Its component map covers the API, UI and test SPAs, so a box that declares demo gets an "unknown component" warning rather than a pin check. And it reads the deploy state from the conventional host path, so a relocated state directory shows as "no state file yet". Neither affects the verdict on a normal install.

forge-deploy --recover — fix and resume, in place

The one to reach for when a box is broken, half-installed, or was interrupted. It is non-destructive: it keeps your database, your files and your configuration.

Its design point is that state is inferred from artefacts, never from a checkpoint file — what setup and previous deploys left behind on disk — so it works regardless of how the box got into its state, including states no sequence of documented commands produces. It deliberately skips the internal preflight that other subcommands run, because a broken box is precisely what it is for.

It scans in dependency order, sorting findings into two piles: things it can fix, and things only you can do. Anything in the second pile stops the run with a plain-language instruction and the command to re-run.

Fixed automatically: starting a stopped Docker daemon; running the CLI installer when state or the log is missing; running first-time setup when there is no .env; fixing ownership after a stray sudo; merging in settings that .env.example has and your .env does not; generating a login signing key that was never generated; pinning the newest release for a component with no tag or a floating one; repairing two historical TLS-override states (a doubly-claimed host port, and a phantom empty directory Docker auto-created where the TLS nginx config file belongs); recreating a container stuck in a restart loop; and bringing up services that are stopped or were never started.

Handed to you, with the exact commands: Docker not installed; Docker running but your user cannot talk to it; the snap-packaged Docker on a cgroup v2 host, which cannot stop or update containers at all; a missing Compose plugin; an incomplete deploy tree; and a host port already held by another program — where it names the port and tells you not to kill docker-proxy processes blindly, because they may belong to something else.

It heals, rescans, and repeats up to three passes, and a fix that failed once is not retried — re-running it only reprints the same error, which is what once turned a single refused sudo into three identical walls of it. The first interactive pass also offers the choice up front: resume in place (the default), fresh start, or quit.

Afterwards, if this box runs the API, it waits on the health gate before declaring success. If the API never becomes healthy it reads the container log and translates known signatures into plain language:

• password authentication failed — the database password in .env does not match the one the database was created with.
• connection refused / name resolution failure — the database or object storage is unreachable; on a split install, check that box first.
• unauthorized from the registry — log in to the registry with a token that has package-read scope, then re-run.

When nothing matches, it says so, hands you the issue URL, and — only if you are logged into the GitHub CLI and it asks first — offers to file the report for you. The report is sanitised: .env is reduced to a key list with credential-shaped values redacted, and log tails are filtered for password-, secret-, token- and key-shaped assignments. Read it before you send it anyway.

A box that comes up healthy but has never been told what it should run flows straight into the topology wizard, so recovery and setup are the same path from the operator's side.

doctor.sh — why can nobody reach it

A different question, and a different tool. ./doctor.sh (also setup.sh --doctor, npx @armoryworks/forge-deploy --doctor, or the console entry "Forge runs here but people cannot reach it — find out why") triages exposure rather than health: stack status, local TLS, host firewall, public IP, NAT-hairpin detection, and a real outside-in reachability probe, ending in plain-language next actions including the literal port-forward rules to type into a router. It takes no arguments, changes nothing, and is safe to re-run until every check reads OK. When the outside-in probe's third-party service is unreachable it says so and explains how to test manually — it never fakes a result.

Do not confuse it with forge-deploy --doctor, which is an alias for --recover and does change things. See The CLI.

forge-deploy --fresh-start — the destructive one

This deletes, on this box: every Forge container, the database and every uploaded file, and the generated configuration — .env, certificates, the compose overrides, the box scope. It requires an interactive terminal and a typed confirmation; anything but the exact word aborts with nothing changed.

It tears down through compose first, then removes the known containers and the project's volumes directly as a belt-and-braces pass for states where compose can no longer resolve the project. Then it runs first-time setup and the topology wizard. If a from-scratch setup also fails on a clean slate, the problem is environmental — Docker, network or registry access — and it says so rather than looping.

There is a middle option the console reaches through the version picker's re-setup entry: it removes this box's containers and pinned images, clears the scope and remote wiring, and re-runs the topology wizard, keeping the data volumes unless you separately type the wipe confirmation. Reach for that when the wiring is wrong and the data is fine.

Failure modes you will meet

Most of these have a fix the tooling already knows; the point of listing them is recognition.

A second install came up against an empty database. Container names are global but volumes are project-prefixed, and the project name defaults to the install directory — so installing to a new path produced a working Forge with none of the data in it. Fresh installs pin the project name now; an existing install must not have it added. See Topologies.

A previous install's containers block setup. Names being global, a stack from another project owning them is a conflict compose can neither rename nor adopt. Setup detects it, stops that stack with its volumes untouched, and adopts its project name so the new tree reuses the same database and files.

A port is already in use. Setup asks Docker first — which names every published mapping without root, and handles the collapsed range form that once made the stack's own storage ports look like a stranger's — and moves the conflicting mapping to the next free port, recording it in .env. Ports 80 and 443 are excluded on purpose.

The stack is unreachable from other machines. Almost always the bind addresses: everything defaults to loopback. That is doctor.sh territory, and the bind table is on Operations and Troubleshooting.

An upgrade fails at the pre-reconcile backup. The backup is taken by executing inside the backup sidecar and the deploy fails closed without it. If a container pruner removed the sidecar, restore it and retry — this is correct behaviour and a surprising way to learn the sidecar was gone.

"relation … does not exist" after an upgrade. The schema reconcile did not run: either the flag was off, or the database is on another box and the reconcile skipped while reporting success. Topologies has the split-install rule; the hub's Upgrades and Rollback has the rest.

A rolled-back API. The health gate timed out. Before assuming the image is bad, raise HEALTHCHECK_TIMEOUT_SECS — first boot on a populated database is legitimately slow.

docs/TROUBLESHOOTING.md in this repo is the long-form catalogue, including host-level problems (storage layout, boot order, thermal throttling on small ARM boards) that never reach the application at all.

What to include when you file something

The recovery doctor's own report is the model: host and Docker facts, the redacted configuration key list, the deploy state, the container table, the health probe, and sanitised log tails from the API and the database. By hand, the equivalent is the running version from the API's anonymous version endpoint, the API's startup database-lifecycle block, forge-deploy --status, and the relevant container log.

File it in the repo that owns the component. Security problems go privately, not into a public issue — see Contributing.