The Reconcile Harness

The harness is a small .NET CLI that decides when, against what, in what order and behind which gates a schema change runs. It does not compute the diff — stripe/pg-schema-diff does that, and deliberately so: reimplementing "desired versus live" for PostgreSQL is the expensive, correctness-critical half of the job, and the coordination around it is the cheap half that actually needs to be Forge-specific. This page is the coordination.

The operator's view — the flag that gates the reconcile during an upgrade, what a destructive halt looks like from the console — is on the hub at Upgrades and Rollback. What follows is what the harness itself does.

The verbs

Verb	What the harness does	Mutates?
assemble	Flatten schema/ into one ordered desired-state file (see Schema Ownership)	no
plan	Render the SQL the engine would run, with its hazard annotations, and warn about pending pre-migrate scripts	no
verify	Assert a live database matches the tree; non-zero on drift. Two independent checks	no
apply	The three-phase reconcile behind the gates, capturing the plan first	yes
dump / import	The clean-rebuild pair — see Dump and Import	import yes
baseline	One-time: ingest a canonical pg_dump --schema-only into the tree	no (writes files)

Every database-touching verb takes --db <postgres url>. The connecting user needs CREATEDB, because pg-schema-diff provisions and tears down its own temporary database on the target server to compute the plan. There is no separate dev-database to stand up and no bootstrap step for one.

Exit codes are a contract, and forge-deploy branches on them:

Code	Meaning
0	Success, or already in sync
1	The verb failed (engine error, script error, drift on verify)
2	dump refused a non-empty output directory
3	Blocked by a gate — a destructive plan without approval, or a non-dev target without confirmation
4	import found foreign-key orphans
64	Usage error

verify is worth calling out because it runs two checks and fails if either does. The first is an empty plan from the engine. The second is SchemaObjectVerifier, an explicit comparison of pg_extension, pg_proc and pg_trigger against the filenames in schema/{extensions,functions,triggers}. The engine does diff all three, so this is belt-and-suspenders — kept because a silent gap on the ledger immutability triggers once shipped undetected, and because pg-schema-diff itself documents that non-SQL function dependencies are untrackable. Extension-owned functions and FK-internal triggers are filtered out so they never read as drift.

The three phases of an apply

apply is not one operation. It is three, in a fixed order, and the ordering is the point.

Phase	What runs	Applied by	Ledgered
0	premigrate/	DataSeedRunner	yes
1	the schema reconcile	pg-schema-diff	no — state, not history
2	data/ then seed/	DataSeedRunner	yes

Phase 0 runs before the plan is computed, not merely before it is applied. The plan is derived from live database state; a rename applied after planning would leave the reconcile working against a shape that no longer exists.

Phase 1 is skipped when the target is already in sync — phase 2 still runs. New reference rows land on an install whose schema did not change, which is usually what you want and occasionally surprising.

Phases 0 and 2 share one ledger, forge_db.data_migration_log, keyed by the directory-qualified script name (premigrate/0010-….sql, seed/0020-….sql). One ledger rather than three means two phases can hold the same filename without either shadowing the other, and no script can be applied twice. The forge_db schema is created defensively by the runner and is excluded from the diff, so it never reads as desired-state drift and never pollutes forge-api's drift check.

Each script runs in its own transaction, ordered lexicographically by filename within its directory — which is why the convention is a zero-padded numeric prefix with gaps of ten. Applied-once means editing an applied script does not re-run it: the harness compares checksums and prints a loud warning when one has changed, but does not fail. Add a new numbered script for the delta instead.

Why premigrate/ exists

pg-schema-diff compares states. That is its whole value, and it is also the one thing it cannot do: it has no way to know that two states are related by a rename. Rename a table in schema/tables/ and the engine sees one table gone and one table new, and plans the only thing those two states justify — a DROP TABLE and a CREATE TABLE. On a dev volume that is a shrug. On an install with a year of correspondence in it, it is the whole table.

It is worse than a plain failure, because the plan succeeds. The operator sees a DELETES_DATA hazard, reasons "yes, I did remove a table", allows it, and the rows are gone.

data/ cannot cover this — those scripts run after the reconcile, by which point the old table is already gone. So renames get their own phase, ahead of everything. Once the rename has landed, the diff sees only the genuinely additive delta, which is exactly what a desired-state model is good at.

Scripts here must be idempotent by authoring, not merely by ledger: ALTER TABLE IF EXISTS, a guard around RENAME COLUMN (which has no IF EXISTS form). This is not belt-and-braces. A script must be safe against a database already at the target shape, because a fresh install provisioned from schema/ is exactly that — it comes up already renamed, and the pre-migrate script still runs against it once.

The authoring contract is in premigrate/README.md.

plan tells the truth about what it cannot see

plan is a pure read and does not run pre-migrate scripts. It checks the ledger, and when any are pending it prints a warning above the plan saying so.

That warning is load-bearing rather than decorative. Without it, plan on a not-yet-renamed target shows a DELETES_DATA hazard on the very table the pending script exists to preserve — and the honest-looking response to a hazard you believe you understand is to reach for --allow-destructive. Do not allow anything destructive on the strength of a plan that carries a pending pre-migrate warning. Run apply; the rename lands first and the reconcile that follows is additive.

The ledger lookup here is deliberately forgiving: an unreachable database, or one that has never been applied to and has no ledger, yields an empty list rather than failing the plan. Planning has to keep working against a target that has never seen the harness.

The gates

DeployGates sits above the engine and decides whether the harness is allowed to run it at all. Two independent rules:

Destructive changes are blocked by default, in every environment. This is parity with dacpac's BlockOnPossibleDataLoss. Destructiveness is read from two sources — pg-schema-diff's own DELETES_DATA hazard annotation, and a scan of the plan SQL for DROP statements. Only --allow-destructive, passed for that run, lifts it.

Non-dev targets require explicit confirmation and a fresh backup. --env dev and --env scratch skip that gate; anything else needs both --yes and --backup-taken. The gate takes --backup-taken at its word — it is an assertion by the caller, not a verification. In the deploy path, forge-deploy earns the right to pass it by taking a backup through the backup sidecar first and aborting the whole upgrade if that fails.

Phase 2 inherits the same posture: pending data//seed/ scripts against a non-dev target without --yes --backup-taken return blocked without touching the database.

One sharp edge in that destructive scan is worth planning around. The pattern matches DROP INDEX and DROP CONSTRAINT alongside DROP TABLE and DROP COLUMN. Dropping an index loses no data, and renaming a constraint plans as a drop plus an add — yet both trip the gate and, in a deploy, halt the upgrade for operator disposition. A release that only reshapes indexes will still stop an upgrade. That is the conservative side to err on, but it means index churn is not free: expect to disposition it, and say so in the release notes.

Whatever hazards pg-schema-diff annotated the plan with are extracted and passed through to apply as --allow-hazards, so the apply allows exactly what the plan showed and nothing more.

Receipts, and where they go

Before applying, the harness captures pg-schema-diff's plan SQL to history/<timestamp>-<env>.sql, with a header stating what it is. These are receipts, not recipes: the tree plus the engine remain the only things that determine state, the harness never reads from history/, and replaying one is not a supported operation. import writes a JSON receipt of its own alongside them.

Two honest caveats:

• The captured plan comes from a separate plan invocation than the one apply computes internally. It is a faithful record only if nothing else changed the schema in the window between them. In the deploy path, where the harness is the only writer, that window is not a practical concern; on a database several people are poking at, do not treat the receipt as a byte-exact transcript.
• In a container run, the receipt is discarded. forge-deploy runs the image as docker run --rm with no volume mounted at /forge-db/history, so the file is written into the container's filesystem and thrown away with it. The plan text still reaches the deploy's own output and log, which is where to look. The repo's .gitignore also excludes history/*.sql, so local receipts stay local. A durable, install-side audit trail is not built yet.

How it ships and how forge-deploy runs it

The Dockerfile bakes three things into one multi-arch image published to GHCR: the pg-schema-diff binary (built from source at a pinned version — the project publishes none), the harness, and the input directories. The image's working directory is the baked repo root, and the harness finds it by walking up for schema/, so a caller only needs to pass --db.

Every input directory is copied in, and premigrate/ is not optional. A missing directory is a silent no-op in the runner — discovery simply yields nothing — so an image built without it would skip phase 0 entirely and let the reconcile plan DROP + CREATE over a rename, with no error to notice. That is the whole reason the COPY lines are enumerated rather than globbed.

forge-deploy invokes the image as a one-shot on the database's own container network, pinning SCHEMA_IMAGE_TAG in lockstep with the release and calling apply --env prod --yes --backup-taken after taking a backup. On exit code 3 it re-runs plan, pulls the destructive statements out of the rendering and enumerates them for disposition. An install that records a support contact is treated as customer-operated and the --allow-destructive bypass is withheld rather than merely discouraged — the person standing in front of the box is often not the person who can price a DROP.

The split install

This is the failure mode to internalise. On an API box whose database lives elsewhere, the PostgreSQL container is not on that box's compose network, so the reconcile skips and the deploy proceeds. It warns; it does not fail. That is deliberate — the API box has no route to the database's network and should not block on it — but the result is that a fully split install will happily upgrade its API against an unreconciled database and report success.

The answer is a subcommand that runs on the box that owns the database:

forge-deploy --reconcile <tag>

It refuses to run anywhere else, pins the schema tag, enables the reconcile flag, and drives the same backup → apply path. For a split install, this is the entire schema-migration story — nothing else on any box will move the schema forward. See Upgrades and Rollback and the forge-deploy wiki.

The pre-migrate halt is not a safe resting place

Phase 0 commits before the plan is computed, and each script commits in its own transaction. So when a destructive plan halts the deploy at phase 1, "nothing was applied and the app was not swapped" is only mostly reassuring: if any pre-migrate script ran, the database has already moved and the still-running application version may not survive it — a rename is precisely the change an old binary cannot tolerate.

The harness says so on every path out that is not success, and forge-deploy repeats it when it sees that line. The two ways out are to disposition the destructive items and roll forward, or to restore the pre-reconcile backup. Stopping and waiting is not one of them.

Running it by hand

For the common "reconcile my local stack" case there is tools/apply-schema.sh, which resolves the database URL from the forge-deploy compose conventions (falling back to the compose defaults), checks the engine is installed, always shows the plan first, applies through the same gates, and finishes with a verify round-trip.

tools/apply-schema.sh                                    # plan → apply → verify, dev target
tools/apply-schema.sh --plan-only                        # look, don't touch
tools/apply-schema.sh --env prod --yes --backup-taken    # gated non-dev apply

It is a convenience wrapper, not a second mechanism: it cannot skip a gate, and --allow-destructive still has to be passed explicitly.

The harness's own tests live in tests/Forge.Db.Tests. Unit tests cover discovery, ordering, glob semantics, COPY-text projection and the dump splitter, and run with no database. The Postgres-backed tests skip silently unless FORGE_DB_TEST_DSN names a reachable superuser DSN whose user has CREATEDB — a green dotnet test with that variable unset proves less than it looks like it does. CI sets it against a pgvector service before it will build or push an image.

Related

• Schema Ownership — what the harness is reconciling to.
• Dump and Import — the other half of the CLI.
• forge-deploy wiki — the caller.
• Backup and Restore — what --backup-taken should mean before you assert it.