feat: unified outpost migrate CLI (#675)

[alexluong]

Summary

Implements the unified migration experience described in #675. Replaces the two separate migration systems (SQL auto-run at startup + outpost-migrate-redis standalone CLI) with a single outpost migrate command and an explicit startup gate that refuses to boot when migrations are pending.

What ships:

• New internal/migrator/coordinator package — wraps both the SQL migrator and the Redis migration runner behind a single API (List, Plan, Apply, Verify, PendingSummary, Unlock). Redis state tracking reuses the same hash-key layout as migratorredis.Runner, so existing Redis state carries over seamlessly.
• New outpost migrate subcommand tree — real list / plan / apply / verify / unlock commands backed by the coordinator. Replaces the old passthrough that delegated to the standalone Redis CLI.
• Startup gate — app.PreRun() no longer auto-applies migrations. Instead it calls the coordinator and refuses to start if anything is pending, with a clear error telling the operator exactly what to run.
• Removal of the standalone migration binaries — cmd/migratesql and cmd/outpost-migrate-redis are deleted along with every reference in the Makefile, goreleaser, Dockerfiles, and container entrypoints. The unified CLI plus the startup gate cover everything they did.

Breaking change

Deployment workflows that relied on auto-migration at startup need to move to an explicit two-step flow:

outpost migrate apply --yes   # run explicitly before deploy
outpost serve                 # refuses to start if anything is still pending

Release notes should flag this and add it to the upgrade guide.

Expected UX — what reviewers should see when testing this locally

The flow below is what an operator gets when starting from a v0.15.0-seeded deployment and upgrading to this branch. Every snippet is real output from a local run.

1. Help output

$ outpost migrate --help
NAME:
   outpost migrate - Database migration tools

USAGE:
   outpost migrate [command [command options]]

COMMANDS:
   list    List all migrations (SQL + Redis) with their status
   plan    Show what migrations would be applied
   apply   Apply all pending migrations (SQL then Redis)
   verify  Verify that migrations were applied correctly
   unlock  Force clear the Redis migration lock (use with caution)

OPTIONS:
   --config string, -c string  Path to config file [$CONFIG]
   --verbose                   Enable verbose logging

2. Startup gate fires when anything is pending

Running the server directly without applying pending migrations:

$ outpost serve
...
{"level":"error","msg":"pending migrations detected, refusing to start","sql_pending":3,"redis_pending":0}
pending migrations detected (3 SQL, 0 Redis) — run 'outpost migrate apply' before starting the server
exit status 1

3. outpost migrate list — unified view of both subsystems

$ outpost migrate list

SQL Migrations:
  [applied       ] sql/000001  init
  [applied       ] sql/000002  delivery_response
  [applied       ] sql/000003  event_delivery_index
  [applied       ] sql/000004  delivery_manual_attempt
  [applied       ] sql/000005  denormalize_attempts
  [applied       ] sql/000006  data_jsonb_to_text
  [pending       ] sql/000007  matched_destination_ids
  [pending       ] sql/000008  attempt_destination_type
  [pending       ] sql/000009  response_data_jsonb_to_text

Redis Migrations:
  [applied       ] redis/001_hash_tags   Migrate from legacy format (tenant:*) to hash-tagged format (tenant:{id}:*) for Redis Cluster support
  [applied       ] redis/002_timestamps  Convert timestamp fields from RFC3339 strings to Unix millisecond timestamps for timezone-agnostic sorting
  [applied       ] redis/003_entity      Add entity field to tenant and destination records for RediSearch filtering

Summary: 3 SQL pending, 0 Redis pending

Rows the coordinator knows aren't applicable (e.g. a Redis migration whose IsApplicable() returns false under the current config) render as [not_applicable] with a reason so list and the startup gate never disagree.

4. outpost migrate plan — what apply would change

$ outpost migrate plan

Planning migrations...

SQL Migrations (3 pending, v6 → v9):
  - sql/000007  matched_destination_ids
  - sql/000008  attempt_destination_type
  - sql/000009  response_data_jsonb_to_text

Redis Migrations: up to date

When Redis does have pending work, plan calls each migration's own Plan() and reports per-migration scope (tenants_need_migration, destinations_need_migration, total_need_migration).

5. outpost migrate apply --yes — applies SQL then Redis

$ outpost migrate apply --yes

Planning migrations...

SQL Migrations (3 pending, v6 → v9):
  - sql/000007  matched_destination_ids
  - sql/000008  attempt_destination_type
  - sql/000009  response_data_jsonb_to_text

Redis Migrations: up to date

{"level":"info","msg":"sql migrations applied","version":9,"count":3}

All migrations applied successfully.

apply always runs SQL first (schema must be ready before Redis data transformations) and then iterates Redis migrations in version order. Each migration goes through the standard plan → lock → apply → mark-applied loop the old Redis CLI used, so it's a drop-in replacement for the old outpost-migrate-redis apply --all --yes flow.

Flags: --yes (skip confirmation), --sql-only, --redis-only.

6. outpost migrate verify — confirms post-apply state

$ outpost migrate verify

Verifying migrations...

SQL: current=9 latest=9 [OK]
Redis:
  redis/001_hash_tags:   OK (0/0 checks passed)
  redis/002_timestamps:  OK (0/0 checks passed)
  redis/003_entity:      OK (0/0 checks passed)

All migrations verified successfully.

7. Server starts cleanly after apply

$ outpost serve
... healthy in ~2s

And the API is fully functional — tenants / destinations / attempts created under the old version are still readable (including response_data that survived the JSONB → TEXT migration in 000009), and freshly published events deliver end-to-end.

8. outpost migrate unlock

Interactive by default; prompts before clearing the Redis migration lock. --yes skips the prompt for scripted recovery from a dead migration process.

What happened to init and cleanup?

The old outpost-migrate-redis binary had two subcommands that are not in the unified CLI. Here's why that's intentional:

init — split into two separate concerns:

• init --current (preflight check) was used by the production container entrypoint to fail fast before starting the server if migrations were pending. This is now handled by the startup gate in app.PreRun() — the server itself refuses to start and prints a clear error. build/entrypoint.sh was simplified to just exec outpost serve.
• Fresh-install marking (the other half of init — detecting an empty Redis and marking all migrations as applied without running them so a brand-new deployment doesn't waste time scanning empty keys) is not implemented in the coordinator. It's not a correctness issue: on a fresh Redis the coordinator iterates each migration and calls its Plan()/Apply(), all of which scan for records needing work, find none, and return. Slightly wasteful on first boot of a brand-new cluster; easy to add later as a fast-path.

cleanup — niche use case, deferred:

Of the three current Redis migrations, only 001_hash_tags leaves legacy keys behind after running (it renames tenant:* → tenant:{id}:*). The other two transform values in place, so there's nothing to delete. And 001_hash_tags only has work to do on deployments that don't set DEPLOYMENT_ID, which is a shrinking set.

So cleanup was mostly dead code in the old CLI too — a safety net for one specific upgrade path. If a future migration genuinely needs post-apply key deletion, it's a small, self-contained addition to the coordinator and the CLI.

Test plan

• go build ./... passes
• go test -short ./... passes, including new unit tests for the SQL introspection helpers and miniredis-backed coordinator tests
• outpost migrate --help shows the new subcommand tree
• End-to-end manual QA of the v0.15.0 → current branch upgrade path (see comment for the full scenario and verification steps)
• outpost serve fails cleanly when migrations are pending and starts cleanly after outpost migrate apply

Open questions for reviewers

• Should the startup gate ship behind an env var (e.g. OUTPOST_STRICT_MIGRATIONS) for one release to give operators an escape hatch before becoming the default?
• Lock retry logic was removed along with auto-migration (it only mattered for concurrent startup races). If operators run outpost migrate apply concurrently from multiple nodes the second will fail fast. Acceptable?

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
outpost-docs	Ready	Preview, Comment	Apr 10, 2026 8:48pm
outpost-website	Ready	Preview, Comment	Apr 10, 2026 8:48pm

[alexluong]

Manual QA: v0.15.0 → current branch upgrade

Walked through the realistic upgrade path an operator will take to go from a released Outpost version to the unified migration CLI.

Scenario

1. Started the hookdeck/outpost:v0.15.0 stack (old auto-migration behavior).
2. Seeded data through the v0.15 API: tenant, webhook destination, published event, confirmed the delivery attempt landed successfully.
3. Stopped v0.15.0 and attempted to start the current-branch server — it refused with pending migrations detected (3 SQL, 0 Redis) — run 'outpost migrate apply' before starting the server, exactly as the new startup gate is supposed to.
4. Ran the new unified CLI end-to-end:
   • outpost migrate list — 6 SQL applied from v0.15, 3 pending (000007/8/9).
   • outpost migrate plan — reported SQL Migrations (3 pending, v6 → v9), Redis "up to date".
   • outpost migrate apply --yes — applied only the 3 SQL migrations, Redis not touched.
   • outpost migrate verify — SQL: current=9 latest=9 [OK].
5. Started the current-branch stack — healthy in ~2s.

Verified

• v0.15-era tenant, destination, and attempt are still readable under the current branch.
• The v0.15 attempt's response_data survived the JSONB → TEXT migration (000009) — body still fully deserializable.
• A freshly published event post-upgrade delivered end-to-end against the same v0.15-era destination within ~10s.
• Legacy cmd/migratesql and cmd/outpost-migrate-redis directories are gone; no references remain in Makefile, goreleaser, Dockerfiles, or entrypoints.

[alexbouchardd]

Very good 👍