Skip to content

v1.8.3 — Deferred data migrations: container up immediately

Choose a tag to compare

View all tags
@clem-field clem-field released this 27 May 19:44
v1.8.3
89d72de

v1.8.3 — Deferred data migrations: container up immediately, body runs in background

Architectural fix. Closes the structural issue that made
v1.8.1 / v1.8.2 deploys vulnerable to a kill loop: v1.8.2 fixed
the SPECIFIC UUID-collision bug in the migration body, but any
future migration body bug would re-trigger the same loop. This
release removes the kill loop entirely.

Any ActiveRecord::Migration marked with include DeferredDataMigration
registers itself at db:migrate time (fast — no actual data work)
and executes its body post-boot via a Solid Queue job (in-Puma).
The container binds its port and ECS health checks pass within
seconds even on multi-minute data migrations.

The structural why

v1.8.0 / v1.8.1 / v1.8.2 ran the back-matter promotion data
migrations synchronously during bundle exec rails db:prepare
(bin/docker-entrypoint:27). The container port stayed closed for
the duration. ECS health checks failed, task drained, replacement
spawned, loop. v1.8.2 fixed the specific UUID-collision bug — but
any future migration body bug would re-trigger the same kill
loop. v1.8.3 makes that class of incident structurally impossible:
migration bodies run after the container is healthy. Bad bodies
fail the JOB (visible in /admin/data_migrations), not the
CONTAINER (silent ECS replacement loop).

Pieces

app/lib/deferred_data_migration.rb
Mixin. In db:migrate context: registers a pending
DataMigrationRun row, returns. In runner context (thread-local
executing flag set): block executes.

app/models/data_migration_run.rb + migration
Tracking row. Lifecycle pending → running → completed / failed.
Mirrors SeedSection shape; name uniquely indexed.

app/services/deferred_data_migration_runner.rb
Acquires PG advisory lock (0x5DA7A_CDA7A_DA7A) so only one
container runs at a time across the fleet. Resets stuck
"running" rows from crashed previous containers. Iterates
pending + failed rows. Per-phase structured JSON logs for
CloudWatch. Emits data_migration_completed AuditEvent on
success.

app/jobs/deferred_data_migration_job.rb
Solid Queue. No internal retry — failures sit visible in the
tracking row; next container boot re-enqueues.

config/initializers/enqueue_data_migrations.rb
after_initialize hook enqueues the job if any pending /
failed rows exist. Guards for test env, console, missing
DB / table, and SPARC_SKIP_DEFERRED_DATA_MIGRATIONS=true.

lib/tasks/sparc_data_migrations.rake
Operator entry points:
sparc:data_migrations:run synchronous trigger
sparc:data_migrations:status text table
sparc:data_migrations:reset NAME=ClassName back to pending

app/controllers/admin/data_migrations_controller.rb + view
/admin/data_migrations — read-only status table. Status badges,
counters, timestamps, duration, records_processed, error
excerpt. Admin-only.

app/models/audit_event.rb
+ data_migration_completed action, + "Data Migrations" category.

Existing v1.8.x back-matter promotions converted

The two back-matter promotion migrations from v1.8.0 / v1.8.2 now
include DeferredDataMigration with their body wrapped in
defer_data_migration { ... }:

db/migrate/20260526120000_promote_cdef_back_matter_imports.rb
db/migrate/20260526200000_promote_remaining_back_matter_imports.rb

bin/docker-entrypoint UNCHANGED. The fix is transparent at the
entrypoint level — db:prepare just runs faster because deferred
migrations register-and-return.

Operator impact

Fresh install
Two pending DataMigrationRun rows registered at db:migrate.
Container boots, hook enqueues the job, work runs in
background (fast — empty tables).

Mid-failed v1.8.x upgrade (state we just stabilized with v1.8.2)
db:prepare completes in seconds. Container binds the port,
ECS health check passes within seconds. Solid Queue worker
picks up the job, back-matter promotion runs in background
while the app serves traffic. Operator watches
/admin/data_migrations for progress.

Already on v1.8.2 cleanly migrated
Both v1.8.x migrations already in schema_migrations — Rails
skips them entirely on v1.8.3 deploy. No DataMigrationRun
rows created. Hook sees nothing pending → no-op.

Test result

bundle exec rspec 2570 examples, 0 failures, 2 pending
(+29 new across 4 spec files)
bin/rubocop on all changed files 0 offenses

Notes for future migration authors

  • Most migrations should stay synchronous (schema changes —
    app crashes if it touches a column that doesn't exist yet).
  • Only mark a migration deferred when its body shuffles
    EXISTING data (INSERT/UPDATE on rows) — never for schema
    changes (ALTER TABLE, ADD COLUMN, etc.).
  • Deferred migrations MUST be idempotent. The runner re-runs
    failed rows on next boot. Design for partial completion +
    retry.

Sequence

v1.8.0 #498, #499, #581, #582, #583, #584
v1.8.1 Okta tab CSP regression, #587 login_failure reasons
v1.8.2 Cross-doc UUID collision in back-matter promotion
(specific bug fix)
v1.8.3 Deferred data migrations (the class of bug)

🤖 Generated with Claude Code

Assets 2
Loading