RFC: Deployment-managed core migrations

[ascorbic]

Background

EmDash currently applies pending core migrations during runtime initialisation. That keeps initial setup simple, but it makes a live request responsible for discovering whether a release's database changes can succeed. The request pays the migration latency, and authentication, connectivity, dialect, or migration failures surface only after the application has been deployed.

The risk and latency are more visible on remote databases. A migration may require many D1 or PostgreSQL round trips, while concurrent cold starts can attempt the same work and D1 provides no general migration lock. CI can build and deploy the application without proving that the intended production database is reachable and current.

Operators need an explicit deployment gate that uses the same migration set and behaviour-changing configuration as the build, identifies the database before mutation, and fails before new application code receives traffic. Adoption must remain optional: existing sites depend on automatic migrations, and the default logged-out request path cannot gain another query.

See #2276

Summary

Move core schema migrations out of the first production request while preserving the current behaviour as the backwards-compatible default.

The build emits a secret-free migration manifest that identifies the exact EmDash version, registered migration set, normalised locale configuration, and adapter-owned migration executor for that artifact. emdash migrate consumes that manifest, resolves an immutable database target, prints it before issuing SQL, and asks the executor to check or apply migrations. The public adapter boundary is a high-level executor rather than a Node-only Kysely connection, so a future executor may run inside a remote Worker without changing the contract.

The first release uses these transports:

Configured database	Migration transport	Operator input
SQLite	Direct Node connection	Writable database path
libSQL	Direct Node connection	URL and token environment variable
PostgreSQL	Direct Node connection	Connection-string environment variable
D1	Cloudflare D1 REST query API	Account ID, database UUID, API token
EmDash Hyperdrive adapter (PostgreSQL)	Direct Node connection to the origin	Separate direct-origin URL and network reachability

D1 uses a private migration-only Kysely transport over the documented REST query endpoint, not an emulation of the Workers D1Database binding or a general-purpose runtime adapter.

Decision

Use three separately versioned boundaries:

1. Migration manifest: the build writes the migration inputs that must match the code about to be deployed.
2. Core executor protocol: the CLI asks an adapter to describe, check, or migrate one immutable target and receives a common report.
3. Private transport: each adapter decides where and how execution happens. Direct adapters construct a dialect in Node; D1 uses its administrative REST API; a later adapter may execute the same request inside a remote Worker.

The generated migration manifest is authoritative in deployment. Evaluating astro.config.* is an explicit local fallback when no manifest exists, not an invisible second source of deployment configuration.

Goals

• Make core schema migration an explicit, automatable deployment gate that fails before application rollout.
• Check or apply the exact migration set and behavior-changing configuration belonging to the build against an operator-verified database target.
• Give operators mutation-free status and actionable failures without exposing database credentials.
• Cover EmDash's built-in production database adapters when their targets are reachable from the deployment environment.
• Preserve existing sites, rolling deployments, and the logged-out performance envelope while deployment-managed migration adoption remains optional.

Non-goals

• Provision, replace, or otherwise manage database infrastructure.
• Provide rollback, down, or a claimed-safe dry run; production migrations remain forward-only.
• Remove runtime automatic migrations in the first release.
• Migrate targets that require execution inside application runtime, such as private Hyperdrive origins or Durable Object storage, or externally manage preview and playground databases.
• Install plugins or apply user-defined content-model changes; the command covers statically registered core migrations only.
• Add database dialects that EmDash does not currently support, including MySQL through Hyperdrive.

Proposed migration manifest

The EmDash Astro integration writes .emdash/migrations.json during astro:config:done for build and sync. The file describes the build being deployed; it is not a cache of arbitrary evaluated configuration.

export interface MigrationManifestV1 {
	schemaVersion: 1;
	emdashVersion: string;
	migrationSet: {
		names: string[];
		fingerprint: string;
	};
	i18n: I18nConfig | null;
	database: {
		type: DatabaseDialectType;
		executorEntrypoint: string;
		executorConfig: unknown;
	};
}

migrationSet.names is the ordered list of statically registered migration names. fingerprint is a stable digest of those names and the EmDash package version. It is an identity check, not a digest of migration source and not a security signature.

The manifest contains normalised I18nConfig, because migrations may read the configured default locale. Omitting it would silently backfill a non-English site as en.

executorConfig must be secret-free. It may contain:

• a relative SQLite path;
• a public libSQL URL and the name of an environment variable containing its token;
• the name of an environment variable containing a PostgreSQL URL;
• a D1 binding name or Wrangler environment selector;
• the name of a Hyperdrive origin-URL environment variable.

It must not contain passwords, API tokens, private keys, connection URLs containing credentials, or values copied from .env or .dev.vars.

The integration validates the generated object against a versioned schema before writing it. It writes atomically so an interrupted build cannot leave a truncated manifest that a later deployment accepts.

The manifest should normally be ignored by source control and included with the deployment workspace or artifact. It is refreshed by the build immediately preceding migration.

Manifest verification

Before resolving a target, the CLI verifies:

1. schemaVersion is supported.
2. The project-local EmDash package version equals emdashVersion.
3. The migration names exported by that package produce the recorded fingerprint.
4. The executor entrypoint resolves from the consuming project's package graph.

A mismatch is a hard error, not a warning. Running migrations from a globally installed or stale EmDash version can otherwise apply a different schema from the artifact that will receive traffic.

Config-evaluation fallback

For local use before a build, emdash migrate --from-config evaluates the discovered astro.config.*, reads symbol-keyed metadata from the EmDash integration, normalizes Astro i18n with the same helper used during integration setup, and constructs an in-memory manifest.

This path executes trusted project configuration but does not run Astro hooks or start a server. It prints that it is using evaluated configuration rather than a build manifest. Deployment documentation uses the default build manifest, not this fallback.

There is no heuristic fallback to ./data.db, the first D1 binding, or whichever of DATABASE_URL and Wrangler configuration happens to exist. Missing or ambiguous configuration is an error requiring an explicit flag.

Database adapter contract

DatabaseDescriptor gains an optional, serializable migration capability:

interface DatabaseDescriptor {
	// Existing runtime fields omitted.
	migrations?: { entrypoint: string; manifestConfig: unknown };
}

manifestConfig is deliberately separate from runtime config; it is always safe to write to the build manifest. It does not default to runtime config, because PostgreSQL, libSQL, and custom adapters may keep credentials there.

Adapters that have no deployment migration executor omit migrations. The CLI fails with an actionable unsupported-adapter error and never imports a Workers-only runtime entrypoint in Node.

The executor boundary carries only the operation, artifact identity, safe target identity, and status sets:

export type MigrationAction = "check" | "apply";

export interface MigrationRequest {
	action: MigrationAction;
	i18n: I18nConfig | null;
	artifact: {
		emdashVersion: string;
		migrationSetFingerprint: string;
	};
}

export interface MigrationTarget {
	kind: string;
	label: string;
	fingerprint: string;
	accountId?: string;
	environment?: string;
	resourceId?: string;
}

export interface MigrationReport {
	target: MigrationTarget;
	knownApplied: string[];
	pending: string[];
	unknownApplied: string[];
	executed: string[];
}

export interface MigrationExecutor {
	target: MigrationTarget;
	execute(request: MigrationRequest): Promise<MigrationReport>;
	dispose?(): Promise<void>;
}

Each adapter's migration module exports a factory that receives its secret-free manifest configuration and CLI context. The factory may resolve credentials and metadata needed to describe a target, but it must not issue SQL. The CLI displays the target before calling execute().

Executors are single-use and disposal is idempotent. Adapter-specific target metadata, construction context, direct-dialect helpers, and lifecycle mechanics are implementation details defined in the implementation plan.

The high-level boundary is load-bearing. A future executor whose database handle exists only inside a Worker can implement the same request and report without returning a Node-side Dialect or changing DatabaseDescriptor again.

CLI behaviour

emdash migrate
emdash migrate --check
emdash migrate --status --json

# Non-standard manifest location
emdash migrate --manifest path/to/migrations.json

# Local fallback
emdash migrate --from-config [--config astro.config.mjs]

By default, the CLI loads .emdash/migrations.json from the project root. --manifest overrides that location. If the default file is missing, the command fails with instructions to build the project or explicitly use --from-config; it never evaluates project configuration implicitly.

migrate applies pending migrations and prints each applied name. --check performs no mutation and exits non-zero unless the database is exactly current for the manifest's migration set. --status prints known applied, pending, and unknown applied migrations; --json emits the stable report shape.

Before apply, the executor performs exact status and refuses to mutate when unknownApplied is non-empty. A database containing migration names unknown to this build may be newer or may belong to a divergent migration history; applying this build's pending names cannot be assumed safe.

There is no down or --dry-run. --check says whether work is required; it does not claim that applying the work would succeed or be reversible.

The CLI verifies the manifest and constructs a safe, immutable target before issuing SQL. It displays that target before remote mutation. Interactive use confirms it; non-interactive use may require an expected fingerprint and fails on mismatch. Credentials are never part of the target or fingerprint.

Exact status

Status compares the ordered registered names with _emdash_migrations in one query and produces three sets. A missing migration table means every known migration is pending.

• knownApplied: registered by this build and present in the database;
• pending: registered by this build and absent from the database;
• unknownApplied: present in the database but unknown to this build.

CLI --check requires pending and unknownApplied both to be empty. Runtime check is directional: it fails only for pending, because an old isolate must tolerate a database already expanded by a newer compatible build during a rolling deployment.

D1 REST executor

Cloudflare documents the built-in D1 REST API as appropriate for administrative use. A deployment migration is an administrative, low-frequency operation, so the REST query endpoint is the first D1 transport.

The executor is a private Kysely driver over the REST query endpoint. It sends compiled SQL and ordered parameters and maps validated results back to Kysely; it does not implement the Workers binding API and cannot be selected as an application runtime adapter.

Target and authentication

The executor must resolve an immutable account ID and database UUID before mutation. It may use explicit overrides or an explicitly selected Wrangler configuration and environment, but preview IDs, placeholders, conflicting accounts, ambiguous bindings, and non-unique names fail closed. The database must already exist, and the printed target includes its safe identity and fingerprint.

CLOUDFLARE_API_TOKEN is read at execution time and must have D1 Edit permission for the target account. Interactive Wrangler OAuth state is not copied.

Failure and concurrency semantics

The transport validates the complete Cloudflare and D1 response and accepts only parameter types established by the live REST conformance suite. Requests have bounded duration and response size, and redirects are disabled.

Writes and DDL are never retried automatically. A lost response after sending a write is an ambiguous outcome: the command fails and directs the operator to --status rather than replaying the statement.

D1 has no advisory migration lock. The existing duplicate migration-record recovery does not make arbitrary migrations safe to run concurrently, so deployment systems must serialize migration jobs by account and database UUID. EmDash does not claim exactly-once D1 execution.

EmDash Hyperdrive adapter and direct PostgreSQL

Cloudflare Hyperdrive supports PostgreSQL, MySQL, and compatible databases, but EmDash's current Hyperdrive adapter is PostgreSQL-specific. Its first migration executor connects directly to the origin using a separately named credential; that connection does not use Hyperdrive pooling, caching, cachedBinding, or private-network reachability.

The deployment runner must be able to reach the origin. Private origins remain on runtime auto until a remote executor is designed.

Future remote-Worker executor

The high-level executor contract leaves room for a separately proposed migration-only Worker for private Hyperdrive origins, bindings without an administrative API, or Durable Object storage. It is outside this RFC's first release and must not become an application Worker route.

Runtime migration policy

Add an additive integration option with auto, check, and manual modes:

emdash({
	database: d1({ binding: "DB" }),
	migrations: {
		runtime: "auto", // "auto" | "check" | "manual"
		dev: "auto", // optional development override
	},
});

• auto is the current behavior and remains the default.
• check performs the one-query directional check and returns a generic 503 with Retry-After before repositories use an old schema.
• manual performs no migration query and assumes deployment enforces emdash migrate --check.

dev defaults to auto but may be set to check or manual when local code uses a remote database. A validated EMDASH_MIGRATIONS_MODE runtime override supports one artifact deployed to multiple environments.

check replaces the existing automatic fast-path query, so it adds no cold-start query; manual removes that query. The effective policy governs runtime initialization, setup, and development bypasses. Commands that create a local database and isolated preview or playground databases remain self-migrating. Existing sites and templates therefore retain their first-run behavior and logged-out query counts.

Deployment sequencing

The recommended pipeline is:

- run: pnpm build
- run: pnpm emdash migrate
- run: pnpm wrangler deploy
- run: pnpm emdash migrate --check

The final check may instead be part of a health check. The migration credential belongs in the migration job and need not be present in the deployed application unless the runtime adapter independently requires it.

Compatibility window

Between migration and deployment, the previous application release serves all traffic against the new schema. If deployment fails, that state may persist indefinitely. Therefore every core migration used in this ordering must be compatible with the previous release.

The compatibility contract below treats rollback to old application code on the newer schema as the same requirement.

Backfill skew

A backfill performed before deployment can miss rows written by the old application after the backfill passes. New code must tolerate the unbackfilled/default value, or the deployment must include a post-deploy idempotent backfill that covers rows created during the window.

A migration must not assume that “all existing rows” remains true between its final query and application deployment.

First deployment

A database must exist before it can be migrated. For D1, automatic resource provisioning during application deployment creates a bootstrap cycle. New sites use one of:

1. provision D1 first with Wrangler or infrastructure as code, record its UUID, migrate, then deploy;
2. perform one bootstrap deployment with runtime auto, then switch later deployments to check or manual.

The CLI detects a missing target and explains these choices. It does not create the database or suggest that retrying migration before provisioning will help.

Migration compatibility contract

Deployment execution does not make migrations transactional, reversible, or isolated from application traffic. These invariants apply equally to CLI execution and runtime auto:

• Published migration history is immutable; corrections are new forward migrations. The manifest fingerprint does not replace this rule because it does not hash migration source.
• Changes are additive across the deployment window. Previous releases must retain valid read and write shapes; incompatible changes use expand, deploy, and later contract releases.
• New code tolerates default or unbackfilled rows written during the deployment window, and old isolates tolerate newer compatible migration records.
• up is resumable after any completed statement, and every intermediate D1 state remains usable.
• Correctness does not depend on a multi-statement transaction, advisory lock, exactly-once execution, stopped traffic, or a single migration job. Work remains bounded for runtime auto and remote databases.
• Migrations perform no external side effects that cannot share the migration record's atomicity.
• Behaviour-changing inputs are declared in the versioned, secret-free manifest and executor request rather than read from ambient configuration.

Failure behaviour

• Manifest mismatch fails before target resolution.
• Connection or authentication failure identifies the safe target without printing credentials.
• Target ambiguity fails before SQL.
• Runtime check returns a generic 503; pending names are logged but not exposed publicly.
• Migration failure exits non-zero and never invokes down.
• A D1 timeout after a request is sent is reported as ambiguous and is not retried.
• A partially applied D1 migration is not described as rolled back.
• Cleanup runs in finally; cleanup errors do not replace migration errors.
• Unknown applied migrations fail CLI --check but do not fail directional runtime check.

Security

• Validate the manifest before loading its executor module.
• Resolve executor entrypoints and dependencies from the consuming project, including under pnpm.
• Never write evaluated runtime database credentials into the manifest.
• Never log token-bearing URLs, authorization headers, API tokens, passwords, or certificate material.
• Require an immutable account/database identity before D1 mutation.
• Use least-privilege D1 credentials scoped to the target account.
• Bound HTTP request duration and response size.
• Disable redirects and automatic write retries.
• Treat astro.config.* as trusted project code only on the explicit config-fallback path.
• Do not expose an application migration route.

Alternatives considered

Cloudflare Vite plugin with a remote D1 binding

Rejected. It preserves exact Workers binding behaviour but repurposes a local-development control plane, making target identity, secret isolation, and process lifecycle harder to prove than with the administrative API. Reconsider it only if the REST transport cannot satisfy the live conformance suite.

A hand-written D1Database shim

Rejected. The migration transport does not emulate Workers binding objects. It adapts Kysely's compiled-query/result boundary directly to the REST API and stays private to deployment migrations.

Wrangler SQL migrations

Rejected. EmDash migrations are TypeScript programs that query data and dynamically discovered tables while executing.

Full Astro or Vite development server

Rejected. It loads unrelated application integrations, routes, plugins, watchers, bindings, and secrets and creates ambiguity about what may mutate the schema.

Production application migration endpoint

Rejected. It requires deploying candidate code before migration, reverses the desired ordering, and creates permanent high-impact remote administrative surface.

Ephemeral remote Worker in the first release

Deferred. It is the likely answer for private Hyperdrive and Durable Object targets, but requires Workers deployment permissions and a secure invocation and cleanup protocol that direct targets do not need.

Acceptance criteria

The design is complete when a deployment job can build an artifact, verify its migration identity, resolve and display the intended database without exposing credentials, apply the same core migrations used at runtime with the artifact's locale configuration, report exact status, and close all resources before deploying.

A D1 job cannot silently select local or preview state, cannot retry an ambiguous write, and does not depend on a development server. A Hyperdrive job cannot imply that a direct PostgreSQL connection is traveling through Hyperdrive. Runtime check and manual cannot be bypassed by setup or development routes. Existing sites retain auto by default, and the default logged-out query path gains no query.

References

• D1 REST API
• D1 outside Workers and administrative API guidance
• D1 Workers binding API
• Workers local development and remote bindings
• Hyperdrive local development