Migrations

A document migration rewrites the documents of a collection — typically to bring every document onto the active schema. It's the eager counterpart to secondary schemas: use secondary schemas so old documents keep working, and a migration when you need them all physically on the new shape (for a new index or query).

Defining a migration

A migration is a DocumentMigration<TModel, TKey>. The simplest form rewrites every document of a repository in place, upgrading each to the active schema (it replaces the document, and a replace serializes with the active schema):

new DocumentMigration<Cat, string>(catsRepository)

Other constructors give you more control:

Constructor	Does
(repository)	Replace every document in place, migrating it to the active schema.
(sourceRepository, Func<TModel, Task> processAsync)	Run your own async processor per document.
(sourceRepository, destinationRepository, Func<TModel, object> convert)	Convert and insert each document into another collection.
(sourceRepository, Func<TModel, IRepository?> destinationSelector, convert)	Per-document destination; return null to skip a document.

A migration scans every document of its source repository, unless DocumentsFilter restricts it to the ones to rewrite. The filter matches the documents as stored, since a migration also addresses documents the current maps don't shape anymore:

new DocumentMigration<Cat, string>(Cats)
{
    DocumentsFilter = new BsonDocument(
        ModelMapSchema.DeprecatedIdElementName,
        new BsonDocument("$exists", true))
}

Registering migrations

Override DocumentMigrationList on your context:

public override IEnumerable<DocumentMigration> DocumentMigrationList =>
[
    new DocumentMigration<Cat, string>(Cats)
];

What a migration run does

Running a db-context migration, under exclusive access, performs three steps:

1. delete old indexes,
2. migrate the documents (each registered DocumentMigration),
3. build the new indexes.

A start can add one more kind of document migration to step 2 — the rewrite of the documents left on a deprecated schema, described below — which runs after the registered ones, inside the same operation.

The index steps skip the context's read-only repositories, whose indexes belong to the collection owner. A dry run — described below — skips both index steps entirely and takes no exclusive access.

Two different locks apply while it runs:

• Exclusive access locks every other flow of this context in this process out of its collections. Other db contexts are not affected, and neither is the regular collection traffic of the other application instances.
• The db context lock is claimed with the operation and kept leased for the whole run, so no other instance can start a migration or a seeding of the same context meanwhile. If that lease is lost, the running steps are cancelled and the operation closes failed rather than writing outside its window.

Failing documents

A migration scans its collection raw and deserializes each document apart, so a document that fails — deserialization, your processor, or serialization — doesn't stop the run: it is skipped, keeping its current content on disk, and recorded with its id and error, while every other document migrates. The recorded detail is capped at DocumentMigration.MaxTrackedDocumentErrors (100) entries per migration; the full count is reported beside it.

Ask for the opposite with stopAtFirstError: the documents migration aborts at its first failing document, which is recorded like the others.

var op = await db.TryStartMigrationAsync(stopAtFirstError: true);

Rewriting the documents left on a deprecated schema

A registered DocumentMigration migrates the documents you point it at. The documents whose stored schema id isn't the active one of their concrete type — written by a previous version of your application, and readable through a secondary schema — are migrated only if one of your migrations selects them.

rewriteDeprecatedSchemas asks the operation to rewrite them all, on every writable repository of the context:

var op = await db.TryStartMigrationAsync(rewriteDeprecatedSchemas: true);

Each selected document is deserialized and written back whole with its current active schema. The documents carrying their schema id under the deprecated _m element are selected too, and the rewrite stamps _s at every level of them — a document nests a schema id as deep as its sub-documents and its summaries go, so renaming the root element wouldn't be enough.

The rewrite runs inside the same operation as your registered migrations, after them, so one exclusive access and one pair of index steps cover both, and the dry run, the failing-document handling and the operation log work on it exactly as they do on yours.

The choice is per run and is carried by the operation (DbMigrationOperation.IsStopAtFirstErrorEnabled); it applies to a dry run too. Aborting a documents migration doesn't skip the rest of the operation: the remaining migrations and the index steps run, and the operation closes failed — as it does whenever a document failed, with the complete report on the document migration log (DocumentMigrationLog.Errors and TotErrorDocs), rendered by the Admin dashboard.

Dry run

A migration can run as a dry run: it simulates the document migrations without persisting anything. Use it to test a migration against real data before the real run.

var op = await db.TryStartMigrationAsync(dryRun: true);   // null when denied, as for a real start

A dry run:

• Persists nothing to the migrated collections. Every write the migration performs — the built-in document replace, or any repository write of a custom processor — executes only its client side: filters and updates render, documents serialize, and the operation returns a simulated acknowledged result without reaching the server. No dependency-update task is enqueued by the simulated writes.
• Handles failing documents like a real migration, running the same scan: a failing document is skipped and recorded, or aborts the migration when the run asks to stop at the first error.
• Skips both index steps: index management has no simulation. For the same reason, index management or an aggregate-to-collection performed by a custom processor throws, and the failure lands in the error report of the documents that reached it.
• Takes no exclusive access. The context stays fully available while the dry run scans, so it can run on live data.
• Holds the db context lock like a real migration, for the whole scan: while it runs, no instance can start another migration or a seeding of the same context. Give a long dry run a lease duration to match — see below.

The operation and its logs persist like for a real migration, marked as dry run (DbMigrationOperation.IsDryRun). A dry run that finds failing documents closes failed: the real migration would fail on those documents.

Running a migration

• From the Admin dashboard. Start a migration — or a dry run, with or without stopping at the first error — for a context from the dashboard UI.

• Programmatically. Queue one and let the task runner execute it:

  var op = await db.TryStartMigrationAsync();   // null when denied

  A start is denied — returning null — on a read-only context, while an exclusive access is running in this process, or when another owner holds the db context lock: a queued or running migration, a dry run, or a seeding, on this instance or on any other. Inspect progress with IsMigrationRunningAsync(), GetMigrationAsync(id) and GetLastMigrationsAsync(page, take). ExecuteMigrationAsync(opId, taskId?, throwOnErrors?) runs one directly when you already hold exclusive access (a dry run operation needs none): it resumes the lock claim the start made, and closes the operation cancelled without migrating when that claim was released or taken over.

Each start claims the db context lock for a lockLeaseDuration, defaulted to ResourceLock.DefaultLeaseDuration (10 minutes); choosing it is covered on Db context lock.

Each DocumentMigration.MigrateAsync accepts a progress callback (callbackEveryTotDocuments, callbackAsync), a dryRun and a stopAtFirstError flag, an evictEveryTotDocuments interval, and returns a MigrationResult: Succeded, MigratedDocuments (the documents processed without errors), ProcessedDocuments (the whole scan), DocumentErrors and TotDocumentErrors, plus the Exception that aborted the scan.

The scan memory

The documents a scan processes between two evictions run inside one transient models scope: at its end everything their flows loaded or tracked leaves the context — the migrated models and the summaries they referenced — so the scan holds an interval at a time, whatever the size of the collection. What the context held before the scan stays, which is how the operation keeps logging its own progress while the scanned models come and go.

The interval is the MigrationEvictEveryTotDocuments db context option (100 documents by default, see Startup and configuration) and it is independent from MigrationCallbackEveryTotDocuments (500), the progress report interval: one tunes the memory the scan retains, the other how often it writes the operation document. Raising the eviction interval holds more memory and lets the documents of an interval share what they load; lowering it does the opposite. The same bounding is available to your own scans through IDbContext.StartTransientModelsScope() — see Best practices and pitfalls.

The operation log

Migrations are logged as DbMigrationOperation documents in the _db_ops collection and driven by the DbMigrationManager. Any failure — unhandled exceptions included — marks the operation failed and logs at error level; it's never left on running, which would misreport a migration in progress. An exception is thrown only when the caller asked for errors (throwOnErrors: true).

An operation keeps one document migration log per collection: the periodic progress replaces it, carrying the running count of migrated documents, and the ended log replaces it in turn with the outcome and the failing documents. The index steps add two entries per repository each. So the operation document stays bounded whatever the size of the collections it migrates, and its final status write always fits the document size limit MongoDB imposes.

DbMigrationOperation.CurrentStatus carries one of five values:

Status	Means
New	Created by the start, waiting for the task runner to execute it.
Running	Executing.
Completed	Every step succeeded. CompletedDateTime is set only here.
Failed	A step failed — a failing document included — or the operation was still running when a later start closed it as orphaned by a dead instance.
Cancelled	Never executed: its task couldn't resume the db context lock claim of the start — released, or taken over by another owner — or it was still new when a later start closed it as orphaned.

---

Next: Versioned schemas for lazy, on-read evolution, Database seeding for one-time initialization, or Admin dashboard to run migrations from a UI.