References and denormalization

This is Scrinium's flagship feature. You store a related entity as a compact summary embedded in the referencing document (so you read the aggregate in one query, no joins), lazy-load any member you didn't denormalize, and Scrinium keeps every denormalized copy in sync automatically when the origin changes.

Reference, don't embed

An entity is stored once, in its own collection. Everywhere else you reference it: the reference stores the entity's id plus whichever members you chose to denormalize (the summary), inline in the owner document. On read you get a summary instance; touching a member you didn't denormalize lazy-loads the full document from the entity's origin repository.

Entity models must be referenced, not embedded: the maps freeze detects every member serializing an entity model through its class map (the full-document serialization — collections and reference configurations included) and fails the engine build with a detailed ScriniumEmbeddedEntityModelException naming each violating member. A custom serializer — set on the member or mapped for the type — opts out, for value-object-like models. (Value objects, by contrast, are always embedded — see Domain models.)

Defining a reference serializer

A reference is a member serialized by a ReferenceSerializer<TModelBase, TKey>. Its configuration is a model map in its own schema-id space that maps only the summary members — and, because that space doesn't infer base types, one map for every type in the referenced hierarchy, up to the last base before object. The common case — one repository per referenced type — needs no source declaration:

using Etherna.MongoDB.Bson;
using Etherna.MongoDB.Bson.Serialization.Serializers;
using Etherna.Scrinium.Core;
using Etherna.Scrinium.Core.Serialization.Serializers;

// Summary of a User: its id plus one denormalized field.
// One AddModelMap per level of User's hierarchy — ModelBase, EntityModelBase<string>, User.
public static ReferenceSerializer<User, string> UserReference(IDbContextEngine engine) =>
    new(engine, config =>
    {
        config.AddModelMap<ModelBase>("1a2b…modelbase", map => { });         // no summary members at this level
        config.AddModelMap<EntityModelBase<string>>("2b3c…entitybase", map =>
        {
            map.MapIdMember(m => m.Id);                                      // required — every summary must carry the id
            map.IdMemberMap.SetSerializer(new StringSerializer(BsonType.ObjectId));  // the same id serializer the origin uses
        });
        config.AddModelMap<User>("3f2c…user-summary", map =>
        {
            map.MapMember(u => u.DisplayName);                               // denormalized into every reference
        });
    });

Every summary must carry the id. The id is the reference's reason to exist — the identity-map key, and the handle each non-denormalized member lazy-loads from; a summary that deserializes without one is dropped, and the reference resolves to null. Map it explicitly, on your entity base: a level you leave out gets an empty default map that drops whatever it declares — the id included. The reference config is a separate schema space, so re-declare the id's serializer here to match the origin; it isn't inherited from the model map. For a polymorphic reference, add a map for each concrete subtype too.

Apply it to a referencing member in the owner's map with SetSerializer:

dbContextEngine.MapRegistry.AddModelMap<Order>("…order", map =>
{
    map.AutoMap();
    map.GetMemberMap(o => o.Owner).SetSerializer(UserReference(dbContextEngine));
});

For a collection of references, wrap the reference serializer in an EnumerableSerializer — for example a User summary applied both to an Order.Owner and, as a list, to a Team.Members.

Source repositories

The source repository is the origin a reference lazy-loads from (and the key for the identity map). Scrinium resolves it for you when it's unambiguous, and lets you declare it when it isn't:

• Implicit (default). With no declaration, it resolves at engine build by walking from the referenced type up its base chain to the nearest level declaring repository properties for the type. Several repositories over the same model type at that level — or only key-incompatible ones — fail fast at startup with ScriniumAmbiguousRepositoryException; repositories at different hierarchy levels don't conflict, the nearest wins.

• Declared. When there are multiple candidates, name the source with the typed factory; the typed selector parameter fixes the generic arguments — no explicit type arguments needed — and compatibility is checked at compile time:

  ReferenceSerializer.Create(
      dbContextEngine,
      // config also declares User's base hierarchy — elided here (see "Defining a reference serializer")
      config => config.AddModelMap<User>("…user-summary", map => { map.MapIdMember(u => u.Id); }),
      sourceRepository: (IMyDbContext dbContext) => dbContext.Users);

  The untyped constructor parameter (sourceRepository: dbContext => ((IMyDbContext)dbContext).Repo) stays as an escape hatch for the one case invariance can't express — a base-typed repository sourcing derived-typed references.

• Cross-context. To reference an entity owned by another db context, name that context's type in the same typed factory, and declare it a child context on the owner's options with ParentFor<TChild>():

  // in the referencing context's map
  ReferenceSerializer.Create(
      dbContextEngine,
      // config also declares User's base hierarchy — elided here (see "Defining a reference serializer")
      config => config.AddModelMap<User>("…user-summary", map => { map.MapIdMember(u => u.Id); }),
      sourceRepository: (IDirectoryDbContext dbContext) => dbContext.Users);
  
  // in startup, on the referencing context's options
  options.ParentFor<IDirectoryDbContext>();

  The reference then behaves like any other: it binds the child's repository, lazy-loads through it, and auto-creates a null-id model into it. The referenced model belongs to the child context — its identity map, change tracking and saves live there — so one document materializes one instance across parent and child, and its changes persist with the child's save, which the parent's SaveChangesAsync() cascades.

  Note. Dependency updates cross the parent–child boundary within one application: a child model changed through any context of the application refreshes the summaries denormalized into the parent's documents. They never cross applications — a process that doesn't host the parent context can't reach its documents — so denormalize members from a child context only when every application writing the child's collections also hosts the referencing context; keep the summary id-only otherwise, unless a stale copy is acceptable.

New referred models auto-create

You can link an entity that was never persisted — its id is still null — and just save the owner: every repository write that serializes references (CreateAsync, SaveChangesAsync, ReplaceAsync) first creates the new referred models into their source repositories, then persists the owner with complete references.

var owner = new User("Ada");            // never persisted: Id is null
var order = new Order(owner);
await db.Orders.CreateAsync(order);     // creates the User first, then the Order

// owner.Id is now assigned, and the User is tracked like an explicitly created model

• Ids are assigned upfront, through the id generator of the mapped id member, before any insert: references between new models serialize complete in any creation order — mutual references included (a new Order holding a new User that references the Order back works).
• Discovery is transitive: a new model referred by another new model is created too.
• Each auto-created model joins the unit of work like an explicit CreateAsync: later mutations are diffed and saved, and the created instance is the instance of its document on the identity map, kept as the reference value by the save refresh. Inside SaveChangesAsync the creations into this context's repositories enlist in the implicit save transaction; a creation into a child context's source repository runs on the child's connection, outside it.
• A model that already has an id is never re-created: auto-creation triggers on the null id only.

Warning. A reference to a model without id fails the write with a detailed InvalidOperationException when it can't auto-create — when its id member configures no id generator. The same guard covers every write that serializes a null-id reference without auto-creation — the upsert helpers and raw collection writes — where the driver surfaces it wrapped in a BsonSerializationException naming the member: a persisted id-less reference would deserialize to null, silently losing the link. Id-less references already persisted keep reading as null.

Summary members and lazy loading

Denormalize the members you read together with the owner; leave the rest to lazy-load. On the identity map:

• a full load upgrades an already-loaded summary in place;
• a freshly deserialized summary merges its denormalized members into an instance already loaded as a summary — so you accumulate denormalized data rather than lose it.

Which members count as loaded is decided by the reference document itself: an element carried by the document marks its member loaded, mapped through the reference schema that deserialized it — regardless of the member's setter accessibility (private setters included). A member the document doesn't carry — even one assigned by a serializer default value during deserialization — is not loaded: its first read lazy-loads the whole document from the origin repository. (A reference read through a custom fallback serializer has no schema mapping elements to members: there the observed sets decide.)

When you only need reference ids, read under the reference serializer modifier (EnableReferenceSerializerModifier(readOnlyId: true)) — see Querying.

Explicit preloading

An implicit lazy load is synchronous over the database call. When you know which members you are about to read, preload them asynchronously with IDbContext.LoadValuesAsync — per instance, or batched over a collection, grouped per source repository:

// one summary
await dbContext.LoadValuesAsync(team.Owner, u => u.Email);

// a whole collection of summaries: the missing documents load with $in queries
await dbContext.LoadValuesAsync(team.Members, u => u.Email, u => u.CreationDateTime);

var email = team.Members.First().Email;   // no lazy load: already full

The members are a precondition, not a projection: a summary already carrying all of them is untouched, any other loads its whole document, merged in place through the identity map. A summary the identity map doesn't hold — read under the no-cache modifier, or evicted by a transient models scope — upgrades from the loaded instance of its document, and invalidates like a held one when the document carries another type. Each repository loads its missing documents with one $in query per chunk of a thousand ids, so a batch never grows a single command nor materializes its whole result at once — a preload of ten thousand summaries still costs ten round trips and ten thousand materialized documents, so page what you preload, and keep a reference array from growing unbounded in a document. IsMemberLoaded(model, m => m.Member) inspects the current state; the id member is definitionally always loaded. Reading the ExtraElements bag of a summary never lazy loads either: the bag lives only inside a load (see Domain models), so a summary has nothing to load into it. How the context reacts to the implicit loads that still happen is configured by the ImplicitLazyLoad option: warn once per member per scope (the default), stay silent, or deny them throwing ScriniumLazyLoadingException. The modes govern the loads a read triggers. Writing a reference serializes every member its schema declares, so a summary missing one of them loads its origin document to complete the document being written: that load always runs, whatever the mode, and reports with a warning of its own.

Missing origin documents

A load has nothing to read when the referred document doesn't exist on the origin collection anymore — deleted, or never created by a flow that persisted the referencing document alone. That is an inconsistency of the database, and each reference declares how its summaries react to it on a read, with its MissingOriginDocument mode (a ReactionMode, from Etherna.Scrinium.Core.Options):

Mode	Reaction
Warn	Log a warning once per model type and source repository, per scope, and give up the summary state. The default.
Silent	Give up the summary state, reporting nothing.
Throw	Deny the load, throwing ScriniumMissingOriginDocumentException with the model type, its id and its source repository.

The default warns instead of throwing because the state is not always an inconsistency: a model deleted through its repository legitimately leaves its references dangling until the background propagation removes them — a read inside that window must not fail.

Tolerating it (Warn and Silent), the model stops being a summary and keeps the members it never loaded at their default values — reading them returns null or 0, indistinguishable from persisted data, and different from the null reference the delete propagation leaves once it completes. Denying it keeps the model a summary, so it still requires its origin document and a later read attempts the load again.

Declare it on the reference configuration, beside its model maps:

config.MissingOriginDocument = ReactionMode.Throw;

Declare Throw on the references your code takes decisions on: a business flow reading default values in place of real data is worse than an exception. The trade is availability inside the deletion window — a Throw reference can fail a read that a Warn one would have degraded.

Both read paths react the same way: the implicit lazy load of a member, and LoadValuesAsync, which reports every summary whose document the preload didn't find — where the inconsistency is, instead of at the next member read. A denormalized member is unaffected: it reads from the summary, and no load runs.

Warning — a save of the referencing model hits this too, and it never degrades. Writing it serializes its reference members, and a reference schema denormalizing a member the summary doesn't carry reads that member, which loads. A dangling reference therefore fails that save, whatever mode the reference declares: the write can't complete the summary it has to persist, and writing the not loaded members at their default values would store them as real data. Repair what dangles before rewriting the documents carrying it — every repository finds and removes such references (Repositories).

Two references to the same model type can declare different modes — one critical, one pointing at documents allowed to disappear. Within a scope one document materializes one instance, whatever the references reaching it, so that instance keeps the strictest mode of the ones that reached it: an opt-out on one reference can't silence another that expects the document to be there.

Finding the dangling references of a whole collection — and removing them — doesn't load anything: every repository scans its reference paths server side and verifies the referenced ids against their origin collections, and the admin dashboard exposes the same scan and removal per collection. See Repositories.

Automatic dependency updates

When you change a member that other documents denormalized, SaveChangesAsync enqueues a background UpdateDocDependenciesTask that propagates the new summary to every document of the application referencing it. This runs off the request path via the task runner and converges even after a transaction abort (it's not part of the save transaction); see Change tracking and saving for what the save enqueues.

The summaries refresh server side, in bulk: one UpdateMany per hosting repository and reference member path rewrites the summary sub-document of every matching document with the active reference schema — so summaries persisted with older reference schemas migrate at their first refresh — and no document content is read back. Edge behaviors to know:

• a referenced model deleted while its update was pending skips the task with a warning; the references then follow the origin delete policy of their reference — removed by default, by the delete propagation the domain delete enqueued;
• a task executed while a flow holds exclusive access (e.g. a running migration) fails and is retried by the task runner, converging on the post-migration state — background propagation never interleaves with a migration;
• summaries hosted behind an unknown document key — a dictionary serialized in document representation writes its keys as element names — sit on a path the update filter can't address (querying unknown document keys is unsupported server side), so the task skips the path and those summaries go stale when their origin changes. The engine build reports every such path, with the reaction declared by the NotPropagatedReferences option: a warning per element path (the default), silent tolerance, or a ScriniumNotPropagatedReferenceException denying the build. Dictionaries in the array of documents or array of arrays representation keep addressable paths, and their summaries refresh like any other collection member, only on the entries of the changed model;
• read-only repositories are skipped: their documents belong to another application, and stay untouched.

Deleting a referenced model

A model deleted through its repository (DeleteAsync) propagates in background to the documents referencing it, applying the origin delete policy each reference declares — the counterpart, at the source, of the missing origin document reaction above. Declare it on the reference configuration (OriginDeleteMode lives in Etherna.Scrinium.Core.Options):

config.OriginDelete = OriginDeleteMode.DeleteReferencingDocument;

Mode	Reaction to the origin delete
RemoveReference	Remove the reference from the referencing documents: a reference hosted as an array item is pulled out of its array, any other one is set to null. The default: a domain delete never leaves its own references dangling.
DeleteReferencingDocument	Delete the referencing documents, with a domain delete that propagates their own reference policies in turn: cascades chain across models, and stop on the documents already deleted — mutual reference cycles included.
KeepReference	Keep the reference dangling: its summary reads react per the MissingOriginDocument mode. The explicit opt-out, for summaries that must survive their origin.

The propagation runs off the request path via the task runner (DeleteDocDependenciesTask), like the dependency updates: removals write server side, in bulk — one update per hosting repository and reference member path, matching the deleted id, so a reference concurrently rewritten to another document stays untouched — while cascades load the referencing models batch by batch and delete them through their repository. Until the task runs, the references dangle: reads inside that window react per the MissingOriginDocument mode, whose Warn default exists exactly for it, while a write that has to complete one of those summaries fails.

What the propagation doesn't cover:

• raw bulk deletes (DeleteManyAsync) and deletes performed by other applications: no domain delete, no propagation — the missing origin references scan (see Repositories and the admin dashboard) finds and removes what they leave behind;
• referencing documents of other applications: the propagation reaches the contexts of the application performing the delete — the deleting context and the parent contexts declaring it a child, like the dependency updates — never the documents of an application that doesn't host them: those references behave as KeepReference whatever they declare;
• reference paths the propagation can't address with a filter — an unknown document key (a dictionary in document representation), or a fixed array position in the path — reported by the same scan as unverifiable;
• read-only repositories: their documents belong to another application, and stay untouched.

Referenced model type changes

A document can change its concrete type over time, keeping its id — say a Web2Account evolving into a Web3Account of the same hierarchy, replaced through its repository:

var account = await dbContext.Accounts.FindOneAsync(accountId);              // Web2Account
await dbContext.Accounts.ReplaceAsync(new Web3Account(account, etherAddress));

Two mechanisms keep references coherent:

• Persisted summaries follow the change. The replace enqueues the dependency update, which rewrites every summary with the reference schema of the new type, discriminator included, so following reads deserialize the reference directly as Web3Account. The reference serializer must map every concrete type of the hierarchy it can host.
• Loaded instances can't upgrade, so they invalidate. The runtime type of an instance can't change: when a full load (implicit lazy load, explicit preload, or a repository read) finds the document with another type of its hierarchy, the fresh instance becomes the loaded one for the scope — returned by that and every next load — and the outdated instance is invalidated: any interaction with it throws ScriniumOutdatedModelTypeException, naming both types. Its id stays readable, and IDbContext.IsOutdatedModel(model) inspects the state:

var author = message.Author;                    // summary, loaded as Web2Account
try
{
    var username = author.Username;             // full load finds a Web3Account document
}
catch (ScriniumOutdatedModelTypeException)
{
    var current = await dbContext.Accounts.FindOneAsync(author.Id);   // fresh Web3Account
}

Saving unrelated changes on a document that still references an outdated instance keeps working: the invalidation guards application interactions, not the library's own serialization.

Unknown schema ids

A reference document whose schema id isn't recognized (or is missing) deserializes by reading only its id; every other member lazy-loads from the origin. You can override this per reference with a fallback schema or serializer on the reference's own model map. Reference maps version like root ones — secondary and fallback schemas included — but accept no fixDeserializedModelFunc: repair logic belongs to the origin document's root schemas, which every lazy load runs through.

Seeing the summaries

The admin dashboard's document structures render each collection with the summary elements marked and expanded: the map of what each reference costs in stored data, and of what an update of a referenced document rewrites.

Common shapes

• Multiple summary tiers — define more than one reference serializer for the same type (a light preview vs a richer summary) and pick per member.
• Id-only summary — an empty summary body persists just _id plus the _s schema id element: a pure reference with everything lazy-loaded.
• Denormalize a hot field — copy a frequently-read value (say a role's normalized name) into the summary so a common lookup never touches the origin collection.

---

Next: Custom serializers for EnumerableSerializer and friends, Background tasks for the dependency-update task, or Migrations to rewrite documents.