Content Loading and Hot Reload

Content Loading and Hot Reload

Audience: Engine Developer Status: ✅ Ready

Content is loaded from Postgres definition rows into in-memory registries at boot, and can be hot-reloaded into a running fleet without a restart: edit an area file, reload, and the change validates, mints a monotonic version, and propagates to every shard — swapping the affected prototypes in place while live players keep playing. This page covers the runtime loading and reload machinery; for the pack format and def-table shapes see Pack Authoring and Pack Entity Reference, and for the operational side see Content-Pack Operations.

Related: Persistence & Durability, Orchestration & Directors, Distributed Systems Model.

Boot load

content.Load reads the enabled packs (from a Postgres Source in production, or an embedded YAML source in tests) and merges them into a LoadedContent — the zones plus ~25 pack-global definition kinds. Each kind accumulates across packs last-write-wins by ref; zones dedup whole-zone last-write-wins by ref. The bare-engine invariant holds: a nil source or an empty enabled list yields empty content, and the engine boots with nothing baked in.

Boot runs several content-lints (non-fatal warnings — reserved core: namespace, ref charset, ref length, trust ladder, channel access), then the world registers the DTOs into per-shard registries. The per-shard prototype cache (protoCache) is mutated only by an atomic whole-table swap (Entity Component Model).

The hot-reload flow

sequenceDiagram
    participant U as Builder (reload)
    participant S0 as triggering shard
    participant PG as Postgres
    participant BUS as content bus (NATS)
    participant SN as every shard
    U->>S0: reload [pack] (rank + builder gated)
    S0->>PG: re-read pack(s)
    S0->>S0: validate (blocks fleet-wide on any problem)
    S0->>PG: BumpContentVersion, monotonic version
    S0->>BUS: PublishPack (per-ref, then KindZone, then VersionComplete)
    BUS->>SN: ordered delivery
    SN->>PG: LoadDefinition(kind, ref), single-ref re-read
    SN->>SN: atomic cache swap, version-guarded reconcile, Lua reload

Loading

Validate before broadcast

validatePacks returns human-readable problems, and a non-empty result blocks the publish fleet-wide — because shared-source convergence means gating at the triggering shard gates the whole fleet. Validation is pure over parsed DTOs and never touches live registries.

Builder-controlled values in those problem strings are length-capped. The DTO parse/validation layer once echoed field values verbatim into build- and reload-time log lines and into the reject/advisory strings — so a malformed 200 KB dice or formula field, or a reference-valued exit/reset target, produced a ~200 KB log line: the same log-poisoning / disk-fill class the builder-Lua log bound closes, arriving through the content-load channel instead (and it matters more now that logs ship into Loki). Every builder value is now clamped to ~1 KB at the source — in the dice/formula parsers and the per-attribute slog attrs — and, crucially, at the two leaf aggregators (capProblems) that every reject log, returned outcome, boot report, and snapshot-gate finding funnels through, so the bound holds regardless of which sub-check produced the value or which consumer reads it. The cap is the shared internal/logcap leaf, byte-identical to the Lua one. (The #454 log-key guard does not catch these — the keys here are area/event/op/value, not its sensitive list — so a length-cap-at-source is the right invariant, not a key-name check.) It is deliberately stricter than boot: boot is fail-safe and degrades, but the reload gate hard-rejects content boot would tolerate (dangling exits, ref collisions, unsafe channel refs, reserved-namespace/charset/trust-ladder violations). reload --check validates and publishes nothing.

A scoped reload validates against the whole merged graph. A reload <onepack> once checked only the in-scope packs, which hid three defect classes: cross-pack attribute cycles, cross-zone dangling exits, and cross-zone reset prototypes. Validation now resolves every ref against the entire enabled content graph, but rejects only a finding whose rooted unit's last-writer pack falls inside the reloaded scope — provenance-by-last-writer (whole-zone last-write-wins, matching the loader's own merge). That buys two invariants at once:

• an unrelated, already-broken pack can never block an otherwise-valid reload — the migration trap you would otherwise walk into the first time someone reloads a healthy pack while a stale one sits in the enabled set; and
• a defect the reloaded pack genuinely contributes to the merged graph is caught, even when the other end of the edge lives in another pack or zone.

Attribute cycles are reported as structured node lists, so a cycle is attributed to the packs that participate in it rather than to whichever pack happened to trigger the reload.

Removing content another pack depends on gets an advisory, not a block. The merged-graph check above catches a reloaded pack's own dangling references, but not the inverse: reload A deletes a room or prototype that a not-reloaded pack B still points at (B's exit into A now dead-ends). Provenance-by-last-writer correctly doesn't hard-block A for B's dependency — the broken edge is rooted in B's out-of-scope zone — so without help the operator is blindsided. So the reload readout (and a WARN log, for both reload and reload --check) appends a non-blocking heads-up: it scans the re-read graph's out-of-scope zones for exits, start rooms, and reset prototypes whose target is absent from the new graph but was live before. The "before" snapshot is the shard's live proto cache read at validate time — sound because the actual removal only lands later, when the publish drives the zone-shape reconcile. Precision needs that pre-vs-post diff, or it would warn on danglers that already existed in B. The wording says B "still references / leads nowhere" rather than promising the live world will dead-end, because the depender side is read from source (matching the validator's basis) and can diverge if B was edited without a reload.

The applier (shard side)

Each shard subscribes to the content bus. Per invalidation it re-reads just the named (kind, ref) and swaps the one prototype into the shared cache — the bus delivers serially, so the applier is the sole runtime writer of the cache. On an infra error it keeps the last-known prototype (never empties a ref on a transient PG blip); on !Found it removes the ref; otherwise it builds and atomically swaps.

The subscriber's pack filter fails closed. It once accepted an invalidation naming no pack (inv.Pack != "" && !r.packs[inv.Pack]), which on an unsigned, single-subject bus with no publisher identity was a knowledge-free fleet-wide primitive — and a destructive one, because the re-read matches on (kind, ref, pack), so an empty pack never resolves a row, returns Found:false, and Found:false is the deletion path. One small message evicted a prototype from every shard's cache (no further spawns; live scripted instances go scriptless) or dropped a channel from every registry. The decision is now a single accepts predicate — split out precisely because the re-widening it guards against is one line — checking three things:

• The kind must be in the closed wire vocabulary (content.AllKinds / KnownKind). An unrecognised kind reached LoadDefinition, whose store dispatch also returns Found:false for anything it doesn't know — the same eviction primitive through a different door.
• The pack must be non-empty and loaded here. The explicit non-empty check is not redundant with the map lookup: the map is built by ranging the configured enabled-pack list, and a YAML content_packs: [reference, ''] survives config load (the env path drops empties, YAML does not), which would put a "" key in the map and silently restore the whole hole. A security property must not depend on the contents of a config-derived map.
• The version-complete sentinel is exempted structurally — kind + version only, no pack, ref, room set or start room. A sentinel carrying content is not a sentinel.

PublishPack also refuses a pack with no name: with subscribers failing closed, an unnamed pack would publish a whole pack's worth of messages that reach nobody — a fleet-wide no-op reported as success.

Scope, stated honestly: this is blast-radius reduction, not authentication. An attacker who names a loaded pack — names that appear in the content repo, the reload readout and the logs — still reaches the zone-shape reconcile, the channel swap and the per-ref re-read. A kind-mismatched invalidation (an item kind naming a room ref) still evicts, and that half is not closable here because it is indistinguishable from a legitimate publish. The sentinel exemption remains forgeable, and the applied-version high-water is monotone with no upper bound, so a forged max-version sentinel still wedges reconcile-on-join's catch-up gate. The deployment-side fix now exists — NATS enforces a per-identity subject-authorization matrix, so only world/director/seed credentials can publish content.invalidate at all and the gate is denied publish outright; a forgery now requires already holding one of those trusted identities. Per-message signing, to distinguish a legitimate publish from a forged one within a trusted identity, remains future work.

A channel reload swaps the channel registry and republishes every live player's comms hear-set. That republish is ref-independent — it recomputes each player's full hear-set regardless of which channel changed — so a burst of channel edits is coalesced per zone: an atomic flag arms one pass, and further channel invalidations arriving while it's queued (or retrying) skip re-posting rather than each triggering an O(players) recompute. A 20-channel edit on a 5k-player shard therefore does one republish per zone, not twenty. The flag is disarmed before the recompute (an edit landing mid-pass re-arms and gets a fresh pass), because disarming after could swallow an edit whose registry swap landed between the read and the disarm — a stale, too-permissive hear-set. Correctness rests on the channel registry being a sequentially-consistent atomic swap. Per-zone Lua reload runs on each hosted zone goroutine: it bumps a chunk generation (old-gen mud.after timers drop), invalidates the shared chunk cache for that ref (colon-segment-bounded, so reloading orc never drops sorcerer), re-registers live scripted instances from the new source preserving self.state, and resets the circuit breaker. Because re-registration re-runs the whole registration body against that preserved state, a top-level state.x = ... seed re-executes on every reload — author it idempotently (state.x = state.x or <default>) so it doesn't clobber live data. See Pack Lua Scripting → self.state.

Zone-shape reconcile + the version guard

Adding/removing rooms is a separate path, reconcileZone, single-writer on the zone goroutine. It:

• refuses an empty desired room set (won't mass-teardown a live zone from a malformed payload);
• drops any reconcile whose version is ≤ z.lastReconciledPackVer — last-writer-wins by version, not arrival, so a racing reload's stale snapshot can't reorder ahead of a newer one;
• applies start_room first (it refuses to tear down the live start room), then adds/updates every desired room, then removes stragglers.

A dropped reconcile (worse than a dropped Lua reload — a lost remove leaves an interactive ghost room) is handed to a bounded-retry loop that re-posts the same immutable message with its original version; the version guard makes that safe. This reconcile retry and the comms hear-set republish retry run on separate goroutine budgets (each capped, sized at parity), not one shared pool. They guard different failure classes — a lost reconcile leaves a ghost room, a lost republish leaves a stale, too-permissive hear-set (security-relevant) — so a storm of dropped reconciles must not be able to starve the republish retries, nor vice-versa; sizing the security-relevant class smaller would make it the first truncated on a many-zone shard, exactly backwards. The decoupling is at the goroutine-pool layer only: both still post into the same per-zone inbox, so a single saturated zone can still fail either.

Zones built after boot use live content

The shard's content snapshot was once written at construction and never again — so every zone built after boot (a HostZone following a rebalance, and every instance mint) was assembled from whatever content the process started with, while its prototypes came from the live reloaded cache. Two halves of one zone from two different versions. A long-lived process drifted further with every mint: a template a reload had deleted still minted, and an instanceable opt-in a builder withdrew never took effect on a running shard.

The snapshot is now an atomic.Pointer refreshed by the hot reloader. markContentStale coalesces the signals (the version-complete sentinel, the zone-shape fork, and reconcile-on-join) and single-flights a debounced, jittered re-read through the same content.LoadWithCore call boot uses. HostZone and MintInstance each take one snapshot and thread it through both validate and build, so the two can no longer straddle a swap. The refresh is version-gated (a replayed or forged invalidation re-reads nothing) and keeps the previous snapshot on a read failure or a zero-real-zone read.

Letting the snapshot move exposed two states that were unreachable while it was frozen:

• A runtime build can be incomplete, so buildZone now reports that and the runtime callers refuse rather than publish. Publishing a roomless zone is not a cosmetic failure: a roomless instance disconnects its entrant after transferOut has already released them, and a roomless HostZone adopts, arms lease renewal, and drops every player a drain hands it — while the directory cheerfully reports a healthy claimed zone.
• Region membership was resolved twice per build (once by seedZone, once by registerZone) with a full build in between, so a swap landing in that gap left a zone holding one region's state while subscribed to another's. It is resolved once and cached.

The refresh is validated before it publishes

The refresh originally published with nothing checking it, while the staff reload command's identical publish was gated by validatePacks. Rows are written by seed/import before any reload runs — so content an operator watched a reload reject ("nothing propagated") went live anyway at the next unrelated invalidation: another builder's reload, a bus reconnect, a version-complete sentinel. "Nothing propagated" had stopped meaning "nothing applied." Gating the refresh makes rejected mean rejected on both publish paths. content.Load is split into LoadPacks + LintPacks + Merge so the refresh reads once and validates the exact slice it merges; reading twice would reintroduce the TOCTOU the gate exists to close.

The refresh gate is deliberately narrower than reload's, by a derived rule: a check is rejectable here iff a finding in it can make the published snapshot unsafe to build a zone from. That follows from the snapshot's three readers (HostZone, MintInstance, regionForZone), which consume it purely as a zone graph. Attribute, channel and trust-ladder findings are therefore demoted to warnings — a refresh cannot deploy any of them by any path (globals are defined at boot only; channels hot-swap through their own route) — so rejecting on them would prevent nothing while handing any single-pack writer the power to freeze every pack's zone graph fleet-wide. Detection is not reduced: demoted findings are still computed and logged, and boot still runs the full validatePacks at Error. validatePacks itself is untouched, so reload keeps full strictness — a reload is a human deliberately propagating an edit and can afford to be refused; the refresh is an automatic reaction to somebody else's event and cannot.

It freezes the whole snapshot rather than dropping the offending pack. Dropping silently promotes the definitions that pack was overriding — precisely the ones validatePacks skipped as inert — publishing content the gate never saw. The stronger objection is that the snapshot is a zone lookup, so dropping a pack removes its existing populated zones and turns a nil lookup into errNoZoneContent for zones with players in them. Freezing is strictly better; per-zone pinning is the principled end state and needs structured zone attribution on findings.

Two details keep a rejection from becoming its own failure mode:

• A rejection must not advance the snapshot's content version, or the shard would treat refused rows as published and stop retrying, so an operator's fix would never land. But that leaves the version gate permanently open, restoring the read-amplification lever it exists to deny — so a rejected version is bounded to one re-read per minute, and an in-place row fix still converges.
• The rejection log memoizes on the problem set, not the version. A raw row edit does not bump content_version, so a version-keyed memo would report the first breakage at Error and downgrade every later, different breakage to Debug — usable to hide a real problem behind a benign one.

Boot reports the same findings at Error but deliberately does not refuse. It has no previous snapshot to fall back on, so refusing would turn a content defect into an outage. That asymmetry is what makes the runtime freeze survivable — and it is the alertable signal that a fleet is now split by uptime, since a rebooted shard will accept content a long-running one froze.

Residual, stated plainly: validatePacks is a health gate, not a provenance one. Staged content that is merely valid — an instanceable: true opt-in, a room-set edit — is still deployed fleet-wide by a reload of a completely different pack, with no reload of the owning pack and an audit record naming the wrong packs. Blocking that needs a per-pack deployed-version notion the schema does not have (content_version is a single global singleton). What exists today makes it legible instead: the refresh is the only place holding both the previous and the incoming content, so it is the only place that can say whose rows moved, and it logs exactly that. The fingerprint it compares includes instanceable deliberately — a staged opt-in changes no room and no exit, so a room-set-only comparison would call the snapshot unchanged and leave the deploy silent, and instanceable is the very flag such a cross-pack deploy turns on (it is the control bounding the instance faucet).

The version authority

The version stamped on invalidations is the fence the reconcile guard compares. It is minted from the single Postgres content_version singleton — not the wall clock. mintReloadVersion calls BumpContentVersion (INSERT ... ON CONFLICT DO UPDATE SET version = version + 1 RETURNING version), monotonic fleet-wide with no clock skew. If the bump errors, the reload fails rather than stamping a wall-clock version that could sit above the un-bumped counter and poison a later reload. Only a source with no PG authority (embedded/mem, in dev/tests) takes a wall-clock fallback floored at pgVersion + 1.

Seam warning: a production source must be a concrete *store.Pool (or a forwarding wrapper), or it silently takes the wall-clock fallback and reintroduces clock-skew risk with a green test suite. The contentVersioner interface is kept structural so the world package needs no store import (store imports world; the reverse would cycle).

Pull, reconcile-on-join, and rolling reboots

A director-coordinated pull <version> (see Content-Pack Operations) mints its version inside ImportVersion under a FOR UPDATE lock — GREATEST(version+1, now_nanos) — with the whole prune→import→bump→registry section atomic, and idempotent by content SHA (a leader-failover redelivery doesn't inflate the version). A trailing content-less KindVersionComplete sentinel is emitted last; a subscriber advances its applied-version cursor only on that sentinel, so a partially-delivered pull never falsely believes it's caught up.

Reconcile-on-join: core NATS buffers nothing while a shard is disconnected, so a pull broadcast during a bus gap is missed (the rows are current — Postgres stays the serving source — but the in-memory prototypes are stale). On reconnect, a shard behind the current version re-reads its enabled packs from Postgres and applies them locally (no fleet re-broadcast), single-flighted, never refusing logins.

What needs a rolling reboot (by design):

Live hot-swappable	Rolling reboot required
rooms / items / mobs (per-ref swap + zone reconcile)	shared defs — attributes, abilities, and the other pack globals do not hot-swap
channels (registry swap + comms republish)	retiring a whole zone
scripted-prototype Lua (self-heals)	stripping a live-hosted pack (refused by the prune guard — drain or roll first)

When a reload (or reload --check) touches content that defines shared defs — attributes, resources, damage types, affects, abilities, combat profiles, progression tracks, bundles, rarity tiers, affixes, loot tables, recipes, wear slots, trust tiers, custom commands, display templates, ruleset formulas, or the pvp policy (every pack global registered by defineGlobals except channels) — the readout appends a reminder — e.g. "Note: this content also defines shared defs (abilities, pvp policy). These are NOT hot-applied — if you changed one, a rolling reboot of the world shards is required for it to take effect." — so a live-edited pvp policy, formula, custom command, or display template is never silently left at its boot value. Rooms/items/mobs/channels in the same reload still hot-swap live.

On a fresh/bare boot (Postgres unreachable) a shard boots the embedded core pack alone — a bootstrap start room so a builder can connect — rather than an empty, login-rejecting world.