Instanced Zones

Instanced Zones

Audience: Engine Developer Status: ✅ Ready

A zone can be copied at runtime into a private, shard-local instance — the classic dungeon-or-party copy. An instance is a full live zone actor running a template's content, minted on demand, isolated from the shared world, and reaped when it empties. It is content opt-in: a zone is not instanceable unless it says so.

Related: Zone Runtime & Actor Model (the actor an instance is), Cross-Shard Handoff (why an instance is never a handoff destination), Persistence & Durability (the anchor), Distributed Systems Model. The audience-shaped views: Building Instanced Zones (authoring) and Running at Scale (operating).

Identity: template vs id

Every zone carries both:

Field	Answers	For a normal zone	For an instance
Zone.template	whose content is this?	its own id	the template it was minted from
Zone.id	which live zone actor is this?	its own id	<template>#<128-bit random>

An instance's rooms keep the template's authored refs. That is what lets every copy share the immutable per-shard protoCache — no content is duplicated in memory — but it has one sharp consequence: a raw parseRef(ref).zone != z.id comparison reads every exit inside an instance as leaving the zone.

So all locality decisions route through a single predicate, ownsZoneRef (and localRoom = parseRef + ownsZoneRef + the room map, in the one order that is correct for an instance). Both live in internal/world/identity.go:

func (z *Zone) ownsZoneRef(zoneID string) bool {
	return zoneID == "" || zoneID == z.id || zoneID == z.template
}

The routing-vs-isolation rule

ownsZoneRef answers the routing question — does this ref stay inside me — and it is the only question that may widen to the template. It is not the isolation question. Anything asking may this actor reach that must stay strict on z.id, or a script in crypt#7 eventually resolves a handle into crypt#9. Two different questions with the same shape; do not collapse them into one helper.

The rule for telling them apart: the room map is the isolation boundary, and the zone-segment test is only a pre-filter in front of it. z.rooms is per-*Zone — two instances of one template hold different maps containing different entities, both keyed by the same authored refs — so widening the pre-filter changes only whether we consult our own map, never which map. No *Zone holds a pointer into another's rooms. A caller that reaches an entity some other way, without that lookup standing behind it, is asking the isolation question and must not use this.

Note the predicate only decides the inside-out direction. An exit in town naming crypt:room:entrance still resolves to the template zone, never to an instance — routing a player into one is the separate entry mechanism below.

Six call sites were migrated: move (both branches), flee, h:move, areaTargets, localRoomByRef, and the resyncRoom ADD gate. Only two were on the original list — the other four were found by audit, and each would have been a silent, separate failure inside an instance:

• flee refused in every room of every copy — and since move() already refuses to walk while fighting, a party that wiped would have had no way out at all.
• Every scripted mob immobilized, as a debug-level no-op.
• room_and_adjacent silently degraded to room-only — the same content quietly behaving differently in a copy than in the template.
• room:exits() handing back bare strings instead of room handles.

The worst was not a refusal: without the intra-shard branch migrated, walking north inside an instance resolved to the shared template zone, claimTransferTarget found it, and the player was silently transferred out of the dungeon into the public copy.

A build-failing AST lint (identity_lint_test.go) now fails the build on any comparison of a parseRef zone result against a *Zone's id or template field, outside identity.go. Both fields are in scope: id is wrong inside an instance, and template alone is wrong for a plain zone whose callers may hold its id — ownsZoneRef is the only correct combination of the two.

It resolves the receiver by type, so z.id, c.z.id, and rt.zone.id are all covered while a content DTO's .Ref is not, and it carries an anti-rot guard: if it finds no parseRef zone bindings at all it fails rather than silently passing forever.

Why a lint and not code review: the six pre-existing violations were individually invisible — destZone != z.id reads as obviously correct. And because template == id for every zone that existed before minting landed, no behavioral test could catch a seventh until an instance was live. That is exactly the shape of bug a structural lint exists for. If it fires, call ownsZoneRef/localRoom — do not add your file to the exemption set.

Why the id looks like that

<template>#<128-bit random>:

• # is outside the authored ref charset, so an instance id can never collide with a content ref. A build-failing parity test pins that the charset can never admit #.
• The serial is unguessable, not a counter. A monotonic serial is enumerable, and combined with the loot RNG below that is a farming oracle.

Unleased by construction

An instance takes no directory lease, is never in the placement pool, and is never a cross-shard handoff destination. This is not a simplification — leasing ephemeral refs would be actively harmful: releaseZone deliberately never deletes a zone hash and a lease gen is immortal, so a lease per dungeon run would leak a permanent Redis key each time and reopen the replay window the handoff fence closed.

Every off-box ingress therefore fails closed on an instance-shaped id: Prepare, AdoptZone, and the durable ZoneRef read all reject one.

The one thing an instance does put in the directory

Being unleased has a cost, and it surfaced in the content pipeline. A dungeon template is typically in no shard's zone pool — the raw template is not meant to be walkable — so nothing ever claims it. A template with forty live copies and parties inside them therefore resolved to ErrNotFound in the prune guard's lease lookup and read as not hosted, so a content pull stripped the pack out from under them.

That is worse than the deferred harm the guard's own doc reasons about. Pruning a leased zone's rows does not yank the running zone, because shard memory is authoritative. But instances are minted continuously, and the very next MintInstance after the prune fails validateMintTemplate with "no such zone" — a runtime failure with no operator action in between.

So each shard heartbeats a TTL'd dir:tmplinuse:<template> claim. It is deliberately the cheapest possible analogue of a lease:

• Keyed by template, never by instance id. A template ref is authored content, so the keyspace is bounded by the pack; instance ids are player-driven and unbounded, which is exactly what this design declined to put in Redis in the first place.
• TTL'd, so a crashed shard's claim expires on its own. Nothing has to reap it.
• No generation counter, and therefore none of the handoff-fence exposure that makes zone leases permanent keys. Nothing is signed against it and nothing is fenced on it; it answers exactly one question — is anybody running copies of this right now — and carries no authority beyond it.

Cadence: renewed every 15 s with a 3× TTL (45 s). Three intervals, not one, because a TTL equal to the cadence lapses on any tick that runs slightly late, and a lapsed claim reads as "nobody is using this" — the one answer that lets a pack be stripped from live parties. Three means two consecutive missed renewals before that can happen, while still expiring a genuinely crashed shard's claim inside a minute.

Two details are load-bearing:

• It runs on its own goroutine and ticker, not the reaper's. It first rode the reaper's tick, which made its renewal cadence interval + sweepDuration — and the sweep was serial over UnhostZone, each call waiting up to 10 s. Five wedged instances stretched the gap past the TTL and lapsed every claim on the shard, including healthy templates with parties inside them. A TTL sized against a cadence that a colocated operation can stretch without bound is a margin on paper.
• A mint kicks the publisher immediately rather than waiting for the next tick. The worst case is a template's first live copy: with no prior claim to fall back on, the guard would read "nobody is using this" for a zone a party is standing in. The lifecycle is advertise-on-create → renew-on-tick → expire-on-death, with no cold-start hole. The kick is non-blocking — a mint must never wait on Redis — and a dropped kick costs at most one interval.

Reserved-but-unpublished records count too: a mint in flight is positive evidence somebody is running copies. The asymmetry decides it — over-advertising delays a legitimate prune by at most one TTL, under-advertising strips a pack out from under a live party.

Mint and reap

The async three-hop entry, and the "never mint on a zone goroutine" contract

MintInstance does a full buildZone — every room spawned, every boot reset run, every proto resolved — plus a scopes.seedZone store round trip, synchronously on the caller's goroutine. That is hundreds of milliseconds to seconds. Its doc comment carries the contract in capitals: NEVER CALL THIS ON A ZONE GOROUTINE.

Calling it from the entrance zone's actor would freeze that zone for the whole build: its heartbeat stops, so combat rounds stop landing and affect ticks stop firing; every other occupant's input sits unread in the inbox; and a long enough build fills that inbox and starts blocking the producers — the stream reader goroutines, the saver's ack drainer, a cross-shard Prepare. One player opening a dungeon door would stall a town square.

So entry is asynchronous, in three hops, and the player-visible contract is a brief pause:

sequenceDiagram
    participant EZ as entrance zone
    participant W as shard mint worker
    participant IZ as new instance
    EZ->>EZ: hop 1 — validate (harm gate, account, template), enqueue, return
    EZ->>W: instanceMintReq
    W->>W: hop 2 — MintInstance: validate, buildZone, store I/O
    W-->>EZ: hop 3 — instanceReadyMsg(zone id or err)
    EZ->>EZ: re-validate everything that could have changed
    EZ->>IZ: ordinary transferOut → transferIn

Loading

The request carries the origin zone pointer, not a zone id. The reply must land on the exact actor that validated it and still holds the session; resolving an id later could find a different zone object under the same id (a teardown plus a re-host), which would post an entry authorization into a zone that never granted it. The pointer also makes "the entrance zone was unhosted mid-flight" self-handling — post selects on z.dead.

Hop 3 re-validates everything, because the gap is unbounded and the player is live for all of it. They may have quit, gone link-dead, been reaped, walked to another zone, started a fight, died, or begun a cross-shard handoff; the mint may have failed; the shard may have begun draining; the instance may already have been reaped. Each produces either a clean player-facing line or a silent, bounded, self-cleaning no-op — never a wedge, and never a session in two places.

Entry deliberately goes through transferOut/transferIn (and therefore rehomeSubtree) rather than a raw Move: two copies of a template allocate RIDs in identical order, so the mob in room 3 of copy A and the mob in room 3 of copy B carry the same rid. A handle resolved against the wrong copy would therefore resolve to a real, plausible, wrong entity — deterministically, every run, rather than as a rare nondeterministic collision that announces itself.

Every hop-3 path that decides not to move the player calls abandonInstance first. That is a flag, not a record deletion: the reaper iterates s.instances and resolves each record through s.zones, so a zone with no record is never visited at all — deleting the record would free the cap slot and permanently orphan the zone, its actor goroutine and its Lua VM, on a path an attacker controls. The flag frees the account's slot immediately and lets the reaper skip the mint grace.

Minting from live content

A mint takes one snapshot of the live content and validates and builds against it. The reloader swaps s.content while the shard runs, so re-reading per check would let validate and build disagree: a template could pass instanceable: true against version N and then be built against N+1, where a reload deleted it — booting the instance empty behind a Debug line and handing a player a zone with no rooms. One run is one version, which is also the semantic a builder expects.

Two consequences worth naming, both live behavior rather than theory:

• A template deleted by a reload no longer mints. "No such zone in loaded content" is now reachable at runtime, not just on a typo.
• Flipping instanceable: false takes effect without a reboot.

The build itself is also now refused if incomplete. Validation proves the snapshot declares a start room; it cannot prove the prototype cache has one, because the snapshot and the cache converge on independent paths after a reload. Since entry lands via transferIn's resolveRoom(""), an instance whose start room did not spawn would disconnect the entrant mid-entry — after transferOut had already released them. A refusal is a "the way will not open" line and the player keeps playing.

Reaping

Reaping is quiescence-driven. That makes Zone.quiescent() load-bearing: it counts population, stashed sessions, and in-flight arrivals, so a copy is never torn down under a player still crossing the queue hop into it. The idle counting in the sweep is only a heuristic for choosing candidates; it is UnhostZone's re-check under the routing mutex that is load-bearing, which is why a refused reap is an ordinary outcome (reset the counter, try next tick) rather than an error.

The sweep is concurrent, bounded at 8. It used to be serial, and UnhostZone waits up to unhostActorGrace (10 s) for a zone's actor to return — so one wedged instance delayed every other reap behind it by 10 s, and k wedged instances delayed the tail by 10k, while the ticker coalesced behind the whole thing. Nothing about the reaps is ordered with respect to each other, so the serialization bought nothing and cost the worst case. The bound is a smoothing bound, not a correctness one: each worker takes and releases the routing mutex before it blocks on the actor wait, so 256 workers would mostly queue on the mutex rather than hold it.

Each teardown runs on a detached context (context.WithoutCancel), not the sweep's. Handing it the shard's context meant a shutdown cancelled every in-flight teardown mid-commit — each had already removed the zone and its record and cancelled the actor, then bailed out of the wait. wg.Wait() returned almost instantly and looked graceful while leaving up to 8 zones half torn down.

The sweep still waits for its batch. Fire-and-forget would let successive ticks pile unbounded goroutines onto the same wedged instance. The cost of waiting is worth naming: rec.idle advances once per completed sweep, not per wall-clock tick, and dead instances count against the per-shard cap until swept, so a shard where sweeps run long can refuse live mints.

Caps are charged to the account, with a mint rate limit and a global per-shard cap. All four are operator-tunable (tunables.* / TELOS_*), validated at boot with 0 meaning the compiled default rather than "unlimited". They are not independent knobs: instances_per_shard is bounded by the drain's instance-eject barrier (every instance ejected under one shared 5 s deadline sized against the default — overshoot silently drops occupants to straggler reclaim on every rolling deploy), and the mint burst is validated with its window as a rate. Note also that the per-account cap does not bound mint churn: slot reservation excludes abandoned records from that count, so mint-abandon-mint is bounded per account by the rate limit alone. The operator view — including that every bound is per process — is in Running at Scale and Sysadmin Reference.

Content opt-in: instanceable

A zone is not instanceable by default. validateMintTemplate originally checked only that the zone existed — which meant a player could mint a private copy of any loaded zone, strip its boot resets, walk out carrying them, and repeat: an uncapped item faucet reaching every zone in content, including ones another builder gated behind a locked door or a level check. A copy has no doorman. Content now opts in explicitly with an instanceable flag (zones), which rides the zones body JSONB.

Isolation: what an instance is excluded from

Each of these was a real defect found in design or code review, not a precaution.

• Persistent resets fail closed. LoadObjects is keyed by the authored room ref — identical across copies — and persistentDone is per-zone, so it does not dedup. N instances would each load the same durable rows, giving N lootable copies of a unique object: a hard durable item dupe. The test asserts zero loader calls, not "no dupe observed."
• Both RNG streams are salted per mint. lootRNG draws from z.lua.rng, seeded by FNV-1a over the zone id — so every mint would restart the loot stream at index 0 from an offline-computable seed. Seeding from the template is equally wrong: every copy would then roll identically. (A zone-id-seeded RNG is fine while zones are permanent; it becomes an oracle the moment they are ephemeral and player-minted.)
• Reserved director schedule events are withheld. One spawn.boss would otherwise spawn the boss — with its full loot table — in the template and every live copy, and each kill would reschedule the shared world timer, last-writer-wins. mud.zone() was added so content can filter correctly, and the demo pack's ev.zone ~= "darkwood" idiom was a worked example of the bug.
• signal_region / signal_world from an instance are refused loudly. The signal envelope drops source, so a director cannot distinguish one party's private progress from the shared world's. Region reads still resolve via the template, so content that gates on region state is not silently inert in every copy.
• Drain: instances are excluded from the handover loop but kept in accounting and the straggler reclaim. Excluding them outright fixed an over-count and introduced an under-count.
• Hot reload is explicitly frozen in both reconcileZone and notifyZones, and pinned instances are reported in the reload advisory.
• Metrics label by template, never by instance id — an instance id as an OTel attribute is unbounded, player-driven cardinality — with a separate per-template instance gauge. The logger keeps the instance id and gains a template= field: logs and metrics want opposite answers here.

The anchor

An instance is shard-local and ephemeral, so it cannot be a player's durable location. The first design projected the template ref into the placement record; two review panels killed it.

The placement record is the gate's routing key, and its invariant is "the recorded zone is the zone that holds the session." Writing a zone owned by a different shard makes a reconnect dial that shard, miss the residency index, and fresh-log the character while the original session still holds it — with nothing to fence the second copy.

The replacement is the anchor: the zone + room the player entered from, recorded on the session at entry and cleared on arrival in any non-instance zone. All three durable write sites write it positively.

• registerPlacement skips an instance entirely, leaving the last good record — which names the entrance — standing.
• clearPlacement still fires its tombstone but drops the zone, because that record outlives the instance and would otherwise dangle at a reaped id.
• zone_ref preservation is enforced at the sink (COALESCE) rather than by writing "". An empty ref maps to SQL NULL, which would clobber the anchor while room_ref kept the instance's authored room — an internally inconsistent row that loses the player's location.

The anchor's invariant decays, and what guards it

placementZoneRef's property is "entry is same-shard by construction, so the anchor names a zone THIS shard hosts." That is true at entry and nothing keeps it true. The anchor zone is an ordinary leased zone and can be rebalanced or drained to a peer while the player is inside the copy. The anchor is not updated when that happens — it names a zone id, and the id does not move — so from then on the placement record routes a reconnect to the peer, which has no session for the character and fresh-logs them from durable state while this shard still holds the live one.

This is not a regression and not specific to instances: before the anchor existed, the record named the last authored zone the player stood in, and a rebalance of that zone produced exactly the same race. It is stated because the anchor is now a load-bearing, documented concept, and a future reader must not take the same-shard property as a maintained safety invariant.

BeginDrain covers the SIGTERM case (the eject is step 0, before the anchor is handed over). A single-zone RebalanceZone did not, so it now settles the anchor question before handoverZoneTo:

State	Outcome
Nobody anchored here	Proceed. One bounded query, only when a directive actually lands
Anchored, defer budget unspent	Refuse with errZoneAnchored — a deferral, not a failure
Anchored, budget spent (3 min)	Eject the occupants to their anchors, then proceed

Where the guard sits is the single most important property, and there is a test that fails if it moves. ShardForZone follows the lease, not the hosting, so the flip is what breaks the routing. A guard placed later — on quiescent(), as the issue originally proposed, or inside UnhostZone — would be worse than none: the flip would land anyway (bug unchanged) and then the pin would block the teardown forever, leaving this shard hosting a zone it no longer owns while runRebalance reported success.

The deferral takes the ordinary retry backoff and deliberately does not clear the directive — telling the coordinator a move happened when the load has not shifted makes it re-plan against a wrong model.

The budget cap is not optional: a dungeon fed by a busy town has somebody inside essentially always, so an uncapped pin defers that town's rebalance forever, and every deferred cycle burns a coordinator cooldown — the imbalance the rebalance exists to correct simply persists.

Two design choices worth carrying forward:

• The anchors are queried on demand, not maintained. Four paths by which an anchor stops being one — a cross-shard exit, a clean quit, a link-death reap, and a failed drain-step-0 eject — destroy the session without passing through transferIn, the single documented clear point. A counter would leak four ways, and each leak is a permanently unmovable zone: a silent wedge in the machinery whose whole purpose is keeping zones movable. Asking costs nothing worth saving — only on a rebalance, bounded by the per-shard instance cap.
• It fails open. A timeout, a wedged instance, a shutting-down shard all answer "not anchored" and the move proceeds.

Failing open is correct because this is not the correctness fence. The double-own is harmful because nothing fences the durable row against a stale owner: finalizeFlush explicitly rebases past the state_version CAS and its zonePresent guard only sees its own shard, so a stale shard's 60-second-old snapshot overwrites the live one's — a duplication primitive reachable with no rebalance at all, via a second login landing on another shard. That fix is tracked separately as an owner epoch at the durable sink, where it cannot be forgotten. What this guard buys is playability: a party mid-dungeon does not get a reconnecting member pulled out to the anchor room.

Exit: every way back out

• Walking out already works through ownsZoneRef.
• Respawn stays inside the instance when it has a start room, and evicts to the anchor when it does not — with mint-time validation, so a template missing one fails loudly at mint rather than silently at death.
• Drain eject lands at the anchor, which is exactly what the anchor is for.

Security posture

• mud.send_to_instance is self-only. The harm gate short-circuits on a non-player actor before the safe-room veto, so any mob-actor script in a temple could otherwise pull a non-consenting player into a private, author-chosen copy where nobody can see or help them. An ordinary teleport moves you within observable space; this moves you outside it.
• The dungeon door is a declared entrance, not a relaxed rule. A room's enter trigger calling send_to_instance was refused, because for a room trigger the invoking actor is the room. The tempting fix — allow the event actor, on the argument that "the player is standing in the room whose script fires" — was rejected: standing in a room is not consent. A greet handler fires precisely because the victim performed the routine act of walking, so co-location is a near-free precondition rather than agreement. It would not even have been a room-only relaxation, since fireRoomEntry hands enter on the room and greet on every scripted mob the identical actor — and the mob-actor path is the one where the PvP check short-circuits ahead of the safe-room veto. It also carries a harm teleport has no analogue for: the mint is billed to the target's account, so a misfiring handler burns a passer-by's instance quota until their own runs are refused. A declared instance_entrances door makes the crossing a movement the player types, so the mover is the actor and self-only holds structurally — no third party in the call frame, no new gate to get wrong. Entrances live in their own map rather than as a sentinel inside exits precisely because every push-the-player path (h:move, directional flee, room-and-adjacent AoE, the cross-shard router) resolves through exits; a build-failing lint pins the single reader.
• Zones are not instanceable by default (above) — the uncapped-faucet class.
• The signed handoff snapshot gained a field-confusion fix. tier had been safe as an append-if-non-empty optional because it was the only one: presence was unambiguous. Adding account on the same terms made single-non-empty digests ambiguous — digest(tier="", account=X) and digest(tier=X, account="") were byte-identical. The handoff wire is plaintext, so an on-path attacker could rewrite an ordinary player's account into the tier field and keep the signature valid. Both optionals are now written unconditionally, so field position is fixed. (Length prefixes solve boundaries, not presence.) This changes the digest for existing tier-only snapshots; the rolling-deploy skew is fail-closed — a handoff is refused, never mis-trusted.

Limits

Instancing is an isolation mechanism, not transparent load-sharding. Occupants of different copies cannot see or interact with each other, so minting copies does not relieve a single crowded public zone — the one-zone-one-core ceiling still stands for shared space. Instances are also shard-local: they never migrate, and they do not survive the shard that minted them.