Distributed Systems Model

Distributed Systems Model

Audience: Engine Developer Status: ✅ Ready

TelosMUD scales a persistent, stateful text world horizontally by sharding it into zones and giving each zone exactly one owner. The invariants that make that safe — single-writer ownership, time-fenced leases, monotonic placement, apply-once event delivery — are collected here. This is the "why it's correct" companion to the mechanisms documented on Zone Runtime & Actor Model, Cross-Shard Handoff, Scoped Event Bus, and Persistence & Durability. For the operator's view see Running at Scale.

The four services and how they scale

Service	Scaling model	Coordination
telos-gate	horizontal, stateless edge	routes a session to its shard via the Redis directory; any gate can serve any player
telos-world	horizontal by zone, single-writer per zone	claims zones from a pool via Redis CAS leases
telos-director	singleton per scope, leader-elected	Redis lease; warm standbys (Orchestration & Directors)
telos-account	horizontal, stateless behind Postgres	off the hot path — the world trusts a signed assertion

The gate holds no authoritative game state, so you add instances freely. Session→shard routing is Redis-directory-driven: the player's placement record names their zone, and ShardForZone resolves whichever shard currently owns it. A cross-shard move carries the destination address in the world's Redirect frame, so there is no gate affinity.

The single-writer spine

Each zone is one goroutine that owns all its entities; every mutation funnels through its inbox. No in-zone locks, deterministic ordering, and a clean unit of placement. Two-writer prevention is enforced at the directory, not in-process:

• Time-fenced lease CAS: claimZone grants ownership only if the zone is unowned, expired, or already yours, using the single Redis TIME clock so a skewed shard can't steal a live lease. Default lease 15 s, renewed at TTL/3.
• Duplicate shard-id refused: registering a shard id that a different live endpoint holds is rejected — the guard that stops two same-id owners both reading as renewals.
• Renewal restarts on re-adoption. A zone the rebalancer moves away and later moves back (A → B → A) must restart lease renewal. If the "handed off" marker were set-only, the shard would host and serve the zone while renewing nothing; once the 15 s lease lapsed, ShardForZone would resolve nobody and ClaimZone — fenced only against a live lease — would grant the zone to any shard that asked, while this one was still writing to it. That is a guaranteed second writer on a routine operation, so the marker is cleared and renewal restarted on re-adoption, and the adopting state is bounded.
• Monotonic placement epoch: epoch is a per-player monotonic fence. Only the handoff coordinator bumps it, and the handoff CAS demands a strictly greater epoch — so a stale or duplicated handoff can't roll a player back to a shard they've left.

Cross-boundary interaction is message-passing only (the gRPC Play stream and the Handoff RPC); no shared mutation ever crosses a shard.

The placement record

The directory keeps one record per player. It is the reconnect-routing spine, and it carries more weight than its name suggests — three subsystems read it:

Field	Read by	Meaning
zone	the gate, on reconnect	the routing key. Resolve it through ShardForZone to get the zone's current owner
epoch	the handoff CAS	the monotonic fence; also the existence key for the tell/mail oracle
shard	nothing, for routing	vestigial for routing; dropped by the logout tombstone

Why zone, not shard. A shard id says where the player was. The moment that zone is rebalanced onto another shard — or that shard exits — the id is stale, and a returning player was dropped into the home zone's start room, losing their durable location. Routing by zone makes an offline rebalance transparent, because ShardForZone always names the current owner.

Every residency writes a placement. Originally the record had exactly one writer, the cross-shard handoff CAS — so a player who simply logged in and stayed put had no placement at all, which made them unroutable on reconnect and invisible to the tell/mail existence oracle ("there is no player by that name"). The world now registers a placement whenever a player becomes resident in a zone: fresh login, link-dead resume, cross-shard arrival, and the intra-shard zone walk — which changes zone without changing shard or epoch, and so never triggers the handoff CAS.

That required a second writer with different semantics. The handoff CAS demands a strictly greater epoch; a login re-registers at the epoch it just resumed from, so reusing the CAS would make every login a silent no-op. registerPlacement therefore accepts an equal epoch, refuses a strictly newer one, and keeps the stored epoch at the maximum. This is safe precisely because an epoch maps to exactly one shard: only the handoff coordinator bumps it, so an equal-epoch write can only ever rewrite the zone within the shard that already owns the player. The hand-off to the background writer coalesces per player rather than dropping a full FIFO.

Logout writes a fenced tombstone, not a delete. Deleting the record would ship three regressions at once: the tell/mail existence oracle would refuse tells to an offline character; a delayed or retried handoff write would find no current value and apply, resurrecting a stale placement; and the returning player would lose their zone. So logout drops only the shard field, and only when the record still names this shard at this epoch — a compare-and-delete, so a clear racing a fast relog or a handoff is a no-op. Existence is therefore keyed on epoch (which every writer sets), and the tombstone also writes the quitting zone, since a logout offered while a zone-change registration is still pending would otherwise leave a stale zone behind. An outright ClearPlayer still exists — as character deletion.

Placement: claim-from-pool, not declare

World servers claim zones from a shared pool rather than statically declaring them. On boot a shard registers its id→endpoint, then walks the configured pool and wins each free zone via the directory CAS; a zone already owned by a live peer is skipped. Liveness is decentralized and works even with no director running — claim-from-pool plus lease expiry is the failover mechanism (a crashed shard's zones become unclaimed when its 15 s leases lapse). A server that wins nothing runs as a warm standby. The director's placement role is an optimizer, not a dependency.

The bootstrap core zone is hosted locally + unleased on every shard, so a fresh/empty fleet still serves a lobby.

The durability ladder as a distributed invariant

State is checkpointed to Redis (~10 s) and Postgres (~60 s), with a state_version optimistic-concurrency guard that fences stale writes: the CAS WHERE state_version = $expected means a mis-fired handoff or a zombie owner loses the race rather than clobbering the durable record. On load the fresher of {row, checkpoint} wins by state_version. See Persistence & Durability.

Event delivery guarantees

The scoped event bus splits transient down-broadcasts (at-most-once, NATS core) from durable signal-up (at-least-once, JetStream). Correctness comes from apply-once over at-least-once: the director consumes the durable stream only while leader (stable consumer id → resume from last ack) and dedups by a per-source monotonic watermark. A capstone test proves exactly-once effect — precisely three boss kills counted across a mid-sequence director restart.

Failure modes & degradation

The world degrades rather than crashing:

• Redis down → single-shard mode; cross-shard exits sealed.
• Postgres down → ephemeral characters; hot-reload disabled.
• NATS down → comms/tells/hot-reload disabled.
• A dead gate → gRPC server keepalive pings reclaim leaked world streams.

Security-sensitive gaps fail closed on boot rather than running open: a gate with no account target, a shard with no handoff verify key, an account service with no caller token, or a pack-set divergence each refuse to start unless TELOS_ALLOW_INSECURE is explicitly set (Deployment).

The scaling ceiling and honest gaps

The fundamental limit is one zone = one core: a zone is a single goroutine with a 250 ms heartbeat budget, and all combat rounds and affect ticks run inline on it. telos.zone.tick_lag_ms (how far past 250 ms a heartbeat fired) is the headline scale signal. A single hot zone cannot exceed one core — it does not shard within itself, and instancing (multiple copies of a zone) is deferred/not implemented. There is no hard players-per-box constant in code; capacity is measured with the bot-swarm load tester plus OpenTelemetry (Running at Scale).

Placement is a maturing subsystem — the load-aware rebalancer is shipped, but a few edges remain:

• Rebalancing is automatic and load-aware. Each shard heartbeats per-zone occupancy; the leader director plans a player-weighted spread and issues rebalance-drain directives that the owning shard's executor drains via the handoff, with a 5-minute per-zone cooldown and locality-aware colocation. It stays an optimizer — if the director is down, zones are still claimed and served, just possibly unbalanced.
• Drain-target selection is director-owned and serialized — a peer is chosen against a soft occupancy ceiling and its headroom is atomically reserved in the directory, so concurrent drains don't pile onto one peer. Each reservation carries its own expiry and the clock is the Redis server's, not the caller's; a stale hold is excluded from the reserved sum and pruned on sight. (Refreshing one whole-key expiry per reserve was the opposite: under exactly the concurrent fleet rollout this guard exists for, a crashed drainer's stale hold survived indefinitely as long as other drainers kept reserving onto the same hot target, inflating the sum and spilling everyone to the fallback.) If every peer is genuinely reservation-full it admits over the soft ceiling rather than stall — a dropped connection is worse than the transient overload the rebalancer then corrects. The reservation is an admission hint, not a durable lock.
• Orphan-zone failover is not instantaneous — it relies on lease expiry (~15 s) plus a re-claiming shard on boot; there is no running-standby auto-adopt loop, so recovery is ~lease-TTL + reclaim.