Distributed Systems Model

Distributed Systems Model

Audience: Engine Developer Status: ✅ Ready

TelosMUD scales a persistent, stateful text world horizontally by sharding it into zones and giving each zone exactly one owner. The invariants that make that safe — single-writer ownership, time-fenced leases, monotonic placement, apply-once event delivery — are collected here. This is the "why it's correct" companion to the mechanisms documented on Zone Runtime & Actor Model, Cross-Shard Handoff, Scoped Event Bus, and Persistence & Durability. For the operator's view see Running at Scale.

The four services and how they scale

Service	Scaling model	Coordination
telos-gate	horizontal, stateless edge	routes a session to its shard via the Redis directory; any gate can serve any player
telos-world	horizontal by zone, single-writer per zone	claims zones from a pool via Redis CAS leases
telos-director	singleton per scope, leader-elected	Redis lease; warm standbys (Orchestration & Directors)
telos-account	horizontal, stateless behind Postgres	off the hot path — the world trusts a signed assertion

The gate holds no authoritative game state, so you add instances freely. Session→shard routing is Redis-directory-driven (per-character placement → home-zone owner → configured fallback); a cross-shard move carries the destination address in the world's Redirect frame, so there is no gate affinity.

The single-writer spine

Each zone is one goroutine that owns all its entities; every mutation funnels through its inbox. No in-zone locks, deterministic ordering, and a clean unit of placement. Two-writer prevention is enforced at the directory, not in-process:

• Time-fenced lease CAS: claimZone grants ownership only if the zone is unowned, expired, or already yours, using the single Redis TIME clock so a skewed shard can't steal a live lease. Default lease 15 s, renewed at TTL/3.
• Duplicate shard-id refused: registering a shard id that a different live endpoint holds is rejected — the guard that stops two same-id owners both reading as renewals.
• Monotonic placement epoch: a per-player CAS makes placement monotonic, so a stale or duplicated handoff can't route a player back to a shard they've left.

Cross-boundary interaction is message-passing only (the gRPC Play stream and the Handoff RPC); no shared mutation ever crosses a shard.

Placement: claim-from-pool, not declare

World servers claim zones from a shared pool rather than statically declaring them. On boot a shard registers its id→endpoint, then walks the configured pool and wins each free zone via the directory CAS; a zone already owned by a live peer is skipped. Liveness is decentralized and works even with no director running — claim-from-pool plus lease expiry is the failover mechanism (a crashed shard's zones become unclaimed when its 15 s leases lapse). A server that wins nothing runs as a warm standby. The director's placement role is an optimizer, not a dependency.

The bootstrap core zone is hosted locally + unleased on every shard, so a fresh/empty fleet still serves a lobby.

The durability ladder as a distributed invariant

State is checkpointed to Redis (~10 s) and Postgres (~60 s), with a state_version optimistic-concurrency guard that fences stale writes: the CAS WHERE state_version = $expected means a mis-fired handoff or a zombie owner loses the race rather than clobbering the durable record. On load the fresher of {row, checkpoint} wins by state_version. See Persistence & Durability.

Event delivery guarantees

The scoped event bus splits transient down-broadcasts (at-most-once, NATS core) from durable signal-up (at-least-once, JetStream). Correctness comes from apply-once over at-least-once: the director consumes the durable stream only while leader (stable consumer id → resume from last ack) and dedups by a per-source monotonic watermark. A capstone test proves exactly-once effect — precisely three boss kills counted across a mid-sequence director restart.

Failure modes & degradation

The world degrades rather than crashing:

• Redis down → single-shard mode; cross-shard exits sealed.
• Postgres down → ephemeral characters; hot-reload disabled.
• NATS down → comms/tells/hot-reload disabled.
• A dead gate → gRPC server keepalive pings reclaim leaked world streams.

Security-sensitive gaps fail closed on boot rather than running open: a gate with no account target, a shard with no handoff verify key, an account service with no caller token, or a pack-set divergence each refuse to start unless TELOS_ALLOW_INSECURE is explicitly set (Deployment).

The scaling ceiling and honest gaps

The fundamental limit is one zone = one core: a zone is a single goroutine with a 250 ms heartbeat budget, and all combat rounds and affect ticks run inline on it. telos.zone.tick_lag_ms (how far past 250 ms a heartbeat fired) is the headline scale signal. A single hot zone cannot exceed one core — it does not shard within itself, and instancing (multiple copies of a zone) is deferred/not implemented. There is no hard players-per-box constant in code; capacity is measured with the bot-swarm load tester plus OpenTelemetry (Running at Scale).

Placement is a maturing subsystem — the load-aware rebalancer is shipped, but a few edges remain:

• Rebalancing is automatic and load-aware. Each shard heartbeats per-zone occupancy; the leader director plans a player-weighted spread and issues rebalance-drain directives that the owning shard's executor drains via the handoff, with a 5-minute per-zone cooldown and locality-aware colocation. It stays an optimizer — if the director is down, zones are still claimed and served, just possibly unbalanced.
• Drain-target selection is director-owned and serialized — a peer is chosen against a soft occupancy ceiling and its headroom is atomically reserved in the directory, so concurrent drains don't pile onto one peer. If every peer is reservation-full it admits over the soft ceiling rather than stall (a dropped connection is worse than transient overload the rebalancer then corrects); the reservation is an admission hint (~15 s TTL), not a durable lock.
• Orphan-zone failover is not instantaneous — it relies on lease expiry (~15 s) plus a re-claiming shard on boot; there is no running-standby auto-adopt loop, so recovery is ~lease-TTL + reclaim.