Running at Scale

Running at Scale

Audience: Sysadmin Status: ✅ Ready

How a TelosMUD fleet scales, how you add and remove capacity without dropping players, and the metrics and logs you watch to run it safely. This page is deliberately honest about the ceilings and the aspirational gaps. The engine-side design rationale is in Distributed Systems Model, Orchestration & Directors, and Cross-Shard Handoff; this page is the operator's view.

How each component scales

Component	Scaling axis	Notes
telos-gate	Horizontal, stateless	Pure edge/proxy: terminates telnet, runs OAuth login, proxies to shards. No authoritative state; add instances freely. A reconnect is routed by the player's zone (from the directory placement record) resolved to that zone's current owner, so any gate can serve any player — no gate affinity, and a rebalance while the player was offline is transparent.
telos-world	Horizontal by zone	A shard hosts N zone actors; each zone is a single-writer goroutine. Two-writer prevention is at the directory (time-fenced Redis lease), not in-process.
telos-director	Singleton per scope, leader-elected	Exactly one live director owns a given scope (region/world) via a Redis lease; a crash lets a warm standby claim within ~lease TTL. A director that can't confirm its lease steps down rather than risk double-leading.
telos-account	Horizontal, stateless behind Postgres	All authoritative state in Postgres; instances interchangeable behind a load balancer. Off the world's hot path (the world trusts the signed assertion). Redis backs device-auth only.

Placement: claim-from-pool

Distribution is claim-from-pool, not declare. On boot a world shard registers its id → endpoint, then walks its configured zone pool and wins each free zone via a directory compare-and-set. It hosts exactly what it wins; a zone already owned by a live peer is skipped (normal in a fleet). The core bootstrap zone is hosted locally and unleased on every shard, so even an empty fleet serves a lobby.

Liveness is decentralized — claim-from-pool plus lease expiry is the entire failover mechanism, and it works with no director running. A crashed shard's zones become unclaimed when its ~15s leases lapse, and a shard reclaims them on boot. Balancing is the director's job and is an optimizer, not a dependency.

Adding capacity

Start another telos-world with the zone pool in its config. On boot it registers and claims any free pool zones and serves them. A shard that wins nothing runs as a warm standby — registered and heartbeating, hosting no zone, ready to receive a drained zone or reclaim an orphaned one. So "scale out" is simply "start more world shards."

Automatic rebalancing

Balancing is the director's job, and it executes, not just advises. Each world shard heartbeats its per-zone occupancy (live player count) into the directory; the leader director reads those weights and plans moves toward an even player-weighted spread — a busy town counts more than an empty wilderness — with hysteresis so the fleet doesn't thrash. For each planned move it issues a rebalance-drain directive: a directory record the owning shard's executor picks up and drains that one zone to the target via the zero-drop handoff. A directive carries a ~90 s TTL, and each moved zone gets a 5-minute cooldown so weights can settle before it's eligible again; a zone already being rebalanced, or a target shard that is itself draining (a rollout), is skipped, and related zones are kept colocated where possible. If the director is down, zones are still claimed and served — just possibly unbalanced (balancing is an optimizer, not a dependency).

Once a rebalanced zone's players and lease have moved, the source tears the emptied zone down (runtime zone teardown) rather than leaving it running. So a long-lived shard that has weathered many rebalances doesn't accumulate a zombie zone per migration — each one's heartbeat, resets, and Lua VM are reclaimed. (A rollout via SIGTERM never needed this, since the process exits; a live rebalance does.)

Drain-target selection is director-owned and serialized: the selector reserves the chosen peer's headroom against a soft occupancy ceiling in the directory, so two concurrent drains don't pile onto the same peer. Each reservation carries its own expiry, timed by the Redis server rather than the caller and sized to outlast the whole drain — the drain deadline plus one presence-reflect window — since the hold is placed once when the target is chosen and never refreshed. That margin keeps a slow-but-alive drain's reservation in place while its players are still migrating (a player landing right at the deadline needs one more presence heartbeat before the target reports the added weight); a clean handover retires the hold early, so only a crashed drainer's hold lingers the full TTL before it's excluded from the reserved sum and pruned — instead of expiring mid-drain and letting a concurrent drainer over-commit onto the target's stale, pre-migration load.

The hard ceiling: one zone = one core

A zone is a single goroutine that runs everything for that zone — commands, combat rounds, affect ticks — inline, on a 250 ms heartbeat. A single hot zone cannot exceed one core; it does not shard within itself. The headline signal for this is telos.zone.tick_lag_ms (how far past the 250 ms budget a heartbeat fired); it rises as that one writer saturates.

There is no hard players-per-box constant in the code — capacity depends on how busy your zones are. Measure with the bot-swarm load harness plus OTel, watching tick_lag_ms (server-side overrun) and bot-observed RTT together. Any "~1–2k per box" figure is an ops observation, not a code invariant. The mitigation for a hot zone is instancing (multiple copies), which is deferred / not implemented — today a single popular zone is a genuine ceiling, so design content to spread load across zones.

Scaling in & rolling upgrades: the zero-drop drain

Removing a shard — or rolling a new build — uses the zero-drop drain, which is implemented and signal-driven. On SIGINT/SIGTERM, before the zone loops tear down, the shard drains:

1. It stops accepting new fresh logins (but still accepts inbound handoff binds).
2. For each hosted zone it picks a peer and atomically flips the zone lease to that peer in a single fenced step, so the directory never sees an ownerless gap.
3. Each zone fans its players off in place — same zone id, same room, now owned by the peer — via a freeze/snapshot handoff. The player's socket stays open; they are redirected with no disconnect.
4. It waits (up to the deadline) for zones to empty; any stragglers are durably flushed and resume from durable state on reconnect (counted as reclaimed, not zero-drop).

What you trigger: an ordinary orchestrator rollout. SIGTERM a world pod (drain timeouts are ~45 s outer / 30 s per-zone); its zones and players migrate to a peer or standby; start the replacement, which claims the now-freed pool zones. Redirected players see no disconnect.

Two shutdown modes: a clean SIGTERM drains; an unexpected lease loss stops immediately without a drain (you can't hand off zones you no longer own). A single shard with no peer does a best-effort durable flush only.

Metrics (OpenTelemetry)

Every service can export OTLP/gRPC metrics, but export is off by default — it turns on only when OTEL_EXPORTER_OTLP_ENDPOINT (or the metrics-specific var) is set; otherwise every record is a negligible no-op. The dev path is: services push OTLP → otel-collector:4317 → re-exposed as Prometheus on :8889. A sysadmin scrapes :8889/metrics — the collector is the scrape surface; the Go services expose no /metrics endpoint. Only metrics are wired; there is no tracing exporter.

The instruments:

Metric	Type	Meaning
telos.zone.tick_lag_ms	histogram	Zone heartbeat overrun past the 250 ms budget — the headline scale signal.
telos.zone.occupancy	gauge (zone label)	Live players per zone.
telos.gate.connections	up/down counter	Live gate connections.
telos.gate.frames_dropped_total	counter	Outbound frames dropped for slow clients (shard-wide).
telos.bus.deliver_lag_ms	histogram	Scoped-event publish→deliver latency, live events only.
telos.bus.catchup_events_total	counter (subject label)	Durable scoped-event backlog events drained by a resuming consumer.
telos.bus.catchup_age_ms	histogram (subject label)	Age of each backlog event at delivery. Its max is how far behind that consumer is.
telos.commbus.durable_parked_total	counter (stream label)	Durable messages parked after exhausting the retry budget — permanent loss. Counted off the broker's MAX_DELIVERIES advisory (via a per-stream queue group, so once cluster-wide), which catches a park from ack-wait expiry or across a restart — not only an in-process final delivery.
telos.commbus.durable_poisoned_total	counter (stream label)	Durable messages dropped as undeliverable (malformed / unroutable).
telos.shard.drain_redirected_total	counter	Players handed to a peer during a graceful drain (socket kept open).
telos.shard.drain_reclaimed_total	counter (fault label)	Players dropped to reconnect at the drain deadline.

Reading the bus metrics during a recovery. deliver_lag_ms samples live deliveries only. A consumer resuming after an outage drains events published hours ago, and folding those into the same histogram would dwarf its distribution exactly when you need to read the live p99. The catch-up pair is where that story lives instead: catchup_events_total is the depth (how much this consumer missed), catchup_age_ms is the age (how far behind it is). Watch the age's max fall toward zero — that is the consumer converging. Both are labeled by subject, so you can see which scope is behind.

Honest observability gaps:

• In the shipped dev compose, only the worlds and gates export OTLP — telos-account and telos-director do not, so fleet dashboards miss those two unless you wire the OTLP env into them.
• There is no tracing.

Logging (slog)

Structured JSON to stdout, one global logger tagged with the service name. Levels are debug/info/warn/error via TELOS_LOG_LEVEL; DEBUG=1 (or true/yes/on) force-lowers to debug regardless of config (dev compose sets it). There is no file sink or log shipper in code — logs go to stdout for your container/orchestrator pipeline to collect.

Log lines worth alerting/keying on:

• Boot & placement: starting; registered shard; claimed zone; won no zones … STANDBY; and the degradation lines postgres ready / redis unavailable; single-shard mode / nats unavailable; … disabled.
• Handoff/drain: signal received: draining before shutdown; drain complete (redirected/reclaimed counts); graceful drain incomplete; lease fence: stopping without drain.
• Leadership: director leadership changed; director lease campaign failed; stepping down; placement: rebalance drain recommended / zone needs (re)assignment.
• Slow clients: slow client wedged … gate write-deadline will reclaim.
• Security refusals: any refusing to start (a fail-closed boot — a missing secret or a pack-set divergence; see Deployment); and TELOS_ALLOW_INSECURE warn lines where an insecure opt-in is active.

Degradation ladder (what breaks first)

The world tolerates dependency loss non-fatally: Redis down → single-shard mode, cross-shard exits sealed; Postgres down → ephemeral characters, hot-reload disabled; NATS down → comms/tells/hot-reload disabled. None of these crash a shard, but each narrows what works — watch the degradation log lines above.

Honest gaps (plan around these)

• Orphaned-zone failover is not instantaneous. There is no running-standby auto-adopt loop; recovery is ~lease TTL (15 s) + a reclaiming shard, not immediate.
• Drain-target admission is a soft ceiling, not a hard cap. The director-owned drain-target selector reserves a peer's headroom against a soft occupancy ceiling (default 1500), but if every live peer is reservation-full it proceeds over the ceiling rather than stall — a dropped connection is worse than the transient overload the rebalancer then corrects. The reservation is an admission hint (each with its own server-timed expiry), not a durable lock.
• The hot-zone ceiling is hard (one zone = one core; instancing deferred), and the observability blind spots above (account/director not exporting OTLP in the shipped compose, no tracing) are real.

Related: Cross-Shard Handoff, Orchestration & Directors, Distributed Systems Model, Deployment.