Running at Scale

Running at Scale

Audience: Sysadmin Status: ✅ Ready

How a TelosMUD fleet scales, how you add and remove capacity without dropping players, and the metrics and logs you watch to run it safely. This page is deliberately honest about the ceilings and the aspirational gaps. The engine-side design rationale is in Distributed Systems Model, Orchestration & Directors, and Cross-Shard Handoff; this page is the operator's view.

How each component scales

Component	Scaling axis	Notes
telos-gate	Horizontal, stateless	Pure edge/proxy: terminates telnet, runs OAuth login, proxies to shards. No authoritative state; add instances freely. Session→shard routing is Redis-directory-driven, so any gate can serve any player — no gate affinity.
telos-world	Horizontal by zone	A shard hosts N zone actors; each zone is a single-writer goroutine. Two-writer prevention is at the directory (time-fenced Redis lease), not in-process.
telos-director	Singleton per scope, leader-elected	Exactly one live director owns a given scope (region/world) via a Redis lease; a crash lets a warm standby claim within ~lease TTL. A director that can't confirm its lease steps down rather than risk double-leading.
telos-account	Horizontal, stateless behind Postgres	All authoritative state in Postgres; instances interchangeable behind a load balancer. Off the world's hot path (the world trusts the signed assertion). Redis backs device-auth only.

Placement: claim-from-pool

Distribution is claim-from-pool, not declare. On boot a world shard registers its id → endpoint, then walks its configured zone pool and wins each free zone via a directory compare-and-set. It hosts exactly what it wins; a zone already owned by a live peer is skipped (normal in a fleet). The core bootstrap zone is hosted locally and unleased on every shard, so even an empty fleet serves a lobby.

Liveness is decentralized — claim-from-pool plus lease expiry is the entire failover mechanism, and it works with no director running. A crashed shard's zones become unclaimed when its ~15s leases lapse, and a shard reclaims them on boot. Balancing is the director's job and is an optimizer, not a dependency.

Adding capacity

Start another telos-world with the zone pool in its config. On boot it registers and claims any free pool zones and serves them. A shard that wins nothing runs as a warm standby — registered and heartbeating, hosting no zone, ready to receive a drained zone or reclaim an orphaned one. So "scale out" is simply "start more world shards."

The hard ceiling: one zone = one core

A zone is a single goroutine that runs everything for that zone — commands, combat rounds, affect ticks — inline, on a 250 ms heartbeat. A single hot zone cannot exceed one core; it does not shard within itself. The headline signal for this is telos.zone.tick_lag_ms (how far past the 250 ms budget a heartbeat fired); it rises as that one writer saturates.

There is no hard players-per-box constant in the code — capacity depends on how busy your zones are. Measure with the bot-swarm load harness plus OTel, watching tick_lag_ms (server-side overrun) and bot-observed RTT together. Any "~1–2k per box" figure is an ops observation, not a code invariant. The mitigation for a hot zone is instancing (multiple copies), which is deferred / not implemented — today a single popular zone is a genuine ceiling, so design content to spread load across zones.

Scaling in & rolling upgrades: the zero-drop drain

Removing a shard — or rolling a new build — uses the zero-drop drain, which is implemented and signal-driven. On SIGINT/SIGTERM, before the zone loops tear down, the shard drains:

1. It stops accepting new fresh logins (but still accepts inbound handoff binds).
2. For each hosted zone it picks a peer and atomically flips the zone lease to that peer in a single fenced step, so the directory never sees an ownerless gap.
3. Each zone fans its players off in place — same zone id, same room, now owned by the peer — via a freeze/snapshot handoff. The player's socket stays open; they are redirected with no disconnect.
4. It waits (up to the deadline) for zones to empty; any stragglers are durably flushed and resume from durable state on reconnect (counted as reclaimed, not zero-drop).

What you trigger: an ordinary orchestrator rollout. SIGTERM a world pod (drain timeouts are ~45 s outer / 30 s per-zone); its zones and players migrate to a peer or standby; start the replacement, which claims the now-freed pool zones. Redirected players see no disconnect.

Two shutdown modes: a clean SIGTERM drains; an unexpected lease loss stops immediately without a drain (you can't hand off zones you no longer own). A single shard with no peer does a best-effort durable flush only.

Metrics (OpenTelemetry)

Every service can export OTLP/gRPC metrics, but export is off by default — it turns on only when OTEL_EXPORTER_OTLP_ENDPOINT (or the metrics-specific var) is set; otherwise every record is a negligible no-op. The dev path is: services push OTLP → otel-collector:4317 → re-exposed as Prometheus on :8889. A sysadmin scrapes :8889/metrics — the collector is the scrape surface; the Go services expose no /metrics endpoint. Only metrics are wired; there is no tracing exporter.

The instruments:

Metric	Type	Meaning
telos.zone.tick_lag_ms	histogram	Zone heartbeat overrun past the 250 ms budget — the headline scale signal.
telos.zone.occupancy	gauge (zone label)	Live players per zone.
telos.gate.connections	up/down counter	Live gate connections.
telos.gate.frames_dropped_total	counter	Outbound frames dropped for slow clients (shard-wide).
telos.bus.deliver_lag_ms	histogram	Scoped-event publish→deliver latency.

Honest observability gaps:

• In the shipped dev compose, only the worlds and gates export OTLP — telos-account and telos-director do not, so fleet dashboards miss those two unless you wire the OTLP env into them.
• telos.bus.deliver_lag_ms is defined but has no call site — instrumented but not emitting yet.
• There is no tracing.

Logging (slog)

Structured JSON to stdout, one global logger tagged with the service name. Levels are debug/info/warn/error via TELOS_LOG_LEVEL; DEBUG=1 (or true/yes/on) force-lowers to debug regardless of config (dev compose sets it). There is no file sink or log shipper in code — logs go to stdout for your container/orchestrator pipeline to collect.

Log lines worth alerting/keying on:

• Boot & placement: starting; registered shard; claimed zone; won no zones … STANDBY; and the degradation lines postgres ready / redis unavailable; single-shard mode / nats unavailable; … disabled.
• Handoff/drain: signal received: draining before shutdown; drain complete (redirected/reclaimed counts); graceful drain incomplete; lease fence: stopping without drain.
• Leadership: director leadership changed; director lease campaign failed; stepping down; placement: rebalance drain recommended / zone needs (re)assignment.
• Slow clients: slow client wedged … gate write-deadline will reclaim.
• Security refusals: any refusing to start (a fail-closed boot — a missing secret or a pack-set divergence; see Deployment); and TELOS_ALLOW_INSECURE warn lines where an insecure opt-in is active.

Degradation ladder (what breaks first)

The world tolerates dependency loss non-fatally: Redis down → single-shard mode, cross-shard exits sealed; Postgres down → ephemeral characters, hot-reload disabled; NATS down → comms/tells/hot-reload disabled. None of these crash a shard, but each narrows what works — watch the degradation log lines above.

Honest gaps (plan around these)

• Orphaned-zone failover is not instantaneous. There is no running-standby auto-adopt loop; recovery is ~lease TTL (15 s) + a reclaiming shard, not immediate.
• Placement rebalancing is advisory-only. The director plans and logs recommended moves but does not execute rebalances yet, and its plan balances by raw zone count (no occupancy signal wired). Rebalancing is a manual/operator action today.
• Drain peer-select is naive — "first live peer," with no serialization of concurrent drains.
• The hot-zone ceiling is hard (one zone = one core; instancing deferred), and the observability blind spots above (account/director not exporting, bus.deliver_lag_ms silent, no tracing) are real.

Related: Cross-Shard Handoff, Orchestration & Directors, Distributed Systems Model, Deployment.