Running at Scale

Running at Scale

Audience: Sysadmin Status: ✅ Ready

How a TelosMUD fleet scales, how you add and remove capacity without dropping players, and the metrics and logs you watch to run it safely. This page is deliberately honest about the ceilings and the aspirational gaps. The engine-side design rationale is in Distributed Systems Model, Orchestration & Directors, and Cross-Shard Handoff; this page is the operator's view.

How each component scales

Component	Scaling axis	Notes
telos-gate	Horizontal, stateless	Pure edge/proxy: terminates telnet, runs OAuth login, proxies to shards. No authoritative state; add instances freely. A reconnect is routed by the player's zone (from the directory placement record) resolved to that zone's current owner, so any gate can serve any player — no gate affinity, and a rebalance while the player was offline is transparent.
telos-world	Horizontal by zone	A shard hosts N zone actors; each zone is a single-writer goroutine. Two-writer prevention is at the directory (time-fenced Redis lease), not in-process.
telos-director	Singleton per scope, leader-elected	Exactly one live director owns a given scope (region/world) via a Redis lease; a crash lets a warm standby claim within ~lease TTL. A director that can't confirm its lease steps down rather than risk double-leading.
telos-account	Horizontal, stateless behind Postgres	All authoritative state in Postgres; instances interchangeable behind a load balancer. Off the world's hot path (the world trusts the signed assertion). Redis backs device-auth only.

Placement: claim-from-pool

Distribution is claim-from-pool, not declare. On boot a world shard registers its id → endpoint, then walks its configured zone pool and wins each free zone via a directory compare-and-set. It hosts exactly what it wins; a zone already owned by a live peer is skipped (normal in a fleet). The core bootstrap zone is hosted locally and unleased on every shard, so even an empty fleet serves a lobby.

Liveness is decentralized — claim-from-pool plus lease expiry is the entire failover mechanism, and it works with no director running. A crashed shard's zones become unclaimed when its ~15s leases lapse, and a shard reclaims them on boot. Balancing is the director's job and is an optimizer, not a dependency.

Adding capacity

Start another telos-world with the zone pool in its config. On boot it registers and claims any free pool zones and serves them. A shard that wins nothing runs as a warm standby — registered and heartbeating, hosting no zone, ready to receive a drained zone or reclaim an orphaned one. So "scale out" is simply "start more world shards."

Automatic rebalancing

Balancing is the director's job, and it executes, not just advises. Each world shard heartbeats its per-zone occupancy (live player count) into the directory; the leader director reads those weights and plans moves toward an even player-weighted spread — a busy town counts more than an empty wilderness — with hysteresis so the fleet doesn't thrash. For each planned move it issues a rebalance-drain directive: a directory record the owning shard's executor picks up and drains that one zone to the target via the zero-drop handoff. A directive carries a ~90 s TTL, and each moved zone gets a 5-minute cooldown so weights can settle before it's eligible again; a zone already being rebalanced, or a target shard that is itself draining (a rollout), is skipped, and related zones are kept colocated where possible. If the director is down, zones are still claimed and served — just possibly unbalanced (balancing is an optimizer, not a dependency).

Once a rebalanced zone's players and lease have moved, the source tears the emptied zone down (runtime zone teardown) rather than leaving it running. So a long-lived shard that has weathered many rebalances doesn't accumulate a zombie zone per migration — each one's heartbeat, resets, and Lua VM are reclaimed. (A rollout via SIGTERM never needed this, since the process exits; a live rebalance does.)

Drain-target selection is director-owned and serialized: the selector reserves the chosen peer's headroom against a soft occupancy ceiling in the directory, so two concurrent drains don't pile onto the same peer. Each reservation carries its own expiry, timed by the Redis server rather than the caller and sized to outlast the whole drain — the drain deadline plus one presence-reflect window — since the hold is not refreshed during the wait for zones to empty (it is rebased each time another zone reserves onto the same target, so exposure runs from the last reserve). That margin keeps a slow-but-alive drain's reservation in place while its players are still migrating (a player landing right at the deadline needs one more presence heartbeat before the target reports the added weight); a clean handover retires the hold early, so only a crashed drainer's hold lingers the full TTL before it's excluded from the reserved sum and pruned — instead of expiring mid-drain and letting a concurrent drainer over-commit onto the target's stale, pre-migration load.

The hard ceiling: one zone = one core

A zone is a single goroutine that runs everything for that zone — commands, combat rounds, affect ticks — inline, on a 250 ms heartbeat. A single hot zone cannot exceed one core; it does not shard within itself. The headline signal for this is telos.zone.tick_lag_ms (how far past the 250 ms budget a heartbeat fired); it rises as that one writer saturates.

There is no hard players-per-box constant in the code — capacity depends on how busy your zones are. Measure with the bot-swarm load harness plus OTel, watching tick_lag_ms (server-side overrun) and bot-observed RTT together. Any "~1–2k per box" figure is an ops observation, not a code invariant. Instanced zones are implemented, but they are an isolation mechanism rather than transparent load-sharding — a minted copy is private, and occupants of different copies cannot see each other — so they solve the dungeon/party case and do not relieve a crowded public zone. A single popular shared zone remains a genuine ceiling, so design content to spread load across zones.

Instanced zones in operation

A world shard can mint private runtime copies of a content zone that opted in with instanceable: true — the dungeon/party copy. Operationally they behave unlike every other zone on the box, and the differences are the point of this section. Design rationale: Instanced Zones; the authoring side is Building Instanced Zones.

They are shard-local and unleased

An instance takes no directory lease. Consequences an operator should plan around:

• It is not in the placement pool and is never rebalanced. The director cannot see it, move it, or plan around it.
• It is never a cross-shard handoff destination. Every off-box ingress fails closed on an instance-shaped zone id.
• It dies with its shard. There is no failover for a copy: if the process goes away ungracefully, its occupants reconnect from durable state into the zone they entered from.
• Its occupancy is invisible to the rebalancer's zone weights, so a shard carrying heavy instanced load can still look "light" to the coordinator. Watch the instance gauge below alongside occupancy.

The caps, and the fact that they are per process

Bound	Default	What it protects
Concurrent live copies per account	3	Per-principal fairness. Charged to the account, never the character (alts route around a per-character cap)
Mints per account per window	6 per minute	The cheap-to-mint / cheap-to-abandon churn a concurrent cap alone does not see
Live copies per shard process	256	The real resource backstop: each copy is a zone object, an actor goroutine and a Lua VM
Concurrent mint builds per shard	2 workers	A mint is CPU- and allocation-heavy plus store I/O; this bounds how much of the box creating them may consume
Pending mint queue depth	64	A refusal bound, not a buffer — a full queue cleanly refuses entry

Every bound is PER PROCESS. There is no cross-shard instance accounting, so an account's real ceiling across the fleet is perAccount × (number of world shards it can reach) — with the defaults, 3 copies per shard, not 3 fleet-wide. That is deliberate: a shard-local cap needs no coordination and cannot fail open on a directory outage. Size perShard for the box, and treat the per-account number as a per-shard fairness knob rather than a global quota.

These are not config knobs today. The defaults are compile-time constants with a Go WithInstanceLimits builder option, and nothing in telos-world wires them to config or an env var — there is no TELOS_INSTANCE_* setting. Changing them means embedding the shard yourself. Note also that raising the per-account cap alone is not enough: the mint rate limit is a fixed window, whose worst case is 2 × burst across a window boundary, and that is only harmless because the concurrent cap bounds how many can be live at once.

Metrics: labeled by template, never by instance id

Metric	Type	Meaning
telos.zone.instances	gauge (template label)	Live runtime-minted copies on this shard. This is where instanced load is visible.
telos.zone.occupancy	gauge (template label)	Players per zone — every copy of a template reports onto the template's series

An instance id is <template>#<128 random bits>, minted per dungeon run. As an OTel attribute that is unbounded, player-driven cardinality: thousands of dead time series, one per run. So metrics label by template only, and telos.zone.occupancy deliberately collapses all copies of a template onto one series to keep its attribute set bounded — use the instance gauge, not occupancy, to see how many copies are live.

Logs go the other way on purpose. A log line keeps the instance id (an operator reading a log needs to know which copy misbehaved) and adds a template= field so lines from every copy can still be grepped together. Metrics want the bounded answer; logs want the specific one.

Drain and rolling upgrades

Instances are excluded from the drain's handover loop — there is no lease to flip and no peer can resolve an instance id — but they are kept in the accounting and straggler reclaim, so their occupants never vanish silently from the drain tally.

Step 0 of a drain walks every instance's occupants back out to their exit anchor — the zone and room they entered from — before any lease moves and before the population snapshot. Once they are standing in the anchor zone they are ordinary residents of an ordinary leased zone and get the normal zero-drop redirect with everybody else. They see an in-world line, not a disconnect warning: The way behind you closes, and you find yourself back where you entered.

The eject is bounded (~5 s per instance). A wedged instance costs the drain that pause once and then degrades to the pre-existing outcome — its occupants are flushed and reclaimed from durable state rather than redirected. Watch for drain: an instance did not finish ejecting its occupants in time and drain: could not even deliver an eject to an instance.

The reload freeze

A live instance is pinned to the content it was minted from: a hot reload performs no room reconcile and no Lua recompile against it. A builder's mid-run edit cannot delete the room a party is standing in, and "the run you started is the run you finish" is the instanced semantic anyway. The edit lands on the next mint — seconds to minutes away.

The reload readout names the pinned instances so this is not silent:

zone "crypt" has 2 live instance(s), which are PINNED to the content they were minted from: this reload does NOT reach them (no room reconcile, no Lua recompile). They pick it up when they are reaped and the next one is minted

Advisories are scoped to the reloaded pack, so an unrelated reload does not narrate every dungeon on the shard. There is a documented residual: the prototype cache is shared and is swapped by a reload, so an entity spawned inside a running instance after a reload gets the new prototype while the room graph stays old. See Content Loading & Hot Reload.

A content pull can be refused while parties are inside copies

A dungeon template is typically in no shard's zone pool — the raw template is not meant to be walkable, so nothing ever leases it. The content-pull prune guard therefore used to read "not hosted" for a template with live copies and parties inside them, and would strip the pack out from under them. That is worse than pruning a leased zone: shard memory is authoritative for a running zone, but instances are minted continuously, so the very next mint after the prune fails with "no such zone" — a runtime failure with no operator action in between.

Each shard now heartbeats a TTL'd in-use claim per template (dir:tmplinuse:<template> in the directory Redis, renewed every 15 s with a 45 s TTL, advertised immediately on the first mint). The prune guard consults it when the lease lookup comes back empty.

What you will see as an operator: a pull refused for a zone that appears in no lease anywhere, with the reason spelled out:

prune guard: zone has no lease but is LIVE as an instance template (parties are inside copies of it right now); the pull is refused until they leave

The remedies differ from the leased case. A leased zone means "drain it and retry." This means wait for the parties to finish, or drain the shard to eject them. The claim clears on its own roughly 45 s after the last copy is reaped, so a brief wait after the dungeon empties is normal and not a stuck state.

The guard fails closed: an unreachable directory propagates as an error rather than degrading to "nothing is using this, go ahead and prune." Also watch for instance template in-use publish failed; the claims may lapse and a content pull could prune a pack that has live instances — a lapsed claim is the precondition for exactly the fail-open this closes. See Content Pack Operations.

A rebalance can be deferred by an exit anchor

While a player is inside a copy, their exit anchor — the zone they entered from — is what the placement record names, and the placement record is the reconnect routing key. Handing that zone's lease to a peer mid-visit would route a reconnect to a shard that holds no session for them.

So a rebalance of a zone that is somebody's exit anchor is deferred, up to 3 minutes, and you will see it on the owning shard:

rebalance deferred: the zone is a live instance occupant's exit anchor; moving it now would route their reconnect to a shard that does not hold their session

The deferral takes the ordinary retry backoff and deliberately does not clear the coordinator's directive — telling the coordinator a move happened when the load has not shifted would make it re-plan against a wrong model — so the move is simply re-attempted. Once the budget is spent, the anchored occupants are ejected to their anchors and the move proceeds:

rebalance defer budget exhausted: ejecting instance occupants anchored to this zone so the move can proceed

The cap is not optional: a dungeon fed by a busy town has somebody inside essentially always, so an uncapped defer would pin that town's rebalance forever while every deferred cycle burns a coordinator cooldown. A steady stream of deferral lines for one zone is therefore expected and self-correcting; what would be worth investigating is deferrals never resolving into either a completed move or a budget-exhausted eject.

This guard fails open — a timeout or a wedged instance answers "not anchored" and the rebalance proceeds — because it buys playability, not correctness. A wedged load balancer would be the worse outcome.

The idle reaper

A copy is retired once it has been quiescent for 4 consecutive 15 s sweeps, after a 2-minute post-mint grace (so a copy nobody has entered yet is not reaped out from under its own party). A copy whose entrant never arrived is marked abandoned, which frees the account's cap slot immediately and skips the grace.

Sweeps retire up to 8 instances concurrently and each teardown runs on a detached context, so one wedged instance no longer delays every other reap behind it by the 10 s actor-teardown grace. A sweep still waits for its batch — successive ticks cannot pile goroutines onto the same wedged instance — which means on a shard where sweeps run long, the idle counter advances once per completed sweep rather than per wall-clock tick. Dead instances still occupy the per-shard cap until they are swept, so a shard full of wedged instances can refuse live mints. Watch instance reap deferred (ordinary: someone entered between the sample and the re-check) and reaped idle zone instance.

Scaling in & rolling upgrades: the zero-drop drain

Removing a shard — or rolling a new build — uses the zero-drop drain, which is implemented and signal-driven. On SIGINT/SIGTERM, before the zone loops tear down, the shard drains:

1. It stops accepting new fresh logins (but still accepts inbound handoff binds).
2. For each hosted zone it picks a peer and atomically flips the zone lease to that peer in a single fenced step, so the directory never sees an ownerless gap.
3. Each zone fans its players off in place — same zone id, same room, now owned by the peer — via a freeze/snapshot handoff. The player's socket stays open; they are redirected with no disconnect.
4. It waits (up to the deadline) for zones to empty; any stragglers are durably flushed and resume from durable state on reconnect (counted as reclaimed, not zero-drop).

What you trigger: an ordinary orchestrator rollout. SIGTERM a world pod (drain timeouts are ~45 s outer / 30 s per-zone); its zones and players migrate to a peer or standby; start the replacement, which claims the now-freed pool zones. Redirected players see no disconnect.

Two shutdown modes: a clean SIGTERM drains; an unexpected lease loss stops immediately without a drain (you can't hand off zones you no longer own). A single shard with no peer does a best-effort durable flush only.

Metrics (OpenTelemetry)

Every service can export OTLP/gRPC metrics, but export is off by default — it turns on only when OTEL_EXPORTER_OTLP_ENDPOINT (or the metrics-specific var) is set; otherwise every record is a negligible no-op. The dev path is: services push OTLP → otel-collector:4317 → re-exposed as Prometheus on :8889. A sysadmin scrapes :8889/metrics — the collector is the scrape surface; the Go services expose no /metrics endpoint. Logs and traces now ride the same OTLP path (into Loki and Tempo), and a busLag spike carries an exemplar to the trace that produced it. The full pipeline, the metric→trace→log pivot, the dev Grafana overlay, and the reference Kubernetes deployment are on the dedicated Observability page — this section is just the metric catalogue.

A bare endpoint silently exports nothing. OTEL_EXPORTER_OTLP_ENDPOINT is parsed as a URL, so otel-collector:4317 with no scheme reads as scheme otel-collector, empty address — the dial fails and metrics never leave the process. Use http://otel-collector:4317.

The instruments:

Metric	Type	Meaning
telos.zone.tick_lag_ms	histogram	Zone heartbeat overrun past the 250 ms budget — the headline scale signal.
telos.zone.occupancy	gauge (zone label)	Live players per zone. Labeled by the zone's template, so every instanced copy reports onto the template's series rather than minting a new one. This is a security boundary, not just tidiness: the instance id is player-mintable, so labeling by it would be a player-triggerable cardinality bomb — pinned by a regression test, and the same rule governs every span attribute (Observability).
telos.zone.instances	gauge (template label)	Live runtime-minted zone instances on this shard. Where instanced load is visible, since occupancy collapses copies onto one series.
telos.gate.connections	up/down counter	Live gate connections.
telos.gate.frames_dropped_total	counter	Outbound frames dropped for slow clients (shard-wide).
telos.world.builder_logs_dropped	counter (zone label)	Builder-Lua log lines dropped past the sustained rate limit. A non-zero rate means content is log-flooding — route source=builder_lua to short retention and look for the offending pack.
telos.bus.deliver_lag_ms	histogram	Scoped-event publish→deliver latency, live events only.
telos.bus.catchup_events_total	counter (subject label)	Durable scoped-event backlog events drained by a resuming consumer.
telos.bus.catchup_age_ms	histogram (subject label)	Age of each backlog event at delivery. Its max is how far behind that consumer is.
telos.commbus.durable_stalled_total	counter (stream label)	Durable messages still being redelivered well past the point routine transients clear — the recoverable window before a park. Counts messages, not attempts (it fires on one specific crossing delivery, so a single stuck message increments once). Scope identity rides the paired WARN log, not a label.
telos.commbus.durable_parked_total	counter (stream label)	Durable messages parked after exhausting the retry budget — permanent loss. Counted off the broker's MAX_DELIVERIES advisory (via a per-stream queue group, so once cluster-wide), which catches a park from ack-wait expiry or across a restart — not only an in-process final delivery.
telos.commbus.durable_poisoned_total	counter (stream label)	Durable messages dropped as undeliverable (malformed / unroutable).
telos.shard.drain_redirected_total	counter	Players handed to a peer during a graceful drain (socket kept open).
telos.shard.drain_reclaimed_total	counter (fault label)	Players dropped to reconnect at the drain deadline.

Reading the bus metrics during a recovery. deliver_lag_ms samples live deliveries only. A consumer resuming after an outage drains events published hours ago, and folding those into the same histogram would dwarf its distribution exactly when you need to read the live p99. The catch-up pair is where that story lives instead: catchup_events_total is the depth (how much this consumer missed), catchup_age_ms is the age (how far behind it is). Watch the age's max fall toward zero — that is the consumer converging. Both are labeled by subject, so you can see which scope is behind.

Honest observability gaps:

• telos-account now exports (metrics via the shared OTLP env, logs via the obs overlay); the dev compose runs no separate telos-director service, so nothing exports for that tier there. On Kubernetes the shared config wires world, gate, and account.
• Tracing is emitted but has no staging backend yet. The engine emits spans (handoff at 100 %, session-attach, the bus/zone-mailbox path) and the dev overlay renders them, but the reference k8s deployment still defers the Tempo backend — so on staging, spans currently go nowhere. See Observability → Traces.

Logging (slog)

Structured JSON to stdout, one global logger tagged with the service name. Levels are debug/info/warn/error via TELOS_LOG_LEVEL; DEBUG=1 (or true/yes/on) force-lowers to debug regardless of config (dev compose sets it). stdout is always the primary sink and your container/orchestrator pipeline can collect it as before. Optionally, setting TELOS_OTEL_LOGS=1 (with an OTLP endpoint) adds a second sink — slog is fanned to stdout and OTLP into the collector's Loki pipeline, independent of DEBUG. On Kubernetes the collector reads container stdout via filelog instead, so the bridge stays off there. See Observability → Logs.

Player input is not logged verbatim, and turning on debug can never change that. Several sites once logged the complete input line — a tell/say body, a channel post, or a mistyped link code / /login URL pasted into the game prompt — at Debug, and the panic-recovery line did it at Error (always on). That is benign as an ephemeral scroll but not once stdout is shipped into a durable, indexed store (the Grafana LGTM stack now pipes it into Loki — see Observability), and DEBUG is the first thing an operator flips to debug staging. So the raw line is dropped from every such site by default — player/character + sequence number remain, which is enough to trace flow — and re-attached only under a separate, explicit opt-in, TELOS_LOG_RAW_INPUT, that is deliberately not coupled to DEBUG. "Turn on debug logging" therefore can never mean "start recording player chat." Where the whole line is a credential (a single-token paste), the default logs only its length, enough to tell a typo from a paste without disclosing it. A CI/make verify guard (check-log-keys.sh) fails the build if a sensitive value (line/body/text/token/secret/ assertion/keywords) is used as a structured-log key — a defense-in-depth backstop for the mechanical single-line case, not a complete invariant (it can't see a value's provenance); the real guarantee is the reviewed gating, and the guard just fails the cheap class fast.

Builder Lua log output is bounded and labeled. Content's print / mud.log / director.log are capped, rate-limited, and tagged source=builder_lua so you can route content output to short-retention independently of engine logs; drops past the sustained rate surface on builder_logs_dropped. See Lua Sandbox Internals.

Log lines worth alerting/keying on:

• Boot & placement: starting; registered shard; claimed zone; won no zones … STANDBY; and the degradation lines postgres ready / redis unavailable; single-shard mode / nats unavailable; … disabled.
• Handoff/drain: signal received: draining before shutdown; drain complete (redirected/reclaimed counts); graceful drain incomplete; lease fence: stopping without drain.
• Leadership: director leadership changed; director lease campaign failed; stepping down; placement: rebalance drain recommended / zone needs (re)assignment.
• Slow clients: slow client wedged … gate write-deadline will reclaim.
• Instanced zones: minted zone instance / reaped idle zone instance; rebalance deferred: the zone is a live instance occupant's exit anchor and rebalance defer budget exhausted; prune guard: zone has no lease but is LIVE as an instance template; and the fail-open warning instance template in-use publish failed.
• Security refusals: any refusing to start (a fail-closed boot — a missing secret or a pack-set divergence; see Deployment); and TELOS_ALLOW_INSECURE warn lines where an insecure opt-in is active.

Degradation ladder (what breaks first)

The world tolerates dependency loss non-fatally: Redis down → single-shard mode, cross-shard exits sealed; Postgres down → ephemeral characters, hot-reload disabled; NATS down → comms/tells/hot-reload disabled. None of these crash a shard, but each narrows what works — watch the degradation log lines above.

Honest gaps (plan around these)

• Orphaned-zone failover is not instantaneous. There is no running-standby auto-adopt loop; recovery is ~lease TTL (15 s) + a reclaiming shard, not immediate.
• Drain-target admission is a soft ceiling, not a hard cap. The director-owned drain-target selector reserves a peer's headroom against a soft occupancy ceiling (default 1500), but if every live peer is reservation-full it proceeds over the ceiling rather than stall — a dropped connection is worse than the transient overload the rebalancer then corrects. The reservation is an admission hint (each with its own server-timed expiry), not a durable lock.
• The hot-zone ceiling is hard (one zone = one core; instancing gives private copies, not a shared zone split across cores), and the observability blind spots above (account/director not exporting OTLP in the shipped compose, no tracing) are real.

Related: Cross-Shard Handoff, Orchestration & Directors, Distributed Systems Model, Deployment.