Sysadmin Reference

Sysadmin Reference

Audience: Sysadmin Status: ✅ Ready

Standing up and operating a TelosMUD fleet in production: deploying the services, wiring OAuth, managing content, and scaling. This is the section landing page; work through the four guides in order, and use the production checklist below as your pre-flight.

The fleet at a glance

Four long-running services plus one-shot tools, all built from one Dockerfile (SERVICE build arg):

• telos-gate — the edge: terminates telnet (TLS), runs OAuth login, proxies players to world shards. Horizontal, stateless.
• telos-world — the simulation: hosts zone shards, one single-writer goroutine per zone. Horizontal by zone.
• telos-director — orchestration: placement planning, scope broadcasts, scheduled spawns. Leader-elected singleton per scope; optional (no published image — build it yourself), and the fleet runs without it.
• telos-account — auth: accounts, characters, trust tiers, the OAuth broker. Horizontal, stateless behind Postgres.
• One-shots/tools: telos-migrate (schema), telos-seed (dev content import), telos-pull (versioned content install), telos-botswarm (load testing).

Dependencies to provision: Postgres (durable state), Redis (directory/leases/presence/ device-auth), NATS + JetStream (comms + events).

The four guides

1. Deployment — images and the service-role model, the port map and the "one public edge" firewall posture, transports & TLS, the fail-closed boot gates, bare-process vs. container, and the cross-link to the cloud IaC repo.
2. OAuth Setup — production GitHub OAuth: a real domain, secure cookies, a persistent session key, and the gate↔account trust secrets.
3. Content Pack Operations — the external versioned store, telos-pull/ImportVersion, the enabled-set registry, and director-coordinated pull/reload.
4. Running at Scale — per-component scaling, claim-from-pool placement, the zero-drop drain for rolling upgrades, metrics, and logging.

Production checklist

Before you go live:

• Supply a TLS cert + key for the gate (TELOS_GATE_TLS_LISTEN/CERT/KEY) and leave plain telnet off — there is no built-in cert.
• Set the trust secrets: the shared TELOS_ACCOUNT_CALLER_TOKEN, the account signing key and the world verify key, and the handoff keypair.
• Set none of the TELOS_ALLOW_INSECURE gates — supply the real secrets instead so the fail-closed boots stay closed.
• Configure production OAuth: the GitHub App on your real domain, TELOS_WEB_PUBLIC_URL, TELOS_WEB_SECURE_COOKIES=1, and a persistent TELOS_WEB_SESSION_KEY.
• Keep the gRPC mesh and datastores private — only the gate's TLS port and the account :8080 broker face the internet.
• Run telos-migrate then telos-seed/telos-pull before any world starts.
• Wire OTel (OTEL_EXPORTER_OTLP_ENDPOINT) if you want metrics — export is off by default.
• Pin the first admin with TELOS_BOOTSTRAP_ADMIN (a GitHub login) — see First Admin Setup.

The real reference deployment (Terraform + Kustomize + k3s, bare-metal or cloud) lives in the separate telosMUD-infra repo; this section is the source of truth for the gomud-side artifacts and configuration.

Cross-shard handoff: authenticated zone adoption

Every inter-shard handoff RPC that mutates state is authenticated with the shared handoff keypair (WithHandoffKeys): Prepare binds the carried player snapshot, and AdoptZone — which makes a shard host a draining peer's zone — binds zone_id, the destination shard, and the zone lease's generation.

A captured AdoptZone is therefore worthless at any other shard, and worthless at its own destination once the handover it authorized has completed. It is a single-use token for one specific handover, not a time-bounded capability: the directory increments a zone's lease generation on every ownership change, and the source's own lease flip is an ownership change. There is no clock anywhere in this path.

• A shard with no verify key refuses inbound handoffs outright unless TELOS_ALLOW_INSECURE is set.
• A keyed shard always enforces the signature. TELOS_ALLOW_INSECURE cannot loosen it.
• Refusals log locally with the reason. adopt zone refused: stale lease generation means the handover this request authorized already completed, or another shard won the flip — normal under a racing drain, and a replay otherwise. adopt zone refused: signature authentication failed means a wrong key or a forgery.
• adopt zone refused: could not read the zone's lease generation means the destination cannot reach Redis. The fence needs a directory read, so a drain now needs the directory reachable from both peers. It fails closed: that zone degrades from a zero-drop redirect to reclaim-from-durable (players reconnect).

The directory Redis should run maxmemory-policy noeviction

telos-world, telos-director and telos-gate read the eviction configuration at startup. What happens next depends on whether you have told them the directory has its own Redis (redis.directory_addr):

Shape	redis.directory_addr	On an evicting policy
Shared (the default)	unset	Error logged, boots anyway — read the caveat below first
Dedicated	set	Fatal — refuses to boot

Dedicated is fatal because nothing on that instance wants eviction: the keyspace is small and bounded, and there is no worse configuration an operator could be ordered into. Shared stays warn-only because the safe-for-coordination remedy is genuinely dangerous on a cache-sized instance — an operator running allkeys-lru there may be right for the Redis they actually have.

Three further behaviours worth knowing:

• An unreadable configuration never refuses. CONFIG GET is disabled on most managed Redis, so refusing there would make the engine unbootable on exactly those platforms; the probe falls back to INFO memory.
• The periodic re-check is warn-only on both shapes. That asymmetry with the boot gate is deliberate: a live shard owns zone leases and player sockets, so exiting over a CONFIG SET would convert an operator mistake into an outage plus an unplanned lease handover, self-inflicted at the worst moment. Boot is the only place where refusing is cheap.
• A configured-but-unreachable coordination Redis is fatal, rather than falling back to the cache instance — a fallback would silently place coordination state on the instance policied for eviction.

The directory is coordination state, not cache. Every key in it is a fact the fleet cannot re-derive, so losing one produces a wrong answer, not a slow one.

Not TTL'd (so only allkeys-*, FLUSHDB, a lossy failover, or an old restore can take them):

key	what its loss does
dir:zone:<id>	the lease owner + its monotonic generation — the fence that makes a signed AdoptZone a single-use token for one handover rather than a standing capability. Stored with PERSIST for exactly this reason
dir:player:<id>	the placement epoch that stops a stale handoff routing a player back to a shard they left

TTL'd — and these are why volatile-* is not an acceptable alternative either. Each one fails open: its absence isn't an error, it's a confident wrong answer.

key	what its loss does
dir:lease:<id>	the director's leader-election lease. The claim script refuses only when a different owner holds a live lease, so an evicted key lets the incumbent's renewal and a standby's claim both succeed — two directors both believing they lead, issuing rebalance directives against the same zones
dir:shard:<id>	the endpoint registration. It refuses a different endpoint only while a live registration exists, so eviction lets two processes sharing a shard id both register — defeating the guard against a duplicated shard id becoming two writers
dir:tmplinuse:<template>	parties are inside live instances of this template. It carries the shortest TTL in the directory, so volatile-ttl evicts it first by construction, and its absence is indistinguishable from "nobody is running copies" — letting a content pull strip the pack out from under them
dir:draining:<shard>	the draining marker; losing it sends peers at a shard on its way out
dir:occ:<zone>, dir:cooldown:<zone>	rebalancer inputs: a busy zone reads as empty, and the anti-thrash guard disappears. Every extra rebalance bumps a zone's lease generation
dir:drainresv:<target>	reserved drain headroom. Listed last on purpose — that ceiling is deliberately soft (a caller proceeds over it rather than stall a drain), so its loss is real but not in the class of the two above

Earlier guidance here allowed volatile-*, reasoning that the zone hash is PERSISTed and therefore immune. That is true of the zone hash and false of the directory as a whole.

This Redis is not only the directory. The same instance carries the checkpoint tier — full character JSON on a 1h TTL, the ~10s rung of the durability ladder. Under any volatile-* policy those are prime eviction candidates, and losing them silently demotes that rung to save-on-logout: a state-rollback surface across a crash.

PERSIST also does not defend against FLUSHDB/FLUSHALL, a failover to a replica that lost writes, or an RDB/AOF restore from an older point. The lease generation is seeded from the Redis clock rather than from zero precisely so a wipe restarts it above every value it ever issued — a backstop, not a first line of defense, and one that assumes the clock never jumps backwards by more than a zone's ownership-change count.

Effects replication is required. redis.call('TIME') in a write script is only safe under effects replication (Redis 5+, mandatory in 7+). The drain-target reservations rely on it, and so does the lease generation's clock seeding.

Read this before setting noeviction

On the shared shape the check warns rather than refusing, because noeviction is not unambiguously safe on a shared instance. With redis.directory_addr unset, the directory shares one Redis with the checkpoint tier, the presence roster and the account service's device-auth codes — and maxmemory-policy is server-wide, so the policy is necessarily chosen by whichever tier is loudest, which is the checkpoint tier (full character JSON on a 1 h TTL), not the directory keys.

• With no maxmemory ceiling, noeviction never returns OOM to clients. Redis grows until the container limit and is OOM-killed, wiping every lease, generation and placement epoch at once — the FLUSHALL-equivalent this whole section is about, caused by the remedy.
• With a ceiling set, writes at the ceiling error. That includes lease renewal, which fences shards — and since they share the instance, fleet-wide and simultaneously.

So the fix is to split, not to flip the policy. Give the directory its own Redis, point redis.directory_addr at it, and set noeviction there (that keyspace is small and bounded); leave the cache instance on an evicting policy with a sensible ceiling. Declaring the split is what upgrades the check from a warning to a boot refusal, so the engine then holds the guarantee for you. Setting noeviction on a shared, cache-dominated instance without a ceiling is worse than the warning you are trying to silence.

Note the asymmetry now that the dedicated shape is fatal: an unreachable shared Redis degrades to single-shard and boots, while a reachable dedicated one with a bad policy refuses. An operator under pressure could "fix" a boot refusal by firewalling Redis and land in a worse state than either branch intends — so fix the policy, never the reachability. (A configured coordination Redis that is unreachable is itself fatal, precisely to close the tempting middle path of silently falling back to the evicting instance.)

Runbook: recovering from directory data loss

A wipe is fail-closed, which is the right direction but is not self-healing. After a reset, a drain reads curGen == 0 for an un-reclaimed zone and is refused — so a Redis data-loss event escalates into stuck drains rather than into a security hole. The remedy is to let each shard re-claim its zones (they do this on their normal lease cadence) and then re-run the drain. If drains stay refused, confirm the shards are actually renewing leases before touching anything else.

Rolling-upgrade note. AdoptZone signing is a fail-closed change, in both directions. While a rollout is in flight, a not-yet-upgraded shard cannot produce a request an upgraded destination will accept, so its drains toward an upgraded shard are refused — the drain errors and the zone is retained by the source (never lost, never double-owned). An upgraded source draining toward an old destination is refused too (the old shard's clock-skew check sees no issue time). There is no way to temporarily loosen a keyed cluster to ride this out. So: do not rely on graceful drain to fully empty a node mid-rollout, and complete the rollout promptly to restore drain liveness.

Tuning the Lua sandbox caps

Two sandbox bounds are settable per deployment. Both default to the compiled-in values, so an untouched deployment is byte-identical to before they existed, and config.example.yaml ships them commented out (setting them pins them, and an explicit deadline defeats the automatic -race scaling).

Setting	Env	Default	Bounds
tunables.lua_instr_budget	TELOS_LUA_INSTR_BUDGET	100,000	must be reachable inside the deadline
tunables.lua_call_deadline_ms	TELOS_LUA_CALL_DEADLINE_MS	5 ms	must stay below one zone pulse

Set them as a pair, or don't set them. They are not independent: at the 5 ms default the instruction budget stops firing at roughly 850k instructions, and past that the wall clock always wins. Raising the budget to 10M therefore does not raise the primary bound — it disables it, and silently weakens the circuit breaker with it (a runaway then aborts on the deadline, which is weighted 0.1 rather than the 0.5 an instruction abort carries, precisely so transient host load can't quarantine a correct script). A script failing four calls in five would stop tripping the breaker at all. The engine refuses such a pair at boot and tells you the deadline the budget would need — see Lua Sandbox Internals.

Operational notes:

• A malformed TELOS_LUA_* value refuses the boot rather than falling back to the default, so a typo can never leave you believing a setting took effect when it didn't.
• A deadline at or above the zone pulse is refused: a Lua call that outlives a heartbeat stops combat rounds and affect ticks for every player in that zone.
• Memory has its own per-call cap (a string-allocation budget), so a single script call can no longer allocate without bound — an important guard, since the worst case at shipping defaults was a s = s .. s doubling loop reaching 64 GB. Note it is a separate dimension: no instruction budget bounds memory (bytes-per-instruction spans five orders of magnitude), so raising lua_instr_budget still raises heap pressure — a call at the ceiling can allocate hundreds of MB in a table-building loop. Raise it only with a container memory limit you have actually tested against.

Tuning the instanced-zone caps

The per-account cap, the per-shard cap and the mint burst/window are settable the same way (tunables.*, TELOS_*), following the shape the Lua caps settled on: fail-closed at boot, 0 means the compiled default and never "unlimited", and a refused set changes nothing. Every malformed TELOS_* value is reported, not just the last — six knobs on one surface makes multiple typos ordinary.

These are not free-standing workload bounds. Two cross-field invariants are enforced, not merely documented:

• instances_per_shard is bounded by the drain's instance-eject barrier, which ejects every instance under one shared 5 s deadline sized against the default. Overshoot doesn't crash — it drops occupants to straggler reclaim on every rolling deploy, which is a far quieter failure and therefore a worse one.
• The mint burst is meaningless without its window, so the pair is validated as a rate.

One correction worth carrying: the per-account cap does not bound mint churn. Slot reservation excludes abandoned records from the per-account count, so mint-abandon-mint is bounded per account by the rate limit alone. If you are tuning against churn, that is the knob.

Every bound is per process, so a fleet's effective ceiling is the per-shard cap times the shard count — see Running at Scale.

Durable-message delivery: what "never lost" covers

Tells (COMMS_TELL) and durable scoped events (WORLD_EVENTS) ride JetStream. A delivery that fails for a transient reason — the target is mid cross-shard handoff, the gate bus blipped, a zone is overloaded — is retried on a ramp-then-hold backoff schedule (200ms, 1s, 3s, 10s, 30s, then holding at 30s) across 10 attempts, ≈ 164 s of covered outage. Anything that fails for the whole window is parked, which is permanent loss.

• Alert on telos.commbus.durable_stalled_total (labeled by stream) — this is the one that gives you time to act, firing while the message is still recoverable rather than after it is lost. It matters most on WORLD_EVENTS, and the reason is MaxAckPending = 1: a stalled message blocks every later message on its consumer for the whole retry window, and that stream has one consumer per scope for the entire fleet. So the incident it signals is not "a message is being retried" but "this scope's orchestration has applied nothing for N seconds" — treat it accordingly. It counts messages rather than attempts, so one increment means one stuck message, not a flurry of routine hiccups; the paired WARN log carries the scope identity.

  Known miss: a wedged handler blocks its consumer entirely, so the callback is never re-entered and the crossing delivery never reaches the counter. The broker still parks the message, so a wedged actor produces the permanent-loss alarm with no early warning — and a wedged actor is a prime stall cause. A restart also under-counts. Don't treat a silent stall counter as proof nothing is stuck.

• Alert on telos.commbus.durable_parked_total (labeled by stream). Any non-zero value means an outage outlived the entire retry schedule — an incident, not routine. The count comes from the broker's authoritative MAX_DELIVERIES advisory (a per-stream queue group counts each park once cluster-wide), so it also catches a park from a hung ack or across a restart, not just an in-process final delivery — but treat it as an alert signal, not an exact ledger: the advisory is ephemeral core NATS, so a park while the whole fleet is down at that instant isn't replayed.
• telos.commbus.durable_poisoned_total counts malformed/unroutable messages dropped without retry. Routine and bounded, but a spike means bad content is being published.
• Consumers run MaxAckPending=1, so one stuck message head-of-line-blocks that consumer for up to the full window before parking. A player whose tells stall for ~2–3 minutes and then resume is this mechanism.

The guarantee is precise: never lost to any transient shorter than the window, from publish through the world→gate handoff. The final world→gate frame is at-most-once — a gate that drops it (slow-consumer overflow, mid-reconnect) loses the message. There is no end-to-end render ack.

Upgrade note (durable consumer config). The per-player durable consumers are long-lived and never deleted, so a build that changes BackOff / MaxAckPending / MaxDeliver updates them in place via CreateOrUpdateConsumer. Confirm your nats-server version accepts those updates on an existing consumer. If you see CreateOrUpdateConsumer config-conflict errors after a rollout, that player simply gets no tell delivery — their tells stay durable in the stream, so it is an availability blip, not data loss. Deleting the COMMS_TELL / WORLD_EVENTS consumers once is safe: the per-sender delivered-cursor makes the re-drain idempotent, so the only visible effect is a one-time backlog replay, paced by MaxAckPending=1 and the login drain pace.