Sysadmin Reference

Sysadmin Reference

Audience: Sysadmin Status: ✅ Ready

Standing up and operating a TelosMUD fleet in production: deploying the services, wiring OAuth, managing content, and scaling. This is the section landing page; work through the four guides in order, and use the production checklist below as your pre-flight.

The fleet at a glance

Four long-running services plus one-shot tools, all built from one Dockerfile (SERVICE build arg):

• telos-gate — the edge: terminates telnet (TLS), runs OAuth login, proxies players to world shards. Horizontal, stateless.
• telos-world — the simulation: hosts zone shards, one single-writer goroutine per zone. Horizontal by zone.
• telos-director — orchestration: placement planning, scope broadcasts, scheduled spawns. Leader-elected singleton per scope; optional (no published image — build it yourself), and the fleet runs without it.
• telos-account — auth: accounts, characters, trust tiers, the OAuth broker. Horizontal, stateless behind Postgres.
• One-shots/tools: telos-migrate (schema), telos-seed (dev content import), telos-pull (versioned content install), telos-botswarm (load testing).

Dependencies to provision: Postgres (durable state), Redis (directory/leases/presence/ device-auth), NATS + JetStream (comms + events).

The four guides

1. Deployment — images and the service-role model, the port map and the "one public edge" firewall posture, transports & TLS, the fail-closed boot gates, bare-process vs. container, and the cross-link to the cloud IaC repo.
2. OAuth Setup — production GitHub OAuth: a real domain, secure cookies, a persistent session key, and the gate↔account trust secrets.
3. Content Pack Operations — the external versioned store, telos-pull/ImportVersion, the enabled-set registry, and director-coordinated pull/reload.
4. Running at Scale — per-component scaling, claim-from-pool placement, the zero-drop drain for rolling upgrades, metrics, and logging.

Production checklist

Before you go live:

• Supply a TLS cert + key for the gate (TELOS_GATE_TLS_LISTEN/CERT/KEY) and leave plain telnet off — there is no built-in cert.
• Set the trust secrets: the shared TELOS_ACCOUNT_CALLER_TOKEN, the account signing key and the world verify key, and the handoff keypair.
• Set none of the TELOS_ALLOW_INSECURE gates — supply the real secrets instead so the fail-closed boots stay closed.
• Configure production OAuth: the GitHub App on your real domain, TELOS_WEB_PUBLIC_URL, TELOS_WEB_SECURE_COOKIES=1, and a persistent TELOS_WEB_SESSION_KEY.
• Keep the gRPC mesh and datastores private — only the gate's TLS port and the account :8080 broker face the internet.
• Run telos-migrate then telos-seed/telos-pull before any world starts.
• Wire OTel (OTEL_EXPORTER_OTLP_ENDPOINT) if you want metrics — export is off by default.
• Pin the first admin with TELOS_BOOTSTRAP_ADMIN (a GitHub login) — see First Admin Setup.

The real reference deployment (Terraform + Kustomize + k3s, bare-metal or cloud) lives in the separate telosMUD-infra repo; this section is the source of truth for the gomud-side artifacts and configuration.

Cross-shard handoff: authenticated zone adoption

Every inter-shard handoff RPC that mutates state is authenticated with the shared handoff keypair (WithHandoffKeys): Prepare binds the carried player snapshot, and AdoptZone — which makes a shard host a draining peer's zone — binds zone_id, the destination shard, and the zone lease's generation.

A captured AdoptZone is therefore worthless at any other shard, and worthless at its own destination once the handover it authorized has completed. It is a single-use token for one specific handover, not a time-bounded capability: the directory increments a zone's lease generation on every ownership change, and the source's own lease flip is an ownership change. There is no clock anywhere in this path.

• A shard with no verify key refuses inbound handoffs outright unless TELOS_ALLOW_INSECURE is set.
• A keyed shard always enforces the signature. TELOS_ALLOW_INSECURE cannot loosen it.
• Refusals log locally with the reason. adopt zone refused: stale lease generation means the handover this request authorized already completed, or another shard won the flip — normal under a racing drain, and a replay otherwise. adopt zone refused: signature authentication failed means a wrong key or a forgery.
• adopt zone refused: could not read the zone's lease generation means the destination cannot reach Redis. The fence needs a directory read, so a drain now needs the directory reachable from both peers. It fails closed: that zone degrades from a zero-drop redirect to reclaim-from-durable (players reconnect).

The directory Redis should run maxmemory-policy noeviction

telos-world, telos-director and telos-gate read the eviction configuration at startup and log an Error if this Redis is set to evict. They do not refuse to start — see the caveat at the end of this section, which you should read before "fixing" a warning.

The directory is coordination state, not cache. Every key in it is a fact the fleet cannot re-derive, so losing one produces a wrong answer, not a slow one.

Not TTL'd (so only allkeys-*, FLUSHDB, a lossy failover, or an old restore can take them):

key	what its loss does
dir:zone:<id>	the lease owner + its monotonic generation — the fence that makes a signed AdoptZone a single-use token for one handover rather than a standing capability. Stored with PERSIST for exactly this reason
dir:player:<id>	the placement epoch that stops a stale handoff routing a player back to a shard they left

TTL'd — and these are why volatile-* is not an acceptable alternative either. Each one fails open: its absence isn't an error, it's a confident wrong answer.

key	what its loss does
dir:lease:<id>	the director's leader-election lease. The claim script refuses only when a different owner holds a live lease, so an evicted key lets the incumbent's renewal and a standby's claim both succeed — two directors both believing they lead, issuing rebalance directives against the same zones
dir:shard:<id>	the endpoint registration. It refuses a different endpoint only while a live registration exists, so eviction lets two processes sharing a shard id both register — defeating the guard against a duplicated shard id becoming two writers
dir:tmplinuse:<template>	parties are inside live instances of this template. It carries the shortest TTL in the directory, so volatile-ttl evicts it first by construction, and its absence is indistinguishable from "nobody is running copies" — letting a content pull strip the pack out from under them
dir:draining:<shard>	the draining marker; losing it sends peers at a shard on its way out
dir:occ:<zone>, dir:cooldown:<zone>	rebalancer inputs: a busy zone reads as empty, and the anti-thrash guard disappears. Every extra rebalance bumps a zone's lease generation
dir:drainresv:<target>	reserved drain headroom. Listed last on purpose — that ceiling is deliberately soft (a caller proceeds over it rather than stall a drain), so its loss is real but not in the class of the two above

Earlier guidance here allowed volatile-*, reasoning that the zone hash is PERSISTed and therefore immune. That is true of the zone hash and false of the directory as a whole.

This Redis is not only the directory. The same instance carries the checkpoint tier — full character JSON on a 1h TTL, the ~10s rung of the durability ladder. Under any volatile-* policy those are prime eviction candidates, and losing them silently demotes that rung to save-on-logout: a state-rollback surface across a crash.

PERSIST also does not defend against FLUSHDB/FLUSHALL, a failover to a replica that lost writes, or an RDB/AOF restore from an older point. The lease generation is seeded from the Redis clock rather than from zero precisely so a wipe restarts it above every value it ever issued — a backstop, not a first line of defense, and one that assumes the clock never jumps backwards by more than a zone's ownership-change count.

Effects replication is required. redis.call('TIME') in a write script is only safe under effects replication (Redis 5+, mandatory in 7+). The drain-target reservations rely on it, and so does the lease generation's clock seeding.

Read this before setting noeviction

The check warns rather than refusing because noeviction is not unambiguously safe on a shared instance, and the directory currently shares one with the checkpoint tier, the presence roster and the account service's device-auth codes. Config exposes a single Redis address, so the policy is necessarily instance-wide, and checkpoints — not directory keys — dominate the memory.

• With no maxmemory ceiling, noeviction never returns OOM to clients. Redis grows until the container limit and is OOM-killed, wiping every lease, generation and placement epoch at once — the FLUSHALL-equivalent this whole section is about, caused by the remedy.
• With a ceiling set, writes at the ceiling error. That includes lease renewal, which fences shards — and since they share the instance, fleet-wide and simultaneously.

So: if this Redis serves only the directory, set noeviction. If it is shared, give the directory its own instance first, then set noeviction on it (the directory keyspace is small and bounded) and leave the cache instance on an evicting policy with a sensible ceiling. Setting noeviction on a shared, cache-dominated instance without a ceiling is worse than the warning you are trying to silence.

Note also the asymmetry if a future release makes this fatal: an unreachable Redis degrades to single-shard and boots, while a reachable one with a bad policy would refuse. An operator under pressure could "fix" a boot refusal by firewalling Redis and land in a worse state than either branch intends.

Runbook: recovering from directory data loss

A wipe is fail-closed, which is the right direction but is not self-healing. After a reset, a drain reads curGen == 0 for an un-reclaimed zone and is refused — so a Redis data-loss event escalates into stuck drains rather than into a security hole. The remedy is to let each shard re-claim its zones (they do this on their normal lease cadence) and then re-run the drain. If drains stay refused, confirm the shards are actually renewing leases before touching anything else.

Rolling-upgrade note. AdoptZone signing is a fail-closed change, in both directions. While a rollout is in flight, a not-yet-upgraded shard cannot produce a request an upgraded destination will accept, so its drains toward an upgraded shard are refused — the drain errors and the zone is retained by the source (never lost, never double-owned). An upgraded source draining toward an old destination is refused too (the old shard's clock-skew check sees no issue time). There is no way to temporarily loosen a keyed cluster to ride this out. So: do not rely on graceful drain to fully empty a node mid-rollout, and complete the rollout promptly to restore drain liveness.

Tuning the Lua sandbox caps

Two sandbox bounds are settable per deployment. Both default to the compiled-in values, so an untouched deployment is byte-identical to before they existed, and config.example.yaml ships them commented out (setting them pins them, and an explicit deadline defeats the automatic -race scaling).

Setting	Env	Default	Bounds
tunables.lua_instr_budget	TELOS_LUA_INSTR_BUDGET	100,000	must be reachable inside the deadline
tunables.lua_call_deadline_ms	TELOS_LUA_CALL_DEADLINE_MS	5 ms	must stay below one zone pulse

Set them as a pair, or don't set them. They are not independent: at the 5 ms default the instruction budget stops firing at roughly 850k instructions, and past that the wall clock always wins. Raising the budget to 10M therefore does not raise the primary bound — it disables it, and silently weakens the circuit breaker with it (a runaway then aborts on the deadline, which is weighted 0.1 rather than the 0.5 an instruction abort carries, precisely so transient host load can't quarantine a correct script). A script failing four calls in five would stop tripping the breaker at all. The engine refuses such a pair at boot and tells you the deadline the budget would need — see Lua Sandbox Internals.

Operational notes:

• A malformed TELOS_LUA_* value refuses the boot rather than falling back to the default, so a typo can never leave you believing a setting took effect when it didn't.
• A deadline at or above the zone pulse is refused: a Lua call that outlives a heartbeat stops combat rounds and affect ticks for every player in that zone.
• Nothing caps memory. A single call at the budget ceiling can allocate ~355 MB in a table-building loop, so raising the ceiling trades a bounded stall for an OOM-kill. Raise it only with a container memory limit you have actually tested against.

Durable-message delivery: what "never lost" covers

Tells (COMMS_TELL) and durable scoped events (WORLD_EVENTS) ride JetStream. A delivery that fails for a transient reason — the target is mid cross-shard handoff, the gate bus blipped, a zone is overloaded — is retried on a ramp-then-hold backoff schedule (200ms, 1s, 3s, 10s, 30s, then holding at 30s) across 10 attempts, ≈ 164 s of covered outage. Anything that fails for the whole window is parked, which is permanent loss.

• Alert on telos.commbus.durable_parked_total (labeled by stream). Any non-zero value means an outage outlived the entire retry schedule — an incident, not routine. The count comes from the broker's authoritative MAX_DELIVERIES advisory (a per-stream queue group counts each park once cluster-wide), so it also catches a park from a hung ack or across a restart, not just an in-process final delivery — but treat it as an alert signal, not an exact ledger: the advisory is ephemeral core NATS, so a park while the whole fleet is down at that instant isn't replayed.
• telos.commbus.durable_poisoned_total counts malformed/unroutable messages dropped without retry. Routine and bounded, but a spike means bad content is being published.
• Consumers run MaxAckPending=1, so one stuck message head-of-line-blocks that consumer for up to the full window before parking. A player whose tells stall for ~2–3 minutes and then resume is this mechanism.

The guarantee is precise: never lost to any transient shorter than the window, from publish through the world→gate handoff. The final world→gate frame is at-most-once — a gate that drops it (slow-consumer overflow, mid-reconnect) loses the message. There is no end-to-end render ack.

Upgrade note (durable consumer config). The per-player durable consumers are long-lived and never deleted, so a build that changes BackOff / MaxAckPending / MaxDeliver updates them in place via CreateOrUpdateConsumer. Confirm your nats-server version accepts those updates on an existing consumer. If you see CreateOrUpdateConsumer config-conflict errors after a rollout, that player simply gets no tell delivery — their tells stay durable in the stream, so it is an availability blip, not data loss. Deleting the COMMS_TELL / WORLD_EVENTS consumers once is safe: the per-sender delivered-cursor makes the re-drain idempotent, so the only visible effect is a one-time backlog replay, paced by MaxAckPending=1 and the login drain pace.