Sysadmin Reference

Sysadmin Reference

Audience: Sysadmin Status: ✅ Ready

Standing up and operating a TelosMUD fleet in production: deploying the services, wiring OAuth, managing content, and scaling. This is the section landing page; work through the four guides in order, and use the production checklist below as your pre-flight.

The fleet at a glance

Four long-running services plus one-shot tools, all built from one Dockerfile (SERVICE build arg):

• telos-gate — the edge: terminates telnet (TLS), runs OAuth login, proxies players to world shards. Horizontal, stateless.
• telos-world — the simulation: hosts zone shards, one single-writer goroutine per zone. Horizontal by zone.
• telos-director — orchestration: placement planning, scope broadcasts, scheduled spawns. Leader-elected singleton per scope; optional (no published image — build it yourself), and the fleet runs without it.
• telos-account — auth: accounts, characters, trust tiers, the OAuth broker. Horizontal, stateless behind Postgres.
• One-shots/tools: telos-migrate (schema), telos-seed (dev content import), telos-pull (versioned content install), telos-botswarm (load testing).

Dependencies to provision: Postgres (durable state), Redis (directory/leases/presence/ device-auth), NATS + JetStream (comms + events).

The four guides

1. Deployment — images and the service-role model, the port map and the "one public edge" firewall posture, transports & TLS, the fail-closed boot gates, bare-process vs. container, and the cross-link to the cloud IaC repo.
2. OAuth Setup — production GitHub OAuth: a real domain, secure cookies, a persistent session key, and the gate↔account trust secrets.
3. Content Pack Operations — the external versioned store, telos-pull/ImportVersion, the enabled-set registry, and director-coordinated pull/reload.
4. Running at Scale — per-component scaling, claim-from-pool placement, the zero-drop drain for rolling upgrades, metrics, and logging.

Production checklist

Before you go live:

• Supply a TLS cert + key for the gate (TELOS_GATE_TLS_LISTEN/CERT/KEY) and leave plain telnet off — there is no built-in cert.
• Set the trust secrets: the shared TELOS_ACCOUNT_CALLER_TOKEN, the account signing key and the world verify key, and the handoff keypair.
• Set none of the TELOS_ALLOW_INSECURE gates — supply the real secrets instead so the fail-closed boots stay closed.
• Configure production OAuth: the GitHub App on your real domain, TELOS_WEB_PUBLIC_URL, TELOS_WEB_SECURE_COOKIES=1, and a persistent TELOS_WEB_SESSION_KEY.
• Keep the gRPC mesh and datastores private — only the gate's TLS port and the account :8080 broker face the internet.
• Run telos-migrate then telos-seed/telos-pull before any world starts.
• Wire OTel (OTEL_EXPORTER_OTLP_ENDPOINT) if you want metrics — export is off by default.
• Pin the first admin with TELOS_BOOTSTRAP_ADMIN (a GitHub login) — see First Admin Setup.

The real reference deployment (Terraform + Kustomize + k3s, bare-metal or cloud) lives in the separate telosMUD-infra repo; this section is the source of truth for the gomud-side artifacts and configuration.

Cross-shard handoff: authenticated zone adoption

Every inter-shard handoff RPC that mutates state is authenticated with the shared handoff keypair (WithHandoffKeys): Prepare binds the carried player snapshot, and AdoptZone — which makes a shard host a draining peer's zone — binds zone_id, the destination shard, and the zone lease's generation.

A captured AdoptZone is therefore worthless at any other shard, and worthless at its own destination once the handover it authorized has completed. It is a single-use token for one specific handover, not a time-bounded capability: the directory increments a zone's lease generation on every ownership change, and the source's own lease flip is an ownership change. There is no clock anywhere in this path.

• A shard with no verify key refuses inbound handoffs outright unless TELOS_ALLOW_INSECURE is set.
• A keyed shard always enforces the signature. TELOS_ALLOW_INSECURE cannot loosen it.
• Refusals log locally with the reason. adopt zone refused: stale lease generation means the handover this request authorized already completed, or another shard won the flip — normal under a racing drain, and a replay otherwise. adopt zone refused: signature authentication failed means a wrong key or a forgery.
• adopt zone refused: could not read the zone's lease generation means the destination cannot reach Redis. The fence needs a directory read, so a drain now needs the directory reachable from both peers. It fails closed: that zone degrades from a zero-drop redirect to reclaim-from-durable (players reconnect).

Run the directory Redis with maxmemory-policy noeviction (or a volatile-* policy — never allkeys-*). A zone's lease generation is load-bearing coordination state, not cache. It is stored without a TTL so it cannot lapse, but an allkeys-* policy will evict it anyway under memory pressure. The counter is seeded from the Redis clock rather than from zero precisely so that a wipe, an eviction, or a restore from an older snapshot restarts it above every value it ever issued — but do not rely on that as the first line of defense.

Rolling-upgrade note. AdoptZone signing is a fail-closed change, in both directions. While a rollout is in flight, a not-yet-upgraded shard cannot produce a request an upgraded destination will accept, so its drains toward an upgraded shard are refused — the drain errors and the zone is retained by the source (never lost, never double-owned). An upgraded source draining toward an old destination is refused too (the old shard's clock-skew check sees no issue time). There is no way to temporarily loosen a keyed cluster to ride this out. So: do not rely on graceful drain to fully empty a node mid-rollout, and complete the rollout promptly to restore drain liveness.

Durable-message delivery: what "never lost" covers

Tells (COMMS_TELL) and durable scoped events (WORLD_EVENTS) ride JetStream. A delivery that fails for a transient reason — the target is mid cross-shard handoff, the gate bus blipped, a zone is overloaded — is retried on a ramp-then-hold backoff schedule (200ms, 1s, 3s, 10s, 30s, then holding at 30s) across 10 attempts, ≈ 164 s of covered outage. Anything that fails for the whole window is parked, which is permanent loss.

• Alert on telos.commbus.durable_parked_total (labeled by stream). Any non-zero value means an outage outlived the entire retry schedule — an incident, not routine.
• telos.commbus.durable_poisoned_total counts malformed/unroutable messages dropped without retry. Routine and bounded, but a spike means bad content is being published.
• Consumers run MaxAckPending=1, so one stuck message head-of-line-blocks that consumer for up to the full window before parking. A player whose tells stall for ~2–3 minutes and then resume is this mechanism.

The guarantee is precise: never lost to any transient shorter than the window, from publish through the world→gate handoff. The final world→gate frame is at-most-once — a gate that drops it (slow-consumer overflow, mid-reconnect) loses the message. There is no end-to-end render ack.

Upgrade note (durable consumer config). The per-player durable consumers are long-lived and never deleted, so a build that changes BackOff / MaxAckPending / MaxDeliver updates them in place via CreateOrUpdateConsumer. Confirm your nats-server version accepts those updates on an existing consumer. If you see CreateOrUpdateConsumer config-conflict errors after a rollout, that player simply gets no tell delivery — their tells stay durable in the stream, so it is an availability blip, not data loss. Deleting the COMMS_TELL / WORLD_EVENTS consumers once is safe: the per-sender delivered-cursor makes the re-drain idempotent, so the only visible effect is a one-time backlog replay, paced by MaxAckPending=1 and the login drain pace.