History / Running at Scale

Observability 3: distributed tracing (gomud #464-#470) The metric->trace->log pivot is real. Fills out the Observability page Traces section (previously "none yet") and corrects the now-stale "no tracing" claims in Running-at-Scale. - Observability (Traces rewrite + intro + diagram): the tracing foundation (TracerProvider on the same OTLP env gate, W3C propagator set unconditionally, head sampling with a 100% carve-out); what is traced — handoff/AdoptZone at 100% (root spans, per-hop children, error-distinguishing failures), session- attach (bounded to the handshake; the Play stream is a context anchor not a span), and the comms bus / zone mailbox (traceparent in the ENVELOPE not NATS headers so MemBus works too; span LINK not parent-child because JetStream is at-least-once; queue_wait_ms at dequeue); the deliberate no-per-command-span decision; the metric->trace exemplar pivot on busLag (and why tick_lag_ms has none); the trace->log traceHandler; and the cardinality-is-a-security-boundary discipline (metricZone template-only, no per-player subjects, gate host not host:port, baggage-free producer links). Documents the dev-vs-staging split: the compose overlay renders the full pivot via otel-lgtm bundled Tempo, but the reference k8s deployment still defers the Tempo backend. - Running-at-Scale: corrected "traces not wired yet"; reframed the occupancy template-label as the #470 security boundary (regression-pinned). - Cross-Shard-Handoff: noted this is the one path traced at 100%.

yfronto committed Jul 21, 2026

Observability 2: the LGTM stack (gomud #458/#459/#481 + infra #13-#20) New Observability page consolidating the telemetry pipeline now that it is real and deployed on staging; the wiki previously had no Loki/Grafana content beyond the forward-references from Observability 1, which are now made concrete. - NEW Observability.md (Sysadmin): OTLP -> collector -> Grafana LGTM pipeline with a flow diagram; metrics (incl. the URL-scheme gotcha #458 that silently dropped all export); logs via the slog OTLP bridge TELOS_OTEL_LOGS (#459), mutually exclusive with the k8s filelog path so nothing double-ships; traces deferred (Tempo empty until tracing ships); the make up-obs dev overlay; and the telosMUD-infra reference k8s deployment (#13-#20) — retention as node protection (Loki 7d, Prometheus 7d+2GB), the collector DaemonSet, guarded public Grafana (own hostname + Traefik basicAuth in front of Grafana login), default-deny NetworkPolicy, and the validate-on-PR + manual-approval-prod CI. Registered in the sidebar and the Sysadmin index. - Running-at-Scale: corrected now-stale claims — "only metrics are wired / no tracing exporter" and "no file sink or log shipper in code" — since logs now ride OTLP into Loki; refreshed the honest-gaps note (account now exports; no director service in dev compose); added the endpoint-scheme warning; made the Observability-1 "planned LGTM" forward-reference concrete. - Content-Loading-and-Hot-Reload + Lua-Sandbox-Internals: the content-DTO log cap (#481) — builder-controlled field values (a 200KB dice/formula field or a reference target) were echoed verbatim into problem strings; now clamped to ~1KB at the source and at the capProblems funnel, via a shared internal/logcap leaf byte-identical to the Lua CapLogMsg. Notes the #454 key-guard does not catch these (keys are area/event/op/value). - Deployment: cross-linked the infra observability layer and the obs overlay.

yfronto committed Jul 21, 2026

Observability 1 milestone: log + port hardening (prelude to the LGTM stack) A security milestone that hardens what the logs and the dev stack expose, ahead of Observability 2 shipping container stdout into Loki — which turns an ephemeral scroll into a durable, indexed archive and makes these exposures real rather than theoretical. - Running-at-Scale (Logging): player input is no longer logged verbatim (#454). Several sites logged the full input line — a tell/say body, a channel post, or a mistyped link code / /login URL — at Debug, and the panic line at Error. Now dropped by default (player+seq remain), re-attached only under a separate TELOS_LOG_RAW_INPUT opt-in deliberately decoupled from DEBUG, with a single- token paste logging only its length. A check-log-keys.sh guard in make verify fails on a sensitive value used as a log key — a backstop for the mechanical case, not a complete invariant. - Lua-Sandbox-Internals (new section) + Pack-Lua-Scripting + Running-at-Scale metric: builder Lua logging is bounded (#456). print/mud.log/director.log had no length cap or rate limit — an always-on write primitive once stdout hits Loki. Length cap (~1KB), a per-CALL line budget reset per FRAME (charges the flooder, not a co-firing victim) that trips the breaker, and a per-runtime wall-clock token bucket (50 lines/s) that bounds a self-rescheduling timer; labeled source=builder_lua; drops on builder_logs_dropped. Notes the two bypasses closed at the source — err.Error() and compile-error echo. - Deployment: the dev Compose stacks now bind every host port to 127.0.0.1 (#455/#479), not 0.0.0.0 — NATS 4222 is a full unauthenticated data plane, plus 8222/4317/8889, redis, postgres default creds, the OAuth bridge and the plaintext-telnet gates. Container-to-container over the compose bridge is unaffected; framed explicitly as dev-stack hardening, not production policy.

yfronto committed Jul 20, 2026

Round 41 milestone: director & durable-bus reliability (#354/#355/#356/#390 + the forged-event fix) Corrects a stale claim: Orchestration said "No region directors are deployed, so signal_region currently has no consuming director". #356 shipped them. - Orchestration-and-Directors (new section) + Pack-Entity-Reference: per-region director scripts (#356). Region scope had been WRITE-DEAD — content published durable region signals and zones applied region deltas, but nothing consumed the subject or wrote region state. Records the three hazards found building it (an omitted ref minted a PHANTOM WORLD director sharing the world lease, instance id and consumer name; the consumer id was not injective across . and :; the tier hardcoded the demo pack) and the disproved premise that a director losing leadership leaks its VM — the proposed fix would have crashed the process on a late signal. - Orchestration-and-Directors: a signal is NAK d when its scope-state write never landed (#354). The ack consumed the event FLEET-WIDE because every director binds the same durable consumer name. Widened to any store error, recorded at the single write chokepoint (a handler return is defeated by pcall, by closure composition, and by the schedule-state path). Includes the prerequisite defect that inverts intuition: a restarted director CASed on a cold cache version of 0, so ErrCASLost meant "cold cache" far more often than "concurrent writer" and NAKing on it would have requeued on every restart. - Orchestration-and-Directors: the down-broadcast carries the store version and replicas fence on it (#355) — with the issue s framing corrected in two directions (churn does not reproduce; and there is NO re-read, so a stale replica stays wrong until the next write or a reseed). Why the version must be the store s, and the three easily-inverted rules (0 = unversioned, unknown key applies, deletes record too). - Scoped-Event-Bus + Pack-Lua-Scripting: a zone script could forge a director-only event, because signal-up and broadcast-down share one subject and shards core-subscribe it. #355 made it worse — a forged high version permanently freezes a key fleet-wide. Notes why boss.died is deliberately NOT in the shared reserved list, and the residual (a NATS-credentialed publisher still reaches it; needs the subjects split). - Running-at-Scale + Sysadmin-Reference + Scoped-Event-Bus: durable_stalled_total (#390) — the recoverable window before a park. Explains why it is an orchestration outage rather than a delayed message (MaxAckPending=1 blocks the whole per-scope consumer) and carries the known miss: a wedged handler never reaches the counter, so the park alarm can fire with no early warning.

yfronto committed Jul 20, 2026

Round 38 milestone: owner_epoch — ownership as a durable fence (#432 + #413/#379/#384) CORRECTS A FALSE SAFETY CLAIM. Three pages asserted that state_version "fences stale writes, so a mis-fired handoff or a zombie owner can never clobber a newer record". It never did: state_version is contention control, and every caller answered a CAS loss by rebasing, so a stale writer rebased exactly as a legitimate one did. - Persistence-and-Durability: "The state_version guard" reworked into "Two guards" — state_version (contention) vs owner_epoch (ownership, at the SINK). Documents the rollback primitive it closed (a stale shard 60s-old logout snapshot force-writing over the live owner, repeatable), why the fence had to be a SECOND column (a rebase moves state_version and cannot reach a separate conjunct), why the mint is atomic (a read-then-bump would look fixed), the three-outcome SaveResult, terminal epoch loss on both save paths, the max(directory,row) floor, fail-closed login, the checkpoint tier ownership guard (it had NO CAS and was a full bypass one rung up; detects a double-own ~6x sooner), the (owner_epoch, state_version) load ordering, and the honest residual (an epoch on row X cannot fence a write to row Y). - Distributed-Systems-Model + Overview: same correction; adds a "do not confuse the two epochs" invariant (directory epoch fences ROUTING and is evictable; owner_epoch fences WRITES and is durable). - Cross-Shard-Handoff: handoff mints the epoch AFTER destination resolution (minting first evicted legitimate players mid-move); an unreadable store now fails the handoff closed; concurrent claims derive distinct epochs hence distinct handoff tokens, closing the shared-pending race structurally. - Zone-Runtime: #413 extends the resolve-then-deliver claim to login attach and Prepare (incl. why neither may reuse claimTransferTarget, and Prepare needing a release on post failure); #379 refuses a reconnect racing an in-flight transfer in TWO layers, since a probe reproduced a dupe the resolve-time check alone cannot see. - #384: documents why the drain-reservation margin is not bound by step 1 duration (the hold is rebased per accumulate and anchored to the redirect it covers) and the WARN added to falsify it; corrects the Round 31 "stamped once and never refreshed" wording on three pages.

yfronto committed Jul 19, 2026

Document instanced zones across all three audiences (#422) Round 36 shipped instanced zones (#72) and Round 37 changed four things about them; only the engine-developer page existed, and it predated the Round 37 work. Builder — new page, Building Instanced Zones: - the `instanceable: true` opt-in and why it is opt-in (a mint runs the zone's boot resets, so every copy is a generation faucet that also routes around every in-world gate: a private copy has no doorman) - the start_room requirement, enforced at mint, and both reasons for it - mud.send_to_instance is self-only, and what "the invoking actor" actually resolves to per attach point: a custom command or an ability acts as the PLAYER, but on("enter")/on("greet") act as the room/mob, so an entity trigger cannot send the entrant in. The dungeon-door idiom is a verb the player types - mud.zone() and why `ev.zone ~= "darkwood"` matches inside every copy - the exhaustive does-not-work list with the why for each: persistent resets (durable item dupe), timed repop (farmable), signal_region / signal_world (the envelope drops source), reserved director schedule events, nesting, the reload freeze - the engine-owned player-facing lines, and the caps to design around Sysadmin — a new section in Running at Scale plus the prune-guard remedy in Content Pack Operations: - the caps ARE PER PROCESS, so an account gets perAccount x shards; and they are not config knobs today (compile-time defaults + a Go option, no TELOS_INSTANCE_* env var wires them) - shard-local and unleased: not in the placement pool, never rebalanced, never a handoff destination, and they die with the shard - metrics label by template because an instance id is unbounded, player-driven cardinality; logs go the other way and keep the id. Adds the missing telos.zone.instances gauge to the metrics table - drain step 0 ejects occupants to their exit anchor - the reload freeze and the pinned-instance advisory - Round 37: a mint builds from the live content snapshot and refuses an incomplete build (#418); the TTL'd dir:tmplinuse claim, so a content pull can be REFUSED while parties are inside copies and clears ~45s after the last one (#416); the concurrent bounded reaper sweep (#419); and the up-to-3-minute rebalance deferral with its "rebalance deferred" log lines (#421) Mudlib — Instanced Zones: - the routing-vs-isolation rule spelled out (the room map is the isolation boundary; the zone-segment test is only a pre-filter), and what the build-failing lint actually covers - the "never mint on a zone goroutine" contract with the reason, the three-hop entry, why the request carries the origin zone POINTER, and why abandonInstance is a flag rather than a record deletion - minting from the live snapshot, and the incomplete-build refusal - the concurrent reaper: the detached per-teardown context, and why the bound is a smoothing bound rather than a correctness one - the dir:tmplinuse claim as the cheapest possible analogue of a lease, its own goroutine, and the mint kick - the anchor's invariant DECAYS, the pre-handoverZoneTo guard and why the placement of that guard is the load-bearing property, why the anchors are queried rather than maintained, and why it fails open

MrG0rd0x committed Jul 19, 2026

Round 36 milestone: instanced zones (#72) New deep-dive page Instanced-Zones, plus corrections to two pages that asserted "instancing is deferred / not implemented". - NEW Instanced-Zones: template-vs-id identity and the ownsZoneRef locality chokepoint (with the build-failing AST lint); why the id is <template>#<128-bit random> (# outside the ref charset, unguessable so it is not a farming oracle); unleased by construction; async 3-hop mint + quiescence reaper; the instanceable content opt-in; the isolation exclusions (persistent resets fail closed vs a durable item dupe, per-mint RNG salting, withheld director schedule events, refused signal_region/world, drain/reload/off-box-ingress); THE ANCHOR (template projection was killed because placement is the gate ROUTING key); exit paths; and the security posture. - Distributed-Systems-Model + Running-at-Scale (x2): instancing now exists but is ISOLATION, not transparent load-sharding — the public hot-zone ceiling stands. - Zone-Runtime: quiescence gains the third `incoming` counter for in-flight intra-shard transfers, claimed under the same mutex teardown checks. - Cross-Shard-Handoff + RPC-and-Protobuf: signed-snapshot optionals now written UNCONDITIONALLY — a second append-if-non-empty optional made digest(tier="",account=X) == digest(tier=X,account="") (length prefixes solve boundaries, not presence); instances rejected at every off-box ingress. - Persistence: an instance is never a durable location; the anchor + COALESCE. - Pack-Entity-Reference: zone `instanceable` flag, start_room respawn semantics. - Pack-Lua-Scripting: mud.zone(), mud.send_to_instance (self-only). - Orchestration: instances withheld from reserved schedule fan-out. - Sidebar + Engine-Developer-Reference index entries.

yfronto committed Jul 19, 2026

Round 32 milestone: comms & JetStream durability residuals - Scoped-Event-Bus: MaxAckPending is now a per-consumer option (default 1, double-clamped vs NATS 0==unlimited) rather than a hardcoded transport constant; both ordered consumers keep the serializing posture, and >1 is the lever a future seen-set reorder-tolerant consumer would pull (#312). - Running-at-Scale + Sysadmin-Reference: durable_parked_total is now counted off the broker MAX_DELIVERIES advisory via a per-stream queue group (once cluster-wide), so it also catches an ack-wait-expiry or across-restart park, not just an in-process final delivery — an alert signal, not an exact ledger (the advisory is ephemeral) (#311). - Distributed-Systems-Model: placement is an existence/routing oracle, not a liveness one; a "currently connected?" decision (e.g. the new-mail ping) reads the presence roster and only to gate, never to route (#325). - Content-Loading-and-Hot-Reload: the comms hear-set republish retry and the zone-shape reconcile retry run on separate goroutine budgets at parity, so a reconcile storm cannot starve the security-relevant republish class (#345). (#371 async-who prompt ordering is an internal fix with no documented surface.)

yfronto committed Jul 13, 2026

Round 31 milestone: handoff phase-2 auth, checkpoint tie-break, drain-reservation TTL - Cross-Shard-Handoff + RPC-and-Protobuf: document that Handoff.Commit/Abort are now Ed25519-signed under the cluster keypair with the digest bound to the destination shard id (#314) — the deterministic handoff_token over public inputs was forgeable, and destination binding stops a captured signed Abort from being replayed against a second destination in a split-brain race. - Persistence-and-Durability + Distributed-Systems-Model: the load freshness check breaks a state_version tie toward the Redis checkpoint (#322); a checkpoint is dumped at the pre-CAS version so it ties the row while carrying newer content — the strict > left the tier inert, so the tie-break is what makes the crash window the ~10s cadence and not the ~60s flush. - Cross-Shard-Handoff + Running-at-Scale + Distributed-Systems-Model: the drain-target reservation TTL now outlasts the whole drain (deadline + one presence-reflect window) so a slow-but-alive drain keeps its hold while players migrate; only a crashed drainer lingers the full TTL (#334).

yfronto committed Jul 13, 2026

Round 27: the AdoptZone fence cluster (#316/#288/#327) The round self-documented #315 (generation fence) on RPC + Sysadmin. Closing the rest: - Cross-Shard-Handoff: new 'Zone adoption: build, confirm, or un-adopt' — the destination requires from_shard_id to be the zone's live owner and refuses before building (#316, a correctness precondition, not a security barrier since owner+gen share one read); adoption is confirmed by observing the lease flip LAND (via ClaimZone), never by the RPC returning; an unconfirmed runtime adoption is un-adopted when its confirm deadline expires, gated on an 'adopted' flag so boot zones fence rather than self-delete (#327). - Zone-Runtime-and-Actor-Model: new 'Runtime zone teardown' — UnhostZone + the per-zone actor context; the four guards (disposable-not-unowned; quiescence is not pop==0; post selects on a dead channel or the saver drainer wedges the whole shard; publish+arm under one lock) (#288). - RPC-and-Protobuf: the from_shard_id line now notes the live-owner precondition. - Running-at-Scale: the source tears down a rebalanced-away zone (no zombie per migration).

yfronto committed Jul 10, 2026

Round 26: the placement record becomes the reconnect-routing spine Track-6 residuals. Round self-documented only #276 (catch-up metrics); the rest of the engine/ops surface was stale. - Distributed-Systems-Model: NEW 'The placement record' section — zone is the routing key (ShardForZone resolves the current owner, so an offline rebalance is transparent), epoch is the fence AND the tell/mail existence key, shard routes nothing. Every residency registers a placement (incl. the intra-shard zone walk); registerPlacement accepts an equal epoch where the handoff CAS demands a strictly greater one. Clean logout writes a fenced compare-and-delete tombstone, not a delete (#320 slice 2, #70). Also: lease renewal restarts on re-adoption, closing an A->B->A split-brain (#288). - Persistence-and-Durability: close the shard-not-zone 'Known gap'; add the saver shutdown sentinel barrier (#282). - Edge-and-Protocol: world-side Send watchdog — keepalive can't see a gate whose transport acks PINGs while its application stopped reading (#274). - Orchestration-and-Directors: a runtime-adopted zone's scope replica is seeded before the zone is exposed (a full-map replace would clobber newer state) (#280). - Cross-Shard-Handoff / Running-at-Scale / DistSys: drain-target reservations now carry per-field, server-timed expiry (#284). - Running-at-Scale: drop the self-contradicting 'deliver_lag_ms silent' gap; the gate row now describes zone-keyed reconnect routing.

yfronto committed Jul 10, 2026

scale: document the durable scoped-event catch-up metrics; drop the stale deliver_lag_ms gap

MrG0rd0x committed Jul 10, 2026

Scaling docs: reflect shipped load-aware placement rebalancer Track 6 drain/scale landed: occupancy-weighted rebalancing that executes (director issues rebalance-drain directives with cooldown + locality colocation) and director-owned, reservation-based drain-target selection. Updates the now-stale 'advisory-only / naive drain-select' language on Running-at-Scale, Distributed-Systems-Model, Cross-Shard-Handoff, and Orchestration-and-Directors.

yfronto committed Jul 9, 2026

Sysadmin Reference: deployment, OAuth, content ops, scale - Deployment: image/service model, port map + exposure posture, TLS, one-public-edge firewall, fail-closed gates; cross-links telosMUD-infra IaC - Sysadmin-OAuth-Setup: production GitHub OAuth + exact trust-secret env keys - Content-Pack-Operations: versioned store, telos-pull, registry, CI gate - Running-at-Scale: per-component scaling, drain/rolling-upgrade, OTel, logging (honest about advisory-only rebalancing, one-zone-one-core, obs gaps) - Sysadmin-Reference landing + production checklist Completes the Sysadmin Reference section (5 pages).

yfronto committed Jul 9, 2026

Scaffold wiki + author About and Getting Started - 48-page skeleton across all audiences (flat files + grouped _Sidebar) - Every page carries an Audience + Status banner (Ready/Draft/Placeholder) - About: Overview, History of MUDs (Ready) - Getting Started: Getting-Started, Running-Locally, First-Admin-Setup, Content-Packs-Intro (Ready) — dev bypass, local GitHub OAuth, bootstrap admin, promote/demote, content-pack model, all grounded in code - SRD5/WoW/D6 pack pages are honest placeholders (packs not yet built)

yfronto committed Jul 9, 2026