Round 49: NATS subject authorization (#554 + #552 slices 1-3 + infra #35)
Closes the unauthenticated-NATS residual the Round 39 (#424) and Round 41
(#355/#474) notes pointed at: a bare TCP connection to :4222 no longer has
full publish rights on every subject.
- Deployment: rewrote the Round-39 "restrict it ... until you have" content-bus
callout into the shipped per-identity authorization matrix (world/gate/
director/seed via TELOS_NATS_USER/PASSWORD). Captures the load-bearing
lessons: gate = deny=[">"] (a denylist leaves $JS.EVENT.ADVISORY.* forgeable);
world AND director both publish content.invalidate (a naive matrix breaks
admin reload); an empty allow=[] is ALLOW-ALL in NATS; NATS doesn't hot-reload
(k8s ships a content-hashed configMapGenerator); the four passwords must be
stable + all set (NATS refuses an unset $VAR). Folded in #554 (the in-process
handle ACL: the gate is denied publish on the whole telos.comms. root).
Updated the NATS ports row + softened the dev-loopback "unauthenticated"
wording (the matrix now defends wrong-identity; dev passwords are shared).
- Content-Loading-and-Hot-Reload: the #424 residual's "deployment-side fix"
now exists — a forgery requires already holding a trusted identity.
- Scoped-Event-Bus: narrowed the #355 forged-down-broadcast residual — the
matrix denies the untrusted gate outright, leaving only a world/director
credential (still needs the up/down subject split to fully close).
- Sysadmin-Reference: the four required-stable NATS passwords in the production
checklist.
Round 45: launch hardening — TLS-port reject, ref-length lint, transfer-audit detectability (#486/#483/#443)
- #486 TLS-only port plaintext reject: Sysadmin-Reference (the "no login prompt"
section now describes the ACTIVE reject, not the old silent hang — with a note
that a still-hanging connection means a stale gate binary) + Edge-and-Protocol
(the first-byte sniff: 0x16 replays via peekConn into tls.Server, else a
one-line "use TLS" + close; the sniff runs off the accept loop, and the
handshake is driven under an explicit HandshakeContext deadline, closing the
resource-exhaustion vector a deadline-less lazy handshake left open).
- #483 identity-token LENGTH bound: Content-Loading (ref length added to the
lint list) + Pack-Authoring (a ref is a store btree PRIMARY KEY, ~2704-byte
page ceiling; RefMaxLen=256; also covers the two content-authored store keys
the charset lint skips — pack name and formula-override name — since an
over-long value fails the import transaction identically; boot warn + reload
hard-reject).
- #443 cross-character transfer audit: Persistence-and-Durability — new
item_transferred event (a transient Released marker records the releaser; a
DIFFERENT saved player's pickup emits the row) reusing the #350 audit trail;
covers drop->get and put->get-from-container; self-pickup/floor-item/mob-move
record nothing; marker cleared on pickup; bound items never reach it.
DETECTION ONLY (no prevention, no conservation invariant). Threaded into the
#432 owner_epoch residual note as the first half ("we cannot know" -> "we can
know") of the cross-row dupe close-out deferred to Launch.
Infra: retarget the reference deployment from Oracle k3s to AWS EKS (infra #27-#33)
The infra repo migrated off single-node Oracle Always-Free k3s onto AWS EKS,
which invalidated several things the wiki asserted — including two dead doc
links and one piece of load-bearing REASONING.
- Deployment: the IaC cross-link now describes AWS EKS and points at the single
README; DEPLOYMENT.md/RUNBOOK.md/PLAN.md were collapsed and DELETED upstream,
so both links were dead. Added a migration note listing what moved
(local-path->gp3, Traefik->ingress-nginx, gate via AWS NLB with
externalTrafficPolicy: Local to preserve client source IP, OCI Object
Storage->S3, TF state on S3 native lockfile, CI via GitHub OIDC->IAM role).
- Observability: the retention rationale CHANGED, not just the platform. The old
argument was node-level (local-path PVCs are directories on the node root FS
and do not enforce size, so a runaway backend takes the game down); on EKS the
gp3 PVCs ARE size-bounded, so the blast radius is now the backend itself —
worse observability, not a downed game. Policy values are unchanged (Loki 7d,
Prometheus 7d+2GB, ~7GiB) and NodeDiskFillingUp now guards emptyDir/image
churn. Grafana basic-auth moved Traefik->ingress-nginx (missing Secret => 503,
fail-closed); NetworkPolicy enforcement is the VPC CNI and the ingress source
is the ingress-nginx namespace, NOT kube-system (a silent breakage if ported
across unchanged). Added the cert split (HTTP-01 for web/Grafana, DNS-01 for
the gate — its own NLB is unreachable to HTTP-01) and the one-click lifecycle
(external-dns + cert-manager via IRSA, per-env delegated Route53 subzone so a
compromised staging pod cannot touch production names).
- Sysadmin-Reference: the data-loss exposure changed — gp3 EBS survives node
replacement (retiring the user_data-forces-replace hazard), but the one-click
`down` drains EBS before destroy, so teardown is the real destructive path.
pg-backup now targets S3 into a Terraform-provisioned versioned bucket;
restore docs moved to the README.
Sysadmin: connecting to a TLS-only gate + backups/DR operations
From the production bring-up: document the 'no login prompt' symptom (a plain
telnet client against a TLS-only gate gets nothing — the top false 'prod is
down' alarm) with how to verify via openssl s_client and connect via a TLS MUD
client, and where the login flow's failure modes actually live (transport vs
OAuth config vs gate->account wiring). Add a Backups & DR section: the single-
node data-loss exposure (incl. user_data-forces-replace), the reference
pg-backup CronJob + dedicated-key guidance, the ~60s RPO, and the load-bearing
restore caveat (plain-SQL replay into the populated DB is a silent no-op; use
pg_restore --clean with the app scaled to zero, or a selective player-state
restore).
Observability 2: the LGTM stack (gomud #458/#459/#481 + infra #13-#20)
New Observability page consolidating the telemetry pipeline now that it is real
and deployed on staging; the wiki previously had no Loki/Grafana content beyond
the forward-references from Observability 1, which are now made concrete.
- NEW Observability.md (Sysadmin): OTLP -> collector -> Grafana LGTM pipeline
with a flow diagram; metrics (incl. the URL-scheme gotcha #458 that silently
dropped all export); logs via the slog OTLP bridge TELOS_OTEL_LOGS (#459),
mutually exclusive with the k8s filelog path so nothing double-ships; traces
deferred (Tempo empty until tracing ships); the make up-obs dev overlay; and
the telosMUD-infra reference k8s deployment (#13-#20) — retention as node
protection (Loki 7d, Prometheus 7d+2GB), the collector DaemonSet, guarded
public Grafana (own hostname + Traefik basicAuth in front of Grafana login),
default-deny NetworkPolicy, and the validate-on-PR + manual-approval-prod CI.
Registered in the sidebar and the Sysadmin index.
- Running-at-Scale: corrected now-stale claims — "only metrics are wired /
no tracing exporter" and "no file sink or log shipper in code" — since logs
now ride OTLP into Loki; refreshed the honest-gaps note (account now exports;
no director service in dev compose); added the endpoint-scheme warning; made
the Observability-1 "planned LGTM" forward-reference concrete.
- Content-Loading-and-Hot-Reload + Lua-Sandbox-Internals: the content-DTO log
cap (#481) — builder-controlled field values (a 200KB dice/formula field or a
reference target) were echoed verbatim into problem strings; now clamped to
~1KB at the source and at the capProblems funnel, via a shared internal/logcap
leaf byte-identical to the Lua CapLogMsg. Notes the #454 key-guard does not
catch these (keys are area/event/op/value).
- Deployment: cross-linked the infra observability layer and the obs overlay.
Round 41 milestone: director & durable-bus reliability (#354/#355/#356/#390 + the forged-event fix)
Corrects a stale claim: Orchestration said "No region directors are deployed,
so signal_region currently has no consuming director". #356 shipped them.
- Orchestration-and-Directors (new section) + Pack-Entity-Reference: per-region
director scripts (#356). Region scope had been WRITE-DEAD — content published
durable region signals and zones applied region deltas, but nothing consumed
the subject or wrote region state. Records the three hazards found building
it (an omitted ref minted a PHANTOM WORLD director sharing the world lease,
instance id and consumer name; the consumer id was not injective across . and
:; the tier hardcoded the demo pack) and the disproved premise that a director
losing leadership leaks its VM — the proposed fix would have crashed the
process on a late signal.
- Orchestration-and-Directors: a signal is NAK d when its scope-state write
never landed (#354). The ack consumed the event FLEET-WIDE because every
director binds the same durable consumer name. Widened to any store error,
recorded at the single write chokepoint (a handler return is defeated by
pcall, by closure composition, and by the schedule-state path). Includes the
prerequisite defect that inverts intuition: a restarted director CASed on a
cold cache version of 0, so ErrCASLost meant "cold cache" far more often than
"concurrent writer" and NAKing on it would have requeued on every restart.
- Orchestration-and-Directors: the down-broadcast carries the store version and
replicas fence on it (#355) — with the issue s framing corrected in two
directions (churn does not reproduce; and there is NO re-read, so a stale
replica stays wrong until the next write or a reseed). Why the version must
be the store s, and the three easily-inverted rules (0 = unversioned, unknown
key applies, deletes record too).
- Scoped-Event-Bus + Pack-Lua-Scripting: a zone script could forge a
director-only event, because signal-up and broadcast-down share one subject
and shards core-subscribe it. #355 made it worse — a forged high version
permanently freezes a key fleet-wide. Notes why boss.died is deliberately NOT
in the shared reserved list, and the residual (a NATS-credentialed publisher
still reaches it; needs the subjects split).
- Running-at-Scale + Sysadmin-Reference + Scoped-Event-Bus:
durable_stalled_total (#390) — the recoverable window before a park. Explains
why it is an orchestration outage rather than a delayed message
(MaxAckPending=1 blocks the whole per-scope consumer) and carries the known
miss: a wedged handler never reaches the counter, so the park alarm can fire
with no early warning.
Round 40 milestone: instance residuals + operator tunables (#436/#438/#435/#429/#425)
Corrects claims from the Round 37 sync: "nothing caps memory" (three places)
and the #340 eviction guidance, whose stated prerequisite now exists.
- Lua-Sandbox-Internals (new section) + Sysadmin-Reference + Pack-Lua-Scripting:
the per-call string-allocation budget (#438). Documents why the filed premise
was impossible — bytes-per-instruction spans FIVE orders of magnitude (25
instructions, 63MB), so no instruction budget bounds memory, and heap sampling
cannot work because a neighbour goroutine s noise floor would let a player
quarantine content they do not own. The real vector is the `..` OPCODE (every
string BUILTIN was already capped; an opcode has no wrapper): at shipping
defaults `s = s .. s` reaches 64 GB. Charged in the fork BEFORE the join.
Also the misclassification fix — a memory bomb used to land in AbortDeadline
(weight 0.1, "transient host load"); new AbortAlloc weighs with AbortBudget.
Keeps the review lesson that "does not amplify" != "does not allocate".
- Sysadmin-Reference + Deployment: redis.directory_addr splits coordination from
cache (#429). The eviction check is now shape-dependent — SHARED stays
warn-only, DEDICATED is FATAL — which supplies the prerequisite the previous
guidance said was missing, so the advice changes from "flip the policy" to
"declare the split". Periodic re-check stays warn-only on both shapes
(exiting over a live CONFIG SET would cause the outage it prevents), and a
configured-but-unreachable coordination Redis is fatal rather than falling
back onto the evicting instance.
- Instanced-Zones + Sysadmin-Reference: the instance caps are operator-tunable
(#436), with the two ENFORCED cross-field invariants that disproved the
issue s "no ordering invariant" premise (instances_per_shard vs the drain
eject barrier; burst validated with its window as a rate), plus the corrected
claim that per-account bounds churn — it does not.
- Instanced-Zones + Building-Instanced-Zones + Pack-Entity-Reference +
Pack-Lua-Scripting: the declared instance entrance (#435). Documents why
option 2 (relaxing self-only) was rejected — standing in a room is not
consent, greet/enter share an actor, and the mint bills the VICTIM s account —
and why entrances live in their own map so no push-the-player path can
traverse a door it cannot see.
Round 37: operator-tunable Lua caps (#368) + runtime zones from live content (#418)
Round 37 was largely self-documenting: #340 landed in 133d670 and #422 (the
"document instanced zones" issue) in cf8a00d, which also covered #416, #419,
#421 and the mint half of #418. Verified those against the merged commits;
this fills the two remaining gaps.
- Lua-Sandbox-Internals (new section) + Pack-Lua-Scripting + Sysadmin-Reference
(new operator section): the instruction budget and call deadline are now
per-deployment tunables, and are a PAIR — at the 5ms default the budget stops
firing near 850k instructions, so raising it alone disables the primary bound
and silently weakens the circuit breaker (an instruction abort weighs 0.5, a
deadline abort 0.1, so every runaway gets reclassified as transient and a
script failing 4-in-5 stops tripping). ValidateCaps rejects such a pair;
New clamps it structurally; validation sits at the injection point to keep
config a leaf; a malformed TELOS_LUA_* refuses boot. InstrPerMS corrected
100k -> 20k after measurement showed real throughput (~90k/ms arithmetic,
~37k/ms allocating) was at or BELOW the old constant — an error in the unsafe
direction that let mis-paired configs pass. Also: a deadline may not reach a
pulse, and memory is uncapped (~355MB at the ceiling = OOM-kill risk).
- Content-Loading-and-Hot-Reload (new subsection): zones built after boot now
use live content. The shard snapshot was written once at construction, so a
post-boot HostZone or instance mint combined boot-version content with
live-cache prototypes; a deleted template still minted and a withdrawn
instanceable never took effect. Now an atomic.Pointer refreshed by the
reloader, one snapshot threaded through validate+build. Documents the two
states this exposed: an incomplete runtime build now refuses to publish
(a roomless HostZone would adopt, renew, and drop drained players while the
directory reported it healthy), and region membership resolved once.
Sysadmin: the directory Redis must not evict coordination state (#340)
Replaces the earlier guidance that allowed a volatile-* policy. That was
right about the zone hash (PERSISTed, so volatile eviction cannot reach
it) and wrong about the directory as a whole: the TTL'd keys each turn a
single-writer guard OFF when evicted — the director's leader-election
lease (two directors both believing they lead), the shard registration
(two processes sharing a shard id both registering), and the
instance-template claim, which carries the shortest TTL in the directory
and is therefore what volatile-ttl takes first.
Adds the shared-instance caveat an operator needs before acting on the
warning: this Redis also carries the checkpoint tier, so noeviction
without a memory ceiling risks an OOM-kill that wipes the whole
directory — worse than the problem. Plus the effects-replication
requirement and a runbook line for the post-wipe stuck-drain tail.
Round 32 milestone: comms & JetStream durability residuals
- Scoped-Event-Bus: MaxAckPending is now a per-consumer option (default 1,
double-clamped vs NATS 0==unlimited) rather than a hardcoded transport
constant; both ordered consumers keep the serializing posture, and >1 is the
lever a future seen-set reorder-tolerant consumer would pull (#312).
- Running-at-Scale + Sysadmin-Reference: durable_parked_total is now counted
off the broker MAX_DELIVERIES advisory via a per-stream queue group (once
cluster-wide), so it also catches an ack-wait-expiry or across-restart park,
not just an in-process final delivery — an alert signal, not an exact ledger
(the advisory is ephemeral) (#311).
- Distributed-Systems-Model: placement is an existence/routing oracle, not a
liveness one; a "currently connected?" decision (e.g. the new-mail ping)
reads the presence roster and only to gate, never to route (#325).
- Content-Loading-and-Hot-Reload: the comms hear-set republish retry and the
zone-shape reconcile retry run on separate goroutine budgets at parity, so a
reconcile storm cannot starve the security-relevant republish class (#345).
(#371 async-who prompt ordering is an internal fix with no documented surface.)
AdoptZone is fenced by the zone's lease generation, not a clock-skew window (#315)
sysadmin: authenticated AdoptZone + rolling-upgrade fail-closed note
Sysadmin: durable-message delivery guarantee, park/poison metrics, and the consumer-config upgrade note (#266)
Sysadmin Reference: deployment, OAuth, content ops, scale
- Deployment: image/service model, port map + exposure posture, TLS,
one-public-edge firewall, fail-closed gates; cross-links telosMUD-infra IaC
- Sysadmin-OAuth-Setup: production GitHub OAuth + exact trust-secret env keys
- Content-Pack-Operations: versioned store, telos-pull, registry, CI gate
- Running-at-Scale: per-component scaling, drain/rolling-upgrade, OTel, logging
(honest about advisory-only rebalancing, one-zone-one-core, obs gaps)
- Sysadmin-Reference landing + production checklist
Completes the Sysadmin Reference section (5 pages).
Scaffold wiki + author About and Getting Started
- 48-page skeleton across all audiences (flat files + grouped _Sidebar)
- Every page carries an Audience + Status banner (Ready/Draft/Placeholder)
- About: Overview, History of MUDs (Ready)
- Getting Started: Getting-Started, Running-Locally, First-Admin-Setup,
Content-Packs-Intro (Ready) — dev bypass, local GitHub OAuth, bootstrap
admin, promote/demote, content-pack model, all grounded in code
- SRD5/WoW/D6 pack pages are honest placeholders (packs not yet built)