Round 49: NATS subject authorization (#554 + #552 slices 1-3 + infra #35)
Closes the unauthenticated-NATS residual the Round 39 (#424) and Round 41
(#355/#474) notes pointed at: a bare TCP connection to :4222 no longer has
full publish rights on every subject.
- Deployment: rewrote the Round-39 "restrict it ... until you have" content-bus
callout into the shipped per-identity authorization matrix (world/gate/
director/seed via TELOS_NATS_USER/PASSWORD). Captures the load-bearing
lessons: gate = deny=[">"] (a denylist leaves $JS.EVENT.ADVISORY.* forgeable);
world AND director both publish content.invalidate (a naive matrix breaks
admin reload); an empty allow=[] is ALLOW-ALL in NATS; NATS doesn't hot-reload
(k8s ships a content-hashed configMapGenerator); the four passwords must be
stable + all set (NATS refuses an unset $VAR). Folded in #554 (the in-process
handle ACL: the gate is denied publish on the whole telos.comms. root).
Updated the NATS ports row + softened the dev-loopback "unauthenticated"
wording (the matrix now defends wrong-identity; dev passwords are shared).
- Content-Loading-and-Hot-Reload: the #424 residual's "deployment-side fix"
now exists — a forgery requires already holding a trusted identity.
- Scoped-Event-Bus: narrowed the #355 forged-down-broadcast residual — the
matrix denies the untrusted gate outright, leaving only a world/director
credential (still needs the up/down subject split to fully close).
- Sysadmin-Reference: the four required-stable NATS passwords in the production
checklist.
Infra: retarget the reference deployment from Oracle k3s to AWS EKS (infra #27-#33)
The infra repo migrated off single-node Oracle Always-Free k3s onto AWS EKS,
which invalidated several things the wiki asserted — including two dead doc
links and one piece of load-bearing REASONING.
- Deployment: the IaC cross-link now describes AWS EKS and points at the single
README; DEPLOYMENT.md/RUNBOOK.md/PLAN.md were collapsed and DELETED upstream,
so both links were dead. Added a migration note listing what moved
(local-path->gp3, Traefik->ingress-nginx, gate via AWS NLB with
externalTrafficPolicy: Local to preserve client source IP, OCI Object
Storage->S3, TF state on S3 native lockfile, CI via GitHub OIDC->IAM role).
- Observability: the retention rationale CHANGED, not just the platform. The old
argument was node-level (local-path PVCs are directories on the node root FS
and do not enforce size, so a runaway backend takes the game down); on EKS the
gp3 PVCs ARE size-bounded, so the blast radius is now the backend itself —
worse observability, not a downed game. Policy values are unchanged (Loki 7d,
Prometheus 7d+2GB, ~7GiB) and NodeDiskFillingUp now guards emptyDir/image
churn. Grafana basic-auth moved Traefik->ingress-nginx (missing Secret => 503,
fail-closed); NetworkPolicy enforcement is the VPC CNI and the ingress source
is the ingress-nginx namespace, NOT kube-system (a silent breakage if ported
across unchanged). Added the cert split (HTTP-01 for web/Grafana, DNS-01 for
the gate — its own NLB is unreachable to HTTP-01) and the one-click lifecycle
(external-dns + cert-manager via IRSA, per-env delegated Route53 subzone so a
compromised staging pod cannot touch production names).
- Sysadmin-Reference: the data-loss exposure changed — gp3 EBS survives node
replacement (retiring the user_data-forces-replace hazard), but the one-click
`down` drains EBS before destroy, so teardown is the real destructive path.
pg-backup now targets S3 into a Terraform-provisioned versioned bucket;
restore docs moved to the README.
Docs: make up now brings up the full stack incl. Grafana LGTM; up-base is the lean path
The Makefile default was inverted so the human-facing 'make up' brings up
everything (base + the LGTM overlay, Grafana on :3000) and 'make up-base' is the
lean no-observability path CI/smoke use. Update Observability, Running-Locally,
and Deployment to match (was: up-obs/down-obs for the overlay).
Observability 2: the LGTM stack (gomud #458/#459/#481 + infra #13-#20)
New Observability page consolidating the telemetry pipeline now that it is real
and deployed on staging; the wiki previously had no Loki/Grafana content beyond
the forward-references from Observability 1, which are now made concrete.
- NEW Observability.md (Sysadmin): OTLP -> collector -> Grafana LGTM pipeline
with a flow diagram; metrics (incl. the URL-scheme gotcha #458 that silently
dropped all export); logs via the slog OTLP bridge TELOS_OTEL_LOGS (#459),
mutually exclusive with the k8s filelog path so nothing double-ships; traces
deferred (Tempo empty until tracing ships); the make up-obs dev overlay; and
the telosMUD-infra reference k8s deployment (#13-#20) — retention as node
protection (Loki 7d, Prometheus 7d+2GB), the collector DaemonSet, guarded
public Grafana (own hostname + Traefik basicAuth in front of Grafana login),
default-deny NetworkPolicy, and the validate-on-PR + manual-approval-prod CI.
Registered in the sidebar and the Sysadmin index.
- Running-at-Scale: corrected now-stale claims — "only metrics are wired /
no tracing exporter" and "no file sink or log shipper in code" — since logs
now ride OTLP into Loki; refreshed the honest-gaps note (account now exports;
no director service in dev compose); added the endpoint-scheme warning; made
the Observability-1 "planned LGTM" forward-reference concrete.
- Content-Loading-and-Hot-Reload + Lua-Sandbox-Internals: the content-DTO log
cap (#481) — builder-controlled field values (a 200KB dice/formula field or a
reference target) were echoed verbatim into problem strings; now clamped to
~1KB at the source and at the capProblems funnel, via a shared internal/logcap
leaf byte-identical to the Lua CapLogMsg. Notes the #454 key-guard does not
catch these (keys are area/event/op/value).
- Deployment: cross-linked the infra observability layer and the obs overlay.
Observability 1 milestone: log + port hardening (prelude to the LGTM stack)
A security milestone that hardens what the logs and the dev stack expose,
ahead of Observability 2 shipping container stdout into Loki — which turns an
ephemeral scroll into a durable, indexed archive and makes these exposures
real rather than theoretical.
- Running-at-Scale (Logging): player input is no longer logged verbatim (#454).
Several sites logged the full input line — a tell/say body, a channel post,
or a mistyped link code / /login URL — at Debug, and the panic line at Error.
Now dropped by default (player+seq remain), re-attached only under a separate
TELOS_LOG_RAW_INPUT opt-in deliberately decoupled from DEBUG, with a single-
token paste logging only its length. A check-log-keys.sh guard in make verify
fails on a sensitive value used as a log key — a backstop for the mechanical
case, not a complete invariant.
- Lua-Sandbox-Internals (new section) + Pack-Lua-Scripting + Running-at-Scale
metric: builder Lua logging is bounded (#456). print/mud.log/director.log had
no length cap or rate limit — an always-on write primitive once stdout hits
Loki. Length cap (~1KB), a per-CALL line budget reset per FRAME (charges the
flooder, not a co-firing victim) that trips the breaker, and a per-runtime
wall-clock token bucket (50 lines/s) that bounds a self-rescheduling timer;
labeled source=builder_lua; drops on builder_logs_dropped. Notes the two
bypasses closed at the source — err.Error() and compile-error echo.
- Deployment: the dev Compose stacks now bind every host port to 127.0.0.1
(#455/#479), not 0.0.0.0 — NATS 4222 is a full unauthenticated data plane,
plus 8222/4317/8889, redis, postgres default creds, the OAuth bridge and the
plaintext-telnet gates. Container-to-container over the compose bridge is
unaffected; framed explicitly as dev-stack hardening, not production policy.
Round 40 milestone: instance residuals + operator tunables (#436/#438/#435/#429/#425)
Corrects claims from the Round 37 sync: "nothing caps memory" (three places)
and the #340 eviction guidance, whose stated prerequisite now exists.
- Lua-Sandbox-Internals (new section) + Sysadmin-Reference + Pack-Lua-Scripting:
the per-call string-allocation budget (#438). Documents why the filed premise
was impossible — bytes-per-instruction spans FIVE orders of magnitude (25
instructions, 63MB), so no instruction budget bounds memory, and heap sampling
cannot work because a neighbour goroutine s noise floor would let a player
quarantine content they do not own. The real vector is the `..` OPCODE (every
string BUILTIN was already capped; an opcode has no wrapper): at shipping
defaults `s = s .. s` reaches 64 GB. Charged in the fork BEFORE the join.
Also the misclassification fix — a memory bomb used to land in AbortDeadline
(weight 0.1, "transient host load"); new AbortAlloc weighs with AbortBudget.
Keeps the review lesson that "does not amplify" != "does not allocate".
- Sysadmin-Reference + Deployment: redis.directory_addr splits coordination from
cache (#429). The eviction check is now shape-dependent — SHARED stays
warn-only, DEDICATED is FATAL — which supplies the prerequisite the previous
guidance said was missing, so the advice changes from "flip the policy" to
"declare the split". Periodic re-check stays warn-only on both shapes
(exiting over a live CONFIG SET would cause the outage it prevents), and a
configured-but-unreachable coordination Redis is fatal rather than falling
back onto the evicting instance.
- Instanced-Zones + Sysadmin-Reference: the instance caps are operator-tunable
(#436), with the two ENFORCED cross-field invariants that disproved the
issue s "no ordering invariant" premise (instances_per_shard vs the drain
eject barrier; burst validated with its window as a rate), plus the corrected
claim that per-account bounds churn — it does not.
- Instanced-Zones + Building-Instanced-Zones + Pack-Entity-Reference +
Pack-Lua-Scripting: the declared instance entrance (#435). Documents why
option 2 (relaxing self-only) was rejected — standing in a room is not
consent, greet/enter share an actor, and the mint bills the VICTIM s account —
and why entrances live in their own map so no push-the-player path can
traverse a door it cannot see.
Round 39 milestone: content-pipeline trust (#423/#424/#427/#366)
- Content-Loading-and-Hot-Reload: the post-boot content refresh is now VALIDATED
before it publishes (#423). Previously it published with nothing checking it
while reload's identical publish was gated — so content an operator watched a
reload REJECT went live at the next unrelated invalidation, and "nothing
propagated" stopped meaning "nothing applied". Documents why the refresh gate
is deliberately NARROWER than reload's (rejectable iff a finding makes the
snapshot unsafe to BUILD A ZONE FROM), why it freezes the whole snapshot
instead of dropping the offending pack (dropping removes populated zones with
players in them), the rejected-version cooldown, the problem-set-keyed memo,
and why boot reports but does not refuse (no fallback snapshot; the asymmetry
is the alertable "fleet split by uptime" signal). Keeps the honest residual:
validatePacks is a HEALTH gate, not a PROVENANCE one — the deploy log makes
cross-pack deploys legible, and its fingerprint includes `instanceable`.
- Content-Loading-and-Hot-Reload + Deployment: the content-bus pack filter now
fails CLOSED (#424). An invalidation naming no pack was accepted by every
shard, and since the re-read matches on (kind, ref, pack) an empty pack
resolves nothing -> Found:false -> the DELETION path: a fleet-wide eviction
primitive. Documents the three-part accepts predicate and states the scope
honestly — blast-radius reduction, NOT authentication. The real remedy is
deployment-side NATS subject permissions, now written up in Deployment.
- Content-Pack-Operations + Builder-Commands: `pull <version> force` (#427).
The guard still runs and still reports; admin-only on top of builder; a stale
override downgrades. Leads with the corrected remedy — REDIRECT CHARACTERS
FIRST, because the reboot IS the harm event (a peer cannot build a stripped
zone, so the handover fails and players are reclaimed to their home room).
- Content-Pack-Operations: the seed->pull ref collision (#366). Definition refs
are GLOBAL, not per-pack, and telos-seed never registers — so seeded rows
collide with the first pulled pack sharing a ref. Detected in-transaction
after the prune; notes that PurgePack must share the LOCK, not just the tx,
and that the old message asserted a registry fact it never checked (a
dead-end remedy is worse than none).
Overworld map, content-defined toggles, and the demo-strip release model
Cross-repo changes (gomud engine+demo, telosMUD-content, telosMUD-infra):
Overworld / minimap (gomud #358/#360/#361/#367):
- Pack-Entity-Reference: new toggle_defs def table (on/off player preference,
verb-flipped, delta-from-default override, never a trust signal); the room
display surface (return nil to fall back → own the render for some rooms only)
- Pack-Lua-Scripting: coord(), has_room_flag (room content flags, vs has_flag =
Living flags), long(), toggle() handle reads
- Builder-Commands + Player-Reference: enter/exit/out named-exit verbs + the
Exits: line order; toggle verbs
- Demo-Pack: the 4th zone (overworld 'The Open Plains', 6x20 grid) + the opt-in
minimap (overworld toggle → room display template); the plains route vs the
kept-for-parity direct market->grove exit
Demo-strip release model (gomud release-images + telosMUD-content/-infra):
- Deployment: published images built with nofixture ship core-only; the world is
pulled from the external content store; telos-pull added to the published matrix
(telos-seed keeps the demo embed for local stacks)
- Content-Packs-Intro + Content-Pack-Operations: the demo is a dev fixture, not a
shipped world; a deployed fleet (e.g. live staging) pulls the reference pack
Sysadmin Reference: deployment, OAuth, content ops, scale
- Deployment: image/service model, port map + exposure posture, TLS,
one-public-edge firewall, fail-closed gates; cross-links telosMUD-infra IaC
- Sysadmin-OAuth-Setup: production GitHub OAuth + exact trust-secret env keys
- Content-Pack-Operations: versioned store, telos-pull, registry, CI gate
- Running-at-Scale: per-component scaling, drain/rolling-upgrade, OTel, logging
(honest about advisory-only rebalancing, one-zone-one-core, obs gaps)
- Sysadmin-Reference landing + production checklist
Completes the Sysadmin Reference section (5 pages).
Scaffold wiki + author About and Getting Started
- 48-page skeleton across all audiences (flat files + grouped _Sidebar)
- Every page carries an Audience + Status banner (Ready/Draft/Placeholder)
- About: Overview, History of MUDs (Ready)
- Getting Started: Getting-Started, Running-Locally, First-Admin-Setup,
Content-Packs-Intro (Ready) — dev bypass, local GitHub OAuth, bootstrap
admin, promote/demote, content-pack model, all grounded in code
- SRD5/WoW/D6 pack pages are honest placeholders (packs not yet built)