Build structured Go server observability for logs, metrics, and traces

[zoeyrose]

Important

This issue is implemented in the fresh MIT-licensed Go server under the replacement program. Its gameplay and content-design decisions remain authoritative. C, CPython, classic packet, file-path, and enum details in the preserved specification are historical evidence only; do not copy, translate, or structurally port GPL implementation code.

Replacement implementation contract

Use Go slog, OpenTelemetry, Prometheus/OpenMetrics, and optional pprof, with bounded labels and explicit privacy rules. Keep operational telemetry separate from durable gameplay metrics.

The server remains authoritative, consumes versioned compiled content, and exposes bounded generated Game Protocol 1 messages. Pure rules may use a specifically approved typed CEL environment. Starlark is not part of this issue unless the separate residual-scripting decision explicitly approves it.

Required verification

• Preserve every observable rule, balance decision, disclosure boundary, and anti-exploit invariant from the specification below.
• Add deterministic Go unit/property tests and wrapper-managed scenario coverage at the appropriate integration boundary.
• Add bounded malformed-input and persistence-failure cases where this feature accepts content, network, or stored data.
• Add Go/Rust protocol conformance fixtures for every new cross-process field; the client must not reconstruct authoritative rules from prose.
• Demonstrate that implementation and tests contain no copied GPL source/test material and execute no runtime Python.

Preserved product/design specification and historical implementation notes

Summary

Replace the old idea of splitting one text log by prefixes (#29) with a deployable, structured observability foundation for the game server and shared networking code.

Atrinik already has timestamped severity-filtered logging in common/toolkit/logger.c, synchronous stdout/file sinks, connection diagnostic IDs, and scattered counters. The proposal should preserve simple local operation while adding stable event fields, bounded runtime metrics, correlation, health diagnostics, and export paths suitable for containers and external tooling.

Gameplay progression metrics in #33 remain authoritative saved player state. Observability data is operational and must never become a gameplay dependency.

Goals

• Make logs machine-readable without losing useful human console output.
• Correlate a connection/session, packet flow, player lifecycle, map load, save, plugin call, and error without logging secrets.
• Expose bounded counters, gauges, and histograms for server health and performance.
• Measure important spans and tick latency without requiring a full distributed-tracing service.
• Use stdout as the default production sink and let deployment tooling own rotation/retention.
• Keep hot gameplay/network paths bounded and non-blocking.

Structured event API

Extend the logger with a typed event entry point while retaining LOG() as a migration wrapper. Each structured record should have a versioned schema containing:

• UTC timestamp and monotonic/process elapsed value from the time-service proposal;
• severity, subsystem, stable event name, source location, and message;
• process/build/protocol identifiers;
• optional connection diagnostic ID, session/character-safe ID, map ID/coordinates, object generation, request/operation ID, and duration;
• a bounded list of typed string/integer/boolean fields.

Emit JSON Lines for machine output and retain a human formatter for development. Escape and bound every field. Stable event names are contracts; prose messages are not.

Never log passwords, salts, authentication tokens, join passwords, private keys, packet payloads containing credentials, or unrestricted chat/dialog contents. Network addresses and account/character identifiers need explicit privacy classification and configurable redaction.

Metrics registry

Add a small in-process registry for named operational metrics:

• counters: connections, authentication outcomes by safe category, packets/bytes, map loads, saves, plugin failures, protocol rejects, queue drops;
• gauges: connected/playing clients, loaded maps, active objects, queue depth/bytes, memory-pool usage;
• histograms: server tick duration, packet handling, map load, save, plugin callback, QUIC handshake/stream latency.

Names, units, labels, and cardinality limits must be registered centrally. Labels may use bounded enums such as command kind or result, never player names, map paths, arbitrary object names, or error strings.

Updates on hot paths should be constant-time. Snapshot/export work must use a bounded per-tick budget.

Export and operator surfaces

Provide three layers:

1. JSON logs to stdout, with optional human text and file sinks for development.
2. A privileged in-game/server-console diagnostics command for concise current health and targeted connection/session correlation.
3. An optional localhost-only health/metrics exporter or sidecar-friendly snapshot using Prometheus/OpenMetrics text.

Do not make the single-threaded game loop serve arbitrary remote HTTP work. If an embedded exporter is used, it must be local-only, authenticated where appropriate, bounded, and fed from immutable snapshots. A sidecar may handle remote access, TLS, dashboards, retention, and alert delivery.

Readiness should mean startup resources/plugins/listeners are usable; liveness should detect a stalled main loop without declaring ordinary game inactivity unhealthy.

Lightweight tracing and profiling

Add span helpers that record start/end monotonic duration and correlation IDs for:

• startup/resource/plugin initialization;
• main-loop ticks and slow-tick contributors;
• connection handshake/authentication/character login;
• map load/transition;
• player/account saves;
• plugin and script callbacks;
• QUIC asset and gameplay streams.

Sample routine successful spans and always retain bounded summaries for errors/slow thresholds. Do not allocate an unbounded trace tree or block on export.

Operational event audit

Define a separately classified audit stream for privileged commands, bans, authentication/security outcomes, and destructive administrative actions. Give it explicit access and retention expectations. Do not mix security records into #33 gameplay metrics or public dashboards.

Migration plan

1. Specify schemas, field bounds, redaction, cardinality, and event naming.
2. Add structured logger/events and JSON stdout alongside current human output.
3. Add the metrics registry, snapshots, and operator health command.
4. Instrument main loop, sockets/QUIC, authentication, maps, saves, and plugins.
5. Add local health/metrics export and example container/dashboard configuration.
6. Migrate high-value LOG() calls; leave low-value prose on the compatibility wrapper until classified.
7. Remove dependence on split prefix files and document external rotation/retention.

Acceptance criteria

• JSON log records are valid, versioned, bounded, and include stable severity/subsystem/event fields.
• Human console output remains available for local development.
• Secrets and high-cardinality player/content text are excluded or explicitly redacted.
• Metrics have registered names/units/labels and enforce cardinality limits.
• Tick, connection, packet/queue, map, save, plugin, and QUIC health are observable.
• Slow/stalled exporters cannot block the game loop or grow memory without limit.
• Liveness/readiness and privileged diagnostics have documented semantics.
• Operational/security audit events have a separate classification and retention model.
• Container deployments can use stdout plus a standard metrics scraper without server-specific log splitting.
• Implement QUIC transport, sessions, and authentication #33 metrics remain authoritative gameplay state and no gameplay rule depends on the observability exporter.

Validation

Unit-test JSON escaping/schema, field truncation, redaction, cardinality rejection, counter overflow, histogram boundaries, snapshot consistency, exporter backpressure, and wall-clock changes. Stress log/metric volume while measuring tick time and memory. Add integration tests for connection IDs across handshake/login/disconnect and for slow map/plugin/save spans.

Supersedes atrinik/atrinik#29. Related: atrinik/atrinik#105, #33, #32, atrinik/atrinik#126 and the deterministic server time proposal.