Skip to content

feat(telemetry): export per-deployment metrics and logs over OTLP#187

Merged
Shine-neko merged 3 commits into
mainfrom
feat/telemetry-metrics-logs
Jul 4, 2026
Merged

feat(telemetry): export per-deployment metrics and logs over OTLP#187
Shine-neko merged 3 commits into
mainfrom
feat/telemetry-metrics-logs

Conversation

@Shine-neko

Copy link
Copy Markdown
Contributor

Adds two OpenTelemetry export signals alongside the existing traces support (#186), each opt-in and independently toggled under [server.telemetry].

[server.telemetry.metrics]
enabled = true
endpoint = "http://collector:4317"
interval_seconds = 15

[server.telemetry.logs]
enabled = true
endpoint = "http://collector:4317"

What's new

  • Metrics (push). Periodically exports the per-deployment resource gauges over OTLP/gRPC: ring.deployment.{cpu.percent,memory.used_bytes,memory.limit_bytes,network.rx_bytes,network.tx_bytes,disk.read_bytes,disk.write_bytes,pids,instances,restarts}, labelled {deployment, namespace, runtime}. Read from the same background stats cache the Prometheus /metrics endpoint uses, so no extra runtime round-trip is added.
  • Logs (push). Bridges the server's log events to an OTLP collector in addition to the console — console logging is never turned off.
  • Each signal (traces / metrics / logs) is independent: enable any combination. All three are off by default, and an unreachable collector degrades gracefully (logged, server keeps running) rather than failing the boot.
  • Standard OTEL_* env vars override the TOML per signal (OTEL_EXPORTER_OTLP_{METRICS,LOGS}_ENDPOINT, OTEL_SERVICE_NAME).

Incidental fix

The telemetry: OTLP ... enabled startup lines were emitted before the tracing subscriber was initialised, so they were silently dropped. They are now buffered and replayed after .init(), which also fixes the traces enable line.

Validation

  • 747 unit tests green, clippy clean, 6 new config tests.
  • t16/t17: operational e2e (boot + graceful degradation against a closed collector port).
  • t18: real ingestion e2e — runs an OpenTelemetry Collector in Docker, deploys a container, and asserts the ring.deployment.* metrics and Ring log records actually reach the collector. Stable across 3 runs; skips cleanly when Docker is absent.

Note

Prometheus scrape export already exists (the /metrics endpoint), so this PR intentionally covers only the OTLP push path for metrics and logs.

@Shine-neko Shine-neko merged commit b93c777 into main Jul 4, 2026
5 checks passed
@Shine-neko Shine-neko deleted the feat/telemetry-metrics-logs branch July 4, 2026 13:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant