Observability

Observability

TinyIce exposes Prometheus metrics, runtime profiles, structured logs, and live SSE so you can watch a deployment in real time and triage one that misbehaves.

Prometheus metrics + pprof (internal port)

Metrics and Go runtime profiles are served on a separate internal port, :8081 — not the public HTTP port.

http://HOST:8081/metrics              # Prometheus exposition
http://HOST:8081/debug/pprof/         # Go runtime profiles

Metrics cover total and per-mount listener counts, bytes in/out, memory, goroutines, GC stats, and uptime.

Firewall :8081. It is intended to bind on the loopback / private side of your deployment. pprof is intentionally not behind auth — treat it like /metrics and restrict it at the network layer (bind it to localhost, scrape over a private network, or front it with your proxy's access rules).

Example scrape config:

scrape_configs:
  - job_name: tinyice
    static_configs:
      - targets: ["10.0.0.10:8081"]

A ready-made Prometheus scrape config and a Grafana dashboard ship in the repo: monitoring/prometheus.yml and monitoring/grafana-dashboard.json.

Profiling a stuck instance

Mutex and block profiling are enabled (sample rate 1), so when a production instance hangs you can capture what's blocked:

curl -o goroutines.txt 'http://HOST:8081/debug/pprof/goroutine?debug=2'
curl -o mutex.txt      'http://HOST:8081/debug/pprof/mutex?debug=2'
curl -o heap.pb        'http://HOST:8081/debug/pprof/heap'

(This is exactly how the lock-contention and goroutine-leak fixes in the 2.5.x / 2.6.x line were diagnosed — see the changelog.)

Live events (SSE)

For dashboards and custom UIs, subscribe to the SSE streams instead of polling:

• /events — public, visible streams only.
• /admin/events — authenticated, full server snapshot every ~500 ms (listeners, bandwidth, per-stream health, AutoDJ state, runtime gauges).

Payload shapes are documented in HTTP API.

Health monitoring

The health monitor auto-removes a mount that goes silent for ~2 minutes (transcoded outputs are exempt — see Transcoding). Per-stream health (0–1) is included in /admin/events and /api/streams.

Logging

Flag	Effect
-log-level	debug · info (default) · warn · error
-json-logs	Structured JSON for ELK / Loki / Vector
-log-file	Write to a file (stdout if unset)
-auth-log-file	Split auth/audit events into a separate trail

For IO-bound boxes under heavy load, drop to -log-level warn and avoid -json-logs. Full flag list in Command Line and Signals.

---

Next: Deployment · Security · HTTP API