Skip to content

Observability and Logging

Jump to bottom
github-actions[bot] edited this page Sep 25, 2026 · 4 revisions

Observability and Logging

Streamline emits structured logs, distributed traces and metrics. All three go over OTLP to a single endpoint; logs additionally go to stderr or a file.

Application logs

log:
  app:
    enabled: true
    level: info        # debug | info | warn | error
    format: text       # text | json
    output: stderr     # stderr, an absolute path, or a path relative to data_dir
    rotate:
      max_size_mb: 100
      max_backups: 5
      max_age_days: 30
      compress: true

text is human-readable and right for docker logs or a terminal. json is right for anything that ships logs onward.

rotate only applies when output is a file path; rotating stderr is meaningless.

Beyond the four standard levels there's one more: CRITICAL (rendered as such, ranked above error). It's reserved for panics, invariant violations and unrecoverable conditions.

Tip

A CRITICAL line is never routine — it's the one to page on.

HTTP access logs

A separate logger with its own switch, so you can keep application logs at debug without drowning in request lines, or vice versa.

log:
  http:
    enabled: true
    format: json       # json | combined
    output: stderr
    rotate: { ... }

combined approximates the Apache combined format with one deliberate deviation: timestamps are RFC3339, not Apache's 02/Jan/2006:15:04:05 -0700. Log parsers expecting strict Apache format need adjusting.

The access logger is mounted as the outermost middleware, so it records every request — including 404s and panics.

Automatic log context

Every log line made with a request context is enriched automatically. You don't configure this and you can't turn it off:

Field Source
request_id chi's request ID middleware
user.id, user.email, user.roles Auth claims, following OTel semantic conventions v1.40.0
http.route The chi route pattern — /api/v1/movies/{id}, not /api/v1/movies/42
trace_id, span_id The active OTel span

http.route carrying the pattern rather than the concrete path is what makes aggregation useful — you can group by route without high-cardinality explosion.

trace_id on every line is the important one: it links a log entry to the trace that produced it, so you can pivot from "this request errored" to the full span tree in one click.

OpenTelemetry

otel:
  endpoint: "alloy.observability.svc.cluster.local:4318"
  insecure: true
  sample_ratio: 0.05
  environment: "prod"

An empty endpoint disables OTel export entirely — no traces, no metrics, no log bridge. That's the default.

Traces, metrics and logs are all batch-exported to that single endpoint.

Important

The SDK defaults to HTTPS. For a plaintext collector set otel.insecure: true (the environment variable OTEL_EXPORTER_OTLP_INSECURE=true does the same). Forgetting this is the usual reason a correctly-configured endpoint receives nothing.

otel.sample_ratio is the head sampling rate for root spans, 0–1, defaulting to 0.05. The DB driver is instrumented, so 1.0 exports every SQL statement. Setting OTEL_TRACES_SAMPLER overrides it — Streamline stands aside and lets the SDK read the standard variables.

otel.environment fills deployment.environment on the resource. Two installs exporting to one collector are otherwise indistinguishable; the resource also carries host and OS attributes, and honours OTEL_RESOURCE_ATTRIBUTES.

Note

log.app.enabled: false disables the stderr sink only. Traces, metrics and OTLP-exported logs continue as long as otel.endpoint is set. (Before v1.1 it disabled the whole pipeline.)

If the collector is unreachable, the export failure is logged to stderr as opentelemetry export failed, at most once a minute. Silence there means export is working — or that log.app.enabled is false.

Traces

Every package has its own tracer, named github.com/datahearth/streamline/internal/<pkg>. Spans are named <pkg>.<op>:

download.grab
rss.process_movie
indexer.query
metadata.tmdb.search_movie
importer.import

Service methods add domain attributes to their spans, so a trace tells you which movie, which indexer, which client — not just that a grab happened.

Outbound HTTP is instrumented via a shared otelhttp-wrapped client, so calls to TMDB, TVDB, your indexers, your download clients and your media servers all appear as child spans with their own timings. When something is slow, the trace shows you whether it's Streamline or the thing it's talking to.

Metrics

The database layer is auto-instrumented — SQL queries are traced via otelsql, and connection-pool statistics are registered as metrics. Go runtime metrics (goroutine count, GC pause, heap size) are exported too.

Beyond those, the series worth building alerts on:

Metric What it answers
streamline.scheduler.job.last_success_unixtime "Has this background job stopped running?" A job that silently dies produces no errors to count — only a timestamp that stops advancing.
streamline.download.client_reachable Whether each download client answered its last call. 0 for more than a few minutes means the client is down.
streamline.bittorrent.stalled_torrents Incomplete torrents with no peers.
streamline.transcoding.running vs .max_concurrent Saturated or stalled. Queue depth is streamline_transcode_jobs_by_status{status="queued"}.
streamline.importer.outcomes Imports by held / failed / terminal / succeeded; streamline.importer.queued is the backlog.
streamline.indexer.queries / .feeds Per-indexer outcome, split into unauthorized, unreachable, bad_response — an expired API key and a dead tracker are different problems.
streamline.mediaserver.refreshes Per-server refresh outcome; catches Plex failing while Jellyfin works.
streamline.auth.api_rejections Rejected /api/v1 authentication by method and reason.
streamline.auth.ratelimit.table_full Any non-zero rate means the login rate limiter has stopped limiting.
streamline_requests_by_status{status="pending"} Request backlog. streamline_movies_by_status and streamline_episodes_by_status cover wanted/downloading counts.

Logs

The otelslog bridge sends application logs down the OTLP logs pipeline in addition to stderr/file. So with an endpoint configured, logs reach your collector already correlated to traces by ID, without a log shipper parsing text.

A ready-made stack

The repository ships a working observability stack so you don't have to assemble one to try this.

Docker Compose:

docker compose -f deploy/compose.observability.yaml up -d
flowchart LR
  S[Streamline] -->|OTLP| A[Grafana Alloy]
  A --> VM[VictoriaMetrics]
  A --> VL[VictoriaLogs]
  A --> VT[VictoriaTraces]
  VM --> G[Grafana]
  VL --> G
  VT --> G
Loading

Point otel.endpoint at Alloy on 4318 and set otel.insecure: true.

Kubernetes: the Helm chart has an optional observability subchart wiring the same components from their upstream charts:

observability:
  enabled: true

It installs into the observability namespace, auto-sets otel.endpoint to the in-cluster Alloy service with otel.insecure: true, and provisions the Streamline dashboard into its Grafana. Two gotchas are already handled in the chart, but worth knowing if you're adapting it:

  • Cross-namespace DNS needs an FQDN. alloy:4318 won't resolve from the streamline namespace; alloy.observability.svc.cluster.local:4318 will.
  • The VictoriaMetrics/Logs/Traces charts (v0.35 / 0.12 / 0.0.7) have a selector/label mismatch — the selector uses app: server but templates drop it. The workaround is server.podLabels.app: server in each subchart's values.

Using your own Grafana and collector

Already running Grafana and a collector? Leave the subchart off and have the chart ship only the dashboard and the OTLP wiring:

config:
  otel:
    endpoint: http://alloy.monitoring.svc.cluster.local:4318
grafanaDashboard:
  enabled: true
  labels:
    grafana_dashboard: "1"        # your sidecar's label / labelValue
  annotations:
    grafana_folder: Media         # your sidecar's folderAnnotation, if any
  logs:
    type: loki                    # or victorialogs
  • The endpoint is OTLP over HTTP, usually port 4318. Not gRPC. The chart accepts host:port or a URL. With http://, the chart also sets otel.insecure: true. With https://, it sets it to false. A URL with a path is refused: the app posts to /v1/traces, /v1/metrics and /v1/logs under host:port and can't take a prefix.
  • The dashboard reads Prometheus-style names: streamline_movies_total, http_server_request_duration_seconds_bucket with http_route and http_response_status_code labels. You get these when the collector converts OTLP through the Prometheus translator, for example Alloy's otelcol.exporter.prometheus into prometheus.remote_write, as the subchart does. VictoriaMetrics' own OTLP endpoint needs -opentelemetry.usePrometheusNaming to produce the same names. Without that flag the HTTP histogram keeps its dotted OTLP name and the request panels stay empty.
  • Metric panels don't need a datasource setting. They use a $datasource variable that defaults to Grafana's default Prometheus-type datasource, and you can switch it from the dashboard.
  • The logs panel does need one, because its backend decides the query language. With loki, it reads container stdout: {namespace="<release namespace>", container="streamline"} | json | level=~"ERROR|WARN|CRITICAL". That query assumes your log shipper labels streams namespace and container, and that log.app.format is json (the chart's default). If your labels differ, set your own query in grafanaDashboard.logs.query. With victorialogs, it reads OTLP logs, the same way the subchart does.
  • The dashboard uid is streamline. If you imported a copy by hand, delete it first, or it keeps the provisioned one from loading.
  • With networkPolicy.egress on, a collector in another namespace needs its own rule in networkPolicy.egress.to.

What to alert on

Reasonable starting points:

Signal Why
Any CRITICAL log line Panic or invariant violation. Always real
/health non-200 Process is up but unhealthy
Scheduled job status: error Something broke; last_error says what
Scheduled job status: skipped, sustained A precondition isn't met — usually no enabled download client, meaning nothing is being grabbed at all
Rising last_duration_ms on drift-check Storage getting slow
Repeated 401/403 in access logs Someone probing, or a broken integration with a revoked key
Disk usage on data_dir SQLite has nowhere to grow

Job state is exposed at GET /api/v1/schedules — an easy target for a scripted check even without a metrics stack:

curl -sS -H "X-API-Key: $KEY" $SL/api/v1/schedules \
  | jq -r '.items[] | select(.status=="error") | "\(.name): \(.last_error)"'

For a bare liveness probe, /health needs no authentication and is deliberately excluded from the OpenAPI spec so it can't drift into being treated as API surface.

🏠 Home

🎬 Operating Streamline

⚙️ Advanced

Clone this wiki locally