feat(runtime): add runtime-owned observability endpoints

[Lythaeon]

Description

Add a runtime-owned observability endpoint for SOF's packaged runtime so operators can scrape metrics and probe runtime liveness/readiness without wrapping the runtime in a separate HTTP layer.

This keeps lifecycle and readiness semantics inside the same runtime composition root that already owns startup, shutdown, plugins, extensions, and derived state.

Changes

Detailed list of what changed:

• crates/sof-observer/src/runtime.rs
  • added typed RuntimeObservabilityConfig
  • added RuntimeSetup::with_observability_bind_addr(...) and with_observability_config(...)
• crates/sof-observer/src/app/config/observability.rs
  • added SOF_OBSERVABILITY_BIND parsing
• crates/sof-observer/src/app/runtime/observability.rs
  • added the runtime-owned TCP listener and /metrics, /healthz, /readyz
  • added tests for metrics formatting and endpoint serving
• crates/sof-observer/src/app/runtime/entrypoints.rs
  • runtime now starts/stops the observability service when configured
• crates/sof-observer/src/app/runtime/runloop/driver.rs
  • runtime readiness now flips after receiver bootstrap and back down on shutdown
• crates/sof-observer/src/runtime_metrics.rs
  • exported ingest/freshness/backlog fields needed for machine-readable ops telemetry
• docs/ops/gitbook/README
  • documented the new endpoint and added ADR-0012

For slice-related changes, include:

• Affected slices
  • infra/runtime composition
  • runtime metrics/ops surface
• Cross-slice communication changes (if any) and why
  • no new domain cross-slice coupling; the endpoint reads existing runtime/extension/derived-state telemetry snapshots
• Migration requirements (if any)
  • none; endpoint is disabled by default and only binds when configured

Motivation

Business motivation:

• serious adopters need a standard scrape/probe surface before they will treat SOF like an operable service

Technical motivation:

• avoid fragmenting readiness and metrics semantics across examples/embedders
• keep observability inside the runtime composition root per ARD-0007 and ARD-0008

Alternative approaches considered:

• example/embedder-owned HTTP servers
• always-on listener
• full web framework dependency

Scope and impact

• Affected slices: infra/runtime composition, runtime metrics surface, operations docs
• Data/API changes: new optional RuntimeSetup observability config and SOF_OBSERVABILITY_BIND
• Backward compatibility: preserved; default behavior is unchanged when unset
• Performance impact: bounded low-cardinality snapshot rendering only when the endpoint is enabled
• Security impact: no default listener; bind address remains an explicit operator choice

Testing

• Unit tests
• Integration tests
• Manual verification
• Performance checks (if applicable)
• Security checks (if applicable)

Commands/results:

cargo make ci
cd docs/gitbook && npm run build
cargo build -p sof --release --example observer_runtime --features gossip-bootstrap

Manual VPS verification on 91.99.102.201:

• started observer_runtime with:
  • SOF_GOSSIP_ENTRYPOINT=64.130.50.23:8001
  • SOF_PORT_RANGE=12000-12100
  • SOF_SHRED_DEDUP_CAPACITY=524288
  • SOF_OBSERVABILITY_BIND=127.0.0.1:9108
• verified:
  • /healthz returns 200 ok immediately
  • /readyz returns 503 starting before bootstrap, then 200 ready
  • /metrics reports live counters (sof_runtime_live, sof_runtime_ready, sof_ingest_packets_seen_total, sof_latest_shred_age_ms, etc.)
  • SIGTERM shuts down cleanly and the endpoint stops accepting connections

Related issues and documentation

• Fixes:
• Related:
• Architecture docs: docs/architecture/README.md
• Relevant ARD/ADR:
  • docs/architecture/ard/0007-infrastructure-composition-and-runtime-model.md
  • docs/architecture/ard/0008-observability-and-operability-standards.md
  • docs/architecture/adr/0012-runtime-owned-observability-endpoints.md
• Operations/runbook updates:
  • docs/operations/advanced-env.md
  • docs/gitbook/operations/tuning-and-env.md
  • docs/gitbook/operations/knob-registry.md

Reviewer checklist

• Code follows project standards and architecture constraints
• Slice boundaries are respected (docs/architecture/ard/0003-slice-dependency-contracts.md)
• Tests added/updated and passing
• Documentation updated (README/docs/operations as needed)
• No undocumented breaking change
• Performance trade-offs documented where relevant
• Security considerations addressed where relevant

Additional notes

The endpoint is intentionally opt-in. Internal metrics remain always available for logs and future adapters, but binding a scrape/probe listener is treated as an explicit operational decision.