FUG-117: HITL Grafana dashboarding — /metrics endpoint + Grafana Cloud wiring + dashboards-as-code

[issuefleet]

FUG-117: HITL Grafana dashboarding

Scopes out and implements Grafana dashboarding for HITL rig status — CPU/memory,
job-queue depth, per-DUT occupancy, reservation lifecycle, and CI job pass/fail —
using Grafana Cloud (free tier) with dashboards kept as code in the repo.

Full design/scoping write-up: pi/hitl/observability/README.md.

Architecture

The rigs are tailnet-only (no inbound), so instead of Grafana scraping in, a
Grafana Alloy agent runs on each Pi and remote_writes metrics out to
Grafana Cloud. CI reports job results separately.

hitl-managerd  ──GET /metrics──►  Alloy  ──remote_write──►  Grafana Cloud
GitHub Actions ──result (GitHub integration and/or Loki push)──►  (Prometheus+Loki)
dashboards/*.json  ──CI POST /api/dashboards/db──►  (dashboards as code)

1. Reporting from the runners (implemented, tested)

• hitl-managerd now serves GET /metrics in Prometheus text-exposition format.
• New internal/metrics package: a stdlib-only exposition writer (the
  pi/hitl Go module is deliberately dependency-free, so no client_golang) plus
  host stat readers (/proc/loadavg, /proc/meminfo, thermal zone) that degrade
  gracefully when a source is unreadable.
• queue.Manager gains monotonic lifecycle counters and a Metrics() snapshot
  with true per-DUT occupancy and queue depth.
• Metrics: hitl_queue_depth, hitl_duts_total/_busy, hitl_dut_busy{device},
  hitl_active_reservations, hitl_lease_seconds, and
  hitl_reservations_total / _activations_total / _releases_total /
  _lease_expirations_total / _start_failures_total, plus host
  hitl_host_load1 / _memory_*_bytes / _temperature_celsius. Every series
  carries a rig label.
• Unit tests for exposition format, host parsing, manager counters, and the
  composed output; smoke-tested against a live daemon.

2. Reporting from GitHub

• Baseline (zero code): Grafana Cloud's GitHub integration — pulls
  workflow/job pass-fail + duration via the API and ships a prebuilt dashboard.
• Optional (richer): observability/push-ci-metrics.sh pushes each HITL run's
  pass/fail to Grafana Cloud Loki (a CI job is an event, not a scrape target),
  wired into hitl.yaml as a final always() step that no-ops without creds.

3. Grafana wiring

• observability/alloy.alloy — the Alloy pipeline (scrape daemon + host node
  metrics, remote_write to Grafana Cloud; creds from env).
• observability/alloy.nix — opt-in NixOS module to run Alloy on a rig, gated on
  /var/lib/hitl/grafana.env (ConditionPathExists) so importing it never breaks
  a rig that isn't wired to Grafana yet.

4. Dashboards as code (bonus)

• observability/dashboards/hitl-rigs.json — the dashboard model (rig +
  datasource template variables; queue/occupancy/lifecycle/host panels).
• .github/workflows/grafana-dashboards.yaml — CI that POSTs dashboards/*.json
  to the Grafana API on change to main. Edit a panel via the JSON in a PR; merge
  syncs it.

Verification

• bazel test //pi/hitl/... — all pass.
• /metrics smoke-tested against a running hitl-managerd (reservation counters
  advance; host mem/load emitted; temperature correctly omitted where absent).
• Pre-commit hooks (shellcheck, prettier, markdownlint, nixpkgs-fmt, buildifier)
  clean.

Operator one-time setup (needs the Grafana Cloud account; can't be done here)

Create the free stack, generate metrics:write / logs:write tokens, seed
grafana.env on the rigs, populate the "HITL"/"Grafana" GitHub Actions
environments, and enable the GitHub integration. The Alloy NixOS module and the
dashboard-sync workflow are provided but not build-verified in-container (no
Nix/Grafana network); they follow the same deploy path as the rest of the rig.

Closes-Linear: FUG-117 (https://linear.app/fughilli/issue/FUG-117/hitl-grafana-dashboarding)

[github-actions]

PR Preview Action v1.8.1
🚀 View preview at https://fughilli.github.io/splanc/pr-preview/pr-90/
Built to branch gh-pages at 2026-08-14 22:34 UTC. Preview will be ready when the GitHub Pages deployment is complete.