Skip to content

v1.20.15 — Grafana dashboard reliability & sub-agent metrics

Choose a tag to compare

@jgouviergmail jgouviergmail released this 29 May 19:47

Observability hardening

A panel-by-panel audit of every Grafana query against the production Prometheus/Loki repaired the dashboards.

Sub-agent metrics wired

metrics_subagent.py was defined but never imported or incremented — the Sub-agents & Skills dashboard was permanently dark. ReactSubAgentRunner (the single sub-agent path, ADR-083) now emits subagent_spawned/active/duration/tokens_in/tokens_out/errors (defensively, labelled by llm_type), and an active-count gauge leak was fixed (try/finally), guarded by a regression test.

Dashboard query fixes (panels stuck on "No data")

  • 05 — LLM error panels repointed to the dedicated llm_api_errors_total / llm_rate_limit_hit_total / llm_context_length_exceeded_total / llm_content_filter_violations_total family
  • 06 — Loki event-name filters (router_*_confidence, response_node_exception)
  • 18 — embedding model / currency filters (Gemini, USD)
  • 09conversation_turns_total Histogram queried as a Counter → average
  • 08 — HITL resumption grouped by strategy
  • 14 — registry edit-iterations rate; dead checkpoints panel removed
  • 16ALERTS{} panels repointed to recording-rule health; redundant row removed

Removed dead / deprecated metrics

checkpoints_total, cache_operation_duration_seconds, redis_connection_pool_size, agent_routing_accuracy / agent_error_rate / agent_latency_p95_seconds, and three sub-agent guard-rail metrics (including a per-user_id PII label) — plus their dashboard panels.

Verification: Ruff / Black / MyPy clean · 8324 fast unit tests green · i18n parity across the 6 locales · tsc --noEmit + ESLint clean · every changed query validated live against the production Prometheus/Loki. No DB migration, no new env var, no endpoint change.

Full details in CHANGELOG.md.