Skip to content

Observability rename: Prometheus metric prefix, Grafana dashboards/provisioning, and alert rules #5330

Description

@JSONbored

Problem: src/selfhost/metrics.ts defines 106 unique Prometheus/OTEL metric names, every one hard-prefixed gittensory_ (examples: gittensory_uptime_seconds, gittensory_http_requests_total, gittensory_queue_pending, gittensory_queue_dead, gittensory_jobs_processed_persisted_total, gittensory_webhook_dedup_total, gittensory_ai_cost_usd_total, gittensory_ai_requests_total, gittensory_ai_review_model_fallback_total, gittensory_agent_action_permission_denied_total, gittensory_qdrant_errors_total, gittensory_clock_skew_seconds, gittensory_backup_acknowledged, gittensory_dlq_dead_lettered_total). These are emitted from roughly 98 distinct call sites across ~20 files (src/services/ai-review.ts, src/services/agent-action-executor.ts, src/github/backfill.ts, src/github/webhook.ts, src/orb/broker-client.ts, src/queue/processors.ts, src/queue/slop-detection.ts, several src/review/*.ts files, several src/selfhost/*.ts files, src/server.ts) — roughly 272 total references — and this whole surface is completely absent from all 17 known rebrand issues.

Area: Observability / Infra

Proposal:

Open question (needs a maintainer decision before starting): Should the metric-name rename ship as a dual-emit window (old and new names both exported for a deprecation period, so operators can migrate dashboards/alerts on their own schedule) or as a hard cutover (single flip, old names gone immediately)? This is the same class of decision as the check-run rename item, and it directly determines the shape of the migration work below, so it needs to be settled before implementation starts.

Once that's decided, rename the gittensory_ metric prefix throughout src/selfhost/metrics.ts and its ~98 call sites, then update every downstream consumer that keys off the old names: prometheus/rules/alerts.yml (42 unique metric names referenced in alert expressions), all 8 checked-in Grafana dashboard JSON files under grafana/dashboards/ (claude-usage.json, codex-usage.json, github-prs.json, gittensory.json, gpu-metrics.json, maintainer-reviews.json, orb-ai-usage.json, resource-hub.json — including their Gittensory-prefixed titles and gittensory--prefixed uids), grafana/provisioning/datasources/sqlite.yml (datasource name GittensoryDB, uid gittensory-db, sqlite file path /reporting/gittensory-reporting.sqlite), grafana/provisioning/dashboards/provider.yml (provider name gittensory, folder Gittensory), scripts/backup-metrics.sh (emits gittensory_backup_latest_timestamp_seconds / gittensory_backup_files), and scripts/smoke-selfhost.sh (greps for gittensory_uptime_seconds as a healthcheck).

This is orthogonal to the environment-variable rename item (which only covers ~20 GITTENSORY_-prefixed shell env vars, not emitted metric names) and to the MCP tool-name item (a visibly separate identifier set already in its own scope). Self-host operators' own Grafana dashboards and Prometheus alert rules key directly off these exact metric-name strings, so renaming the prefix without a migration strategy silently blanks every existing dashboard panel and disables every alert rule referencing the old names.

Deliverables:

  • src/selfhost/metrics.ts and all ~98 call sites updated to the decided naming scheme (dual-emit or hard cutover, per the open question above).
  • All 42 alert-rule metric references in prometheus/rules/alerts.yml updated to match.
  • All 8 Grafana dashboard JSON files under grafana/dashboards/ updated (panel queries, titles, uids).
  • grafana/provisioning/datasources/sqlite.yml and grafana/provisioning/dashboards/provider.yml updated (datasource name/uid, sqlite file path, provider name/folder).
  • scripts/backup-metrics.sh and scripts/smoke-selfhost.sh updated to emit/check the new metric names.
  • If a dual-emit window is chosen: a written deprecation timeline for when the old metric names stop being emitted.

Acceptance criteria:

  • Every one of the 106 metric names is updated consistently across src/selfhost/metrics.ts, all call sites, alert rules, dashboards, and provisioning — no mismatched old/new name left behind.
  • All 8 Grafana dashboards load without broken panels and all 42 alert rules evaluate correctly against the post-change metric names.
  • scripts/smoke-selfhost.sh's healthcheck grep passes against the post-change metric name.
  • The dual-emit-vs-hard-cutover decision from the open question is recorded and implemented as decided.

Test scenarios:

  • Stand up self-host with the changes applied and verify each of the 8 Grafana dashboards renders data in every panel, not just loads without error.
  • Fire each of the 42 alert conditions (or replay historical data through them) to confirm the alert rules still trigger correctly against the new metric names.
  • Run scripts/backup-metrics.sh and scripts/smoke-selfhost.sh end-to-end and confirm they pass.
  • If dual-emit is chosen, verify both old and new metric names are simultaneously queryable during the deprecation window.

Resources:

  • src/selfhost/metrics.ts
  • prometheus/rules/alerts.yml
  • grafana/dashboards/ (claude-usage.json, codex-usage.json, github-prs.json, gittensory.json, gpu-metrics.json, maintainer-reviews.json, orb-ai-usage.json, resource-hub.json)
  • grafana/provisioning/datasources/sqlite.yml
  • grafana/provisioning/dashboards/provider.yml
  • scripts/backup-metrics.sh
  • scripts/smoke-selfhost.sh

Boundaries:

  • Leave every reference to the separate gittensor network's own domain/name untouched.
  • Do not touch the environment-variable renames or the MCP tool-name renames — those are separate, already-scoped items; this issue covers emitted Prometheus/OTEL metric names, Grafana dashboard/provisioning config, and alert rules only.
  • Do not pick the dual-emit-vs-hard-cutover answer unilaterally — get the maintainer decision first.

Part of #4761.

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.roadmapOn the Wave-2 agent-layer roadmap board (project 9)

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions