GitHub Copilot Telemetry Dashboard

A self‑hosted, near real‑time observability stack for GitHub Copilot usage and cost. GitHub Copilot Chat in VS Code emits OpenTelemetry signals (traces, metrics, and events) as you work. This project captures all three, stores each in a purpose‑built backend, and renders live dashboards in Grafana — entirely on your own machine, with no cloud dependency.

It answers three questions:

1. What am I actually paying GitHub? (plan‑aware cost, offset against your allowance)
2. How am I using Copilot? (operations, tokens, models, tools, per‑session cost)
3. What would this cost at raw model API prices? (value / model comparison)

---

Screenshots

Actual Cost (your plan) — plan‑aware spend; here an Enterprise seat with no usage limit shows a flat $39/month.

Usage Overview — operations, tokens, models, tools, and per‑session cost with a session name / UUID toggle.

Value & Model Comparison — shadow API‑price cost, cache savings, and per‑model comparison.

---

Table of contents

• Architecture
• Prerequisites
• Setup
• What data is captured
• The dashboards & how to read them
• Configuration
• Metric reference
• Privacy
• Troubleshooting
• Stopping & data retention

---

Architecture

VS Code Copilot exposes a single OTLP endpoint but emits three signal types. An OpenTelemetry Collector receives everything on one port and fans each signal out to the backend built to store it. The collector also derives extra metrics from spans (cost inputs, per‑session usage, session names).

GitHub Copilot Chat  (OTLP/HTTP :4318)
        │
        ▼
  OTel Collector ── transform (session name) ── sum connector (cost & session metrics)
        ├── traces      ─▶ Tempo        (spans / trace tree, prompt content)
        ├── metrics     ─▶ Prometheus   (counters, histograms, derived cost)
        └── events/logs ─▶ Loki         (structured event stream)
                                  │
                                  ▼
                              Grafana   (Tempo + Prometheus + Loki datasources)

Service	Port	Purpose
OTel Collector	4318	OTLP/HTTP receiver — point VS Code here
OTel Collector	4317	OTLP/gRPC receiver (optional)
OTel Collector	8889	Prometheus scrape endpoint (Copilot metrics)
Tempo	3200	Trace query API
Prometheus	9090	Metrics store + query UI
Loki	3100	Event/log store
Grafana	3000	Dashboards (admin / admin)

---

Prerequisites

• Docker Desktop (or Docker Engine + Compose v2).
• VS Code with GitHub Copilot Chat (the OTel export requires a recent Copilot Chat version — the setting github.copilot.chat.otel.enabled must exist).

---

Setup

1. Start the stack

docker compose up -d
docker compose ps    # confirm all 5 containers are healthy

2. Point VS Code at the collector

Copy the keys from vscode-settings-snippet.jsonc into your User settings.json (Ctrl+Shift+P → Preferences: Open User Settings (JSON)):

{
  "github.copilot.chat.otel.enabled": true,
  "github.copilot.chat.otel.exporterType": "otlp-http",
  "github.copilot.chat.otel.otlpEndpoint": "http://localhost:4318",
  "github.copilot.chat.otel.captureContent": true
}

Use User settings, not workspace settings — the OTel SDK initializes early in VS Code startup and workspace settings can load too late.

captureContent controls whether prompt/response text is captured. It is required for readable session names (derived from the first prompt) and for content in Loki/Tempo. Set it to false for metadata‑only (metrics still work; session names fall back to the conversation UUID). See Privacy.

3. Reload VS Code

Ctrl+Shift+P → Developer: Reload Window. The OTel SDK only initializes at startup, so settings changes require a reload.

4. Use Copilot, then open Grafana

Use Copilot Chat / agent mode as normal. Then open http://localhost:3000 → Dashboards → GitHub Copilot folder. Data appears within ~30–60s (Prometheus scrapes every 15s; dashboards refresh every 10s–5m).

---

What data is captured

Copilot Chat emits three OTel signal types. Content (prompts/responses) is only captured when captureContent is true.

Traces → Tempo

A hierarchical span tree per interaction:

invoke_agent            (whole turn: model, total tokens, git repo/branch)
  ├── chat <model>      (one LLM call: tokens, cache, latency, finish reason)
  ├── execute_tool …    (one per tool: tool name, type, edit type, MCP info)
  └── execute_hook …    (hook decision/duration)

Key span attributes: gen_ai.request.model, gen_ai.response.model, gen_ai.usage.input_tokens/output_tokens, gen_ai.usage.cache_read.input_tokens/cache_creation.input_tokens, gen_ai.usage.reasoning.output_tokens, copilot_chat.copilot_usage_nano_aiu (Copilot AI Units), gen_ai.conversation.id, gen_ai.tool.name, error.type, and — with content capture — gen_ai.input.messages / gen_ai.output.messages.

Metrics → Prometheus

Native Copilot metrics (token usage, operation duration, time‑to‑first‑token, tool call count/duration, sessions, lines of code, edit acceptance, …) plus derived metrics this project adds in the collector:

• Cost inputs (from chat spans): copilot_tokens_cache_read, copilot_tokens_cache_creation, copilot_tokens_reasoning, copilot_usage_aiu_nano — each by gen_ai_request_model.
• Per‑session usage (from chat spans): copilot_session_input_tokens, copilot_session_output_tokens, copilot_session_aiu_nano — each by gen_ai_conversation_id, gen_ai_request_model, and copilot_session_name.

These are restricted to chat spans so values are counted once (the aggregate invoke_agent span repeats them, which would otherwise double‑count).

Events → Loki

Session start, tool calls, agent turns, edit accept/reject, feedback, etc., as a structured log stream ({service_name="copilot-chat"}). With content capture, prompt/response text is included here.

Derived: session names

A collector transform extracts the first user prompt from gen_ai.input.messages into a copilot_session_name label (best‑effort; falls back to the conversation UUID when content is unavailable). The Usage Overview has a "Show sessions as" toggle to switch between the readable name and the UUID.

---

The dashboards & how to read them

1. GitHub Copilot — Actual Cost (your plan) (the main cost view)

Pick your plan from the Copilot plan dropdown. Pinned to the current month.

Section	What it means
Seat cost / month	Fixed price you pay GitHub regardless of usage.
Monthly AIU allowance	Included AI Units for the plan. Shows ∞ Unlimited for plans with no usage limit.
AIU used / remaining / used %	Consumption offset against the allowance this month.
Overage AIU / cost	Consumption beyond the allowance × overage rate (0 if capped/unlimited).
Total actual cost (MTD)	Seat cost + overage. For unlimited plans this equals the seat price.
Projection	Straight‑line month‑end estimate: used × days_in_month ÷ day_of_month.
AI Units / tokens by model	Where consumption goes.
Cost by session	Chat conversations ranked by AI Units — find the expensive tasks.

What you actually pay GitHub. Paid plans have unlimited base usage (no token cap). The only variable cost is AI Unit / premium‑request overage beyond your allowance. Tokens are usage, not a spend cap.

2. GitHub Copilot — Usage Overview

Operational view + per‑session analysis.

• Activity at a glance — sessions, LLM calls, tool calls, tokens, lines of code.
• Usage over time — operations, token consumption by model, model distribution.
• Latency — LLM response p95 by model, time‑to‑first‑token p95.
• Cache, tools & quality — prompt‑cache hits vs creation, top tools, tool latency, errors, edit‑acceptance activity.
• Recent agent operations — a live trace table (drill into any span).
• Per‑session breakdown:
  • "Show sessions as" toggle — session name or conversation UUID.
  • Selected session stats (AIU / input / output) — driven by the Session dropdown, or by clicking a session id in the table.
  • Chat sessions table — every session ranked by AIU/tokens.
  • Session drill‑down — Tempo traces for the selected session; expand to see the exact tools and models that session used.

3. GitHub Copilot — Value & Model Comparison (shadow / not your bill)

What the tokens would cost at raw model API list prices — a value/ROI and model‑comparison view, honoring the time picker (5m/15m/6h). Includes gross/net shadow cost, prompt‑cache savings, spend burn rate, per‑model comparison, and cache/reasoning token detail.

Net vs Gross note. Net cost is derived as increase(gross) − increase(savings) because net is not monotonic (a heavily‑cached call increases savings faster than gross), so applying increase()/rate() directly to net would be wrong.

---

Configuration

All cost/plan config lives in config/prometheus-rules.yml. After editing, reload Prometheus: docker compose kill -s SIGHUP prometheus.

• Model prices (copilot-model-prices group) — USD per 1M tokens, one input/output/cached trio per model. The model label must match gen_ai_request_model exactly.
• Plans (copilot-plans group) — per plan: seat_usd (accurate), included_aiu (calibrate to your billing page; large sentinel = unlimited), overage_usd_per_aiu (default 0).
• AIU price for the shadow view — the * 0.04 factor in copilot_cost_aiu_usd.

Session‑name vs UUID: the collector extraction lives in config/otel-collector-config.yaml (transform/session_name). The dashboard toggle is the "Show sessions as" variable on the Usage Overview.

Where to check your Copilot plan

• Plan + usage: https://github.com/settings/copilot (Usage shows month‑to‑date premium‑request / AI‑credit consumption).
• Billing: https://github.com/settings/billing (or your enterprise's billing page).

---

Metric reference

Metrics use add_metric_suffixes: false, so PromQL names are the OTel names with dots → underscores; histograms expose _bucket / _sum / _count.

Metric	Source	Key labels
copilot_chat_session_count	native	—
gen_ai_client_token_usage_*	native	gen_ai_token_type, gen_ai_request_model
gen_ai_client_operation_duration_*	native	gen_ai_request_model, error_type
copilot_chat_time_to_first_token_*	native	—
copilot_chat_tool_call_count	native	gen_ai_tool_name, error_type
copilot_tokens_cache_read / _cache_creation / _reasoning	derived	gen_ai_request_model
copilot_usage_aiu_nano	derived	gen_ai_request_model
copilot_session_{input,output}_tokens, copilot_session_aiu_nano	derived	gen_ai_conversation_id, gen_ai_request_model, copilot_session_name
copilot_cost_{input,output,cache_savings,gross,net}_usd	rule	gen_ai_request_model
copilot_cost_aiu_usd	rule	gen_ai_request_model
copilot_plan_{seat_usd,included_aiu,overage_usd_per_aiu}	rule	plan

_nano metrics are AI Units × 10⁹ — divide by 1e9 for AI Units.

Verify data is flowing:

# metrics at the collector
(Invoke-WebRequest http://localhost:8889/metrics).Content -split "`n" |
  Where-Object { $_ -match 'copilot_|gen_ai_' } | Select-Object -First 20

Tempo (Grafana → Explore): { resource.service.name = "copilot-chat" } Loki (Grafana → Explore): {service_name="copilot-chat"}

---

Privacy

• Content capture is a choice. With captureContent: true, prompt/response text is captured — it flows to Loki/Tempo and short first‑prompt snippets become the copilot_session_name label in Prometheus. Everything stays on your machine (no cloud). Set captureContent: false for metadata‑only; session names then fall back to the conversation UUID.
• No data leaves the local Docker network unless you change the exporters.

---

Troubleshooting

Symptom	Fix
Grafana panels show No data	Confirm VS Code User settings + a window reload, then use Copilot. Check docker compose logs otel-collector.
A panel is empty but others work	That metric only appears when the matching activity occurs (e.g. lines of code after you accept an agent edit).
Cost $0	Set your model prices / plan values in config/prometheus-rules.yml and SIGHUP Prometheus.
Session names show UUIDs	Enable captureContent, reload VS Code, and generate a new chat. Extraction is best‑effort.
Numbers look inflated	Test traffic accumulates; use the Actual Cost dashboard (plan‑aware) for real spend.

---

Stopping & data retention

docker compose down          # stop, keep data (Prometheus retains 120 days)
docker compose down -v       # stop and delete all stored telemetry