-
Notifications
You must be signed in to change notification settings - Fork 3
Copilot Usage Analytics GPT 5
Niclas Olofsson edited this page Aug 18, 2025
·
1 revision
A developer’s playbook for measuring, visualizing, and improving effectiveness and cost-efficiency across tasks and models
Audience: Hands-on developers who switch models depending on the task (architecture & design, code generation, refactoring/cleanup, project maintenance) and want data-driven guidance on when to use which model, and how to continuously improve.
Version: 1.0
- Make model choices data-driven, not habitual.
- Improve flow by reducing latency where speed matters and investing in quality where outcomes matter more.
- Control costs by matching model capability to task complexity and monitoring request volume.
- Build a personal feedback loop to learn which patterns and setups deliver the best results for you.
Our pipeline combines:
- Raw session events (chat sessions, turns, tool call rounds, metadata).
- Aggregations that transform these events into actionable statistics.
Key raw fields available per turn or request (derived via mapping logic similar to the one used by analytics and usage panels):
- Time, session/workspace identifiers
- Agent (chat/edit/workspace/inline)
- Model identifier/label
- File path(s) and inferred language
- Latency
- Requests per turn via toolCallRounds (backend LLM calls)
- Turn type (ask vs edit-like modes)
Ground rules for counting:
- One backend LLM call = one toolCallRound. If missing, fall back to 1/request for compatibility. This is how we track real “model usage.”
Use these foundational metrics everywhere (KPI chips, summaries, and filters). All periods are filterable by time range and workspace.
- Sessions: unique chat sessions in the period.
- Turns: total user turns.
- Requests: sum of backend LLM calls (toolCallRounds).
- Files: unique files referenced.
- Models: unique models used.
- Agents: unique agents used.
- Median latency (ms): median end-to-end request latency.
- Edits: count of edit-type turns.
- Edit ratio: share of turns that were edits.
Formulas (KaTeX):
- Edit ratio:
$edit_ratio = \frac{edits}{turns}$ - Requests:
$requests = \sum_{t \in turns} \left|toolCallRounds(t)\right|$ (fallback: 1 if rounds missing) - Median latency:
$median({latency_ms_i})$
Optional cost-aware extensions (future-ready if you add cost metadata):
- Cost per request:
$cpr = \frac{\sum cost}{requests}$ - Cost-efficiency index (example):
$CEI = \frac{edit_ratio}{cpr}$
Make insights obvious with a small set of high-signal views:
- KPI Chips (top-level snapshot)
- Sessions, Turns, Requests, Files, Models, Agents, Median Latency, Edits, Edit Ratio.
- Purpose: A quick “health check” for the selected time range.
- Daily Requests Trend (bar/line chart)
- Requests/day for the chosen period with a 7-day rolling average.
- Purpose: See spikes, streaks, regressions; correlate with project phases or deadlines.
- Model Distribution (ranked list or stacked bar)
- Requests by model with trend arrows (up/down/stable) period-over-period.
- Purpose: Reveal overreliance on a single model; identify candidates for cheaper/faster alternatives.
- Agent Mix (ranked list)
- Requests by agent (chat, edit, workspace, inline).
- Purpose: Match workflow patterns to productivity strategies (e.g., heavy inline usage → optimize latency).
- Activity Feed (recent interactions)
- A compact feed of recent turns with agent/model labels and basic context.
- Purpose: Debug behaviors, confirm scanning works, spot anomalies fast.
- Language/File Heat (optional)
- Requests by language and by top files/folders.
- Purpose: Identify hotspots in the codebase; measure refactor campaigns or documentation sprints.
Different tasks value different trade-offs. Use stats to route requests to the right model.
A) Architecture & Design (deep reasoning, long context)
- Priorities: quality, coherence, fewer but more capable calls
- Ideal signals: higher edit ratio on complex files, acceptable higher latency
- Guidance: Prefer premium models for long-form reasoning. Accept higher
$latency$ if outcomes (edit ratio, reduced rework) are strong.
B) Code Generation (implementation, boilerplate)
- Priorities: speed and decent accuracy, volume throughput
- Ideal signals: high requests with low latency, steady edit ratio
- Guidance: Use fast, cost-efficient models when median latency spikes or cost per request rises.
C) Refactoring & Cleanup (multi-file hygiene, correctness)
- Priorities: balanced reasoning and speed, consistent acceptance of edits
- Ideal signals: stable edit ratio, improved file touch rate with minimal rework
- Guidance: If refactors stall (low edit ratio, high rework), step up to a more capable model for these turns only.
D) General Project Activities (reviews, small fixes, browsing)
- Priorities: frictionless flow, low latency, predictable responses
- Ideal signals: quick round-trip times, minimal context needs
- Guidance: Default to fast models; escalate only when the task reveals deeper complexity.
Practical heuristics (rules of thumb):
- If
$latency_{median}$ > target for simple tasks, switch to a faster model. - If
$edit_ratio$ falls below a threshold for complex tasks, try a higher-capability model. - If model usage concentrates >80% on one model across all tasks, experiment with a cascade (start cheap/fast, escalate only when needed).
At the moment of asking Copilot, apply a short checklist:
- What task type is this?
- simple completion, non-critical → prefer fast/cheap
- refactor/migration, correctness-critical → prefer balanced/higher capability
- architecture/design → prefer premium reasoning
- What does my recent data say?
- For this task type, which model gave higher
$edit_ratio$ under acceptable$latency$ ? - Did the daily trend show cost/latency spikes for my default choice?
- Choose model:
- “Baseline”: fastest viable model
- “Escalate”: if quality indicators lag (low acceptance, high rework), move to next tier for this turn only
- “Fallback”: low-cost option when nearing daily budgets
- Log and learn:
- Watch how KPIs respond; iterate on thresholds weekly.
Recommended thresholds (tune to your tolerance):
- High Latency (for fast loops):
$latency_{median} > 2000\text{ ms}$ for code-gen/inline work - Low Effectiveness (complex work):
$edit_ratio < 0.5$ over last 10+ turns of refactoring/design - Cost Spike (if tracked): daily cost > 2× rolling average
- Context Pressure (if tracked): repeated context overflow events in a session
Each alert should recommend an action:
- “High latency on inline edits” → switch to faster model for short turns
- “Low effectiveness on refactors” → escalate to higher-capability model
- “Cost spike” → reduce capability for low-stakes tasks today
Morning (setup)
- Review KPI chips for your default range (e.g., 7d).
- Scan the daily requests chart—are we trending up/down? Any anomalies yesterday?
During work
- For each turn, use the decision framework (Section 6).
- Watch the activity feed for quick validation of agent/model routing.
End of day
- Check model distribution—are you overusing a single model?
- If latency spiked for routine tasks, adjust defaults to a faster model tomorrow.
Weekly review (30 minutes)
- Compare edit ratio and latency across your top 2–3 models.
- Capture lessons learned: “For refactors in Typescript, Model X consistently yields higher acceptance with ~+0.5s latency—worth it.”
- Update personal model cascade playbook.
Track the least you need:
- DO: counts, latency, filenames (or redacted paths), languages, agent/model labels.
- DO NOT: persist source code or PII unless explicitly allowed.
Retention and redaction:
- Keep aggregated metrics for 30–90 days.
- Redact sensitive file paths on export or share aggregated-only data.
You already have the building blocks to deliver the analytics and views described above:
Data and KPIs
- KPI computation and filters come from (
AnalyticsService.getKpis), with helper accessors like (AnalyticsService.getDailyRequests), (AnalyticsService.getAgents), (AnalyticsService.getModels), and (AnalyticsService.getActivity). - The turn mapping that feeds these metrics includes sessionId, agent, model, filePath, language, latencyMs, and modelRequests (toolCallRounds).
Accurate request counting
- The real backend call counting via toolCallRounds is already leveraged in the usage model: model usage is computed from raw session results.
Panels and components (how to present)
- History panel model/view orchestrate component-level updates:
- Usage History Model
- Usage History View
- Recommended components match your micro-architecture:
- KPI chips → Kpis from
getKpis - Daily requests chart →
getDailyRequests - Agents/models lists →
getAgents,getModels - Activity feed →
getActivity
- KPI chips → Kpis from
Filters and ranges
- Use the panel’s global filters (time range, workspace) to scope metrics. History model applies a user-friendly time range mapping and refreshes downstream view models.
- Default to fast models for short-turn tasks; escalate by exception.
- For design/refactor work, quality beats speed—measure by edit ratio and reduced rework.
- Don’t fight your trends: if a model is consistently slow in your environment, switch for latency-sensitive tasks.
- Separate “thinking” turns (higher-capability) from “doing” turns (faster/cheaper). Your stats will reflect clean gains.
- Keep your daily chart and KPI chips in view—behavior follows what you measure.
- Cost tracking per model (adds cost-per-request and cost-efficiency indices).
- Task tagging per turn (architecture, code-gen, refactor) for per-task model benchmarking.
- Rolling percentiles (P50/P90/P95) for latency stability insights: visualize
$P50, P90, P95$ side-by-side. - Anomaly hints (sudden latency spikes, dips in edit ratio) with automatic “try-faster/try-stronger” suggestions.
- Team view (if shared): compare model preferences and identify best practices to broadcast.
- Measure a compact set of KPIs that tie directly to choices you can make at the keyboard.
- Visualize them in a way that makes “what to do next” obvious.
- Apply a lightweight decision framework per request to route to the right model for the task at hand.
- Review trends weekly; codify your personal model cascade and evolve it as projects change.
Outcome: Higher effectiveness at lower cost, with fewer regressions and a clearer understanding of where each model shines for your work.