Glass Box - AI DevTools | Real-time LLM request inspector with context, token, performance, and replay debugging

[shukla10]

Summary of change : Adds a new webview-based developer panel for debugging and understanding how Copilot's AI system constructs and executes LLM requests — inspect context assembly, token budgets, and performance in real time. Accessible via Copilot: Open Glass Box AI DevTools command.

Why this matters: Today, developers building with LLMs have zero visibility into why AI responses behave the way they do — what context was assembled, how tokens were budgeted, or where latency comes from. This makes debugging AI behavior, diagnosing regressions, and evaluating model alternatives entirely guesswork across any LLM-powered experience

How it helps: Glass Box AI DevTools brings AI debuggability to VS Code by providing visiblity for the full request lifecycle — context assembly, token allocation, model reasoning, and timing — in a single inspectable panel, Turning interaction with AI from Blackbox to GlassBox. The built-in Replay feature lets developers re-send any captured query to a different model and compare responses side-by-side, enabling data-driven prompt tuning, model evaluation, and confident performance optimization.

What it does

• Context Inspector — view every context item sent to the model (system prompt, user message, history, tool results) with token counts and relevance scores
• Token Budget Visualizer — stacked bar chart and breakdown table showing prompt/completion/cached/reasoning token usage against the model's context window, with
  plain-English explanations of the math
• Performance Dashboard — latency, TTFT, tool call breakdown, cache hit rates, and session-level aggregates across all requests
• Replay — re-send a captured user query to a different model and compare the response side-by-side with the original (latency, token counts, output)
• Call Flow Timeline — visual strip showing the sequence of LLM calls in a session (title gen → intent → main → tool calls)

How to Test

1. Pull the changes from PR
2. Press Ctrl+Shift+D → select "VS Code" → press F5
3. In the Code OSS window: Ctrl+Shift+P → "Copilot: Open Glass Box AI DevTools"
4. Make a Copilot chat request — the panel shows context, tokens, performance, and replay tabs

30 sec Demo

GlassBox-AI.Debug.tools.mp4

[Copilot]

Pull request overview

Adds a new “Glass Box AI DevTools” surface to the Copilot extension that collects sanitized request-logger data and displays it in a dedicated webview panel for inspecting context assembly, token usage, performance, and replay behavior.

Changes:

• Introduces a new IGlassBoxService that builds bounded, sanitized aggregates from the existing request logger (context items, token budget, performance, response text).
• Adds a webview-based panel (tabs + request selector + replay) and a contribution/command to open it, plus a new configuration setting to enable capture.
• Adds unit tests for the aggregation logic and sensitive-data redaction utilities.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 10 comments.

Show a summary per file

File	Description
extensions/copilot/src/platform/configuration/common/configurationService.ts	Adds a new config key for enabling Glass Box capture.
extensions/copilot/src/extension/glassBox/vscode-node/glassBoxServiceImpl.ts	Implements request aggregation/sanitization from IRequestLogger.
extensions/copilot/src/extension/glassBox/vscode-node/glassBoxPanel.ts	Implements the Glass Box webview panel UI and replay orchestration.
extensions/copilot/src/extension/glassBox/vscode-node/glassBoxContribution.ts	Registers the open command, reacts to setting changes, manages panel lifecycle.
extensions/copilot/src/extension/glassBox/test/vscode-node/glassBoxService.spec.ts	Adds unit tests for the Glass Box service aggregation behavior.
extensions/copilot/src/extension/glassBox/test/node/sensitiveDataFilter.spec.ts	Adds unit tests for redaction/sanitization helpers.
extensions/copilot/src/extension/glassBox/common/types.ts	Defines shared types for aggregates and host/webview messaging.
extensions/copilot/src/extension/glassBox/common/sensitiveDataFilter.ts	Implements centralized redaction + preview/label sanitization.
extensions/copilot/src/extension/glassBox/common/glassBoxService.ts	Defines the IGlassBoxService interface/identifier.
extensions/copilot/src/extension/extension/vscode-node/services.ts	Registers IGlassBoxService in the extension DI container.
extensions/copilot/src/extension/extension/vscode-node/contributions.ts	Adds the Glass Box contribution to the extension’s contribution list.
extensions/copilot/package.nls.json	Adds setting description text for Glass Box enablement.
extensions/copilot/package.json	Contributes the command and the configuration property for Glass Box.

[shukla10]

@microsoft-github-policy-service agree