VS Code Extension

VS Code Extension

The official AgentBench VS Code extension brings agent testing directly into your editor. Write, run, debug, and replay agent tests without leaving VS Code. The extension integrates with AgentBench's core engine and provides visual feedback for test results, execution traces, coverage data, and snapshot diffs.

---

Installation

Install from the VS Code Marketplace or via the command line:

code --install-extension agentbench.agentbench-vscode

Or search for "AgentBench" in the VS Code Extensions panel (Cmd+Shift+X / Ctrl+Shift+X).

Requirements:

• VS Code 1.90+
• AgentBench v0.3.0+ installed in your project (npm install -g agentbench or pnpm add @agentbench/core)
• An agentbench.config.ts file in your workspace root

---

Features

13 Commands via Command Palette

All extension functionality is accessible through the Command Palette (Cmd+Shift+P / Ctrl+Shift+P). Prefix commands with "AgentBench" to see all options:

Command	Description
AgentBench: Run Test	Run the test at the current cursor position
AgentBench: Debug Test	Run the test with the VS Code debugger attached
AgentBench: Replay Test	Replay the test deterministically from a stored snapshot
AgentBench: Run All Tests	Run the entire test suite for the current project
AgentBench: Run Test Suite	Run a specific test suite
AgentBench: Show Test History	Open the test history panel
AgentBench: View Trace	Open the trace viewer for the last test run
AgentBench: Compare Snapshots	Diff the current output against a stored snapshot
AgentBench: View Coverage Report	Open the 4D coverage report
AgentBench: Open Dashboard	Open the AgentBench web dashboard
AgentBench: Initialize Project	Run agentbench init in the current workspace
AgentBench: Generate CI Workflow	Generate .github/workflows/agentbench.yml
AgentBench: Clear Test Results	Clear cached test results and diagnostics

---

CodeLens: Inline Test Actions

Above each test definition, the extension displays interactive CodeLens buttons:

• Run -- Execute the test and show results inline
• Debug -- Attach the VS Code debugger and step through the test
• Replay -- Replay the test from a stored snapshot (zero LLM cost)

//  Run |  Debug |  Replay          <-- CodeLens appears above this line
test('customer support handles refund requests', async () => {
  const result = await agentbench.run({ /* ... */ })
  await expect(result).tool('search').toBeCalled().run()
})

Clicking "Run" executes the test and displays pass/fail status with assertion results as inline decorations. Clicking "Debug" launches the VS Code debugger with breakpoints working inside test code. Clicking "Replay" runs the test in deterministic mode against a stored snapshot.

---

Sidebar Tree View

The AgentBench sidebar (accessible via the AgentBench icon in the activity bar) provides a tree view with five sections:

Run -- Execute tests with one click. Shows:

• Individual test files with run/debug/replay buttons
• Test suites grouped by project
• "Run All" button at the top

History -- Browse past test runs:

• Chronological list of test executions
• Each entry shows: timestamp, pass/fail/skip counts, duration
• Click any entry to view its trace
• Filter by project, status, or date range

Suites -- Manage test suites:

• Tree view of all test suites with their test cases
• Add, remove, and reorder test cases
• Run individual suites directly from the tree

Coverage -- View 4D coverage:

• Overall coverage percentage at the top
• Four dimension bars: Prompt, Workflow, Tool, Edge Case
• Expand each dimension to see covered/uncovered items
• Click "Improve Coverage" to see suggested test cases

Snapshots -- Manage stored snapshots:

• List all snapshots grouped by project
• Each snapshot shows: name, creation date, tags
• Right-click to restore, compare, or delete
• "Create Snapshot" button to capture the current state

---

Status Bar Integration

The AgentBench status bar item shows the current project's test status at a glance:

• Checkmark -- All tests passing
• Warning triangle -- Some tests failing
• Error circle -- Tests cannot run (configuration issue)
• Hourglass -- Tests currently running

Click the status bar item to open the History view. The tooltip shows: last run time, pass/fail/skip counts, and overall coverage percentage.

---

Diagnostics from Test Failures

When a test fails, AgentBench publishes VS Code diagnostics directly into the editor:

• Failed assertions appear as error diagnostics on the relevant line
• Hover over a diagnostic to see the full assertion result (expected vs. actual)
• Diagnostics are cleared when tests pass again
• In the Problems panel, filter by "AgentBench" to see all test failures

This means you see test failures the same way you see TypeScript errors -- with red squiggly underlines and hover details.

---

Trace Viewer (Webview Panel)

The trace viewer is a dedicated VS Code webview panel that visualizes the complete execution trace of an agent test run:

• Timeline View -- Horizontal timeline showing every step (LLM calls, tool calls) with duration bars
• Step Details -- Click any step to see the full request/response payload, including messages, tools, temperature, and token usage
• Tool Call Inspector -- See tool names, arguments (pretty-printed JSON), and results/errors
• Token Breakdown -- Bar chart showing prompt vs. completion tokens per LLM call, with cost estimates
• Streaming Replay -- For streaming responses, a simulated character-by-character replay of the agent's output
• Scores Panel -- If the run was evaluated, shows all dimension scores with the judge's reasoning

The trace viewer is invaluable for debugging: you can trace exactly what the LLM saw (input messages, tools), what it returned (content, tool calls), and what tools produced (results, errors) -- all without adding console.log statements to your agent code.

---

Configuration

The extension reads your agentbench.config.ts automatically. You can customize extension behavior in VS Code settings:

Setting	Default	Description
agentbench.autoRunOnSave	false	Auto-run tests when you save a test file
agentbench.showCodeLens	true	Show Run/Debug/Replay CodeLens above tests
agentbench.traceViewer.theme	"dark"	Theme for the trace viewer webview ("light", "dark", "auto")
agentbench.cliPath	"agentbench"	Path to the AgentBench CLI binary
agentbench.clearDiagnosticsOnClose	true	Clear diagnostics when closing a test file
agentbench.notificationLevel	"error"	Minimum level for notifications ("info", "warning", "error")

---

Getting Started

1. Install the extension from the VS Code Marketplace
2. Open a project that has agentbench.config.ts at the root
3. Open any test file (e.g., tests/refund.test.ts)
4. Click the "Run" CodeLens above any test
5. View results in the sidebar and the trace viewer

The extension automatically detects your AgentBench configuration and loads the correct project setup. No additional configuration is needed.

---

Related pages: Getting-Started | Core-Concepts | Guides | CLI-Reference