Web Dashboard Guide

Web Dashboard Guide

Target audience: First-time AgentBench web dashboard users Reading time: 10 minutes Prerequisites: None

---

One-Sentence Summary

AgentBench is a testing framework, like Jest. The difference: Jest tests code, AgentBench tests AI Agents.

It has two parts:

Part	Purpose	Analogy
Web Dashboard	Manage tests, view results	Like Jest's HTML Report
CLI	Execute tests	Like jest or npx jest

Web = view, CLI = run. When you create a Test Suite in the Dashboard, you are only defining what to test -- nothing has been executed yet.

---

Each Dashboard Page Explained

Home: Dashboard

Project overview showing:

• Pass Rate -- percentage of completed tests that passed
• Test Suites -- how many test suites you have created
• Total Runs -- total number of test executions
• Avg Cost -- average cost per test

Where does the data come from? From the results of running agentbench test via CLI.

Tests Page

Where you create test suites. A Test Suite contains multiple Test Cases:

Test Suite: "Customer Support"
  ├── Test Case 1: Greeting Test
  │    Input: "Hello, who are you?"
  │    Expected: reply contains "assistant"
  │
  ├── Test Case 2: Refund Test
  │    Input: "What is your refund policy?"
  │    Expected: reply contains "refund, policy"
  │
  └── Test Case 3: Escalation Test
       Input: "I want to speak to a manager!"
       Expected: reply contains "escalate, human"

Creating a Test Suite = writing the test definition. Nothing has been executed yet.

Runs Page

Test execution history. Each agentbench test invocation adds a record here.

Run statuses:

• PENDING -- created but not yet executed (most common for new users)
• RUNNING -- executing right now
• PASSED -- all assertions passed
• FAILED -- some assertions failed

Compare Page

Compare two Runs side by side. For example:

• Did the score improve after changing the prompt?
• Did token usage increase after switching models?

Datasets Page

Manage test datasets. Import CSV/JSON/JSONL files to batch-manage test inputs.

Experiments Page

A/B testing. Compare the performance of two prompts or two models.

Snapshots Page

Snapshot management. Records the complete state of an agent execution for zero-cost regression testing.

Coverage Page

4D coverage analysis: Prompt coverage, Workflow coverage, Tool coverage, Edge Case coverage.

---

New User 3-Step Walkthrough

Step 1: Create a Test Suite in the Web Dashboard

1. Open http://localhost:3000/tests
2. Click New Suite
3. Fill in:
   • Suite Name: My First Test
   • Project: select E2E Final Test
   • Add a Test Case:
     • Name: Greeting
     • Input: {"messages": [{"role": "user", "content": "Hello"}]}
     • Expected Keywords: hello, assistant
4. Click Create Test Suite

Step 2: Execute Tests via CLI

# Navigate to your agentbench project root
cd /your-agentbench-project

# Run the tests (requires an OpenAI API Key)
agentbench test

The CLI reads your Test Suite, calls the LLM, checks assertions, and writes results to the database.

Step 3: View Results in the Web Dashboard

Refresh http://localhost:3000/runs -- you will see a new Run with status PASSED or FAILED. Click into it to see:

• Each assertion's result
• Token usage and cost
• Execution duration

---

Common Questions

Q: Why can't I see my Test Suite in the Runs page?

A: Creating a Test Suite is not the same as executing tests. A Test Suite is a "definition"; a Run is an "execution". You need to run agentbench test via CLI to create a Run.

Q: Why do all Runs show PENDING?

A: PENDING means "created but never executed". These runs are seed data or records created via the API that have never called an LLM.

Q: Why does the data show 0 or Unknown?

A: Because the Run is in PENDING status -- there is no actual execution data (tokens, latency, scores).

Q: Do I need both the Dashboard and CLI?

A: Yes. The Web Dashboard is the management layer (create tests, view results); the CLI is the execution layer (actually runs the tests). Think of it like Jest's config file vs the jest command.

Q: Can I run tests directly in the Web Dashboard without the CLI?

A: Not currently. The Web Dashboard does not execute agent code -- that requires your agent runtime environment and API keys. The CLI runs on your machine, calls LLMs, and writes results to the database.

---

Full Guide

This is a condensed wiki version. Read the complete guide in the repo: Web Dashboard Guide

---

Related pages: Home | Getting-Started | FAQ | Core-Concepts