Home
The Regression Testing Framework for AI Agents.
First time using AgentBench? Start with the Web-Dashboard-Guide -- it explains what every page does, how the Web Dashboard and CLI work together, and walks you through your first test in 3 steps.
npm install -g agentbench
agentbench init
agentbench test5 minutes to your first agent test.
AgentBench brings software engineering rigor to AI agents — replay, assertions, coverage, regression detection, and CI — the same tools you expect when testing code, now for your AI layer.
AgentBench is to AI Agents what Jest is to JavaScript.
| Without AgentBench | With AgentBench |
|---|---|
| "I think the new prompt is better" | Score improved 7.2 → 9.1 (+26%) |
| Manual spot-checking | Automated test suites with assertions |
| No idea if GPT → Claude breaks things | Cross-model replay catches drift instantly |
| Can't reproduce failures | Full trace — every call, every tool use |
console.log(agentResponse) as test suite |
agentbench test in CI |
- Replay — Record and replay agent executions deterministically (zero LLM cost)
-
Assertions — Chainable DSL:
expect(run).tool("search").toBeCalled().output().toContain("answer") - Evaluation — 14 rule evaluators + LLM-as-Judge across 8 dimensions
- Coverage — 4D analysis: Prompt, Workflow, Tool, Edge-case
- Regression Detection — Auto-flag score/token/cost/latency regressions
- CI-Ready — GitHub Actions, GitLab CI, CircleCI, JUnit export
- 15 Packages — Core, Providers, Adapters, Config, VS Code Extension
| Section | |
|---|---|
| Getting-Started | Install, init, and first test in 5 minutes |
| Core-Concepts | Replay, Assertions, Evaluation, Coverage, Snapshots |
| Guides | Testing patterns, CI/CD, Custom Providers, Migration |
| CLI-Reference | All 13 CLI commands with flags |
| API-Reference | REST API endpoints with curl examples |
| Config-Reference | Complete agentbench.config.ts reference |
| Assertion-DSL | All 22 matchers API reference |
| Cookbook | Recipes for common testing scenarios |
| Examples | 14 official examples indexed by difficulty |
| Architecture | System design and data flow |
| Provider-Ecosystem | 12+ LLM providers + build your own |
| VS-Code-Extension | Extension features and setup |
| FAQ | Frequently asked questions |
| Roadmap | v0.3 → v1.0 plan |
| Contributing | How to contribute |
15 packages including 8 provider plugins: OpenAI, Anthropic, Gemini, DeepSeek, Azure OpenAI, OpenRouter, Groq, Ollama — plus LangGraph, MCP, CrewAI, LlamaIndex adapters.
14 official examples — from simple Hello Agent to complex Multi-Agent Workflows.