Getting Started

Getting Started

AgentBench is the regression testing framework for AI agents. This guide gets you from zero to your first passing test in under five minutes.

5 Minutes to First Test

# 1. Install AgentBench globally
npm install -g agentbench

# 2. Create a new project (interactive CLI)
agentbench init

# 3. Run your first test
agentbench test

That is it. The interactive agentbench init walks you through project setup, detects your API keys, and generates a working "Hello Agent" test. Run agentbench test and you will see green checkmarks in your terminal.

What agentbench init Does

When you run agentbench init, you go through an interactive walkthrough:

$ agentbench init

 █████╗  ██████╗ ███████╗███╗  ██╗████████╗██████╗ ███████╗███╗  ██╗ ██████╗██╗  ██╗
██╔══██╗██╔════╝ ██╔════╝████╗ ██║╚══██╔══╝██╔══██╗██╔════╝████╗ ██║██╔════╝██║  ██║
███████║██║  ██╗ █████╗  ██╔██╗██║   ██║   ██████╦╝█████╗  ██╔██╗██║██║     ███████║
██╔══██║██║  ╚██╗██╔══╝  ██║╚████║   ██║   ██╔══██╗██╔══╝  ██║╚████║██║     ██╔══██║
██║  ██║╚██████╔╝███████╗██║ ╚███║   ██║   ██████╦╝███████╗██║ ╚███║╚██████╗██║  ██║
╚═╝  ╚═╝ ╚═════╝ ╚══════╝╚═╝  ╚══╝   ╚═╝   ╚═════╝ ╚══════╝╚═╝  ╚══╝ ╚═════╝╚═╝  ╚═╝

                    The Regression Testing Framework for AI Agents
                                   v0.3.0

  Welcome to AgentBench! Let's set up your first agent test suite.

▶ Step 1/5: Project Setup
  Project name: (my-agent-tests)
  Language: TypeScript (default) / JavaScript
  Package manager: npm / pnpm / yarn

▶ Step 2/5: Provider Detection
  🔍 Scanning environment for API keys...

  ✔ OpenAI       OPENAI_API_KEY        → Auto-configured
  ✔ Anthropic    ANTHROPIC_API_KEY     → Auto-configured
  ✔ Gemini       GEMINI_API_KEY        → Auto-configured
  ✗ DeepSeek     DEEPSEEK_API_KEY      (not found)

▶ Step 3/5: Choose a Starter Template
  ▶ Hello Agent     Minimal single-agent test — best for learning
    Customer Support   Multi-turn support agent with tool calling
    RAG Agent          Retrieval-augmented generation agent
    Empty              Blank project — for experienced users

▶ Step 4/5: Configuration
  ▶ Test directory:     tests/
  ▶ Output format:      terminal + JSON + HTML
  ▶ CI mode:            GitHub Actions

▶ Step 5/5: Generating Project...
  ✔ Created agentbench.config.ts
  ✔ Created tests/hello-agent.test.ts
  ✔ Created .github/workflows/agentbench.yml
  ✔ Installed dependencies

  ✨ AgentBench is ready!

Non-Interactive Mode

For CI or scripting, pass flags to skip the prompts:

agentbench init --yes                          # Accept all defaults
agentbench init --template hello-agent         # Pick a specific template
agentbench init --provider openai,anthropic    # Specify providers
agentbench init --ci                           # CI-optimized setup

Understanding Your Project

After agentbench init, your project looks like this:

my-agent-tests/
├── agentbench.config.ts          # Main configuration (generated, works out of the box)
├── src/
│   └── agent.ts                  # Your agent under test
├── tests/
│   ├── hello-agent.test.ts       # Your first test (generated by init)
│   └── __snapshots__/            # Auto-generated snapshots
├── dataset/
│   └── hello-agent.queries.csv   # Sample test inputs
├── report/                       # Generated test reports (gitignored)
├── .agentbench/                  # Internal state (gitignored)
├── .github/workflows/
│   └── agentbench.yml            # CI workflow (GitHub Actions)
└── .env                          # Your API keys (gitignored)

The Config File

The generated agentbench.config.ts uses defineConfig for full TypeScript support:

import { defineConfig } from 'agentbench'

export default defineConfig({
  name: 'my-agent-tests',
  description: 'Test suite for my AI agent',

  agent: {
    entry: './src/agent.ts',
  },

  providers: {
    openai: {
      apiKey: process.env.OPENAI_API_KEY,
      defaultModel: 'gpt-4o',
    },
    anthropic: {
      apiKey: process.env.ANTHROPIC_API_KEY,
      defaultModel: 'claude-sonnet-4-5',
    },
  },

  test: {
    testDir: './tests',
    timeout: 30000,
    retry: 2,
    maxConcurrency: 4,
  },

  assertions: {
    scoreThreshold: 7,
    maxTokens: 4096,
    maxLatency: 30000,
  },

  replay: {
    storage: '.agentbench/snapshots',
    mode: 'deterministic',
  },

  evaluation: {
    judges: ['correctness', 'faithfulness', 'safety'],
    judgeModel: 'openai/gpt-4o-mini',
  },

  coverage: {
    dimensions: ['prompt', 'workflow', 'tool', 'edge-case'],
    thresholds: {
      prompt: 0.8,
      workflow: 0.7,
      tool: 0.9,
      'edge-case': 0.5,
    },
  },

  report: {
    formats: ['terminal', 'json', 'html', 'junit'],
    outputDir: './report',
  },

  ci: {
    provider: 'github-actions',
    commentOnPR: true,
    failOnRegression: true,
  },
})

Configuration is resolved in this priority order (highest to lowest):

1. CLI flags (--timeout, --concurrency, etc.)
2. agentbench.config.ts (local project)
3. agentbench.config.js / agentbench.config.mjs / agentbench.config.json
4. "agentbench" key in package.json
5. Environment variables (AGENTBENCH_*)
6. Built-in defaults

Your First Test

AgentBench tests follow a Jest-like API that feels familiar:

import { expect, test, suite } from 'agentbench'
import { myAgent } from '../src/agent'

suite('Hello Agent', () => {
  test('should respond to a simple greeting', async () => {
    const result = await myAgent.run('Hello! Who are you?')

    await expect(result)
      .status().toBeCompleted()
      .output().toContain('assistant')
      .tokens().toBeLessThan(1000)
      .latency().toBeLessThan(15000)
      .score('correctness').toBeGreaterThan(7)
      .run()
  })

  test('should handle a factual question', async () => {
    const result = await myAgent.run('What is the capital of France?')

    await expect(result)
      .status().toBeCompleted()
      .output().toContain('Paris')
      .score('correctness').toBeGreaterThan(8)
      .run()
  })

  test.replay('greeting should be consistent', async () => {
    const result = await myAgent.run('Hello! Who are you?')
    await expect(result).toMatchSnapshot()
  })
})

Running Tests

Default Output (Jest-style)

$ agentbench test

▶ AgentBench v0.3.0 — Running tests for "my-agent-tests"

  Model: openai/gpt-4o  |  Concurrency: 4  |  Timeout: 30s

  Running 5 test(s) in 2 suite(s)...

  ✓  Customer Support Agent › Greeting
     should welcome the customer (1.2s, 342 tokens, $0.0034)

  ✓  Customer Support Agent › Refund Policy
     should call the refund_policy tool (0.8s, 256 tokens, $0.0026)

  ✓  Customer Support Agent › Refund Policy
     should not hallucinate the refund period (1.5s, 412 tokens, $0.0041)

  ✓  Customer Support Agent › Escalation
     should escalate when unable to help (0.9s, 198 tokens, $0.0020)

  ✗  Customer Support Agent › Escalation
     should not escalate for simple queries (FAILED)

     ┌─────────────────────────────────────────────────────────────┐
     │  Assertion failed: tool("escalate_to_human").not.toBeCalled() │
     │                                                               │
     │  Expected: tool "escalate_to_human" was NOT called              │
     │  Received: tool "escalate_to_human" was called 1 time           │
     │                                                               │
     │  ▶ View full trace: agentbench replay --run run_abc123         │
     └─────────────────────────────────────────────────────────────┘

──────────────────────────────────────────────────────────────────
  Tests:  4 passed, 1 failed, 5 total
  Suites: 1 passed, 1 failed, 2 total
  Time:   5.6s
  Tokens: 1,756 (prompt: 1,012, completion: 744)
  Cost:   $0.0176
──────────────────────────────────────────────────────────────────

  ✗ 1 test failed. Check the report for details:
    ▶ HTML Report:  report/index.html
    ▶ JSON Report:  report/results.json

Watch Mode

Re-run tests automatically when files change:

agentbench test --watch

  👀 Watching for changes in tests/ and src/...

  ── File changed: src/agent.ts ──

  Re-running...

  ✓  Customer Support Agent › Greeting             (1.3s)
  ✓  Customer Support Agent › Refund Policy         (0.9s)
  ...

  All 5 tests passed. (5.5s, $0.0182)

  Press 'q' to quit, 'r' to re-run, 'f' to filter

Key CLI Flags

Flag	Description
--suite <name>	Run a specific test suite
--test <name>	Run a specific test case
--grep <pattern>	Run tests matching a pattern
--watch	Watch mode -- re-run on file changes
--ui	Open the interactive test UI
--json	Output results as JSON
--junit	Output results as JUnit XML
--replay	Run in replay mode (cached responses, zero LLM cost)
--update-snapshots	Update stored snapshots
--coverage	Generate coverage report
--verbose	Show full trace on failure
--debug	Show debug output (request/response bodies)
--concurrency <n>	Max parallel tests
--timeout <ms>	Per-test timeout in milliseconds
--retry <n>	Retry failed tests n times
--ci	CI mode: JSON + JUnit output, non-interactive
--no-cost	Do not display cost estimates

Running in CI

AgentBench is built for CI/CD. The agentbench init command generates a GitHub Actions workflow at .github/workflows/agentbench.yml that runs on every PR:

name: AgentBench

on:
  pull_request:
    paths:
      - 'src/agent/**'
      - 'prompts/**'
      - 'tools/**'
      - 'agentbench.config.*'
      - 'tests/**'
      - 'dataset/**'
  push:
    branches: [main]

jobs:
  agent-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
      - name: Install dependencies
        run: npm ci

      - name: Run AgentBench
        id: agentbench
        uses: agentbench/github-action@v0.3
        with:
          mode: pr-check
          fail-on-regression: true
          comment-on-pr: true
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}

For other CI providers, use agentbench test --ci --json --junit to get machine-readable output. The JUnit XML report is compatible with GitLab CI, CircleCI, Jenkins, and any tool that consumes JUnit format.

Testing Without an API Key

Two ways to run tests without calling an LLM provider:

1. Replay mode: Record a run once, then replay it deterministically without any API calls:

   agentbench test --replay

2. Ollama (local models): Use the @agentbench/ollama provider to run tests against models running on your own machine. No API key required.

Next Steps

• Core-Concepts — Understand agent testing, replay, assertions, evaluation, coverage, snapshots, and how AgentBench handles non-determinism.
• Guides — Learn to test OpenAI agents, Anthropic agents, set up CI/CD, write custom providers and judges, manage datasets, and migrate from older versions.
• CLI-Reference — Every command and flag, documented in detail.
• Config-Reference — All defineConfig options explained.
• Assertion-DSL — The complete chainable assertion API.
• Cookbook — Real-world recipes: prompt regressions, model migration, cost budgets, safety testing, A/B experiments.
• Examples — 14 official reference implementations covering every major agent framework.