Guides

How-To Guides

Step-by-step guides for common AgentBench tasks. Each guide links to the full documentation in the main repository. Use these to learn how to test specific agent types, integrate with CI/CD pipelines, build custom integrations, manage test data, and migrate from v0.2.0.

Whether you are testing your first agent, setting up automated CI regression testing, or building a custom provider for an unsupported LLM API, these guides provide the practical steps you need.

---

Testing OpenAI Agents

A complete guide to testing AI agents built with OpenAI's GPT and o-series models. Covers everything from basic setup with @agentbench/openai to advanced patterns.

What you'll learn:

• Installing and configuring @agentbench/openai with your API key
• Automatic tracing of every LLM call -- no code changes required
• Streaming response testing with time-to-first-token measurement
• Tool/function calling verification with tool().toBeCalledWith()
• Structured output validation against JSON schemas
• Reasoning model support for o1, o3, and o4-mini
• Cost tracking per run with token-level accuracy
• Common troubleshooting: rate limits, streaming timeouts, schema mismatches

The @agentbench/openai package wraps the official OpenAI SDK and intercepts every API call transparently -- your agent code stays exactly the same. The only change is the import:

// Before: direct OpenAI SDK
import OpenAI from 'openai'
const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY })

// After: AgentBench wraps it for automatic tracing
import { OpenAIProvider } from '@agentbench/openai'
const provider = new OpenAIProvider({ apiKey: process.env.OPENAI_API_KEY })

Read the full guide

---

Testing Anthropic Claude Agents

A complete guide to testing agents built with Anthropic's Claude models. Claude's API format differs from OpenAI's in important ways -- this guide covers the specifics.

What you'll learn:

• Installing and configuring @agentbench/anthropic with your API key
• Claude-specific message format (system + messages block vs. systemPrompt)
• Extended thinking/reasoning support with visible thinking budgets
• Tool use testing with tool_use blocks and content block types
• Streaming response handling and chunk assembly
• Cost tracking for Claude models with Anthropic's pricing tiers
• Troubleshooting: tool_use block parsing errors, multi-turn message ordering, empty content blocks

import { AnthropicProvider } from '@agentbench/anthropic'

const provider = new AnthropicProvider({
  apiKey: process.env.ANTHROPIC_API_KEY,
  maxRetries: 2,
})

// AgentBench automatically traces every Messages API call

Read the full guide

---

CI/CD Integration

A comprehensive guide to integrating AgentBench into CI/CD pipelines across all major platforms: GitHub Actions, GitLab CI, CircleCI, and Jenkins. This is the single highest-leverage practice for maintaining AI agent quality over time -- every PR is validated before merge.

What you'll learn:

• The --ci flag: how it changes AgentBench behavior for pipeline environments (non-interactive output, strict thresholds, JUnit XML reports, no TTY escape codes)
• GitHub Actions workflow: auto-generated .github/workflows/agentbench.yml via agentbench init --ci
• GitLab CI pipeline: .gitlab-ci.yml with parallel test execution
• CircleCI configuration: .circleci/config.yml with resource class sizing for LLM testing
• Cost budget enforcement in CI: prevent PRs that increase token usage beyond thresholds
• Regression detection automation: AgentBench auto-flags regressions in scores, tokens, cost, and latency
• Flaky test handling: retry strategies, flaky test quarantine, and reporting
• Docker-based execution: reproducible CI environments with pinned provider SDK versions
• Artifact upload: JUnit XML for test result dashboards, HTML reports for PR previews

Read the full guide

---

GitHub Actions Setup

A deep dive into GitHub Actions integration for AgentBench. Goes beyond the auto-generated workflow to cover advanced patterns.

What you'll learn:

• Generating the complete workflow: agentbench init --ci
• The auto-generated workflow structure with replay-based regression (zero LLM cost) and periodic full evaluation with LLM judges
• Composite action at .github/actions/agentbench/: reusable across multiple workflows
• PR comment templates: automated comment with test results, coverage change, and regression warnings
• Check run integration: test results appear in GitHub's Checks tab
• Secret management: storing API keys as repository secrets and referencing them securely
• Matrix testing: running tests across multiple models/providers in parallel
• Artifact handling: uploading JUnit XML, HTML reports, and trace files as workflow artifacts
• Cost estimation: the GitHub Actions summary includes estimated LLM costs for the run
• Troubleshooting: rate limiting, timeout configuration, runner resource limits, working directory issues

Read the full guide

---

Building Custom Providers

A complete tutorial for building third-party LLM provider integrations for AgentBench. Two approaches based on your API's format.

What you'll learn:

• Approach 1 (90% of cases): Extend OpenAICompatibleProvider from @agentbench/provider-utils. Override just 3 methods: getBaseURL(), getDefaultHeaders(), and getProviderId(). Suitable for any API that speaks the OpenAI Chat Completions format.
• Approach 2 (non-OpenAI format): Implement AgentBenchProvider interface from scratch. Handle createChatCompletion(), createChatCompletionStream(), countTokens(), calculateCost(), healthCheck(), and listModels().
• Full walkthrough: building an "AcmeAI" provider from scratch, step by step
• Provider lifecycle: initialization, authentication, chat completion, streaming, token counting, cost calculation
• Registration and auto-discovery: how to make AgentBench CLI and config recognize your provider
• Token counting: tiktoken for OpenAI-format, character-based fallback, custom tokenizers
• Cost calculation: per-model pricing tables, input vs. output token pricing, image/audio token costs
• Testing your provider: test suite patterns for provider correctness
• Publishing to npm: package structure, peer dependencies, versioning

Read the full guide

---

Building Custom LLM Judges

A guide to creating custom evaluation judges beyond the 8 built-in dimensions (correctness, faithfulness, safety, relevance, completeness, reasoning, conciseness, tool_usage). Use this when you need domain-specific evaluation criteria.

What you'll learn:

• The Judge interface: receiving the execution trace and returning scored dimensions
• Writing effective evaluation prompts: scoring rubrics with explicit anchors at each level (0-10), bias mitigation, avoiding leading language
• Domain-specific dimensions: brand tone compliance, legal/regulatory safety, medical accuracy, internal style guide adherence, code quality, translation quality
• Meta-evaluation: testing your judge against known-good and known-bad outputs to calibrate scoring
• Hybrid mode: combining custom judges with rule evaluators using rule_first, llm_first, or parallel strategies
• Performance considerations: judge model selection (GPT-4o-mini vs. GPT-4o vs. Claude Haiku), prompt token optimization, caching repeated evaluations

Read the full guide

---

Dataset Management

A complete guide to managing datasets for systematic agent testing. Datasets are the foundation of scalable agent evaluation -- define inputs and expected outputs once, and run your agent against every item.

What you'll learn:

• Creating datasets from scratch: item structure, metadata, schema definition
• Importing from 7 formats: CSV, JSON, JSONL, HuggingFace datasets, OpenAI Evals format, DeepEval format, LangSmith exports
• Dataset validation: schema checks, duplicate detection, required field enforcement
• Splitting: train/test/validation splits with stratified sampling
• Sampling: random subsets for cost-efficient testing in CI
• Versioning: dataset tags, changelog, and reproducibility
• CLI management: agentbench dataset create, import, list, validate, split, sample, export, delete
• Best practices: dataset size guidelines, diversity requirements, avoiding benchmark contamination

Read the full guide

---

Migration Guide: v0.2.0 to v0.3.0

A step-by-step migration guide for existing AgentBench users upgrading from v0.2.0 to v0.3.0 ("Adoption"). v0.3.0 maintains backward compatibility -- your existing configs, tests, and snapshots will continue to work. This guide helps you adopt the new capabilities.

What the migration covers:

• New defineConfig() helper from @agentbench/config with IDE autocompletion and Zod validation
• New snapshot storage path (.agentbench/snapshots/ instead of .snapshots/)
• Updated report output directory (./agentbench-report/ instead of ./report/)
• Two new CLI commands: agentbench dataset for dataset management, agentbench benchmark for performance benchmarking
• Six new provider packages: Gemini, DeepSeek, Azure OpenAI, OpenRouter, Groq, Ollama
• New adapter packages: LangGraph, MCP
• Template structure changes for examples
• Updated .gitignore patterns
• Complete migration checklist with an automated migration shell script

Read the full guide

---

What to Read Next

• New to AgentBench? Start with Getting-Started and the Hello Agent example
• Setting up CI? Read the CI/CD Integration and GitHub Actions guides above
• Building a provider? Read the Building Custom Providers guide and see Provider-Ecosystem
• Migrating from v0.2.0? Read the Migration Guide and check Migration-Guide

Related pages: Core-Concepts | Cookbook | Examples | Config-Reference