Skip to content

Guides

Jump to bottom
Mickl edited this page Jul 10, 2026 · 1 revision

How-To Guides

Step-by-step guides for common AgentBench tasks. Each guide links to the full documentation in the main repository. Use these to learn how to test specific agent types, integrate with CI/CD pipelines, build custom integrations, manage test data, and migrate from v0.2.0.

Whether you are testing your first agent, setting up automated CI regression testing, or building a custom provider for an unsupported LLM API, these guides provide the practical steps you need.

Testing OpenAI Agents

A complete guide to testing AI agents built with OpenAI's GPT and o-series models. Covers everything from basic setup with @agentbench/openai to advanced patterns.

What you'll learn:

  • Installing and configuring @agentbench/openai with your API key
  • Automatic tracing of every LLM call -- no code changes required
  • Streaming response testing with time-to-first-token measurement
  • Tool/function calling verification with tool().toBeCalledWith()
  • Structured output validation against JSON schemas
  • Reasoning model support for o1, o3, and o4-mini
  • Cost tracking per run with token-level accuracy
  • Common troubleshooting: rate limits, streaming timeouts, schema mismatches

The @agentbench/openai package wraps the official OpenAI SDK and intercepts every API call transparently -- your agent code stays exactly the same. The only change is the import:

// Before: direct OpenAI SDK
import OpenAI from 'openai'
const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY })

// After: AgentBench wraps it for automatic tracing
import { OpenAIProvider } from '@agentbench/openai'
const provider = new OpenAIProvider({ apiKey: process.env.OPENAI_API_KEY })

Read the full guide

Testing Anthropic Claude Agents

A complete guide to testing agents built with Anthropic's Claude models. Claude's API format differs from OpenAI's in important ways -- this guide covers the specifics.

What you'll learn:

  • Installing and configuring @agentbench/anthropic with your API key
  • Claude-specific message format (system + messages block vs. systemPrompt)
  • Extended thinking/reasoning support with visible thinking budgets
  • Tool use testing with tool_use blocks and content block types
  • Streaming response handling and chunk assembly
  • Cost tracking for Claude models with Anthropic's pricing tiers
  • Troubleshooting: tool_use block parsing errors, multi-turn message ordering, empty content blocks
import { AnthropicProvider } from '@agentbench/anthropic'

const provider = new AnthropicProvider({
  apiKey: process.env.ANTHROPIC_API_KEY,
  maxRetries: 2,
})

// AgentBench automatically traces every Messages API call

Read the full guide

CI/CD Integration

A comprehensive guide to integrating AgentBench into CI/CD pipelines across all major platforms: GitHub Actions, GitLab CI, CircleCI, and Jenkins. This is the single highest-leverage practice for maintaining AI agent quality over time -- every PR is validated before merge.

What you'll learn:

  • The --ci flag: how it changes AgentBench behavior for pipeline environments (non-interactive output, strict thresholds, JUnit XML reports, no TTY escape codes)
  • GitHub Actions workflow: auto-generated .github/workflows/agentbench.yml via agentbench init --ci
  • GitLab CI pipeline: .gitlab-ci.yml with parallel test execution
  • CircleCI configuration: .circleci/config.yml with resource class sizing for LLM testing
  • Cost budget enforcement in CI: prevent PRs that increase token usage beyond thresholds
  • Regression detection automation: AgentBench auto-flags regressions in scores, tokens, cost, and latency
  • Flaky test handling: retry strategies, flaky test quarantine, and reporting
  • Docker-based execution: reproducible CI environments with pinned provider SDK versions
  • Artifact upload: JUnit XML for test result dashboards, HTML reports for PR previews

Read the full guide

GitHub Actions Setup

A deep dive into GitHub Actions integration for AgentBench. Goes beyond the auto-generated workflow to cover advanced patterns.

What you'll learn:

  • Generating the complete workflow: agentbench init --ci
  • The auto-generated workflow structure with replay-based regression (zero LLM cost) and periodic full evaluation with LLM judges
  • Composite action at .github/actions/agentbench/: reusable across multiple workflows
  • PR comment templates: automated comment with test results, coverage change, and regression warnings
  • Check run integration: test results appear in GitHub's Checks tab
  • Secret management: storing API keys as repository secrets and referencing them securely
  • Matrix testing: running tests across multiple models/providers in parallel
  • Artifact handling: uploading JUnit XML, HTML reports, and trace files as workflow artifacts
  • Cost estimation: the GitHub Actions summary includes estimated LLM costs for the run
  • Troubleshooting: rate limiting, timeout configuration, runner resource limits, working directory issues

Read the full guide

Building Custom Providers

A complete tutorial for building third-party LLM provider integrations for AgentBench. Two approaches based on your API's format.

What you'll learn:

  • Approach 1 (90% of cases): Extend OpenAICompatibleProvider from @agentbench/provider-utils. Override just 3 methods: getBaseURL(), getDefaultHeaders(), and getProviderId(). Suitable for any API that speaks the OpenAI Chat Completions format.
  • Approach 2 (non-OpenAI format): Implement AgentBenchProvider interface from scratch. Handle createChatCompletion(), createChatCompletionStream(), countTokens(), calculateCost(), healthCheck(), and listModels().
  • Full walkthrough: building an "AcmeAI" provider from scratch, step by step
  • Provider lifecycle: initialization, authentication, chat completion, streaming, token counting, cost calculation
  • Registration and auto-discovery: how to make AgentBench CLI and config recognize your provider
  • Token counting: tiktoken for OpenAI-format, character-based fallback, custom tokenizers
  • Cost calculation: per-model pricing tables, input vs. output token pricing, image/audio token costs
  • Testing your provider: test suite patterns for provider correctness
  • Publishing to npm: package structure, peer dependencies, versioning

Read the full guide

Building Custom LLM Judges

A guide to creating custom evaluation judges beyond the 8 built-in dimensions (correctness, faithfulness, safety, relevance, completeness, reasoning, conciseness, tool_usage). Use this when you need domain-specific evaluation criteria.

What you'll learn:

  • The Judge interface: receiving the execution trace and returning scored dimensions
  • Writing effective evaluation prompts: scoring rubrics with explicit anchors at each level (0-10), bias mitigation, avoiding leading language
  • Domain-specific dimensions: brand tone compliance, legal/regulatory safety, medical accuracy, internal style guide adherence, code quality, translation quality
  • Meta-evaluation: testing your judge against known-good and known-bad outputs to calibrate scoring
  • Hybrid mode: combining custom judges with rule evaluators using rule_first, llm_first, or parallel strategies
  • Performance considerations: judge model selection (GPT-4o-mini vs. GPT-4o vs. Claude Haiku), prompt token optimization, caching repeated evaluations

Read the full guide

Dataset Management

A complete guide to managing datasets for systematic agent testing. Datasets are the foundation of scalable agent evaluation -- define inputs and expected outputs once, and run your agent against every item.

What you'll learn:

  • Creating datasets from scratch: item structure, metadata, schema definition
  • Importing from 7 formats: CSV, JSON, JSONL, HuggingFace datasets, OpenAI Evals format, DeepEval format, LangSmith exports
  • Dataset validation: schema checks, duplicate detection, required field enforcement
  • Splitting: train/test/validation splits with stratified sampling
  • Sampling: random subsets for cost-efficient testing in CI
  • Versioning: dataset tags, changelog, and reproducibility
  • CLI management: agentbench dataset create, import, list, validate, split, sample, export, delete
  • Best practices: dataset size guidelines, diversity requirements, avoiding benchmark contamination

Read the full guide

Migration Guide: v0.2.0 to v0.3.0

A step-by-step migration guide for existing AgentBench users upgrading from v0.2.0 to v0.3.0 ("Adoption"). v0.3.0 maintains backward compatibility -- your existing configs, tests, and snapshots will continue to work. This guide helps you adopt the new capabilities.

What the migration covers:

  • New defineConfig() helper from @agentbench/config with IDE autocompletion and Zod validation
  • New snapshot storage path (.agentbench/snapshots/ instead of .snapshots/)
  • Updated report output directory (./agentbench-report/ instead of ./report/)
  • Two new CLI commands: agentbench dataset for dataset management, agentbench benchmark for performance benchmarking
  • Six new provider packages: Gemini, DeepSeek, Azure OpenAI, OpenRouter, Groq, Ollama
  • New adapter packages: LangGraph, MCP
  • Template structure changes for examples
  • Updated .gitignore patterns
  • Complete migration checklist with an automated migration shell script

Read the full guide

What to Read Next

Related pages: Core-Concepts | Cookbook | Examples | Config-Reference

AgentBench v0.3.0 · GitHub · Report Issue · Changelog

AgentBench v0.3.0

Home

Getting Started

Core Concepts

  • Core-Concepts
  • Replay & Snapshots
  • Assertions & Evaluation
  • Coverage & Non-Determinism

How-To Guides

Reference

Cookbook

  • Cookbook
  • Prompt Regressions
  • Model Migration
  • Cost Budgets
  • Safety Testing
  • A/B Testing

Community

Ecosystem

Clone this wiki locally