Migration Guide

Migration Guide: v0.2.0 to v0.3.0

AgentBench v0.3.0 ("Adoption") is a major feature release that maintains backward compatibility with v0.2.0. Your existing agentbench.config.ts, tests, snapshots, and project data will continue to work. However, v0.3.0 introduces new capabilities and reorganized defaults that you should adopt to get the full benefit.

Read the full migration guide: Migration Guide

---

Summary of Changes

Area	v0.2.0	v0.3.0	Impact
Config format	Manual agentbench.config.ts	New defineConfig() helper from @agentbench/config	Optional; old format still works
Snapshot path	<project>/.snapshots/	.agentbench/snapshots/	Move your snapshots or keep old path via config
Report output	./report/	./agentbench-report/	Update CI scripts
CLI commands	11 commands	13 commands (dataset, benchmark added)	New capabilities
Provider packages	2 providers (OpenAI, Anthropic)	8 providers (+ Gemini, DeepSeek, Azure OpenAI, OpenRouter, Groq, Ollama)	New provider options
Adapter packages	1 adapter (generic)	3 adapters (+ LangGraph, MCP)	Framework-specific tracing
Config validation	Manual checks at runtime	Zod schemas + defineConfig() with IDE autocomplete	Better DX
Test format	Custom test functions	Jest/Vitest-style test() / describe() blocks	Familiar patterns
Dataset system	None	Full dataset management (7 import formats)	Systematic testing

---

Step-by-Step Migration Checklist

Step 1: Update Dependencies

Update your AgentBench packages to v0.3.0:

pnpm add @agentbench/core@^0.3.0 @agentbench/config@^0.3.0

# If using providers:
pnpm add @agentbench/openai@^0.3.0
pnpm add @agentbench/anthropic@^0.3.0

# Optional new providers:
pnpm add @agentbench/gemini@^0.3.0
pnpm add @agentbench/deepseek@^0.3.0

The new @agentbench/config package is optional but strongly recommended. It provides defineConfig() with full TypeScript autocompletion and Zod validation.

---

Step 2: Update Configuration Format

v0.2.0 configs were plain objects. v0.3.0 introduces defineConfig() from @agentbench/config:

Before (v0.2.0):

// agentbench.config.ts
module.exports = {
  agent: {
    provider: 'openai',
    model: 'gpt-4o',
    systemPrompt: 'You are a helpful assistant.',
  },
  test: {
    testDir: './tests',
  },
}

After (v0.3.0):

// agentbench.config.ts
import { defineConfig } from '@agentbench/config'

export default defineConfig({
  name: 'my-agent-project',
  description: 'Test suite for my customer support agent',
  agent: {
    provider: 'openai',
    model: 'gpt-4o',
    systemPrompt: 'You are a helpful assistant.',
    temperature: 0.3,
    maxTokens: 4096,
  },
  test: {
    testDir: './tests',
    timeout: 30000,
    retry: 2,
  },
})

The old format still works. You can migrate at your own pace. The benefit of defineConfig() is IDE autocompletion for every option and Zod validation at startup that catches misconfigurations before tests run.

---

Step 3: Learn New CLI Commands

v0.3.0 adds two new top-level CLI commands:

agentbench dataset -- Full dataset management:

agentbench dataset create --name "customer-queries" --format json
agentbench dataset import --file queries.csv --format csv
agentbench dataset list
agentbench dataset validate --name "customer-queries"
agentbench dataset split --name "customer-queries" --train 80 --test 20

agentbench benchmark -- Performance benchmarking:

agentbench benchmark --project customer-support --runs 10
agentbench benchmark --project customer-support --models gpt-4o,claude-sonnet-4 --runs 5

All existing v0.2.0 commands (init, test, run, replay, snapshot, evaluate, coverage, report, serve, dev, doctor) continue to work unchanged.

---

Step 4: Move Snapshots to New Path

v0.2.0 stored snapshots at <project>/.snapshots/. v0.3.0 stores them at .agentbench/snapshots/ by default.

# Move existing snapshots to the new location
mkdir -p .agentbench/snapshots
mv .snapshots/* .agentbench/snapshots/ 2>/dev/null || echo "No old snapshots to move"

# Or keep the old path via config:
# In agentbench.config.ts, set: replay: { storageDir: '.snapshots' }

If you don't move them, old snapshots will not be automatically discovered by v0.3.0. Use the config override if you prefer to keep the old path.

---

Step 5: Update .gitignore

v0.3.0 uses different paths. Update your .gitignore:

# AgentBench
- .snapshots/
+ .agentbench/snapshots/          # Keep this -- commit snapshots!
  .agentbench/replays/            # Optional to commit
+ .agentbench/cache/
+ reports/
+ agentbench-report/
+ agentbench-artifacts/
  .env.agentbench                 # Never commit API keys

Important: Snapshots should be committed to version control (they are small, typically 10-50 KB each, and are essential for CI and team collaboration). The .env.agentbench file containing API keys should never be committed.

---

Step 6: Update Report Paths in CI Scripts

The report output directory changed from ./report/ to ./agentbench-report/. Update any CI scripts that reference the old path:

Before:

- name: Upload test results
  uses: actions/upload-artifact@v4
  with:
    name: agent-test-results
    path: ./report/junit.xml

After:

- name: Upload test results
  uses: actions/upload-artifact@v4
  with:
    name: agent-test-results
    path: ./agentbench-report/junit.xml

---

Step 7: Adopt New Provider Packages

If you were using the OpenAI or Anthropic SDKs directly in v0.2.0, v0.3.0 provides dedicated provider packages that wrap the SDKs and intercept calls automatically for tracing:

Before (v0.2.0):

import OpenAI from 'openai'

const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY })
const response = await openai.chat.completions.create({ /* ... */ })

After (v0.3.0):

import { OpenAIProvider } from '@agentbench/openai'

const provider = new OpenAIProvider({ apiKey: process.env.OPENAI_API_KEY })
// The provider intercepts calls for automatic tracing, token counting, and cost tracking

If you do not want to change your code, the old approach still works. The provider packages are an opt-in improvement for automatic tracing.

---

Step 8: Adopt Template Structure for Examples

If you maintain example projects based on AgentBench, v0.3.0 standardizes the template structure:

examples/<name>/
├── agentbench.config.ts
├── agent/
│   └── index.ts
├── tests/
│   ├── basic.test.ts
│   ├── replay.test.ts
│   └── dataset.json
├── .env.example
├── .github/workflows/
│   └── agentbench.yml
├── package.json
└── README.md

Old example structures continue to work but won't benefit from the new auto-discovery conventions.

---

Backward Compatibility Guarantee

v0.3.0 maintains full backward compatibility with v0.2.0. Specifically:

• Existing agentbench.config.ts files without defineConfig() still work (a deprecation warning is shown)
• Old snapshot paths continue to be discoverable
• All v0.2.0 CLI commands work with the same flags
• Test files written in the old format continue to execute
• The v0.2.0 assertion DSL (expect(result).tool().toBeCalled()) is identical

The changes in v0.3.0 are additive: new features are available if you choose to adopt them, but nothing breaks if you don't.

---

Quick Migration Script

Run this script to automate the most common migration steps:

#!/bin/bash
# migrate-v0.2-to-v0.3.sh

echo "=== AgentBench v0.2.0 -> v0.3.0 Migration ==="

# 1. Update packages
pnpm add @agentbench/core@^0.3.0 @agentbench/config@^0.3.0

# 2. Move snapshots
mkdir -p .agentbench/snapshots
if [ -d ".snapshots" ]; then
  mv .snapshots/* .agentbench/snapshots/ 2>/dev/null
  echo "Moved snapshots from .snapshots/ to .agentbench/snapshots/"
fi

# 3. Create new directories
mkdir -p .agentbench/cache

# 4. Update .gitignore
if ! grep -q ".agentbench/cache/" .gitignore 2>/dev/null; then
  cat >> .gitignore << 'EOF'

# AgentBench (v0.3.0)
.agentbench/cache/
reports/
agentbench-report/
agentbench-artifacts/
.env.agentbench
EOF
  echo "Updated .gitignore"
fi

echo "=== Migration complete ==="
echo "Next steps:"
echo "  1. Review your agentbench.config.ts -- consider adopting defineConfig()"
echo "  2. Run 'agentbench test' to verify everything works"
echo "  3. Check the new CLI commands: 'agentbench dataset' and 'agentbench benchmark'"
echo "  4. See new provider packages: pnpm list @agentbench/*"

---

Getting Help

If you encounter any issues during migration:

• Check the FAQ for common questions
• Open a GitHub Issue with the migration label
• Run agentbench doctor to diagnose configuration issues

---

Related pages: Guides | Config-Reference | Getting-Started | FAQ