Config Reference

Configuration Reference

AgentBench is configured through a single agentbench.config.ts file at your project root. The file exports a default config using the defineConfig helper from @agentbench/config, which provides full TypeScript autocompletion and Zod-powered validation.

Read the complete reference: Configuration Reference

---

Quick Start

import { defineConfig } from '@agentbench/config'

export default defineConfig({
  name: 'my-agent-project',
  description: 'Test suite for my customer support agent',
  agent: {
    provider: 'openai',
    model: 'gpt-4o',
    systemPrompt: 'You are a helpful assistant.',
    temperature: 0.3,
    maxTokens: 4096,
  },
  test: {
    testDir: './tests',
    timeout: 30000,
    retry: 2,
  },
})

---

Top-Level Structure

The configuration is organized into the following sections:

Section	Description
name	Project name (string)
description	Project description (string)
agent	Agent-under-test settings (model, prompts, tools, sampling)
providers	LLM provider connection details (API keys, base URLs)
test	Test execution settings (discovery, timeout, concurrency)
assertions	Default assertion thresholds applied to all tests
replay	Replay recording and playback settings
evaluation	LLM-as-Judge configuration
coverage	Coverage analysis dimensions and thresholds
report	Report output formats and directories
ci	CI/CD integration settings

---

agent -- Agent Under Test

Configuration for the agent you are testing. This is the most important section.

Option	Type	Default	Description
entry	string	undefined	Path to the agent entry file
provider	string	undefined	Provider ID (must match a key in providers or a built-in ID)
model	string	undefined	Model name (e.g., gpt-4o, claude-sonnet-4-20250514)
systemPrompt	string	undefined	System prompt injected at the start of every conversation
temperature	number	0	Sampling temperature (0-2). Lower = more deterministic
maxTokens	number	4096	Maximum tokens per LLM completion
topP	number	undefined	Nucleus sampling parameter (0-1)
frequencyPenalty	number	undefined	Frequency penalty (-2.0 to 2.0). Positive reduces repetition
presencePenalty	number	undefined	Presence penalty (-2.0 to 2.0). Positive encourages new topics
stop	string[]	undefined	Stop sequences that halt generation
tools	ToolConfig[]	undefined	Tools available to the agent

Each tool in the tools array has name, description, and parameters (JSON Schema for the tool's arguments).

agent: {
  provider: 'openai',
  model: 'gpt-4o',
  systemPrompt: `You are a customer support agent.
Always verify the customer's identity before processing refunds.`,
  temperature: 0.3,
  maxTokens: 4096,
  topP: 0.95,
  tools: [
    {
      name: 'search_orders',
      description: 'Search customer orders by email or order ID',
      parameters: {
        type: 'object',
        properties: {
          email: { type: 'string', description: 'Customer email address' },
          order_id: { type: 'string', description: 'Order ID (optional)' },
        },
        required: ['email'],
      },
    },
  ],
}

---

providers -- LLM Provider Configuration

Configuration for each LLM provider. Keys are provider IDs, values are ProviderConfig objects.

Option	Type	Default	Description
provider	ProviderId	undefined	Provider identifier ('openai', 'anthropic', 'gemini', etc.)
apiKey	string	undefined	API key (prefer process.env)
apiBase	string	undefined	Base URL override (for proxies or self-hosted)
organization	string	undefined	Org ID (OpenAI-specific)
headers	Record<string, string>	undefined	Extra HTTP headers
timeout	number	undefined	Request timeout in ms
maxRetries	number	undefined	Max retries on transient failures

providers: {
  openai: {
    provider: 'openai',
    apiKey: process.env.OPENAI_API_KEY,
    timeout: 60000,
    maxRetries: 3,
  },
  anthropic: {
    provider: 'anthropic',
    apiKey: process.env.ANTHROPIC_API_KEY,
    maxRetries: 2,
  },
  self_hosted: {
    provider: 'custom',
    apiKey: process.env.LLAMA_API_KEY,
    apiBase: 'http://localhost:8080/v1',
  },
}

---

test -- Test Execution

Test execution configuration following Jest/Vitest naming conventions.

Option	Type	Default	Description
testDir	string	'./tests'	Directory containing test files
testMatch	string | string[]	undefined	Glob patterns to match test files within testDir
timeout	number	30000	Per-test timeout in milliseconds
retry	number	2	Number of retries for failed tests
maxConcurrency	number	4	Maximum concurrent test executions
bail	boolean	undefined	Stop after the first test failure
setupFiles	string[]	undefined	Files to run before each test suite
globalSetup	string	undefined	Path to a global setup module
globalTeardown	string	undefined	Path to a global teardown module

test: {
  testDir: './tests',
  testMatch: ['**/*.test.ts', '**/*.spec.ts'],
  timeout: 60000,
  retry: 1,
  maxConcurrency: 2,
  bail: true,
  setupFiles: ['./tests/setup.ts'],
}

---

assertions -- Default Assertion Thresholds

Default assertion thresholds applied to every test case. Individual tests can override these.

Option	Type	Default	Description
scoreThreshold	number	7	Minimum score on 1-10 scale for a passing test
maxTokens	number	4096	Maximum allowed tokens per test run
maxLatency	number	30000	Maximum allowed end-to-end latency (ms)
requiredTools	string[]	undefined	Tools the agent must call at least once
forbiddenTools	string[]	[]	Tools the agent must never call

assertions: {
  scoreThreshold: 8,
  maxTokens: 2048,
  maxLatency: 15000,
  requiredTools: ['search_docs', 'verify_identity'],
  forbiddenTools: ['delete_record', 'sudo_execute', 'raw_sql_query'],
}

---

replay -- Replay Configuration

Option	Type	Default	Description
enabled	boolean	false	Enable replay recording and playback
storageDir	string	'.agentbench/replays'	Directory for replay snapshots
maxReplays	number	undefined	Maximum replay snapshots to retain
ttl	number	undefined	Time-to-live for snapshots in seconds (0 = indefinite)
mode	'deterministic' | 'llm' | 'mixed'	'deterministic'	Replay strategy

Replay Modes:

Mode	Behavior
deterministic	Play back exact recorded LLM responses. No API calls. Fastest and cheapest
llm	Re-send prompts to the LLM with the same parameters. Useful for testing model behavior changes
mixed	Deterministic for tools, LLM for text responses. Balances speed with freshness

replay: {
  enabled: true,
  storageDir: '.agentbench/replays',
  maxReplays: 100,
  ttl: 86400 * 30,   // 30 days
  mode: 'deterministic',
}

---

evaluation -- LLM-as-Judge Configuration

Option	Type	Default	Description
judges	JudgeDimension[]	['correctness', 'faithfulness', 'safety']	Judge dimensions to evaluate
judgeModel	string	'openai/gpt-4o-mini'	Model for LLM-as-Judge (should be cheap and fast)
scoreThreshold	number	7	Minimum score (1-10) for a passing test
rubric	string	undefined	Custom scoring rubric for the LLM judge

8 Judge Dimensions: correctness, faithfulness, safety, relevance, completeness, reasoning, conciseness, tool_usage

evaluation: {
  judges: ['correctness', 'faithfulness', 'safety', 'completeness'],
  judgeModel: 'openai/gpt-4o-mini',
  scoreThreshold: 7,
  rubric: `Score from 1-10 where:
1-3: Completely incorrect or harmful
4-6: Partially correct with significant errors
7-8: Mostly correct with minor issues
9-10: Excellent, comprehensive, and helpful`,
}

---

coverage -- Coverage Analysis

Option	Type	Default	Description
dimensions	CoverageDimensionName[]	['prompt', 'workflow', 'tool', 'edge-case']	Coverage dimensions to analyze
thresholds	Record<string, number>	{}	Per-dimension coverage thresholds (0-100)
exclude	string[]	undefined	Patterns to exclude from coverage analysis

Coverage Dimensions: prompt, workflow, tool, state, edge, edge-case

coverage: {
  dimensions: ['prompt', 'workflow', 'tool', 'edge-case'],
  thresholds: {
    prompt: 80,
    workflow: 70,
    tool: 90,
    'edge-case': 60,
  },
  exclude: ['**/deprecated/**', '**/experimental/**'],
}

---

report -- Report Generation

Option	Type	Default	Description
formats	ReportFormat[]	['terminal', 'json', 'html']	Output formats
outputDir	string	'./agentbench-report'	Report output directory
includeTrace	boolean	undefined	Include full execution traces in reports
includeMetrics	boolean	undefined	Include detailed metrics breakdowns

ReportFormat: 'terminal' | 'json' | 'html' | 'markdown' | 'junit'

report: {
  formats: ['terminal', 'json', 'html', 'junit'],
  outputDir: './agentbench-report',
  includeTrace: true,
  includeMetrics: true,
}

---

ci -- CI/CD Configuration

Option	Type	Default	Description
provider	CIProvider	'github-actions'	CI platform
failOnThreshold	boolean	true	Fail the CI run if any threshold is breached
commentOnPR	boolean	true	Post a PR comment with test results
artifactsDir	string	'./agentbench-artifacts'	Directory for CI artifacts

CIProvider: 'github-actions' | 'gitlab-ci' | 'circleci' | 'jenkins' | 'none'

ci: {
  provider: 'github-actions',
  failOnThreshold: true,
  commentOnPR: true,
  artifactsDir: './agentbench-artifacts',
}

---

Minimal Configuration

import { defineConfig } from '@agentbench/config'

export default defineConfig({
  agent: {
    provider: 'openai',
    model: 'gpt-4o-mini',
    systemPrompt: 'You are a helpful assistant.',
  },
})

---

Production Configuration

import { defineConfig } from '@agentbench/config'

export default defineConfig({
  name: 'customer-support-tests',
  description: 'Test suite for the customer support AI agent',

  providers: {
    openai: {
      provider: 'openai',
      apiKey: process.env.OPENAI_API_KEY,
      timeout: 60000,
      maxRetries: 3,
    },
    anthropic: {
      provider: 'anthropic',
      apiKey: process.env.ANTHROPIC_API_KEY,
      maxRetries: 2,
    },
  },

  agent: {
    provider: 'openai',
    model: 'gpt-4o',
    systemPrompt: `You are a customer support agent.
Always greet the customer, verify their identity, and provide accurate information.
Never share personal data or make promises you cannot keep.`,
    temperature: 0.3,
    maxTokens: 4096,
    topP: 0.95,
    tools: [
      {
        name: 'search_kb',
        description: 'Search the knowledge base',
        parameters: {
          type: 'object',
          properties: { query: { type: 'string' } },
          required: ['query'],
        },
      },
    ],
  },

  test: {
    testDir: './tests',
    testMatch: ['**/*.test.ts'],
    timeout: 45000,
    retry: 2,
    maxConcurrency: 2,
    bail: true,
  },

  assertions: {
    scoreThreshold: 7,
    maxTokens: 4096,
    maxLatency: 30000,
    requiredTools: ['search_kb'],
    forbiddenTools: ['delete_record', 'execute_sql'],
  },

  replay: {
    enabled: true,
    storageDir: '.agentbench/replays',
    maxReplays: 50,
    ttl: 86400 * 14,
    mode: 'deterministic',
  },

  evaluation: {
    judges: ['correctness', 'faithfulness', 'safety', 'completeness', 'tool_usage'],
    judgeModel: 'openai/gpt-4o-mini',
    scoreThreshold: 7,
  },

  coverage: {
    dimensions: ['prompt', 'workflow', 'tool', 'edge-case'],
    thresholds: { prompt: 80, workflow: 70, tool: 85, 'edge-case': 50 },
  },

  report: {
    formats: ['terminal', 'json', 'html', 'junit'],
    outputDir: './agentbench-report',
    includeMetrics: true,
  },

  ci: {
    provider: 'github-actions',
    failOnThreshold: true,
    commentOnPR: true,
    artifactsDir: './agentbench-artifacts',
  },
})

---

Important Notes

• API keys: Never hardcode. Always use process.env.YOUR_API_KEY.
• Provider ID matching: The agent.provider value must match a key in providers or be a built-in provider ID.
• Timeout vs maxLatency: test.timeout cancels the test; assertions.maxLatency marks it failed but does not cancel. Set test.timeout higher than assertions.maxLatency.
• Judge model cost: Always use a cheap model for judging (gpt-4o-mini, gpt-4.1-nano, or claude-haiku-4-5).
• Coverage thresholds: Start with 70-80%. Setting 100% will cause CI failures for any untested edge case.

---

Related pages: CLI-Reference | Assertion-DSL | Core-Concepts | Guides