Core Concepts

Core Concepts

AgentBench brings software engineering rigor to AI agents through a set of interconnected core concepts. Each concept addresses a fundamental challenge in testing non-deterministic, LLM-powered systems. This page provides an overview of all core concepts with links to the detailed documentation in the main repository.

---

Replay: Deterministic, Cross-Model, and Batch

Replay is AgentBench's mechanism for recording a complete agent execution trace -- including every LLM response, tool call, token usage, and timing data -- and replaying it on demand. This is the foundation that makes reliable AI agent testing possible.

Deterministic Replay records a run once and replays it exactly, forever, with zero LLM API cost. The Tracer in @agentbench/core intercepts every API call and captures the full request/response cycle. When you replay, the engine returns the recorded responses instead of calling the real API. This means you can run your full agent test suite on every PR in CI without spending a cent on LLM calls -- and the results are 100% deterministic.

import { buildDeterministicReplay, applyReplayOverrides } from '@agentbench/core'

const replayConfig = buildDeterministicReplay(originalRunConfig, { seed: 42 })
const newRunConfig = applyReplayOverrides(replayConfig)

Cross-Model Replay records a run with one model (e.g., GPT-4o) and replays it against a different model (e.g., Claude Sonnet). The engine sends the same inputs to the new model and compares behavior, metrics, and scores against the original. This is invaluable for model migration testing, cost optimization (does GPT-4o-mini produce the same quality?), and provider compatibility detection.

Batch Replay runs the same scenario N times and aggregates statistics with mean, standard deviation, min, and max. This is essential for measuring variance in non-deterministic agents, establishing statistical significance for A/B experiments, and understanding the true reliability of your agent (not just a single lucky run).

Read the full guide: Replay

---

Assertions: Chainable DSL with 25 Matchers

AgentBench provides a fluent, chainable assertion DSL that reads like English and composes like functional code. At its heart is the expect() function, which builds a pipeline of assertions against a RunResult. Each assertion evaluates a specific aspect of the agent's execution trace.

The DSL is organized into 6 categories of matchers:

1. Status Matchers (2) -- Assert on the run's final status (completed, errored)
2. Tool Matchers (7) -- Assert on which tools were called, with what arguments, how many times
3. Output Matchers (12) -- Assert on the agent's final text output (contains, regex, schema, snapshot)
4. Token Matchers (5) -- Assert on token consumption (under budget, within range)
5. Latency Matchers (4) -- Assert on execution timing (total, time-to-first-token)
6. Score Matchers (4) -- Assert on evaluation scores from LLM judges (above threshold, within range)

import { expect } from '@agentbench/core'

const result = await expect(runResult)
  .status().toBeCompleted()
  .tool('search_docs').toBeCalled()
  .tool('search_docs').toBeCalledWith({ query: 'refund' })
  .output().toContain('30 days')
  .output().not.toContain('hallucinated fact')
  .tokens().toBeLessThan(4096)
  .latency().toBeLessThan(5000)
  .latency().firstToken().toBeLessThan(500)
  .score('correctness').toBeGreaterThan(7)
  .run()

Assertions come in two fundamental flavors: rule-based (deterministic, zero-cost, instant -- checking objective trace facts) and score-based (LLM-evaluated, measuring subjective qualities like correctness and safety). The best practice is to always pair them: check the basics with rules first, then validate quality with scores. The DSL also supports compound assertions with .all() (AND) and .any() (OR) for complex logic without nested callbacks.

Read the full guide: Assertions See the complete DSL reference: Assertion-DSL

---

Evaluation: 14 Rules + 8 LLM Judges + Hybrid

AgentBench provides three distinct evaluation strategies that can be used individually or combined.

Rule Evaluators (14 total) are deterministic functions that inspect the agent's execution trace and return pass/fail with a score of 0 or 1. They check objective, measurable facts: exact match, contains substring, regex match, JSON schema validation, tool called/not called/called with/called N times, status code, latency thresholds, token thresholds, and cost thresholds. These are instant, free, and never flaky.

LLM Judges (8 dimensions) use a separate LLM (the "judge model") to score subjective qualities on a 0-10 scale. The 8 built-in dimensions are: correctness, faithfulness, safety, relevance, completeness, reasoning, conciseness, and tool_usage. Each dimension has a dedicated prompt template with a detailed scoring rubric. For cost efficiency, use GPT-4o-mini or Claude Haiku as the judge model -- evaluating all 8 dimensions costs approximately $0.008 per test case.

Hybrid Judges blend rule evaluators and LLM judges with three configurable strategies:

• rule_first (default): Rules run first. If all pass, the LLM judge is skipped entirely -- saving cost. If any fail, the LLM judge provides a blended score.
• llm_first: LLM judge runs first. If confident (score >= 7 or <= 3), rules are skipped. If uncertain, rules break the tie.
• parallel: Both run simultaneously, producing the most robust evaluation.

For high-stakes evaluations, you can run a judge pool with multiple judges and voting strategies (majority, unanimous, weighted) to reduce individual judge bias.

import { runHybridJudge, runJudgePool } from '@agentbench/core'

const result = await runHybridJudge({
  strategy: 'rule_first',
  rules: [
    { type: 'contains', params: { substring: 'refund' } },
    { type: 'tokens_lt', params: { threshold: 4096 } },
  ],
  llmJudge: {
    provider: 'openai', model: 'gpt-4o-mini',
    dimensions: ['correctness', 'faithfulness'],
  },
}, context, callLLM)

Read the full guide: Evaluation

---

Coverage: 4D Model

Traditional code coverage (line, branch, statement) does not apply to AI agents. Agents have no single control-flow graph -- their behavior depends on prompts, model choice, available tools, and LLM non-determinism. AgentBench replaces this with a 4-dimensional coverage model:

1. Prompt Coverage -- How many of your defined prompt variables and their possible values have been tested. If your agent supports tone, language, user_type, and urgency variables but only 7 of 12 combinations have been tested, prompt coverage is 58%.

2. Workflow Coverage -- The diversity of execution paths your agent takes. A "path" is the ordered sequence of step types in a trace (e.g., llm_call -> tool_call -> llm_call -> response). Two runs following the same path contribute coverage of 1 unique path. Low workflow coverage signals that error-recovery, multi-tool chains, and early-termination paths are untested.

3. Tool Coverage -- What percentage of your agent's available tools have been called in at least one test run. Also tracks which tool combinations appear together and identifies gaps in complex workflow testing.

4. Edge Case Coverage -- How many defined edge cases have been explicitly tested. AgentBench provides 6 default edge cases (empty input, max length, unicode, timeout, error tool, no tools) and lets you define domain-specific ones like PII handling, SQL injection attempts, prompt injection, and contradictory requests.

const report = calculateCoverage({
  projectId: 'proj_abc123',
  runs: allCompletedRuns,
  promptVariables: { tone: ['professional', 'casual', 'empathetic'], language: ['en', 'zh'] },
  availableTools: ['search', 'lookup', 'create_ticket', 'escalate'],
  edgeCases: [{ name: 'empty_input', description: 'Empty user input', testHint: 'Test with empty string' }],
})
console.log(`Overall coverage: ${report.overall}%`)

You can set minimum coverage thresholds that cause CI failures when not met, and track coverage trends over time. Each uncovered path in the report includes a suggestedTest that tells you exactly what to test next.

Read the full guide: Coverage

---

Snapshots

Snapshot testing is inspired by Jest's toMatchSnapshot() but designed for the unique challenges of AI agents. A snapshot captures the complete agent state: system prompt, model configuration, tools and their parameters, input messages, execution trace (every LLM response and tool call), environmental metadata, and final output. This creates a comprehensive, version-controlled baseline for regression detection.

Snapshots are stored in .agentbench/snapshots/ organized by project. They should be committed to your repository so every developer and CI runner has access to the same baselines. When a snapshot comparison fails, AgentBench's compareSnapshots() function detects changes across multiple dimensions -- prompt text, model name/temperature, tool additions/removals, execution options -- and produces a detailed diff.

import { expect } from '@agentbench/core'

// Assert output matches the stored snapshot
await expect(runResult).output().toMatchSnapshot('refund-baseline').run()

# Update snapshots when behavior changes intentionally
agentbench test --project customer-support --update-snapshots

Snapshots underpin the replay system. Use restoreConfigFromSnapshot() to recreate a run configuration from any snapshot, enabling you to re-run or re-evaluate historical agent executions.

Read the full guide: Snapshots

---

Non-Determinism: 5-Layer Strategy

Testing AI agents is fundamentally harder than testing traditional software because LLMs are probabilistic systems. The same prompt can produce different responses due to temperature, sampling strategies, GPU floating-point non-determinism, and provider-side model updates. AgentBench's entire architecture is designed around this central challenge with a 5-layer strategy:

1. Layer 1: Deterministic Replay -- Eliminate non-determinism entirely. Record once, replay deterministically with zero LLM cost. Use in CI for every PR.

2. Layer 2: Tool-Based Assertions -- Assert on deterministic facts from the trace (tool calls, token counts, latency). These are objective and never flaky.

3. Layer 3: Fuzzy Output Assertions -- Use toContain(), toMatchRegex(), and toMatchSchema() instead of exact string matching. Tolerate natural language variation.

4. Layer 4: Score-Based Assertions -- Use LLM judge scores with tolerance bands (>= 7, not == 10). Accept that judges themselves have some variance.

5. Layer 5: Batch Replay -- Measure variance statistically. Run N times, use means and standard deviations. Replace "latency is 3200ms" with "P95 latency is 4100ms."

The core principle: test behavior, not output. Test THAT the agent called the right tools, THAT its response contains key facts, and THAT its answer is approximately correct -- do NOT test THAT the agent said exact words.

Read the full guide: Non-Determinism

---

Putting It All Together

These core concepts work together as an integrated system:

Record baseline with Snapshots
    -> Replay deterministically in CI (zero cost)
    -> Assert with the DSL (tool calls, output, tokens, latency)
    -> Evaluate with rules + LLM judges (quality dimensions)
    -> Measure Coverage across 4 dimensions
    -> Handle Non-Determinism with the 5-layer strategy

AgentBench is to AI Agents what Jest is to JavaScript. Each concept mirrors a Jest concept adapted for the unique challenge of testing non-deterministic, LLM-powered systems.

---

Related pages: Guides | Cookbook | Examples | Config-Reference | Assertion-DSL