Cookbook

Cookbook -- Recipes for Common Scenarios

Practical, step-by-step recipes for solving common AI agent testing challenges with AgentBench. Each recipe provides a complete workflow you can adapt to your own project, with links to the full documentation in the main repository.

---

Catching Prompt Regressions

Prompt engineering is iterative. You tweak wording, add instructions, adjust the tone -- and sometimes you make things worse. This recipe shows you how to set up automated prompt regression detection so you catch quality drops before they reach production.

The workflow:

1. Baseline -- Record snapshots of your agent's behavior with the current prompt using agentbench test --suite <name> --snapshot-on-pass
2. Change -- Modify the prompt and re-run tests against the saved snapshots
3. Compare -- AgentBench's compareSnapshots() detects changes across system prompt text, tool call patterns, output content, token usage, and evaluation scores
4. Gate -- Block the PR if any snapshot comparison fails or quality scores drop below configured thresholds

The recipe covers how to integrate this into a GitHub Actions PR workflow with automated comments showing the before/after comparison, so reviewers can see exactly what changed in agent behavior with every prompt edit.

Read the full recipe

---

Model Migration Testing

Switching LLM models -- whether upgrading, cross-grading, or cost-optimizing -- carries risk. The new model may interpret prompts differently, miss tool calls, or produce different-quality outputs. This recipe provides a systematic, measurable approach to model migration testing.

The workflow:

1. Record baseline -- Run your full test suite against the current model and tag runs for easy filtering
2. Cross-model replay -- Replay the same inputs against the candidate model using agentbench replay <run-id> --mode cross-model --model <new-model>
3. Identify failure patterns -- AgentBench auto-detects regressions in token usage, cost, latency, and scores, flagging which test cases broke on the new model
4. A/B experiment -- Run a statistically valid experiment with N >= 30 samples per variant using the experiment engine
5. Gradual rollout -- Use batched cross-model replay to compare behavior on a percentage of production traffic before fully switching

The recipe includes acceptance criteria templates (e.g., "correctness score must not drop more than 1 point, cost must not increase more than 20%, no tool call regressions") and a decision matrix for go/no-go on model migration.

Read the full recipe

---

Cost Budget Enforcement in CI

LLM API costs add up fast. A seemingly innocent prompt change can double token usage across thousands of daily requests. This recipe shows you how to enforce cost budgets in CI, so no PR merges without passing cost checks.

The workflow:

1. Configure per-test token budgets -- Set maxTokens on test cases and tokens().toBeLessThan() assertions
2. Set cost thresholds -- Configure cost budgets in agentbench.config.ts using the assertions.maxTokens and cost-related rule evaluators
3. Track cost trends -- Use agentbench coverage --project <name> --trend to monitor token/cost trends over time and detect stealth cost creep
4. Block PRs -- CI fails the build if any test exceeds its token or cost budget, with a clear report showing which test cases are over budget and by how much
5. Optimize prompts -- Use the cost report to identify the most expensive test cases and iteratively refine prompts to reduce token usage without sacrificing quality

The recipe includes strategies for setting realistic per-test budgets (use P95 from historical data + 20% headroom) and how to handle legitimate cost increases (approve with a cost-budget-bypass label on the PR).

Read the full recipe

---

Safety Testing for AI Agents

AI agents that interact with users, call APIs, or access data are vulnerable to prompt injection, jailbreaking, and unsafe behavior. This recipe shows you how to build a comprehensive safety testing suite using AgentBench's safety judge dimension, forbidden tool configuration, and adversarial test datasets.

The workflow:

1. Enable the safety judge -- Configure the safety evaluation dimension in agentbench.config.ts with a strict score threshold (>= 9)
2. Build adversarial datasets -- Create test datasets with prompt injection attempts ("Ignore previous instructions..."), jailbreaking ("You are now DAN..."), role-playing attacks, and PII extraction attempts
3. Configure forbidden tools -- Set forbiddenTools in assertions config to ensure dangerous tools (delete_record, sudo_execute, raw_sql_query) are never called, even under adversarial input
4. Content moderation testing -- Verify the agent's output does not contain harmful, biased, or inappropriate content using output assertions with .not.toMatchRegex() patterns
5. CI safety gates -- Block deployments that fail any safety test, with a detailed report showing which adversarial inputs bypassed safety measures

The recipe includes a starter set of 50 adversarial test inputs and templates for custom safety test cases relevant to your domain (healthcare, finance, legal, etc.).

Read the full recipe

---

A/B Testing AI Agents

A/B testing for AI agents is more nuanced than for traditional software. LLM non-determinism means you cannot compare single runs -- you need statistical methods to determine whether differences are real or noise. This recipe covers the full experiment workflow.

The workflow:

1. Define experiment variants -- Create two or more variants that differ in prompt, model, temperature, tools, or any combination using the experiment config
2. Run the experiment -- Execute each variant N times across your test suite. The recipe recommends N >= 30 per variant for statistical significance
3. Analyze results -- AgentBench applies statistical tests (Welch's t-test for means, confidence intervals, Cohen's d effect size) to determine whether observed differences are statistically significant
4. Interpret findings -- The experiment report indicates whether there is a meaningful winner or if the difference is within the noise floor
5. Determine sample size -- The recipe includes a sample size calculator to know how many runs you need before starting the experiment

Key statistical concepts covered: p-values, confidence intervals, effect size (Cohen's d), statistical power, and how to avoid p-hacking by defining success criteria before running the experiment. Includes a decision tree for choosing the right statistical test based on your data distribution.

Read the full recipe

---

Choosing the Right Recipe

Your Challenge	Start With
"I changed a prompt and things feel... worse?"	Catching Prompt Regressions
"We're switching from GPT-4o to Claude"	Model Migration Testing
"Our LLM bill doubled this month"	Cost Budget Enforcement in CI
"We're deploying to production and worried about safety"	Safety Testing for AI Agents
"Is Prompt A actually better than Prompt B?"	A/B Testing AI Agents

---

Related pages: Core-Concepts | Guides | Examples | Config-Reference