AgenticTestPyramid
title: Agentic Test Pyramid type: technique created: 2026-06-23 last_updated: 2026-06-23 related: ["LLM Evaluation Methodology", "Frequently Asked Questions About AI Evals", "Unit Testing Principles", "Building Effective Agents (Anthropic)"] sources: ["https://matthewboston.com/blog/the-agentic-test-pyramid.html"] radar_quadrant: Techniques radar_ring: Assess radar_position: inner
A six-layer test strategy for AI agent systems that extends Martin Fowler's traditional test pyramid with a second axis: determinism and cost. Proposed by Matthew Boston, the framework separates tests that can run cheaply and deterministically from those that require live model calls and produce non-deterministic results.
Traditional test pyramids assume deterministic systems: identical inputs yield identical outputs. Language models violate this assumption. The same prompt may produce varying outputs across runs, making exact-match assertions brittle and binary pass/fail meaningless for subjective quality.
The pyramid stacks from cheapest and most deterministic (bottom) to most expensive and least deterministic (top):
| Layer | Type | Description |
|---|---|---|
| 1. Pure unit tests | Deterministic | Parsing, validation, pure logic -- no I/O or model calls |
| 2. Tripwires | Deterministic | Static analysis checks that catch contract violations in source code |
| 3. In-process integration | Deterministic | Real components wired together; no external calls |
| 4. Real-dependency E2E | Deterministic | Actual browsers or processes; external services mocked |
| 5. Behavioral E2E | Non-deterministic | Prompts and policies tested against a live model via API |
| 6. Quality evals | Non-deterministic | Subjective correctness graded by a model-as-judge |
Layers 1-4 are unchanged from Fowler's model. Layers 5-6 are the agentic additions.
The most novel layer is tripwires: executable architecture documentation that enforces contracts through static pattern matching rather than runtime behaviour. The author argues that most regressions in LLM agent systems are contract violations -- a tool signature changes, a prompt variable is removed, a schema field is renamed -- and these are cheapest to catch at the static layer. Tripwires run in CI with no model cost.
For the expensive non-deterministic layers (5 and 6), the framework distinguishes two modes:
- Gates -- block merges; only used for stable, high-confidence checks where a failure unambiguously signals a regression
- Periodic monitoring -- run on a schedule; used for fuzzy or subjective evaluations where model variance makes per-merge gating noisy
Tests at layer 6 use tolerance ranges rather than exact thresholds to absorb model variance without producing false alarms.
LLM Evaluation Methodology focuses on defining measurable success criteria and building automated eval suites before shipping. The Agentic Test Pyramid is complementary: it provides the structural layer taxonomy that determines where each eval type belongs in the CI/CD pipeline and how it should be gated.
Agentic Test Pyramid sits in the Assess ring of the Techniques quadrant, at inner position. First studied via Matthew Boston's blog on 2026-06-23; no personal production use to date. The framework is structurally clear and solves a real problem: teams building LLM agents typically lack a principled way to organise tests across the determinism boundary. Inner position reflects the high practical value of the tripwires concept alone -- a concrete, cost-free technique for catching contract regressions that is immediately adoptable even without investing in the full pyramid.
