Assertion DSL

Assertion DSL Reference

Quick reference for the AgentBench chainable assertion DSL. The DSL provides a fluent, type-safe API for writing expressive assertions against agent run results. It reads like English and provides rich, typed feedback on failures.

Read the complete reference: Assertion DSL Reference Read the conceptual guide: The Assertion Model

---

Entry Point: expect()

import { expect } from '@agentbench/core'

function expect(context?: AssertionContext | RunResult): AssertionBuilder

Creates a new AssertionBuilder with an optional context. If a RunResult is passed, it is automatically converted into an AssertionContext. If no context is provided, you must pass one to .run().

// From a RunResult (auto-converted)
const builder = expect(runResult)

// From a plain AssertionContext
const builder = expect(assertionContext)

// Build first, run later
const builder = expect()
builder.tool('search').toBeCalled()
const results = builder.run(runResult)

---

The Fluent API Chain

Each chain entry point returns a specialized sub-builder, and every chained assertion returns AssertionBuilder so you can keep chaining:

Method	Returns	Chains to
.status()	StatusAssertionBuilder	.toBe(), .toBeCompleted()
.tool(name)	ToolAssertionBuilder	.toBeCalled(), .toBeCalledWith(), .toBeCalledTimes(), .not.toBeCalled()
.output()	OutputAssertionBuilder	.toContain(), .toEqual(), .toMatchRegex(), .toMatchSchema(), .toMatchSnapshot(), .not.*
.tokens()	TokenAssertionBuilder	.toBeLessThan(), .toBeGreaterThan(), .toBeBetween(), .prompt().toBeLessThan()
.latency()	LatencyAssertionBuilder	.toBeLessThan(), .toBeGreaterThan(), .firstToken().toBeLessThan()
.score(dimension?)	ScoreAssertionBuilder	.toBeGreaterThan(), .toBeLessThan(), .toBeBetween()
.all(builders)	AssertionBuilder	Nested sub-assertions (AND logic)
.any(builders)	AssertionBuilder	Nested sub-assertions (OR logic)
.run(context?)	AssertionRunResult	Terminal -- executes all assertions

---

All 22 Matchers by Category

Status Matchers (2)

Assert on the overall run status.

DSL Method	Config type	Description
.status().toBe(status)	completed_successfully / completed_with_error	Assert the run ended with a specific status
.status().toBeCompleted()	(inline)	Assert the run completed (status is passed or completed)

// The run must complete without error
expect(runResult).status().toBeCompleted()

// The run must have errored (e.g., testing error handling)
expect(runResult).status().toBe('error')

---

Tool Matchers (7)

Assert on which tools were called, with what arguments, and how many times.

DSL Method	Config type	Description
.tool(name).toBeCalled()	tool_called	Tool was called at least once
.tool(name).not.toBeCalled()	tool_not_called	Tool was never called
.tool(name).toBeCalledWith(args)	tool_called_with	Tool was called with specific arguments (deep equality)
.tool(name).toBeCalledTimes(n)	tool_called_times	Tool was called exactly N times

// Assert search_docs was called with the right query
expect(runResult)
  .tool('search_docs').toBeCalledWith({
    query: 'refund policy',
    max_results: 5,
  })

// Assert the agent NEVER called a dangerous tool
expect(runResult).tool('delete_customer_data').not.toBeCalled()

// Assert exactly 3 calls
expect(runResult).tool('search_docs').toBeCalledTimes(3)

---

Output Matchers (12)

Assert on the agent's final text output.

DSL Method	Config type	Description
.output().toContain(substring)	contains	Output contains the substring (case-insensitive by default)
.output().not.toContain(substring)	not_contains	Output does NOT contain the substring
.output().toEqual(expected)	exact_match	Output matches exactly (with optional normalization)
.output().not.toEqual(expected)	(negation)	Output does NOT equal the expected string
.output().toMatchRegex(pattern, flags?)	matches_regex	Output matches the regex pattern
.output().not.toMatchRegex(pattern, flags?)	(negation)	Output does NOT match the regex pattern
.output().toMatchSchema(schema)	matches_schema	Output is valid JSON matching a JSON schema
.output().toMatchSnapshot(snapshot)	matches_snapshot	Output matches a stored snapshot string

// Substring matching (case-insensitive)
expect(runResult).output().toContain('30-day')
expect(runResult).output().not.toContain('no refunds')

// Regex matching
expect(runResult).output().toMatchRegex(/refund.*(30|thirty)\s*days/i)

// JSON schema validation
expect(runResult).output().toMatchSchema({
  type: 'object',
  properties: {
    eligible: { type: 'boolean' },
    refund_window_days: { type: 'number', minimum: 0 },
  },
  required: ['eligible', 'refund_window_days'],
})

// Snapshot matching
expect(runResult).output().toMatchSnapshot('baseline-refund-response')

---

Token Matchers (5)

Assert on token consumption.

DSL Method	Config type	Description
.tokens().toBeLessThan(threshold)	tokens_lt	Total tokens under threshold
.tokens().toBeGreaterThan(threshold)	tokens_gt	Total tokens above threshold
.tokens().toBeBetween(min, max)	tokens_between	Total tokens within a range
.tokens().prompt().toBeLessThan(threshold)	(inline)	Prompt tokens under threshold

// Stay within budget
expect(runResult).tokens().toBeLessThan(4096)

// Ensure the agent didn't give a one-word answer
expect(runResult).tokens().toBeGreaterThan(50)

// Target token range
expect(runResult).tokens().toBeBetween(200, 2000)

// Check prompt tokens specifically
expect(runResult).tokens().prompt().toBeLessThan(2048)

---

Latency Matchers (4)

Assert on execution timing.

DSL Method	Config type	Description
.latency().toBeLessThan(threshold)	latency_lt	Total duration under threshold (ms)
.latency().toBeGreaterThan(threshold)	latency_gt	Total duration above threshold (ms)
.latency().firstToken().toBeLessThan(threshold)	first_token_lt	Time-to-first-token under threshold (ms)

// SLA: respond within 10 seconds
expect(runResult).latency().toBeLessThan(10000)

// User-facing: first token within 500ms for streaming UX
expect(runResult).latency().firstToken().toBeLessThan(500)

// Latency floor (ensure no caching artifact)
expect(runResult).latency().toBeGreaterThan(100)

---

Score Matchers (4)

Assert on evaluation scores produced by LLM judges.

DSL Method	Config type	Description
.score(dimension).toBeGreaterThan(threshold)	score_gt	Score above threshold
.score(dimension).toBeLessThan(threshold)	score_lt	Score below threshold
.score(dimension).toBeBetween(min, max)	score_between	Score within range

// LLM judge scored correctness at 7 or above
expect(runResult).score('correctness').toBeGreaterThan(7)

// Safety is non-negotiable
expect(runResult).score('safety').toBeGreaterThan(9)

// Overall composite score
expect(runResult).score('overall').toBeBetween(7, 10)

When the dimension name is omitted, .score() matches against all scores:

// Any score dropped below 5?
expect(runResult).score().toBeGreaterThan(5)

---

Compound Assertions

Logical composition with .all() (AND) and .any() (OR):

// All conditions must pass (AND)
expect(runResult).all([
  (b) => b.output().toContain('refund'),
  (b) => b.output().toContain('30'),
  (b) => b.tokens().toBeLessThan(4096),
])

// At least one condition must pass (OR)
expect(runResult).any([
  (b) => b.output().toContain('30 days'),
  (b) => b.output().toContain('thirty days'),
  (b) => b.output().toContain('one month'),
])

---

The run() Result

interface AssertionRunResult {
  assertions: AssertionResult[]  // One result per matcher
  passed: number                 // Count of passed
  failed: number                 // Count of failed
  errored: number                // Count of errors
  skipped: number                // Count of skipped
  allPassed: boolean             // Convenience: true if failed == 0 && errored == 0
  duration: number               // Total evaluation time in ms
}

Each assertion result includes type, status ('passed' | 'failed' | 'error'), expected, actual, and a human-readable message.

---

Complete Example

import { expect } from '@agentbench/core'

async function validateCustomerSupportAgent(runResult: RunResult) {
  const results = await expect(runResult)
    // 1. Operational checks
    .status().toBeCompleted()
    .tokens().toBeLessThan(4096)
    .latency().toBeLessThan(10000)
    .latency().firstToken().toBeLessThan(1000)

    // 2. Tool usage checks
    .tool('search_knowledge_base').toBeCalled()
    .tool('escalate_to_human').not.toBeCalled()
    .tool('lookup_order').toBeCalledWith({ order_id: 'ORD-12345' })

    // 3. Output content checks
    .output().toContain('refund')
    .output().toContain('30 days')
    .output().not.toContain("I don't know")
    .output().toMatchRegex(/\$\d+\.\d{2}/)

    // 4. Quality checks
    .score('correctness').toBeGreaterThan(7)
    .score('faithfulness').toBeGreaterThan(8)
    .score('safety').toBeGreaterThan(9)

    // 5. Execute
    .run()

  if (!results.allPassed) {
    console.error(`${results.failed} assertion(s) failed:`)
    for (const a of results.assertions) {
      if (a.status === 'failed') {
        console.error(`  - [${a.type}] ${a.message}`)
      }
    }
  }

  return results
}

---

Rule-Based vs Score-Based Assertions

• Rule-based assertions: Deterministic, evaluated locally against captured trace data. Check objective facts -- "did the agent call tool X?", "is latency under 5 seconds?", "does the output contain 'refund'?". Instant, free, never flaky.
• Score-based assertions: Depend on evaluation scores produced by LLM judges. Measure subjective qualities -- "is the answer correct?", "is the reasoning sound?", "is the response safe?". Require a prior LLM judge evaluation.

Best practice: Always pair score-based assertions with rule-based assertions. Check the basics first (status, tools, tokens, latency), then validate quality (correctness, faithfulness, safety).

---

Related pages: Core-Concepts | Config-Reference | Guides