AgentSpec

Testing framework for AI agents — Jest for non-deterministic AI behavior

AI agents are non-deterministic. When you change a prompt, swap a model, or update a tool, behavior shifts in ways you can't predict. AgentSpec lets you write tests that catch those shifts before they reach production.

Quick start

npm install -g @ozperium/agentspec
agentspec init

This creates agentspec.yaml:

name: "my-agent-tests"
tests:
  - name: "handles greeting"
    input: "hello"
    expect:
      contains: "help"

Run tests:

agentspec run

Assertions for non-deterministic output

AgentSpec provides assertions designed for AI output, not just string equality:

tests:
  - name: "handles expired token"
    input: "my token expired"
    expect:
      contains: "refresh token"
      not_contains: "I don't know"
      max_latency_ms: 5000

  - name: "routes billing question"
    input: "I want a refund"
    expect:
      contains_any: ["billing", "refund", "support"]
      tool_called: "transfer_to_billing"

  - name: "response is semantically on-topic"
    input: "how do I reset my password?"
    expect:
      semantically_similar: "reset password credentials"
      min_confidence: 0.5

  - name: "JSON output has correct structure"
    input: "get user profile"
    expect:
      json_path: "data.email"
      json_value: "user@example.com"

  - name: "matches error code pattern"
    input: "simulate error"
    expect:
      regex: "ERR-\\d{5}"

LLM-as-judge

Use a local model to evaluate response quality — no API costs, full privacy:

# Use local Ollama as judge
agentspec run --judge-endpoint http://127.0.0.1:11434/v1/chat/completions --judge-model qwen2.5:7b

tests:
  - name: "response is helpful"
    input: "How do I reset my password?"
    expect:
      llm_judge: "The response should explain how to reset a password"

Behavior diff reports

AgentSpec stores the last passing output per test. When behavior changes, you see exactly what shifted:

⚠️ REGRESSION support agent > handles expired token
  ⚠  Behavior REGRESSED — test was passing, now failing
  + added: your, token, has, expired, please, contact, support
  - removed: you, need, refresh, the, token, settings, security

Testing real agents

Use --endpoint to test any HTTP-accessible agent:

agentspec run --endpoint https://my-agent.example.com/chat

AgentSpec POSTs {input: "..."} to your endpoint and expects {output: "..."} in the response.

CI/CD integration

# Exit code 1 on failure
agentspec run --ci

# JUnit XML for CI systems
agentspec run --junit > results.xml

# JSON output
agentspec run --json

GitHub Action

- uses: agentspec/action@v1
  with:
    test-dir: tests
    format: junit

CLI commands

agentspec run [--dir tests] [--ci] [--json] [--junit] [--watch] [--test "pattern"]
agentspec init
agentspec list [--dir tests]
agentspec version

Why AgentSpec?

Problem	AgentSpec
Agent behavior changes with model updates	Regression tests catch the shift
"It worked yesterday" debugging	Diff reports show what changed
Manual testing before each deploy	CI/CD gates with --ci exit codes
Jest/Vitest don't handle non-deterministic output	contains_any, regex, llm_judge, semantically_similar
No CI integration for AI features	GitHub Action + JUnit XML

Roadmap

• YAML test definitions
• Core assertions (contains, not_contains, contains_any, contains_all, regex)
• Metadata assertions (max_latency_ms, max_tokens, tool_called)
• Semantic similarity (word overlap)
• JSON path assertions
• CLI with run/init/list commands
• CI mode with exit codes
• JUnit XML and JSON output
• Test filtering
• GitHub Action
• Behavior diff reports — see what changed between runs
• LLM-as-judge — use a local model (Ollama) to evaluate output quality
• HTTP agent adapter — test real agents via --endpoint
• Auto-test generation from conversation history
• Python agent support
• Watch mode with file watcher
• Web dashboard
• GitHub PR integration (comment on PRs with behavior diffs)

Part of the AI Dev Workflow Stack

AgentSpec is one tool in a three-part observability stack for AI development:

Tool	What it does	Install
AgentSpec	Test AI agent behavior — catch regressions before production	npm i -g @ozperium/agentspec
AICostTracker	Track token usage and costs across projects	npm i -g @ozperium/aicost-tracker
quota	Monitor AI rate limits — know what's left before it stops you	npm i -g @ozperium/quota

License

MIT