agenteval

pytest for AI agents.

Did I break my agent?

Agents are production code now — they need CI too.

pip install agent-eval   # package & CLI name

What check shows you

# before (baseline)
  retrieve → search_orders → answer
  ✓ uses internal DB

# after your PR
- retrieve → search_orders → answer
+ search_web → answer

❌ FAIL: agent behavior changed (CI blocked PR)

Root cause:
  prompt.py:42  system instruction changed

Answer still looks fine. LLM evals said OK. Your PR is blocked anyway.

Real-world failure

We changed a prompt.

The agent still answered correctly.

But silently stopped using internal DB and switched to web search.

This change passed all LLM-based evals.

agenteval caught it in CI.

---

Three steps (like pytest)

Trace	Record what your agent did
Baseline	Save a run when it works
Check	Did the last run break it?

explain shows the diff. check in GitHub Actions blocks the PR. Same flow as pytest snapshots.

Everything else → Advanced features.

---

Quick start

No API keys. No YAML test suite.

git clone https://github.com/agent-eval/agent-eval
cd agent-eval
pip install -e ".[dev]"

python examples/order_bot/run.py
agent-eval baseline create --agent order-bot

# break SYSTEM_PROMPT in examples/order_bot/run.py
python examples/order_bot/run.py
agent-eval check --agent order-bot   # exit 1
agent-eval explain --agent order-bot

Trace (5 lines)

from agent_eval.core.trace import Trace

with Trace(agent_name="order-bot") as tr:
    tr.set_prompt(SYSTEM_PROMPT)
    tr.step("retrieve", type="retrieval", input=q, output=docs)
    tr.step("search_orders", type="tool_call", input={"order_id": "12345"}, output=order)
    tr.step("answer", type="llm", output=answer)

Run once → baseline. Run again → check.

---

Agent CI

Commit .agent-eval/baseline.json to your repo.

- run: pip install agent-eval
- run: python my_agent/run.py
- run: agent-eval check --agent order-bot

Or the composite action from this repo:

- uses: ./
  with:
    agent: order-bot
    install: .

Exit 0 = pass. Exit 1 = broken. Like pytest.

---

Advanced features

Explain & JSON

agent-eval explain --agent order-bot
agent-eval check --agent order-bot --json

Baseline versioning

agent-eval baseline history
agent-eval baseline diff v1 v2
agent-eval baseline promote v2
agent-eval baseline create --agent order-bot --branch feature/x

Framework integrations

import agent_eval
bot = agent_eval.instrument(bot, agent_name="order-bot", framework="pydanticai")

Extras: [pydanticai], [langgraph], [openai], [langchain], [integrations].

RAG pipelines

agent-eval baseline create --agent docs-bot --mode rag

Optional Ragas metrics: --with-quality-metrics (requires [rag]).

LLM judge (second opinion, does not change exit code)

[judge]
model = gpt-4o-mini

pip install agent-eval[judge]
agent-eval check --agent order-bot --with-judge

Config, viewer, server, export

agent-eval init
agent-eval view
agent-eval serve          # [server]
agent-eval run-tests tests/agents/*.yaml

See docs/ and PROJECT_SUMMARY.md.

---

vs other tools

Tool	Focus
agenteval	Agent CI — did behavior change?
Promptfoo	Authored test cases, prompt A/B
DeepEval / Ragas	LLM-as-judge, quality scores
Langfuse / LangSmith	Dashboards

COMPETITION.md — honest comparison.

---

Development

pip install -e ".[dev]"
pytest
ruff check .

License

MIT