v0.5.0 — The Testing Methodology Release 🧪

🧪 Agent Testing Goes Mainstream

v0.4.0 made it possible to test an agent in 30 seconds. v0.5.0 teaches you how to test well.

---

📖 Agent Testing Pyramid

The definitive mental model for testing AI agents — 3 layers, 3 cadences, 1 framework.

           ┌─────────────┐
           │  Quality    │  Nightly / Release
           │ Assertions  │  LLM-as-Judge
           ├─────────────┤
           │    Flow     │  Push to Main
           │ Assertions  │  Multi-step Replay
           ├─────────────┤
           │ Single-Step │  Every PR
           │ Assertions  │  Tool / Output / Token
           └─────────────┘

Read the full pyramid →

🚫 8 Agent Testing Anti-Patterns

#	Anti-Pattern	What It Misses
1	只走阳光道 (Happy Path Only)	Edge cases silently break
2	只盯输出不看工具 (Output-Only)	Wrong tool, right words
3	改了 Prompt 就祈祷 (Prompt-and-Pray)	No regression guard
4	单模型幻觉 (Single Model Fallacy)	Works on GPT, fails on Claude
5	Token 盲症 (Token Blindness)	Budget blown in production
6	快照失忆症 (Snapshot Amnesia)	Can't reproduce failures
7	人肉 CI (Manual QA)	Slow, inconsistent, skipped
8	覆盖率表演 (Coverage Theater)	Tests that assert nothing

Read all 8 →

---

🔌 Ecosystem Integrations

Platform	Guide
Vercel AI SDK	12-line integration
Claude Code	Custom agent testing
LangChain / CrewAI	Zero-refactor integration

---

🎛️ Dashboard: Guided Failure Triage

No more staring at a red FAIL badge wondering what to do.

• What failed — clear summary
• Why — plain-language explanation
• How to fix — assertion-type-specific suggestions (26 types)
• Expected vs Actual — side-by-side diff

📊 Dashboard: Quality Trends

SVG-based line chart on the main dashboard. Track correctness, faithfulness, safety, and more across 7 / 14 / 30 days.

---

🚀 GitHub Marketplace Action

One YAML block. Zero config.

- uses: agentbench/action@v0.5
  with:
    api-key: ${{ secrets.OPENAI_API_KEY }}

• Inline PR annotations for every failed assertion
• Outputs: score, pass-rate, has-regression
• Uploads test report as artifact

action.yml · Example workflow

---

📦 Cumulative: v0.4.0 + v0.5.0

Feature	v0.4.0	v0.5.0
agentbench init --quick	✅	✅
Agent Testing Pyramid	—	✅
8 Anti-Patterns	—	✅
Ecosystem Guides (3)	—	✅
Failure Guidance UI	—	✅
Quality Trends Chart	—	✅
GitHub Marketplace Action	—	✅
462 Tests · 0 TS Errors	✅	✅

🔗 Links

• Full Changelog
• Testing Pyramid
• Anti-Patterns
• Documentation