Skip to content

Releases: 1304674612/agentbench

Release list

v0.3.1 — Web Dashboard Refinements & New Models 🎛️

v0.3.1 — Web Dashboard Refinements & New Models 🎛️ Latest
Latest

Choose a tag to compare

View all tags
@1304674612 1304674612 released this 10 Jul 07:35
v0.3.1
c972511

What's New

🤖 Latest Models

  • Added GPT-5, GPT-5 Mini, GPT-5 Nano, O5 models
  • Added DeepSeek Chat and DeepSeek Reasoner to all model selectors
  • Unified model labels — no more hardcoded version numbers

🖥️ Web Dashboard

  • Working search bar — Cmd+K opens command palette to navigate all pages
  • Compare page redesigned — dropdown run selectors instead of manual ID input
  • Dashboard stats fixed — Pass Rate now excludes PENDING runs
  • Expected output field fixed — accepts plain text keywords, not JSON
  • React errors eliminated — insertBefore / removeChild bugs fixed
  • 8 loading.tsx added — no more blank screens during navigation
  • 3 error boundaries added — page crashes now show recovery UI
  • API auth secured — all 14 API route groups now protected
  • 32 swallowed errors fixed — proper logging and error responses

📚 Documentation

  • Web Dashboard Guide — new user walkthrough explaining every page
  • FAQ expanded — 5 new-user questions (Web vs CLI, PENDING explained)
  • GitHub Wiki synced — 22 pages
  • Star History chart — static image, no API rate limits

🧹 Code Quality

  • 247 lines of duplicate code removed
  • 82 new tests — config and provider-utils packages
  • Naming unified — consistent class names across all providers

Full Changelog: v0.3.0...v0.3.1

Assets 2
Loading

v0.3.0 — Developer Experience & Ecosystem 🚀

v0.3.0 — Developer Experience & Ecosystem 🚀

Choose a tag to compare

View all tags
@1304674612 1304674612 released this 10 Jul 03:00
v0.3.0
4b3a5d4

🎯 AgentBench v0.3.0 — The Testing Framework for AI Agents

5 分钟从安装到第一次测试

npm install -g agentbench
agentbench init
agentbench test

✨ 核心更新

🔥 全新 CLI 开发者体验

  • agentbench init 完全重写 — ASCII Logo + 5步交互引导 + 9 个 API Key 自动检测 + 模板选择
  • agentbench test Jest 风格输出 — watch/coverage/replay/json/junit/ci 全模式支持
  • 零配置理念，渐进式披露

📦 2 个新基础 Package

Package 说明
@agentbench/config defineConfig()、Jest 风格配置解析、Zod 校验、异步配置
@agentbench/provider-utils Provider SDK、OpenAICompatibleProvider 基类、CostCalculator(50+ 模型)、TokenCounter(tiktoken+启发式)

🌐 6 个新 LLM Provider（tsc --noEmit 通过）

Provider 说明 特点
@agentbench/gemini Google Gemini 原生 Google AI API、Vision、Tool Calling
@agentbench/deepseek DeepSeek OpenAI 兼容、Reasoning Models
@agentbench/azure-openai Azure OpenAI Azure AD 认证、Deployment 端点
@agentbench/openrouter OpenRouter 200+ 模型透传
@agentbench/groq Groq 超快推理、LPU 优化
@agentbench/ollama Ollama 本地运行、零成本、自动检测

📚 14 个官方 Examples（从 3 个扩展）

hello-agent · customer-support-agent · research-agent · code-review-agent · rag-agent · sql-agent · coding-agent · tool-calling-agent · mcp-agent · langgraph-agent · openai-agent-sdk · crewai-agent · llamaindex-agent · multi-agent-workflow

每个 Example 包含：README + Agent 源码 + 测试套件 + Dataset + CI 配置

📊 Dataset 系统

  • 7 种格式导入导出 — CSV / JSON / JSONL / HuggingFace / OpenAI Evals / DeepEval / LangSmith
  • 完整操作 — validate / split（分层抽样）/ sample / version / diff / compare
  • 600+ 行测试 · CLI 10 个子命令

🛠️ VS Code 插件（15 文件）

  • 13 条命令 · CodeLens（▶Run · 🐛Debug · 📊Replay）
  • TreeView 侧边栏 · StatusBar 状态栏 · Diagnostics 问题面板
  • TraceViewer 轨迹可视化 · Coverage 覆盖率展示

🔗 GitHub 集成

  • Actions CI/CD 工作流 · 复合 Action · PR 评论模板
  • 自动回归检测 · 成本追踪 · 评分对比

🏆 Benchmark Marketplace

  • 完整类型系统 · CLI 7 个子命令 · 14 个分类
  • Mock 数据支持 · 搜索/安装/运行/提交排行榜

📖 品牌 & 文档

  • 全新 README — Jest 类比定位 · 竞品对比 · 14 个 Example 索引
  • CHANGELOG v0.3.0 — 完整更新日志
  • ROADMAP v0.3→v1.0 — 路线图
  • 15 章节产品设计文档 — 系统架构、模块设计、信息架构

📊 数据对比

指标 v0.2.0 v0.3.0
Packages 8 15
Examples 3 14
CLI 命令 11 13
Providers 2 8
TypeScript 文件 ~100 274
VS Code 插件 无 15 文件
GitHub 集成 基础 CI Action + PR 模板

🔗 链接

AgentBench is to AI Agents what Jest is to JavaScript.

Loading

v0.2.0 — Production Release: From Demo to Product

v0.2.0 — Production Release: From Demo to Product

Choose a tag to compare

View all tags
@1304674612 1304674612 released this 10 Jul 02:15
v0.2.0
40c74f2

AgentBench v0.2.0 — Production Release

This release transforms AgentBench from an alpha prototype into a production-grade product. 12 work streams delivered simultaneously across 182 files (21,179 additions).

🔴 P0: Critical Foundations

Authentication (S1) — Full NextAuth.js v5 integration

  • Credentials provider (email + password) + GitHub + Google OAuth
  • Route protection middleware, sign-in/sign-up pages, registration API
  • Real API key validation against database (replaces mock ab-* prefix)
  • Seed admin user: admin@agentbench.dev

Testing (S2) — 342 unit tests across 14 test files (was 51 in 3)

  • Every core module now tested: runner, tracer, assertion DSL, evaluators, snapshot, diff, replay, experiment, coverage, reporter
  • 31 expanded API integration tests

Streaming (S3) — SSE stream capture

  • New StreamCapture class: OpenAI + Anthropic SSE parsing
  • Tracer.traceLLMCallStream() — end-to-end streaming trace
  • SDK streaming with tee'd streams for simultaneous user + trace consumption

🟡 P1: Ecosystem Expansion

SDK Adapters (S4) — New @agentbench/langgraph package

  • Real LangGraphAdapter with run() and stream() support
  • Duck-typing design — no hard dependency on LangChain

Production Hardening (S5)

  • GET /api/health — DB + Redis health check
  • Token bucket rate limiter — 100 req/min per API key/IP
  • Centralized Zod schemas for all API inputs
  • CORS middleware, unified ApiError handler, request logging

Examples (S6) — 3 complete example projects (21 files)

  • Customer Support Agent, Code Review Agent, Research Agent
  • 1066-line seed script with demo data

Python SDK (S7) — pip install agentbench

  • 15 files: Runner, Tracer, Assertion DSL, CLI, HTTP client
  • OpenAI + Anthropic wrappers for automatic tracing
  • 49 tests, all passing

🟢 P2: Polish & Delivery

CLI Modularization (S8) — 734 lines → 16 files

  • 11 command modules + 4 shared libraries
  • New agentbench dev command with file watcher
  • Progress spinners, --json on all commands, --debug flag

Feature Refactoring (S9)

  • 8 shared UI components + 7 feature components
  • Loading skeletons, barrel exports from @/features

Email Notifications (S10)

  • nodemailer + 4 HTML templates + notification API
  • In-app notification dropdown with unread badge
  • Auto-notify on run completion + regression detection

Performance & SEO (S11)

  • Full metadata on all 12+ pages, OG image API, sitemap, robots.txt
  • PWA manifest, bundle optimization

Publishing (S12)

  • All 5 packages build to dist/ ready for npm
  • Multi-stage Dockerfile + docker-compose.prod.yml
  • Changesets + GitHub Release workflow

📊 Stats: v0.1.0 → v0.2.0

Metric v0.1.0 v0.2.0
Tests 51 391+
Test files 3 17
Packages 8 9
API Endpoints 18 20
CLI Commands 8 11
Shared UI Components 3 11
Examples 0 3
Python Support None Full SDK (49 tests)
Auth Mock NextAuth.js + OAuth
Streaming No Full SSE capture
Docker DB only Full production

Full Changelog: https://github.com/1304674612/agentbench/blob/main/CHANGELOG.md

Loading

v0.1.0 — Initial Alpha Release 🎉

v0.1.0 — Initial Alpha Release 🎉

Choose a tag to compare

View all tags
@1304674612 1304674612 released this 09 Jul 13:43
v0.1.0
dca9f0c

Changelog

v0.1.0 (2026-07-09) — Initial Alpha Release

🎉 First Public Release

AgentBench v0.1.0 is the first public alpha — a complete regression testing framework for AI agents, built over 7 milestones.

✨ Features

Core Engine

  • Agent Runner with execution context, timeout, and concurrency management
  • Execution Tracer with OpenAI, Anthropic, and generic interceptors
  • Token counting (heuristic) and cost calculation for 15+ models
  • Storage abstraction (PostgreSQL + in-memory) via Prisma

Evaluation & Assertion

  • 14 rule-based evaluators (exact match, contains, regex, JSON schema, tool checks, metrics)
  • LLM-as-Judge across 8 dimensions (correctness, faithfulness, safety, relevance, completeness, reasoning, conciseness, tool usage)
  • Hybrid Judge with rule_first/llm_first/parallel strategies
  • Chained Assertion DSL: expect(run).tool().tokens().latency().output().score().run()

Regression & Replay

  • Snapshot Manager (create, list, restore, compare)
  • Replay Engine (deterministic, cross-model, batch)
  • Regression Detection (score, token, latency, cost thresholds)
  • Diff Engine (text, metric, trace, score comparison)

Experiments & Coverage

  • A/B Experiment Engine (t-test, bootstrap, Cohen's d effect size)
  • Coverage Analysis (prompt, workflow, tool, edge-case dimensions)

SDK Ecosystem

  • @agentbench/openai — OpenAI wrapper with automatic tracing
  • @agentbench/anthropic — Anthropic Claude wrapper with tracing
  • @agentbench/mcp — MCP client wrapper for tool calls
  • @agentbench/adapter — Generic adapter for LangGraph, CrewAI, LlamaIndex

Platform

  • Report generation (JSON, Markdown, HTML, JUnit XML)
  • Dataset management (CSV/JSON/JSONL import, train/test/validation split)
  • CI/CD: GitHub Actions workflow + webhook endpoint
  • 18 REST API endpoints with Zod validation

Web Dashboard

  • Dark-first Linear-inspired UI (Next.js 15 + Radix + Tailwind v4)
  • Landing page with code demo and feature grid
  • Dashboard with real-time stats and pass/fail distribution
  • Run detail with scores, assertion results, and execution timeline
  • Test suites & cases management
  • Experiments listing with variant previews
  • Coverage dashboard with dimension progress bars
  • Compare page for side-by-side run analysis
  • Snapshot listing with metadata

CLI

  • 8 commands: init, run, test, evaluate, replay, compare, snapshot, report
  • Colored output with chalk, progress spinners with ora
  • JSON and table output formats

Quality

  • TypeScript strict mode across all packages (0 errors)
  • 97 source files, ~14,500 lines of code
  • E2E tested at 95% pass rate
  • 4 documentation files (Architecture, Schema, Roadmap, Deployment)

🏗️ Architecture

agentbench/
├── apps/
│   ├── web/          Next.js 15 Dashboard + API
│   └── cli/          Commander.js CLI
├── packages/
│   ├── core/         @agentbench/core (Engine)
│   ├── openai/       @agentbench/openai
│   ├── anthropic/    @agentbench/anthropic
│   ├── mcp/          @agentbench/mcp
│   ├── adapter/      @agentbench/adapter
│   └── typescript-config/
├── docs/             Architecture, Schema, Roadmap, Deployment
└── .github/          CI workflow

📊 Stats

Metric Value
TypeScript Files 97
Lines of Code 14,500
Packages 8
API Endpoints 18
CLI Commands 8
TS Errors 0
E2E Test Coverage 95%

🔜 Roadmap to v1.0

  • Unit tests for core engine modules
  • Integration tests for all API routes
  • Full NextAuth.js authentication
  • Rate limiting
  • Real LLM integration tests
  • Performance benchmarking
  • Community contribution guide

Versioning

AgentBench follows Semantic Versioning.

Stage Version Pattern
Alpha v0.1.x — Feature development
Beta v0.5.x — Feature complete, stabilization
RC v0.9.x — Release candidate
GA v1.0.0 — Production ready
Loading