Roadmap

AgentBench — Roadmap & Milestones

Overview

• Current Version: v0.3.0 (July 2026)
• Next: v0.4.0 (Q4 2026) → v0.5.0 (Q1 2027) → v1.0.0 (Q2 2027)
• Methodology: Milestone-driven, one module at a time
• Quality Gate: Each milestone requires review before proceeding

---

v0.3.0 — Provider Ecosystem & Platform [✅ COMPLETED — July 2026]

Goal: Expand the provider ecosystem, introduce Datasets and Benchmarks, add VS Code and GitHub integrations.

Achievements

Provider Ecosystem (15 packages)

• ✅ @agentbench/core — enhanced with Dataset engine, Config system, Provider Plugin architecture
• ✅ @agentbench/openai — OpenAI SDK interception/wrapping
• ✅ @agentbench/anthropic — Anthropic SDK interception/wrapping
• ✅ @agentbench/mcp — MCP client wrapping and tracing
• ✅ @agentbench/gemini [NEW] — Google Gemini SDK wrapping
• ✅ @agentbench/deepseek [NEW] — DeepSeek API SDK wrapping
• ✅ @agentbench/azure-openai [NEW] — Azure OpenAI Service SDK wrapping
• ✅ @agentbench/openrouter [NEW] — OpenRouter multi-provider gateway wrapping
• ✅ @agentbench/groq [NEW] — Groq fast inference SDK wrapping
• ✅ @agentbench/ollama [NEW] — Ollama local model wrapping
• ✅ @agentbench/provider-utils [NEW] — Shared base classes (AgentBenchProvider interface, OpenAICompatibleProvider)
• ✅ agentbench-cli — 13 commands (added dataset, benchmark, experiment, dev, config)
• ✅ agentbench-web — Next.js Dashboard
• ✅ agentbench-vscode [NEW] — VS Code Extension
• ✅ agentbench-github [NEW] — GitHub Integration (Actions, PR comments, CI badges)

Dataset Module

• ✅ Dataset CRUD + import/export (CSV, JSON, JSONL)
• ✅ Dataset validation (schema, integrity, quality)
• ✅ Dataset splitting (random, stratified, k-fold)
• ✅ Dataset sampling (random, stratified, weighted, reservoir)
• ✅ Dataset versioning (create, diff, compare)
• ✅ CLI: agentbench dataset — 10 subcommands
• ✅ API: Independent dataset endpoints

Benchmark System

• ✅ Benchmark registry and search
• ✅ Benchmark install/run/submit/publish
• ✅ Public leaderboard per benchmark
• ✅ CLI: agentbench benchmark — 7 subcommands
• ✅ API: Benchmark and leaderboard endpoints

Config System

• ✅ defineConfig with multi-provider support
• ✅ Multi-source configuration resolution (defaults → config file → local overrides → env vars → CLI flags)
• ✅ CLI: agentbench config — show, validate, set, get, edit

Platform Integrations

• ✅ VS Code Extension — test explorer, inline results, trace viewer, run launcher, coverage gutter
• ✅ GitHub Integration — Actions workflow, PR comments, status checks, CI badges
• ✅ agentbench dev — interactive development mode with hot reload, REPL, trace inspector

---

v0.4.0 — GitHub Integration, VS Code, Datasets, Benchmarks [Q4 2026]

Goal: Deepen platform integrations, advanced dataset features, collaborative benchmarks.

GitHub Integration (Enhanced)

• GitHub App (marketplace listing)
• PR regression auto-block (fail the PR if agent regresses)
• Multi-repo agent testing
• GitHub-native trace viewer (render trace in PR comments)
• Scheduled benchmark runs via GitHub Actions
• Slack/Discord notification integration

VS Code Extension (Enhanced)

• Agent debugging — breakpoints on tool calls, step-through execution
• Prompt editor with syntax highlighting and autocomplete
• Inline coverage heatmap overlay
• Integration with VS Code Test Runner API
• Remote development support (SSH, Codespaces, Dev Containers)

Datasets (Enhanced)

• Dataset augmentation and synthesis (LLM-assisted)
• Dataset annotation workflow (labeling interface)
• Dataset sharing and discovery (public registry)
• Dataset version branching and merging
• Streaming dataset support for large files (>1GB)

Benchmarks (Enhanced)

• Custom benchmark builder (visual editor)
• Benchmark versioning and changelog
• Benchmark badges (shields.io integration)
• Automated benchmark scheduling
• Cross-benchmark comparison (compare models across different benchmarks)

---

v0.5.0 — Team Collaboration, Cloud, Enterprise [Q1 2027]

Goal: Multi-user collaboration, cloud-hosted option, enterprise features.

Team Collaboration

• Team/Organization management
• Role-based access control (Owner, Admin, Editor, Viewer)
• Shared test libraries and prompt templates
• Review and approval workflows for test changes
• Activity feed and notifications

Cloud Platform

• AgentBench Cloud (agentbench.dev) — hosted SaaS
• Managed PostgreSQL + Redis
• Usage-based pricing (Free tier → Pro → Enterprise)
• Auto-scaling worker infrastructure
• Global region deployment (US, EU, Asia)

Enterprise Features

• SSO (SAML/OIDC)
• Audit logging and compliance reports
• Data residency controls
• Custom SLA
• Dedicated infrastructure
• Priority support

---

v1.0.0 — Production Ready, Plugin Ecosystem, Marketplace [Q2 2027]

Goal: Stable API, plugin marketplace, production-grade maturity.

Stability & Performance

• Stable public API (no breaking changes without major version)
• Performance benchmarks and optimization targets
• Horizontal scaling to 10,000+ concurrent runs
• 99.9% uptime SLA for cloud
• Comprehensive error handling and recovery
• Graceful degradation under load

Plugin Ecosystem

• Plugin marketplace (agentbench.dev/marketplace)
• Community provider plugins
• Community evaluator plugins
• Community benchmark plugins
• Plugin SDK and documentation
• Plugin review and certification process

Documentation & Community

• Interactive tutorial (learn.agentbench.dev)
• Video course (Agent Testing 101)
• Community forum (Discord/Discourse)
• Regular community calls
• Guest blog posts and case studies

Certification

• AgentBench Certified — official certification for quality-tested agents
• AgentBench Verified Provider — certification for provider plugins
• Badge program for open-source projects using AgentBench

---

Versioning

Version	Type	Status
v0.1.0	Alpha — Core MVP (Runner, Tracer, basic Web UI)	✅ Released
v0.2.0	Alpha — Evaluation, Assertion, Replay, Snapshots, Experiments, Coverage	✅ Released
v0.3.0	Alpha — 8 Providers, Datasets, Benchmarks, VS Code, GitHub Integration	✅ Released (July 2026)
v0.4.0	Beta — Enhanced Integrations, Advanced Datasets, Collaborative Benchmarks	🚧 Planned (Q4 2026)
v0.5.0	Beta — Team Collaboration, Cloud, Enterprise	📋 Planned (Q1 2027)
v1.0.0	GA — Production Release, Plugin Ecosystem, Marketplace	📋 Planned (Q2 2027)

---

Risk Register

Risk	Impact	Mitigation
LLM API instability	High	Abstract all LLM calls, support fallback judges
Streaming capture complexity	Medium	Start with non-streaming, add streaming later
Performance at scale (1000+ runs)	Medium	JSONB for traces, partition by project, use Redis cache
Multi-framework SDK maintenance	High	Focus on P0 (OpenAI, Anthropic, MCP), rest as community adapters
Competition (LangSmith, Braintrust)	Medium	Differentiate on regression testing + self-hosted + open source

---

Next Step: Task List (actionable development tasks)