-
Notifications
You must be signed in to change notification settings - Fork 0
Architecture
Mickl edited this page Jul 10, 2026
·
2 revisions
┌─────────────────────────────────────────────────────────────────────┐
│ AgentBench System │
├─────────────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ CLI Tool │ │ Web Dashboard│ │ SDKs │ │
│ │ (Commander) │ │ (Next.js) │ │ (@agentbench│ │
│ │ │ │ │ │ /openai, │ │
│ │ │ │ │ │ /anthropic,│ │
│ │ │ │ │ │ /mcp...) │ │
│ └──────┬──────┘ └──────┬───────┘ └──────┬──────┘ │
│ │ │ │ │
│ └────────────────┼─────────────────┘ │
│ │ │
│ ▼ │
│ ┌──────────────────────────────────────────────┐ │
│ │ API Layer (REST + WS) │ │
│ │ ┌──────────┐ ┌──────────┐ ┌──────────────┐ │ │
│ │ │ REST API │ │Server │ │ WebSocket │ │ │
│ │ │(Next.js) │ │Actions │ │ (Realtime) │ │ │
│ │ └──────────┘ └──────────┘ └──────────────┘ │ │
│ └──────────────────────┬───────────────────────┘ │
│ │ │
│ ▼ │
│ ┌──────────────────────────────────────────────┐ │
│ │ @agentbench/core │ │
│ │ │ │
│ │ ┌─────────┐ ┌─────────┐ ┌────────────────┐ │ │
│ │ │ Runner │ │ Tracer │ │ Storage │ │ │
│ │ │ Engine │ │ Engine │ │ Layer │ │ │
│ │ └────┬────┘ └────┬────┘ └───────┬────────┘ │ │
│ │ │ │ │ │ │
│ │ ┌────┴────┐ ┌────┴────┐ ┌──────┴────────┐ │ │
│ │ │Replay │ │Evaluator│ │ Snapshot │ │ │
│ │ │Engine │ │Engine │ │ Manager │ │ │
│ │ └─────────┘ └─────────┘ └───────────────┘ │ │
│ │ │ │
│ │ ┌─────────┐ ┌─────────┐ ┌────────────────┐ │ │
│ │ │Diff │ │Coverage │ │ Reporter │ │ │
│ │ │Engine │ │Engine │ │ Engine │ │ │
│ │ └─────────┘ └─────────┘ └────────────────┘ │ │
│ └──────────────────────┬───────────────────────┘ │
│ │ │
│ ▼ │
│ ┌──────────────────────────────────────────────┐ │
│ │ Data & Infra Layer │ │
│ │ │ │
│ │ ┌──────────┐ ┌──────────┐ ┌──────────────┐ │ │
│ │ │PostgreSQL│ │ Redis │ │ File Storage │ │ │
│ │ │(Primary) │ │(Cache/Q) │ │ (S3/Local) │ │ │
│ │ └──────────┘ └──────────┘ └──────────────┘ │ │
│ │ │ │
│ │ ┌──────────┐ ┌──────────┐ │ │
│ │ │ BullMQ │ │ External │ │ │
│ │ │(Workers) │ │ LLM APIs │ │ │
│ │ └──────────┘ └──────────┘ │ │
│ └──────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────┘
┌─────────────┐
│ @agentbench│
│ /core │
└──────┬──────┘
│
┌────────────────────────┼────────────────────────────┐
│ │ │
▼ ▼ ▼
┌────────────┐ ┌────────────┐ ┌────────────┐
│@agentbench │ │@agentbench │ │@agentbench │
│/openai │ │/anthropic │ │/mcp │
└────────────┘ └────────────┘ └────────────┘
▼ ▼ ▼
┌────────────┐ [NEW] ┌────────────┐ [NEW] ┌────────────┐ [NEW]
│@agentbench │ │@agentbench │ │@agentbench │
│/gemini │ │/deepseek │ │/azure-openai│
└────────────┘ └────────────┘ └────────────┘
▼ ▼ ▼
┌────────────┐ [NEW] ┌────────────┐ [NEW] ┌────────────┐ [NEW]
│@agentbench │ │@agentbench │ │@agentbench │
│/openrouter │ │/groq │ │/ollama │
└────────────┘ └────────────┘ └────────────┘
│
▼
┌────────────────────┐ [NEW]
│ @agentbench │
│ /provider-utils │
└─────────┬──────────┘
│
┌─────────────────┼─────────────────┐
│ │ │
▼ ▼ ▼
┌────────────┐ ┌────────────┐ ┌────────────┐
│ agentbench │ │ agentbench │ │ agentbench │
│ CLI │ │ web │ │ vscode [NEW]│
└────────────┘ └────────────┘ └────────────┘
| Package | Responsibility | Dependencies | Status |
|---|---|---|---|
@agentbench/core |
Core engine: Runner, Tracer, Evaluator, Assertion, Snapshot, Diff, Coverage, Reporter, Storage abstraction, Dataset engine, Config system | None (except DB driver) | Updated |
@agentbench/openai |
OpenAI SDK interception/wrapping | @agentbench/core |
Stable |
@agentbench/anthropic |
Anthropic SDK interception/wrapping | @agentbench/core |
Stable |
@agentbench/mcp |
MCP client wrapping and tracing | @agentbench/core |
Stable |
@agentbench/gemini |
Google Gemini SDK wrapping |
@agentbench/core, @agentbench/provider-utils
|
[NEW] |
@agentbench/deepseek |
DeepSeek API SDK wrapping |
@agentbench/core, @agentbench/provider-utils
|
[NEW] |
@agentbench/azure-openai |
Azure OpenAI Service SDK wrapping |
@agentbench/core, @agentbench/provider-utils
|
[NEW] |
@agentbench/openrouter |
OpenRouter multi-provider gateway wrapping |
@agentbench/core, @agentbench/provider-utils
|
[NEW] |
@agentbench/groq |
Groq fast inference SDK wrapping |
@agentbench/core, @agentbench/provider-utils
|
[NEW] |
@agentbench/ollama |
Ollama local model wrapping |
@agentbench/core, @agentbench/provider-utils
|
[NEW] |
@agentbench/provider-utils |
Shared base classes and utilities for building providers (AgentBenchProvider interface, OpenAICompatibleProvider base class) | @agentbench/core |
[NEW] |
agentbench-cli |
CLI tool (commander.js + ink) — 13 commands including dataset, benchmark |
@agentbench/core, all providers |
Updated |
agentbench-web |
Next.js Dashboard | @agentbench/core |
Updated |
agentbench-vscode |
VS Code Extension — inline test results, trace viewer, run launcher | @agentbench/core |
[NEW] |
agentbench-github |
GitHub Integration — Actions, status checks, PR comments, CI badges | @agentbench/core |
[NEW] |
@agentbench/core/src/
├── index.ts # Public API surface
├── types/ # Core type definitions
│ ├── index.ts
│ ├── trace.ts # ExecutionTrace, TraceStep
│ ├── run.ts # Run, RunConfig, RunResult
│ ├── evaluator.ts # Evaluator, Score, JudgeConfig
│ ├── assertion.ts # Assertion, AssertionResult
│ ├── snapshot.ts # Snapshot
│ ├── test.ts # TestCase, TestSuite
│ ├── experiment.ts # Experiment, Variant
│ ├── coverage.ts # CoverageReport
│ ├── dataset.ts # Dataset, DatasetItem
│ └── provider.ts # Provider, Model configs
│
├── runner/ # Agent Runner Engine
│ ├── index.ts
│ ├── runner.ts # Main runner class
│ ├── context.ts # Execution context builder
│ ├── sandbox.ts # Execution sandbox
│ ├── timeout.ts # Timeout management
│ └── concurrency.ts # Concurrent execution
│
├── tracer/ # Execution Tracer
│ ├── index.ts
│ ├── tracer.ts # Main tracer class
│ ├── interceptors/ # LLM SDK interceptors
│ │ ├── base.ts
│ │ ├── openai.ts
│ │ ├── anthropic.ts
│ │ └── generic.ts
│ ├── stream-capture.ts # Streaming response capture
│ └── tool-capture.ts # Tool call capture
│
├── evaluator/ # Evaluation Engine
│ ├── index.ts
│ ├── rule-evaluator.ts # Rule-based evaluation
│ ├── llm-judge.ts # LLM-as-Judge
│ ├── hybrid-judge.ts # Hybrid evaluation
│ ├── judge-pool.ts # Judge model pool
│ └── scoring/ # Scoring functions
│ ├── correctness.ts
│ ├── faithfulness.ts
│ ├── safety.ts
│ ├── relevance.ts
│ ├── completeness.ts
│ └── reasoning.ts
│
├── assertion/ # Assertion Engine
│ ├── index.ts
│ ├── assert.ts # Main assertion builder
│ ├── matchers/ # Matcher library
│ │ ├── tool-matchers.ts
│ │ ├── token-matchers.ts
│ │ ├── latency-matchers.ts
│ │ ├── output-matchers.ts
│ │ └── score-matchers.ts
│ └── result.ts # Assertion result types
│
├── snapshot/ # Snapshot Manager
│ ├── index.ts
│ ├── create.ts # Snapshot creation
│ ├── restore.ts # Snapshot restoration
│ ├── compare.ts # Snapshot comparison
│ └── store.ts # Snapshot storage
│
├── diff/ # Diff Engine
│ ├── index.ts
│ ├── text-diff.ts # Text/prompt diff
│ ├── trace-diff.ts # Execution trace diff
│ ├── metric-diff.ts # Numeric metric diff
│ └── renderer.ts # Diff visualization data
│
├── replay/ # Replay Engine
│ ├── index.ts
│ ├── replay.ts # Main replay class
│ ├── deterministic.ts # Deterministic replay
│ └── cross-model.ts # Cross-model replay
│
├── coverage/ # Coverage Engine
│ ├── index.ts
│ ├── prompt-coverage.ts # Variable combination coverage
│ ├── workflow-coverage.ts # Execution path coverage
│ ├── tool-coverage.ts # Tool call coverage
│ └── calculator.ts # Coverage percentage calculation
│
├── experiment/ # Experiment Engine
│ ├── index.ts
│ ├── experiment.ts # Experiment runner
│ ├── variant.ts # Variant management
│ └── statistics.ts # Statistical tests
│
├── reporter/ # Report Generation
│ ├── index.ts
│ ├── json-reporter.ts
│ ├── html-reporter.ts
│ ├── markdown-reporter.ts
│ └── junit-reporter.ts
│
├── storage/ # Storage Abstraction Layer
│ ├── index.ts
│ ├── adapter.ts # Storage adapter interface
│ ├── postgres-adapter.ts # PostgreSQL adapter
│ ├── sqlite-adapter.ts # SQLite adapter (local dev)
│ ├── memory-adapter.ts # In-memory (ephemeral)
│ └── file-adapter.ts # File-based (CLI local mode)
│
├── config/ # Configuration
│ ├── index.ts
│ ├── loader.ts # Config file loader
│ └── schema.ts # Zod schema for config
│
└── utils/ # Utilities
├── token-counter.ts # Token counting (tiktoken)
├── cost-calculator.ts # Cost calculation
├── latency.ts # Timing utilities
├── id-generator.ts # Run ID generation
└── logger.ts # Structured logging
// @agentbench/core/src/types/trace.ts
interface ExecutionTrace {
id: string
runId: string
steps: TraceStep[]
metadata: TraceMetadata
createdAt: Date
}
interface TraceStep {
id: string
sequence: number
type: 'llm_call' | 'tool_call' | 'response' | 'error'
// Timing
startedAt: Date
endedAt: Date
duration: number // ms
// LLM Call specific
llmRequest?: {
provider: string
model: string
messages: Message[]
tools?: ToolDefinition[]
temperature: number
maxTokens: number
}
llmResponse?: {
content: string
toolCalls?: ToolCall[]
finishReason: string
usage: TokenUsage
}
// Tool Call specific
toolRequest?: {
name: string
arguments: Record<string, unknown>
}
toolResponse?: {
result: unknown
error?: string
}
// Cost
cost: number
// Status
status: 'success' | 'error' | 'timeout'
error?: Error
}
interface TokenUsage {
promptTokens: number
completionTokens: number
totalTokens: number
}// @agentbench/core/src/types/run.ts
interface RunConfig {
name: string
description?: string
projectId: string
testCaseId?: string
// Agent configuration
agent: {
provider: 'openai' | 'anthropic' | 'gemini' | 'custom'
model: string
temperature: number
maxTokens: number
systemPrompt: string
tools: ToolDefinition[]
}
// Input
input: {
messages: Message[]
variables?: Record<string, string>
context?: Record<string, unknown>
}
// Execution options
options: {
timeout: number // ms
maxSteps: number
retries: number
concurrency: number
seed?: number // For deterministic replay
}
// Tags
tags: string[]
metadata: Record<string, unknown>
}
interface RunResult {
id: string
config: RunConfig
status: 'passed' | 'failed' | 'error' | 'timeout'
// Trace
trace: ExecutionTrace
// Metrics
metrics: RunMetrics
// Evaluations
scores: Score[]
assertionResults: AssertionResult[]
// Timing
startedAt: Date
endedAt: Date
duration: number
// Summary
summary: string
}
interface RunMetrics {
totalTokens: number
promptTokens: number
completionTokens: number
totalCost: number
totalLatency: number
firstTokenLatency: number
toolCallCount: number
toolSuccessCount: number
toolFailureCount: number
stepCount: number
llmCallCount: number
}# Projects
GET /api/v1/projects
POST /api/v1/projects
GET /api/v1/projects/:id
PATCH /api/v1/projects/:id
DELETE /api/v1/projects/:id
# Test Suites
GET /api/v1/projects/:projectId/suites
POST /api/v1/projects/:projectId/suites
GET /api/v1/suites/:id
PATCH /api/v1/suites/:id
DELETE /api/v1/suites/:id
# Test Cases
GET /api/v1/suites/:suiteId/cases
POST /api/v1/suites/:suiteId/cases
GET /api/v1/cases/:id
PATCH /api/v1/cases/:id
DELETE /api/v1/cases/:id
# Runs
POST /api/v1/runs # Execute a run
GET /api/v1/runs # List runs (with filters)
GET /api/v1/runs/:id # Run detail
GET /api/v1/runs/:id/trace # Execution trace
GET /api/v1/runs/:id/scores # Evaluation scores
POST /api/v1/runs/:id/replay # Replay this run
POST /api/v1/runs/:id/evaluate # Re-evaluate
GET /api/v1/runs/:id/artifacts # Run artifacts (streaming)
# Compare / Diff
POST /api/v1/compare # Compare two runs
GET /api/v1/compare/:id # Comparison result
# Snapshots
GET /api/v1/projects/:projectId/snapshots
POST /api/v1/snapshots
GET /api/v1/snapshots/:id
POST /api/v1/snapshots/:id/restore
DELETE /api/v1/snapshots/:id
# Experiments
GET /api/v1/projects/:projectId/experiments
POST /api/v1/experiments
GET /api/v1/experiments/:id
POST /api/v1/experiments/:id/run
GET /api/v1/experiments/:id/results
# Coverage
GET /api/v1/projects/:projectId/coverage
GET /api/v1/projects/:projectId/coverage/trend
# Datasets
GET /api/v1/projects/:projectId/datasets
POST /api/v1/datasets
GET /api/v1/datasets/:id
POST /api/v1/datasets/:id/import
GET /api/v1/datasets/:id/export
DELETE /api/v1/datasets/:id
# API Keys
GET /api/v1/api-keys
POST /api/v1/api-keys
DELETE /api/v1/api-keys/:id
# Dashboard
GET /api/v1/dashboard/stats # Aggregated statistics
GET /api/v1/dashboard/trends # Time-series data
GET /api/v1/dashboard/models # Per-model stats
# Auth
POST /api/v1/auth/sign-up
POST /api/v1/auth/sign-in
POST /api/v1/auth/sign-out
POST /api/v1/auth/refresh
# Webhook
POST /api/v1/webhooks/github
POST /api/v1/webhooks/gitlab
POST /api/v1/webhooks/ci
# Real-time Run Updates
ws://localhost:3000/ws/runs/:runId
Events:
- run.started { runId, timestamp }
- run.step { runId, step: TraceStep }
- run.completed { runId, result: RunResult }
- run.error { runId, error }
# Dashboard Live Updates
ws://localhost:3000/ws/dashboard
Events:
- stats.updated { stats: DashboardStats }
- run.created { run: RunSummary }
- run.completed { run: RunSummary }
app/
├── (auth)/
│ ├── sign-in/page.tsx
│ ├── sign-up/page.tsx
│ ├── forgot-password/page.tsx
│ └── layout.tsx # Auth layout (centered card)
│
├── (dashboard)/
│ ├── layout.tsx # Dashboard layout (sidebar + header)
│ │
│ ├── dashboard/
│ │ └── page.tsx # Main dashboard
│ │
│ ├── projects/
│ │ ├── page.tsx # Project list
│ │ └── [projectId]/
│ │ ├── page.tsx # Project overview
│ │ ├── tests/
│ │ │ ├── page.tsx # Test suites & cases
│ │ │ └── [testId]/
│ │ │ └── page.tsx # Test case detail
│ │ ├── runs/
│ │ │ ├── page.tsx # Run history
│ │ │ └── [runId]/
│ │ │ ├── page.tsx # Run detail (timeline)
│ │ │ └── trace/
│ │ │ └── page.tsx # Full trace view
│ │ ├── experiments/
│ │ │ ├── page.tsx # Experiment list
│ │ │ └── [experimentId]/
│ │ │ └── page.tsx # Experiment detail
│ │ ├── coverage/
│ │ │ └── page.tsx # Coverage dashboard
│ │ ├── datasets/
│ │ │ ├── page.tsx # Dataset list
│ │ │ └── [datasetId]/
│ │ │ └── page.tsx # Dataset detail
│ │ └── settings/
│ │ └── page.tsx # Project settings
│ │
│ ├── snapshots/
│ │ ├── page.tsx # Snapshot list
│ │ └── [snapshotId]/
│ │ └── page.tsx # Snapshot detail
│ │
│ ├── compare/
│ │ └── page.tsx # Compare runs
│ │
│ ├── settings/
│ │ ├── page.tsx # Account settings
│ │ ├── api-keys/
│ │ │ └── page.tsx # API keys management
│ │ ├── team/
│ │ │ └── page.tsx # Team management (Pro)
│ │ └── billing/
│ │ └── page.tsx # Billing (Pro)
│ │
│ └── docs/
│ └── page.tsx # In-app documentation
│
├── api/
│ └── v1/
│ ├── projects/...
│ ├── runs/...
│ ├── snapshots/...
│ ├── experiments/...
│ ├── datasets/...
│ ├── compare/...
│ └── ...
│
├── layout.tsx # Root layout
├── page.tsx # Landing page
├── error.tsx # Global error boundary
├── not-found.tsx # 404 page
└── loading.tsx # Global loading
agentbench-cli/src/
├── index.ts # Entry point
├── cli.ts # CLI bootstrap
│
├── commands/
│ ├── init.ts # agentbench init
│ ├── run.ts # agentbench run
│ ├── test.ts # agentbench test
│ ├── replay.ts # agentbench replay
│ ├── compare.ts # agentbench compare
│ ├── snapshot.ts # agentbench snapshot
│ ├── experiment.ts # agentbench experiment
│ ├── report.ts # agentbench report
│ ├── config.ts # agentbench config
│ └── serve.ts # agentbench serve (launch dashboard)
│
├── ui/ # CLI UI (Ink/React)
│ ├── run-progress.tsx # Live run progress
│ ├── results-table.tsx # Results table
│ ├── trace-view.tsx # Trace viewer
│ └── report-view.tsx # Report viewer
│
├── config/
│ ├── loader.ts # agentbench.config.ts loader
│ └── default-config.ts # Default configuration
│
└── utils/
├── output.ts # Formatted output
├── spinner.ts # Loading spinner
└── errors.ts # Error formatting
┌─────────────────────────────────────────────────────────────┐
│ Database ER Diagram │
├─────────────────────────────────────────────────────────────┤
│ │
│ ┌──────────┐ ┌──────────────┐ ┌──────────┐ │
│ │ User │1────N│ Project │1────N│ TestSuite│ │
│ └──────────┘ └──────┬───────┘ └────┬─────┘ │
│ │ │ │
│ │ 1│ │
│ │ │ │
│ ┌─────────────┼─────────┐ N│ │
│ │ │ │ │ │
│ ▼ ▼ ▼ ▼ │
│ ┌─────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐│
│ │ Dataset │ │Experiment│ │ Run │ │TestCase ││
│ └────┬────┘ └────┬─────┘ └────┬─────┘ └────┬─────┘│
│ │ │ │ │ │
│ 1│ 1│ 1│ 1│ │
│ │ │ │ │ │
│ N│ N│ N│ N│ │
│ ┌────┴────┐ ┌────┴─────┐ │ ┌──────┴─────┐│
│ │Dataset │ │Variant │ │ │ Assertion ││
│ │Item │ └──────────┘ │ │ Evaluator ││
│ └─────────┘ │ └────────────┘│
│ │ │
│ ┌───────────────────────┼───────────────┐ │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │TraceStep │ │ Score │ │Assertion │ │
│ │ │ │ │ │Result │ │
│ └──────────┘ └──────────┘ └──────────┘ │
│ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ Snapshot │ │RunMetrics│ │ APIKey │ │
│ └────┬─────┘ └──────────┘ └──────────┘ │
│ │ │
│ 1│ │
│ │ │
│ 1│ │
│ ┌────┴─────────┐ │
│ │SnapshotData │ (JSONB - full snapshot payload) │
│ └──────────────┘ │
│ │
│ ┌──────────┐ ┌──────────────┐ │
│ │Team/Org │1────N│ TeamMember │ (Pro/Enterprise) │
│ └──────────┘ └──────────────┘ │
│ │
│ ┌──────────┐ │
│ │AuditLog │ (Enterprise) │
│ └──────────┘ │
│ │
└─────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────┐
│ Production Deployment │
├─────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────────┐ ┌─────────────────┐ │
│ │ CDN / Proxy │ │ Docker Hub │ │
│ │ (CloudFlare) │ │ / ghcr.io │ │
│ └────────┬────────┘ └─────────────────┘ │
│ │ │
│ ▼ │
│ ┌────────────────────────────────────────┐ │
│ │ Docker Compose / K8s │ │
│ │ │ │
│ │ ┌──────────┐ ┌──────────┐ │ │
│ │ │ Next.js │ │ Worker │ │ │
│ │ │ (Web) │ │ (BullMQ) │ │ │
│ │ │ :3000 │ │ │ │ │
│ │ └──────────┘ └──────────┘ │ │
│ │ │ │
│ │ ┌──────────┐ ┌──────────┐ │ │
│ │ │PostgreSQL│ │ Redis │ │ │
│ │ │ :5432 │ │ :6379 │ │ │
│ │ └──────────┘ └──────────┘ │ │
│ │ │ │
│ │ ┌──────────┐ │ │
│ │ │ MinIO │ (File storage) │ │
│ │ │ :9000 │ │ │
│ │ └──────────┘ │ │
│ └────────────────────────────────────────┘ │
│ │
│ For Self-Hosted (Community): │
│ - Single docker-compose.yml │
│ - 4 containers: web, worker, postgres, redis │
│ - 2GB RAM minimum │
│ │
│ For Enterprise: │
│ - Kubernetes Helm Chart │
│ - Horizontal scaling for workers │
│ - Managed PostgreSQL / Redis │
│ - SSO (SAML/OIDC) │
│ - Audit logging │
│ │
└─────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────┐
│ Security Layers │
├─────────────────────────────────────────────────────────────┤
│ │
│ Layer 1: Transport │
│ - HTTPS everywhere │
│ - HSTS headers │
│ - TLS 1.3 minimum │
│ │
│ Layer 2: Authentication │
│ - NextAuth.js v5 (JWT + Database sessions) │
│ - OAuth 2.0 (Google, GitHub) │
│ - API Key auth for programmatic access │
│ - MFA/2FA (Pro/Enterprise) │
│ │
│ Layer 3: Authorization │
│ - RBAC: Owner > Admin > Editor > Viewer │
│ - Project-level permissions │
│ - API Key scopes: read, write, admin │
│ │
│ Layer 4: Data Protection │
│ - API Key hashing (SHA-256) │
│ - Sensitive data encryption at rest │
│ - LLM API keys encrypted (AES-256-GCM) │
│ - PII masking in traces (configurable) │
│ │
│ Layer 5: Rate Limiting │
│ - Per-user rate limiting │
│ - Per-API-Key rate limiting │
│ - Per-IP rate limiting │
│ - LLM Judge rate limiting (cost control) │
│ │
│ Layer 6: Audit │
│ - All mutations logged │
│ - Sensitive operations logged │
│ - Audit log retention policy │
│ │
└─────────────────────────────────────────────────────────────┘
The pluggable provider architecture allows any LLM provider to be integrated with minimal code. All providers implement the AgentBenchProvider interface:
// @agentbench/core/src/types/provider.ts
interface AgentBenchProvider {
/** Unique provider identifier (e.g., 'openai', 'gemini', 'deepseek') */
readonly id: string
/** Human-readable display name */
readonly name: string
/** Initialize the provider with credentials and options */
initialize(config: ProviderConfig): Promise<void>
/** Create a chat completion (non-streaming) */
createChatCompletion(params: ChatCompletionParams): Promise<ChatCompletionResult>
/** Create a streaming chat completion */
createStreamingChatCompletion(
params: ChatCompletionParams,
callbacks: StreamingCallbacks
): Promise<ChatCompletionResult>
/** List available models for this provider */
listModels(): Promise<ModelInfo[]>
/** Validate API credentials */
validateCredentials(): Promise<boolean>
/** Get token count for messages (for cost estimation) */
countTokens(messages: Message[]): Promise<number>
/** Clean up resources */
dispose(): Promise<void>
}
interface ProviderConfig {
apiKey?: string
baseURL?: string
organization?: string
defaultModel?: string
timeout?: number
maxRetries?: number
extra?: Record<string, unknown>
}For providers using OpenAI-compatible APIs (DeepSeek, OpenRouter, Groq, Ollama, Azure), the @agentbench/provider-utils package provides a base class:
// @agentbench/provider-utils/src/openai-compatible-provider.ts
abstract class OpenAICompatibleProvider implements AgentBenchProvider {
protected client: OpenAI // OpenAI-compatible client
constructor(
protected readonly id: string,
protected readonly name: string,
protected readonly defaultBaseURL: string
) {}
// Default implementations for all AgentBenchProvider methods
async createChatCompletion(params: ChatCompletionParams): Promise<ChatCompletionResult> { ... }
async createStreamingChatCompletion(params, callbacks): Promise<ChatCompletionResult> { ... }
async listModels(): Promise<ModelInfo[]> { ... }
async countTokens(messages: Message[]): Promise<number> { ... }
async validateCredentials(): Promise<boolean> { ... }
// Subclasses override only what's different
protected abstract mapMessages(messages: Message[]): OpenAI.Chat.Completions.ChatCompletionMessageParam[]
protected abstract mapResponse(response: OpenAI.Chat.Completions.ChatCompletion): ChatCompletionResult
}// Providers are auto-discovered and registered
import { registerProvider, getProvider, listProviders } from '@agentbench/core'
// Manually register a custom provider
registerProvider(new MyCustomProvider())
// Discover and initialize all installed providers
const providers = await discoverProviders()
for (const provider of providers) {
await provider.initialize(config)
}
// CLI auto-discovers providers from installed packages
// $ agentbench run --provider groq --model llama-3.1-70bThe Dataset module supports full dataset lifecycle management — import, export, validation, splitting, versioning, diffing, and comparison.
@agentbench/core/src/dataset/
├── index.ts
├── dataset-manager.ts # Main dataset lifecycle manager
├── import/ # Dataset import
│ ├── index.ts
│ ├── csv-importer.ts # CSV import
│ ├── json-importer.ts # JSON import
│ ├── jsonl-importer.ts # JSONL import
│ └── auto-detect.ts # Format auto-detection
├── export/ # Dataset export
│ ├── index.ts
│ ├── csv-exporter.ts
│ ├── json-exporter.ts
│ └── jsonl-exporter.ts
├── validation/ # Dataset validation
│ ├── index.ts
│ ├── schema-validator.ts # Schema conformance
│ ├── integrity-check.ts # Data integrity
│ └── quality-check.ts # Quality metrics (missing, duplicates)
├── split/ # Dataset splitting
│ ├── index.ts
│ ├── random-split.ts # Random train/test/val
│ ├── stratified-split.ts # Stratified by label
│ └── cross-fold.ts # K-fold cross-validation
├── sample/ # Sampling
│ ├── index.ts # Random, stratified, weighted sampling
│ └── reservoir.ts # Reservoir sampling for large files
├── version/ # Dataset versioning
│ ├── index.ts
│ ├── version-store.ts # Version metadata storage
│ └── diff-engine.ts # Dataset version diff
└── compare/ # Dataset comparison
├── index.ts
└── comparator.ts # Statistical comparison of datasets
interface Dataset {
id: string
name: string
description?: string
format: 'csv' | 'json' | 'jsonl'
schema: DatasetSchema
version: number
versions: DatasetVersion[]
items: DatasetItem[]
metadata: DatasetMetadata
createdAt: Date
updatedAt: Date
}
interface DatasetItem {
id: string
input: {
messages: Message[]
variables?: Record<string, string>
context?: Record<string, unknown>
}
expected?: {
output?: string
toolCalls?: ToolCall[]
score?: number
labels?: string[]
}
metadata: Record<string, unknown>
tags: string[]
}v0.3.0 introduces a unified configuration system with defineConfig and a multi-source resolution order:
// agentbench.config.ts
import { defineConfig } from '@agentbench/core'
export default defineConfig({
name: 'my-agent-project',
version: '0.3.0',
// Provider configuration (new in v0.3.0)
providers: {
openai: {
apiKey: process.env.OPENAI_API_KEY,
defaultModel: 'gpt-4o',
},
anthropic: {
apiKey: process.env.ANTHROPIC_API_KEY,
defaultModel: 'claude-sonnet-4-20250514',
},
gemini: {
apiKey: process.env.GEMINI_API_KEY,
defaultModel: 'gemini-2.5-pro',
},
deepseek: {
apiKey: process.env.DEEPSEEK_API_KEY,
baseURL: 'https://api.deepseek.com/v1',
},
ollama: {
baseURL: 'http://localhost:11434',
defaultModel: 'llama3',
},
},
// Test configuration
tests: {
timeout: 30000,
maxSteps: 10,
retries: 1,
concurrency: 4,
},
// Dataset configuration (new in v0.3.0)
datasets: {
directory: './datasets',
defaultFormat: 'jsonl',
autoValidate: true,
},
// Benchmark configuration (new in v0.3.0)
benchmarks: {
directory: './benchmarks',
registry: 'https://registry.agentbench.dev',
},
// Reporting
reports: {
formats: ['json', 'html', 'markdown', 'junit'],
outputDir: './reports',
},
})Config is resolved from multiple sources, with later sources overriding earlier ones:
1. Default config — @agentbench/core built-in defaults
2. agentbench.config.ts — Project-level config file
3. agentbench.config.local.ts — Local overrides (gitignored)
4. Environment variables — AGENTBENCH_* prefixed env vars
5. CLI flags — Command-line arguments (highest priority)
# Environment variable overrides
AGENTBENCH_PROVIDERS_OPENAI_API_KEY=sk-...
AGENTBENCH_TESTS_TIMEOUT=60000
AGENTBENCH_DATASETS_DIR=./custom-dataThe AgentBench VS Code extension (agentbench-vscode) provides in-editor testing for AI agents:
agentbench-vscode/
├── package.json # Extension manifest
├── src/
│ ├── extension.ts # Activation entry point
│ │
│ ├── providers/ # VS Code data providers
│ │ ├── test-tree.ts # Test Explorer tree provider
│ │ ├── run-history.ts # Run history view
│ │ └── coverage-view.ts # Coverage visualization
│ │
│ ├── panels/ # Webview panels
│ │ ├── trace-viewer.ts # Interactive trace viewer
│ │ ├── diff-viewer.ts # Side-by-side diff viewer
│ │ └── benchmark-view.ts # Benchmark results
│ │
│ ├── commands/ # VS Code commands
│ │ ├── run-test.ts # Run test at cursor
│ │ ├── run-suite.ts # Run entire suite
│ │ ├── replay.ts # Replay a run
│ │ └── snapshot.ts # Create snapshot
│ │
│ ├── decorations/ # Editor decorations
│ │ ├── coverage-gutter.ts # Coverage in gutter
│ │ ├── inline-results.ts # Inline assertion results
│ │ └── status-bar.ts # Run status in status bar
│ │
│ └── tasks/ # VS Code task integration
│ ├── test-task.ts
│ └── benchmark-task.ts
│
└── media/ # Webview resources
├── trace-viewer/
└── diff-viewer/
- Test Explorer: Browse and run test suites/cases from the VS Code sidebar
- Inline Results: Pass/fail assertions shown inline in the editor gutter
- Trace Viewer: Interactive flame graph and step-by-step execution replay
- Run Launcher: Right-click to run tests with different models/providers
- Coverage Gutter: See which code paths are covered/untested
- Status Bar: Live test status, run progress, token usage
agentbench-github/
├── action.yml # GitHub Action definition
├── src/
│ ├── action.ts # Action entry point (post-commit)
│ │
│ ├── checks/ # GitHub Checks API
│ │ ├── check-run.ts # Create/update check runs
│ │ ├── annotations.ts # Line-level annotations
│ │ └── summary.ts # Check run summary markdown
│ │
│ ├── comments/ # PR Comments
│ │ ├── comment-builder.ts # Build comparison tables
│ │ ├── regression-alert.ts # Regression detection comment
│ │ └── benchmark-report.ts # Benchmark run report
│ │
│ ├── status/ # Commit Statuses
│ │ ├── status-check.ts # Set commit status
│ │ └── badge.ts # CI badge generation
│ │
│ ├── app/ # GitHub App (Pro)
│ │ ├── auth.ts # App authentication
│ │ ├── webhook.ts # Webhook event handling
│ │ └── permissions.ts # Permission scoping
│ │
│ └── config/ # Integration config
│ └── workflow.ts # Workflow template generation
│
└── templates/ # Workflow templates
├── agent-test.yml # Basic test workflow
├── agent-benchmark.yml # Benchmark workflow
└── agent-regression.yml # Regression detection workflow
# .github/workflows/agent-test.yml
name: Agent Tests
on:
pull_request:
paths:
- 'agent/**'
- 'prompts/**'
- 'agentbench.config.ts'
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: agentbench/github-action@v0.3
with:
config: agentbench.config.ts
mode: test
fail-on-regression: true
comment-pr: trueWhen a PR modifies agent code or prompts, the GitHub integration:
- Runs the test suite against the PR branch
- Runs the same tests against the base branch
- Posts a comparison comment showing:
- Pass/fail status changes
- Token usage delta
- Cost delta
- Latency delta
- New regressions detected
- Adds line annotations for failed assertions
- Sets commit status (green/red check)
[](https://agentbench.dev)Next Step: Database Detailed Design (Prisma Schema)
AgentBench v0.3.0 · GitHub · Report Issue · Changelog
- Core-Concepts
- Replay & Snapshots
- Assertions & Evaluation
- Coverage & Non-Determinism
- Guides
- Testing OpenAI / Anthropic
- CI/CD Integration
- Custom-Providers
- Migration-Guide
- Cookbook
- Prompt Regressions
- Model Migration
- Cost Budgets
- Safety Testing
- A/B Testing