Contributing

Contributing to AgentBench

Thanks for your interest in contributing. AgentBench is a monorepo with 15 packages, a CLI, a web dashboard, and a VS Code extension. This guide covers everything you need to get started.

Read the full contributing guide: CONTRIBUTING.md

---

Development Setup

Prerequisites

• Node.js >= 20
• pnpm >= 9 -- npm install -g pnpm
• Docker -- for PostgreSQL + Redis (optional, you can point at existing instances)
• Git

Clone and Install

git clone git@github.com:1304674612/agentbench.git
cd agentbench
pnpm install

Start Infrastructure

# Start PostgreSQL + Redis
docker compose up -d

# Verify services are healthy
docker compose ps

Initialize the Database

cp .env.example .env
pnpm db:generate    # Generate Prisma Client
pnpm db:push        # Push schema to database

Start Development

pnpm dev            # Starts all packages in watch mode + web dashboard

The dashboard is available at http://localhost:3000.

Run Tests

pnpm test           # Run all unit tests across packages
pnpm test:watch     # Watch mode
pnpm lint           # Biome linting
pnpm format         # Biome formatting
pnpm typecheck      # TypeScript strict mode check

---

Monorepo Structure

agentbench/
├── packages/                    # 15 packages
│   ├── core/                    #   @agentbench/core -- Engine, types, runner, tracer, evaluator
│   ├── config/                  #   @agentbench/config -- defineConfig, Zod schemas
│   ├── provider-utils/          #   @agentbench/provider-utils -- Base classes for providers
│   ├── openai/                  #   @agentbench/openai
│   ├── anthropic/               #   @agentbench/anthropic
│   ├── gemini/                  #   @agentbench/gemini
│   ├── deepseek/                #   @agentbench/deepseek
│   ├── azure-openai/            #   @agentbench/azure-openai
│   ├── openrouter/              #   @agentbench/openrouter
│   ├── groq/                    #   @agentbench/groq
│   ├── ollama/                  #   @agentbench/ollama
│   ├── mcp/                     #   @agentbench/mcp
│   ├── langgraph/               #   @agentbench/langgraph
│   └── typescript-config/       #   @agentbench/typescript-config -- Shared tsconfig presets
├── apps/
│   ├── cli/                     #   agentbench-cli -- Commander.js + Ink CLI
│   └── web/                     #   agentbench-web -- Next.js dashboard
├── examples/                    #   14 official reference implementations
├── docs/                        #   Documentation
├── docker-compose.yml           #   PostgreSQL + Redis for local dev
├── pnpm-workspace.yaml          #   pnpm workspace config
├── biome.json                   #   Biome formatter + linter config
├── vitest.workspace.ts          #   Vitest workspace config
└── turbo.json                   #   Turborepo pipeline config

Dependency Flow

@agentbench/core          ← No framework dependencies (pure TypeScript)
  ↑
@agentbench/provider-utils ← Depends on core
  ↑
@agentbench/openai, anthropic, gemini, deepseek, ...  ← Depends on provider-utils
  ↑
agentbench-cli             ← Depends on core + all provider packages
  ↑
agentbench-web             ← Depends on core

---

How to Add a Provider

To add support for a new LLM provider:

1. Choose your approach:

   • Extend OpenAICompatibleProvider if the API speaks OpenAI Chat Completions format (most common -- ~90% of APIs). Only need to override getBaseURL(), getDefaultHeaders(), and getProviderId().
   • Implement AgentBenchProvider interface from scratch if the API format is fundamentally different.

2. Create the package:

   cp -r packages/_template packages/<provider-name>

   Update package.json with the correct name (@agentbench/<name>), description, and peer dependencies.

3. Implement the provider class:

   import { OpenAICompatibleProvider } from '@agentbench/provider-utils'
   
   export class MyProvider extends OpenAICompatibleProvider {
     getBaseURL(): string { return 'https://api.example.com/v1' }
     getDefaultHeaders(): Record<string, string> {
       return { 'Authorization': `Bearer ${this.apiKey}` }
     }
     getProviderId(): string { return 'my-provider' }
   }

4. Add tests for the provider in packages/<name>/__tests__/.

5. Update the CLI to register the new provider in apps/cli/src/providers/.

6. Add documentation in docs/guides/ following the existing provider guide template.

7. Submit a PR with the new package and docs.

---

How to Add a Judge

To add a custom evaluation judge dimension:

1. Create the judge prompt in packages/core/src/evaluator/judge-prompts.ts. Each dimension needs a system prompt with a detailed scoring rubric (0-10 scale with explicit anchors) and a user prompt template with placeholders for input, output, and expected.

2. Register the dimension by adding it to the JudgeDimension type union in packages/core/src/types/evaluation.ts.

3. Add validation for the new dimension in the evaluator's score() handler -- the score must be parsed from JSON, validated to be a number 0-10, and include a reasoning string.

4. Add tests in packages/core/src/__tests__/evaluator/. Test that the judge prompt produces valid JSON, scores are within 0-10, and meta-evaluation (testing the judge against known outputs) passes.

5. Update documentation in docs/guides/custom-judges.md with the new dimension's rubric and use cases.

6. Submit a PR with the dimension, prompt, tests, and docs.

---

How to Add an Example

To add a new official example:

1. Create the example directory in examples/ with the standard structure:

   examples/<example-name>/
   ├── agentbench.config.ts
   ├── agent/
   │   └── index.ts              # Agent implementation
   ├── tests/
   │   ├── basic.test.ts         # Basic test suite (8+ test cases)
   │   ├── replay.test.ts        # Replay test suite
   │   └── dataset.json          # 20+ test inputs
   ├── .env.example
   ├── .github/workflows/
   │   └── agentbench.yml
   ├── package.json
   └── README.md                 # Standard template
   

2. Implement the agent following the relevant framework pattern.

3. Write tests with at least 3 different assertion types (tool, output, score, latency, tokens).

4. Create a replay test suite that uses snapshots for deterministic regression testing.

5. Write the README following the standard template (what it demonstrates, quick run instructions, link to source).

6. Add to the examples index in docs/examples/index.md.

7. Verify quality standards: passes agentbench test with 100%, at least 3 test suites, 8 test cases, 20 dataset items.

8. Submit a PR with the complete example.

---

Code Standards

AgentBench enforces code quality through automated tooling:

• Formatting: Biome -- pnpm format
• Linting: Biome -- pnpm lint
• Type Checking: TypeScript strict mode -- pnpm typecheck
• Testing: Vitest -- pnpm test

Code Style Guidelines

• Use type imports for type-only imports (import type { Foo } from './bar')
• Prefer interfaces over type aliases for public API surfaces
• Every exported function must have a JSDoc comment with @param and @returns
• Error messages must be descriptive and actionable (include what was expected and what was received)
• Use const assertions for string literal unions where applicable
• No any types in public API surfaces -- use unknown and narrow with type guards
• File names: kebab-case for modules, PascalCase for classes/components

Commit Messages

Follow Conventional Commits:

feat(scope): add feature description
fix(scope): fix bug description
docs(scope): update documentation
chore(scope): maintenance work
test(scope): add or update tests
refactor(scope): code restructuring

---

Pull Request Process

1. Fork the repository and create a feature branch from main.
2. Make your changes following the code standards above.
3. Add tests for any new functionality. All existing tests must continue to pass.
4. Run the full validation suite:

   pnpm lint && pnpm format && pnpm typecheck && pnpm test

5. Update documentation if your change adds, removes, or modifies public API.
6. Write a clear PR description:
   • What problem does this solve?
   • What changes were made and why?
   • How was it tested?
   • Screenshots or CLI output if relevant.
7. Submit the PR against the main branch.
8. Address review feedback -- PRs require at least one maintainer approval.
9. Squash and merge once approved.

---

Release Process

AgentBench uses Changesets for versioning and releases:

1. Add a changeset when making a change:

   pnpm changeset

   Select the affected packages and choose the bump type (patch, minor, major).

2. Changesets are reviewed alongside PRs. The changeset file describes the change in user-facing language.

3. On merge to main, the Changesets GitHub Action opens a "Version Packages" PR that bumps versions and updates changelogs automatically.

4. Merging the Version Packages PR triggers the release workflow, which publishes packages to npm, creates a GitHub Release, and updates the documentation site.

---

Getting Help

• Questions? Open a GitHub Discussion
• Bug report? Open a GitHub Issue with a minimal reproduction
• Feature request? Open a GitHub Discussion first to discuss before implementing
• Security issue? Email zhoujiankai1014@gmail.com -- do not open a public issue

---

Related pages: Architecture | Provider-Ecosystem | Examples | Roadmap