Provider Ecosystem

Provider Ecosystem

AgentBench provides a rich ecosystem of 12+ LLM provider plugins, enabling you to test AI agents built on any model, from any provider, with a uniform testing interface. All providers implement the same AgentBenchProvider interface from @agentbench/provider-utils, so your tests work identically regardless of which model powers your agent.

Read the full guide on building custom providers: Building Custom Providers

---

Official Provider Plugins

@agentbench/openai -- OpenAI

The most widely used provider. Wraps the official OpenAI SDK to intercept every API call transparently -- capturing traces, counting tokens, and calculating costs automatically. Supports all GPT models (GPT-4o, GPT-4o-mini, GPT-4.1), o-series reasoning models (o1, o3, o4-mini), streaming, function/tool calling, structured outputs, and vision.

pnpm add @agentbench/openai

import { OpenAIProvider } from '@agentbench/openai'

Package documentation

---

@agentbench/anthropic -- Anthropic Claude

Wraps the Anthropic SDK for Claude models. Supports the Messages API with system + messages format, extended thinking/reasoning (Claude Opus 4, Sonnet 4), tool use with tool_use blocks, streaming, and computer use. Handles Claude-specific message format differences from the OpenAI format automatically.

pnpm add @agentbench/anthropic

import { AnthropicProvider } from '@agentbench/anthropic'

Package documentation

---

@agentbench/gemini -- Google Gemini [NEW in v0.3.0]

Full support for Google's Gemini models (Gemini 2.5 Pro, Gemini 2.5 Flash). Handles Gemini's native API format including streaming, embeddings, vision inputs, and tool calling. Converts between Gemini's format and AgentBench's normalized trace format.

pnpm add @agentbench/gemini

import { GeminiProvider } from '@agentbench/gemini'

Package documentation

---

@agentbench/deepseek -- DeepSeek [NEW in v0.3.0]

Provider for DeepSeek models. Use deepseek-chat for DeepSeek-V3 (general-purpose chat) and deepseek-reasoner for DeepSeek-R1 (advanced reasoning). Extends OpenAICompatibleProvider with special support for reasoning models that produce reasoning_content fields and have distinct pricing for reasoning vs. output tokens.

pnpm add @agentbench/deepseek

import { DeepSeekProvider } from '@agentbench/deepseek'

Package documentation

---

@agentbench/azure-openai -- Azure OpenAI [NEW in v0.3.0]

Provider for Azure OpenAI Service. Extends OpenAICompatibleProvider with Azure AD authentication support, resource-specific endpoint configuration, and API version management. Maps Azure's deployment-based model naming to AgentBench's unified model identifiers.

pnpm add @agentbench/azure-openai

import { AzureOpenAIProvider } from '@agentbench/azure-openai'

Package documentation

---

@agentbench/openrouter -- OpenRouter [NEW in v0.3.0]

Provider for OpenRouter, a multi-model pass-through gateway that gives access to 200+ models from a single API endpoint. Handles OpenRouter-specific headers for model routing, fallback configuration, and provider-specific pricing. Ideal for model comparison and migration testing.

pnpm add @agentbench/openrouter

import { OpenRouterProvider } from '@agentbench/openrouter'

Package documentation

---

@agentbench/groq -- Groq [NEW in v0.3.0]

Provider for Groq's ultra-fast inference platform. Extends OpenAICompatibleProvider to handle Groq's low-latency API characteristics. Great for testing agents that require rapid iteration or have strict latency SLAs. Supports Llama, Mixtral, and Gemma models hosted on Groq's hardware.

pnpm add @agentbench/groq

import { GroqProvider } from '@agentbench/groq'

Package documentation

---

@agentbench/ollama -- Ollama [NEW in v0.3.0]

Provider for Ollama, the local LLM runner. Auto-detects running Ollama instances, enumerates available models, and handles Ollama's streaming format. Enables completely free, local agent testing with no API keys and no network dependency. Ideal for offline development and privacy-sensitive workflows.

pnpm add @agentbench/ollama

import { OllamaProvider } from '@agentbench/ollama'

Package documentation

---

Adapter Packages

In addition to providers, AgentBench ships adapter packages for popular agent frameworks:

Package	Description
@agentbench/adapter	Generic adapter -- wrap any agent framework for automatic tracing
@agentbench/langgraph	Trace and evaluate LangGraph agents with state graph awareness
@agentbench/mcp	MCP (Model Context Protocol) -- automatic tool call tracing

---

Building a Custom Provider

You can add support for any LLM API by implementing the AgentBenchProvider interface. There are two approaches:

Approach 1: Extend OpenAICompatibleProvider (Recommended for most APIs)

If your API speaks the OpenAI Chat Completions format, you only need to override 3 methods:

1. getBaseURL() -- Return your API's base URL
2. getDefaultHeaders() -- Return authentication and custom headers
3. getProviderId() -- Return a unique provider identifier string

import { OpenAICompatibleProvider } from '@agentbench/provider-utils'

export class MyCustomProvider extends OpenAICompatibleProvider {
  getBaseURL(): string {
    return 'https://api.myprovider.com/v1'
  }

  getDefaultHeaders(): Record<string, string> {
    return {
      'Authorization': `Bearer ${this.apiKey}`,
      'X-Custom-Header': 'value',
    }
  }

  getProviderId(): string {
    return 'my-provider'
  }
}

Approach 2: Build From Scratch

For APIs with completely different formats (like Anthropic), implement all methods of the AgentBenchProvider interface directly:

interface AgentBenchProvider {
  // Identity
  getProviderId(): string
  getProviderName(): string

  // Core LLM interaction
  createChatCompletion(params: ChatCompletionParams): Promise<ChatCompletionResponse>
  createChatCompletionStream(params: ChatCompletionParams): AsyncIterable<ChatCompletionChunk>

  // Token counting and cost
  countTokens(messages: Message[], model: string): Promise<number>
  calculateCost(model: string, promptTokens: number, completionTokens: number): CostBreakdown

  // Health and metadata
  healthCheck(): Promise<HealthStatus>
  listModels(): Promise<ModelInfo[]>
}

Read the complete tutorial: Building Custom Providers

---

Provider Utility Package

The @agentbench/provider-utils package provides the shared foundation for all providers:

• AgentBenchProvider interface -- the contract every provider implements
• OpenAICompatibleProvider base class -- for any OpenAI-format API
• Token counting utilities (tiktoken-based for OpenAI models, character-based fallback)
• Cost calculation helpers with per-model pricing tables
• SSE streaming parser for OpenAI-compatible streaming endpoints
• Provider registration and auto-discovery

pnpm add @agentbench/provider-utils

---

Choosing a Provider

Your Situation	Recommended Provider
Using OpenAI GPT models	@agentbench/openai
Using Anthropic Claude models	@agentbench/anthropic
Using Google Gemini	@agentbench/gemini
Using DeepSeek models	@agentbench/deepseek
Using Azure-hosted OpenAI	@agentbench/azure-openai
Comparing many models via a gateway	@agentbench/openrouter
Need ultra-fast inference	@agentbench/groq
Want free, local, offline testing	@agentbench/ollama
Using a provider not listed above	Build a custom provider

---

Related pages: Guides | Core-Concepts | Config-Reference | Examples