SDK Guide

SDK 使用指南

AgentBench v0.3.0 提供 10 个 SDK 包，覆盖 8 个主流 LLM 提供商，以及通用适配器和 Provider 构建工具。

安装

pnpm add @agentbench/core

# 按需安装 SDK（8 个提供商）
pnpm add @agentbench/openai
pnpm add @agentbench/anthropic
pnpm add @agentbench/gemini        # [NEW] Google Gemini
pnpm add @agentbench/deepseek      # [NEW] DeepSeek
pnpm add @agentbench/azure-openai  # [NEW] Azure OpenAI
pnpm add @agentbench/openrouter    # [NEW] OpenRouter
pnpm add @agentbench/groq          # [NEW] Groq
pnpm add @agentbench/ollama        # [NEW] Ollama（本地模型）
pnpm add @agentbench/mcp
pnpm add @agentbench/adapter

# 构建自定义 Provider
pnpm add @agentbench/provider-utils   # [NEW] Provider 构建工具

---

@agentbench/openai

OpenAI SDK 包装器，自动截获 API 调用并生成 Trace。

import { AgentBenchOpenAI, createOpenAIClient, runWithOpenAI } from '@agentbench/openai'

// 方式 1：直接使用包装器
const client = new AgentBenchOpenAI({
  apiKey: process.env.OPENAI_API_KEY!,
  tracing: true,
})

const result = await client.createChatCompletion({
  model: 'gpt-4o',
  messages: [
    { role: 'system', content: '你是一个客服' },
    { role: 'user', content: '如何退款？' },
  ],
  temperature: 0.7,
  max_tokens: 4096,
})

console.log(result.content)           // Agent 的回复
console.log(result.usage.total_tokens) // Token 用量
console.log(result.cost)               // 费用（USD）
console.log(result.trace)              // TraceStep

流式输出

const stream = client.createStreamingChatCompletion({
  model: 'gpt-4o',
  messages: [{ role: 'user', content: '讲个故事' }],
})

for await (const chunk of stream) {
  process.stdout.write(chunk.content)
}

与 Runner 集成

const client = createOpenAIClient({ apiKey: '...' })

const { output, trace, cost } = await runWithOpenAI({
  client,
  agent: {
    provider: 'openai',
    model: 'gpt-4o',
    temperature: 0.7,
    maxTokens: 4096,
    systemPrompt: '你是一个客服 Agent',
  },
  messages: [{ role: 'user', content: '如何退款？' }],
  tools: [{
    type: 'function',
    function: {
      name: 'search_docs',
      description: '搜索文档',
      parameters: { query: { type: 'string' } },
    },
  }],
  maxSteps: 10,
})

---

@agentbench/anthropic

Anthropic Claude SDK 包装器。

import { AgentBenchAnthropic, createAnthropicClient } from '@agentbench/anthropic'

const client = new AgentBenchAnthropic({
  apiKey: process.env.ANTHROPIC_API_KEY!,
})

const result = await client.createMessage({
  model: 'claude-sonnet-4-20250514',
  system: '你是一个客服 Agent，帮助用户解决退款问题。',
  messages: [{ role: 'user', content: '如何退款？' }],
  max_tokens: 4096,
  temperature: 0.7,
  tools: [{
    name: 'search_docs',
    description: '搜索文档',
    input_schema: {
      type: 'object',
      properties: { query: { type: 'string' } },
    },
  }],
})

console.log(result.content)   // Claude 的回复
console.log(result.cost)      // 费用

流式输出

const stream = client.createStreamingMessage({
  model: 'claude-sonnet-4-20250514',
  messages: [{ role: 'user', content: '讲个故事' }],
  max_tokens: 2048,
})

for await (const chunk of stream) {
  process.stdout.write(chunk.content)
}

---

@agentbench/gemini [NEW in v0.3.0]

Google Gemini SDK 包装器。

import { AgentBenchGemini, createGeminiClient } from '@agentbench/gemini'

const client = new AgentBenchGemini({
  apiKey: process.env.GEMINI_API_KEY!,
})

const result = await client.createChatCompletion({
  model: 'gemini-2.5-pro',
  messages: [
    { role: 'system', content: '你是一个客服 Agent' },
    { role: 'user', content: '如何退款？' },
  ],
  temperature: 0.7,
  maxTokens: 4096,
})

console.log(result.content)
console.log(result.usage.totalTokens)
console.log(result.cost)

流式输出

const stream = client.createStreamingChatCompletion({
  model: 'gemini-2.5-flash',
  messages: [{ role: 'user', content: '讲个故事' }],
})

for await (const chunk of stream) {
  process.stdout.write(chunk.content)
}

多模态输入

const result = await client.createChatCompletion({
  model: 'gemini-2.5-pro',
  messages: [
    {
      role: 'user',
      content: [
        { type: 'text', text: '这张图片里有什么？' },
        { type: 'image_url', image_url: { url: 'https://example.com/photo.jpg' } },
      ],
    },
  ],
})

---

@agentbench/deepseek [NEW in v0.3.0]

DeepSeek API SDK 包装器（兼容 OpenAI API 格式）。

import { AgentBenchDeepSeek, createDeepSeekClient } from '@agentbench/deepseek'

const client = new AgentBenchDeepSeek({
  apiKey: process.env.DEEPSEEK_API_KEY!,
  // baseURL 默认 https://api.deepseek.com/v1
})

const result = await client.createChatCompletion({
  model: 'deepseek-v3',
  messages: [
    { role: 'system', content: '你是一个代码审查 Agent' },
    { role: 'user', content: '审查以下代码的安全性：...' },
  ],
  temperature: 0.3,
  maxTokens: 8192,
})

console.log(result.content)
console.log(result.cost)  // DeepSeek 以极具竞争力的价格闻名

推理模式（DeepSeek-R1）

const result = await client.createChatCompletion({
  model: 'deepseek-r1',
  messages: [
    { role: 'user', content: '解这道数学题：...' },
  ],
  maxTokens: 16384,
})

// R1 模型会在 response 中包含推理链
console.log(result.reasoning)   // DeepSeek-R1 的思维链
console.log(result.content)      // 最终回答

---

@agentbench/azure-openai [NEW in v0.3.0]

Azure OpenAI Service SDK 包装器。

import { AgentBenchAzureOpenAI, createAzureOpenAIClient } from '@agentbench/azure-openai'

const client = new AgentBenchAzureOpenAI({
  apiKey: process.env.AZURE_OPENAI_API_KEY!,
  endpoint: process.env.AZURE_OPENAI_ENDPOINT!,  // https://<resource>.openai.azure.com
  deployment: 'gpt-4o',                           // Azure 部署名
  apiVersion: '2024-08-01-preview',
})

const result = await client.createChatCompletion({
  model: 'gpt-4o',  // 映射到 Azure 部署名
  messages: [
    { role: 'system', content: '你是一个企业客服 Agent' },
    { role: 'user', content: '如何升级服务套餐？' },
  ],
  temperature: 0.7,
  maxTokens: 4096,
})

console.log(result.content)

多部署管理

const client = new AgentBenchAzureOpenAI({
  apiKey: process.env.AZURE_OPENAI_API_KEY!,
  endpoint: process.env.AZURE_OPENAI_ENDPOINT!,
  deployments: {
    'gpt-4o': 'gpt-4o-prod',
    'gpt-4o-mini': 'gpt-4o-mini-prod',
    'gpt-4-turbo': 'gpt-4-turbo-staging',
  },
})

// 使用不同的部署
await client.createChatCompletion({ model: 'gpt-4o-mini', messages: [...] })

---

@agentbench/openrouter [NEW in v0.3.0]

OpenRouter 多提供商网关 SDK 包装器——通过统一 API 访问 200+ 模型。

import { AgentBenchOpenRouter, createOpenRouterClient } from '@agentbench/openrouter'

const client = new AgentBenchOpenRouter({
  apiKey: process.env.OPENROUTER_API_KEY!,
  // 可选：设置默认的 HTTP Referer 和应用名（OpenRouter 排行榜要求）
  appName: 'my-agent-app',
  appUrl: 'https://myagent.app',
})

// 通过 OpenRouter 访问 Anthropic Claude
const claudeResult = await client.createChatCompletion({
  model: 'anthropic/claude-sonnet-4-20250514',
  messages: [{ role: 'user', content: '分析这段文本的情感' }],
})

// 通过 OpenRouter 访问 Google Gemini
const geminiResult = await client.createChatCompletion({
  model: 'google/gemini-2.5-pro',
  messages: [{ role: 'user', content: '总结这篇文章' }],
})

// 通过 OpenRouter 访问 Meta Llama
const llamaResult = await client.createChatCompletion({
  model: 'meta-llama/llama-3.1-405b-instruct',
  messages: [{ role: 'user', content: '写一首诗' }],
})

console.log(`Provider: ${claudeResult.provider}`)  // 'anthropic'
console.log(`Model: ${claudeResult.model}`)          // 'claude-sonnet-4-20250514'

跨模型对比（一键测试多个模型）

const models = [
  'openai/gpt-4o',
  'anthropic/claude-sonnet-4-20250514',
  'google/gemini-2.5-pro',
  'deepseek/deepseek-v3',
]

for (const model of models) {
  const result = await client.createChatCompletion({
    model,
    messages: [{ role: 'user', content: '如何提升客户满意度？' }],
  })
  console.log(`${model}: ${result.content.slice(0, 50)}...`)
  console.log(`  Tokens: ${result.usage.totalTokens}, Cost: $${result.cost}`)
}

---

@agentbench/groq [NEW in v0.3.0]

Groq 高速推理 SDK 包装器——以极低延迟著称。

import { AgentBenchGroq, createGroqClient } from '@agentbench/groq'

const client = new AgentBenchGroq({
  apiKey: process.env.GROQ_API_KEY!,
})

const result = await client.createChatCompletion({
  model: 'llama-3.1-70b-versatile',
  messages: [
    { role: 'system', content: '你是一个快速的客服 Agent' },
    { role: 'user', content: '退款政策是什么？' },
  ],
  temperature: 0.7,
  maxTokens: 4096,
})

console.log(result.content)
console.log(`Latency: ${result.duration}ms`)  // Groq 通常比 OpenAI 快 3-5x
console.log(`Tokens/sec: ${result.usage.totalTokens / (result.duration / 1000)}`)

高速流式输出

const stream = client.createStreamingChatCompletion({
  model: 'llama-3.1-8b-instant',  // 最快的模型
  messages: [{ role: 'user', content: '讲个笑话' }],
})

let fullContent = ''
const startTime = Date.now()

for await (const chunk of stream) {
  fullContent += chunk.content
}
console.log(`Total time: ${Date.now() - startTime}ms`)
console.log(fullContent)

---

@agentbench/ollama [NEW in v0.3.0]

Ollama 本地模型 SDK 包装器——零 API 费用，完全离线。

import { AgentBenchOllama, createOllamaClient } from '@agentbench/ollama'

const client = new AgentBenchOllama({
  baseURL: 'http://localhost:11434',  // Ollama 默认地址
})

// 列出本地已安装的模型
const models = await client.listModels()
console.log('本地模型:', models.map(m => m.id))
// ['llama3:latest', 'qwen2:7b', 'mistral:latest']

const result = await client.createChatCompletion({
  model: 'llama3',
  messages: [
    { role: 'system', content: '你是一个客服 Agent' },
    { role: 'user', content: '如何退款？' },
  ],
  temperature: 0.7,
  maxTokens: 4096,
})

console.log(result.content)
console.log(result.cost)  // $0.00 — 本地模型无 API 费用

流式输出

const stream = client.createStreamingChatCompletion({
  model: 'llama3',
  messages: [{ role: 'user', content: '写一篇关于 AI 测试的文章' }],
})

for await (const chunk of stream) {
  process.stdout.write(chunk.content)
}

自定义模型参数

const result = await client.createChatCompletion({
  model: 'qwen2:7b',
  messages: [{ role: 'user', content: '你好' }],
  temperature: 0.8,
  maxTokens: 2048,
  extra: {
    // Ollama 特有参数
    num_ctx: 4096,        // 上下文窗口大小
    num_predict: -1,      // 不限输出 token
    repeat_penalty: 1.1,  // 重复惩罚
    top_k: 40,
    top_p: 0.9,
  },
})

多模型本地对比

const models = ['llama3', 'qwen2:7b', 'mistral', 'phi3:mini']

for (const model of models) {
  const result = await client.createChatCompletion({
    model,
    messages: [{ role: 'user', content: '1+1等于几？' }],
  })
  console.log(`${model}: ${result.content} (${result.duration}ms)`)
}

---

MCP（Model Context Protocol）客户端包装器。

import { AgentBenchMCP, createMCPClient } from '@agentbench/mcp'

const client = new AgentBenchMCP({
  serverUrl: 'http://localhost:8080/mcp',
  authToken: 'your-token',
})

// 连接并初始化
const { tools } = await client.connect()
console.log('可用工具:', tools.map(t => t.name))

// 调用工具
const result = await client.callTool({
  name: 'search_docs',
  arguments: { query: '退款政策' },
})
console.log(result.result)

// 读取资源
const resource = await client.readResource('docs://refund-policy')
console.log(resource.contents)

// 断开连接
client.disconnect()

---

@agentbench/adapter

通用适配器，用于包装自定义 Agent 或其他框架。

import { createAdapter, GenericAgentAdapter } from '@agentbench/adapter'

const adapter = createAdapter({
  name: 'my-custom-agent',
  provider: 'custom',
  run: async (input) => {
    // 你的自定义 Agent 逻辑
    const response = await myAgent.run(input.messages)

    return {
      output: response.text,
      toolCalls: response.toolCalls.map(tc => ({
        name: tc.name,
        arguments: tc.args,
        result: tc.result,
      })),
    }
  },
  hooks: {
    onStart: (config) => console.log('Agent started:', config.model),
    onStep: (step) => console.log('Step:', step.type),
    onEnd: (result) => console.log('Done:', result.status),
    onError: (err) => console.error('Error:', err),
  },
  tools: [
    { type: 'function', function: { name: 'search', description: 'Search', parameters: {} } },
  ],
})

// 使用适配器运行
const output = await adapter.run({
  messages: [{ role: 'user', content: 'Hello' }],
  systemPrompt: 'You are helpful.',
})

// 转换为 Runner 可用的 AgentConfig
const config = adapter.toAgentConfig()

框架适配器（Placeholder）

LangGraph、CrewAI、LlamaIndex 的适配器接口已预留，等待对应 SDK 生态成熟后实现。

import { createLangGraphAdapter, createCrewAIAdapter, createLlamaIndexAdapter } from '@agentbench/adapter'

// LangGraph（需要 @langchain/langgraph 依赖）
// const adapter = createLangGraphAdapter({ name: 'my-graph', graph: compiledGraph })

// CrewAI（需要 crewai Python SDK）
// const adapter = createCrewAIAdapter({ name: 'my-crew', crew: crewConfig })

// LlamaIndex（需要 llama-index Python SDK）
// const adapter = createLlamaIndexAdapter({ name: 'my-agent', agent: llamaAgent })

适配器注册表

import { registerAdapter, getAdapter, listAdapters } from '@agentbench/adapter'

registerAdapter(adapter)

const found = getAdapter('my-custom-agent')
console.log(listAdapters())

---

@agentbench/provider-utils [NEW in v0.3.0]

构建自定义 LLM Provider 的工具包。提供 AgentBenchProvider 接口和 OpenAICompatibleProvider 基类。

方式一：继承 OpenAICompatibleProvider（推荐）

如果目标提供商的 API 兼容 OpenAI 格式（大多数现代 LLM 提供商都兼容），只需继承 OpenAICompatibleProvider：

import { OpenAICompatibleProvider, type ProviderConfig, type ChatCompletionParams, type ChatCompletionResult } from '@agentbench/provider-utils'

class MyCustomProvider extends OpenAICompatibleProvider {
  constructor() {
    super(
      'my-provider',                        // 唯一 ID
      'My Custom Provider',                  // 显示名称
      'https://api.myprovider.com/v1'       // 默认 API 地址
    )
  }

  // 大多数方法已由基类实现，只需重写差异部分
}

方式二：实现 AgentBenchProvider 接口

如果提供商的 API 不是 OpenAI 兼容格式，需要完整实现接口：

import type { AgentBenchProvider, ProviderConfig, ChatCompletionParams, ChatCompletionResult, ModelInfo, StreamingCallbacks, Message } from '@agentbench/provider-utils'

class MyCustomProvider implements AgentBenchProvider {
  readonly id = 'my-provider'
  readonly name = 'My Custom Provider'
  private apiKey: string
  private initialized = false

  async initialize(config: ProviderConfig): Promise<void> {
    this.apiKey = config.apiKey!
    // 初始化连接、验证凭证...
    this.initialized = true
  }

  async createChatCompletion(params: ChatCompletionParams): Promise<ChatCompletionResult> {
    this.ensureInitialized()
    const startTime = Date.now()

    // 调用你的 API
    const response = await fetch('https://api.myprovider.com/v1/chat', {
      method: 'POST',
      headers: {
        'Authorization': `Bearer ${this.apiKey}`,
        'Content-Type': 'application/json',
      },
      body: JSON.stringify({
        model: params.model,
        messages: params.messages,
        temperature: params.temperature,
        max_tokens: params.maxTokens,
        tools: params.tools,
      }),
    })

    const data = await response.json()
    const duration = Date.now() - startTime

    return {
      id: data.id,
      model: data.model,
      content: data.choices[0].message.content,
      toolCalls: data.choices[0].message.tool_calls?.map(tc => ({
        id: tc.id,
        name: tc.function.name,
        arguments: JSON.parse(tc.function.arguments),
      })),
      finishReason: data.choices[0].finish_reason,
      usage: {
        promptTokens: data.usage.prompt_tokens,
        completionTokens: data.usage.completion_tokens,
        totalTokens: data.usage.total_tokens,
      },
      duration,
      cost: this.calculateCost(data.model, data.usage),
      raw: data,
    }
  }

  async createStreamingChatCompletion(
    params: ChatCompletionParams,
    callbacks: StreamingCallbacks
  ): Promise<ChatCompletionResult> {
    // 实现流式输出...
  }

  async listModels(): Promise<ModelInfo[]> {
    const response = await fetch('https://api.myprovider.com/v1/models', {
      headers: { 'Authorization': `Bearer ${this.apiKey}` },
    })
    const data = await response.json()
    return data.models.map(m => ({
      id: m.id,
      name: m.name,
      contextWindow: m.context_length,
      pricing: {
        input: m.pricing.input_per_1k_tokens,
        output: m.pricing.output_per_1k_tokens,
      },
    }))
  }

  async validateCredentials(): Promise<boolean> {
    try {
      await this.listModels()
      return true
    } catch {
      return false
    }
  }

  async countTokens(messages: Message[]): Promise<number> {
    // 实现 token 计数逻辑
    const text = messages.map(m => m.content).join(' ')
    return Math.ceil(text.length / 4)  // 粗略估算
  }

  async dispose(): Promise<void> {
    // 清理资源
    this.initialized = false
  }

  private ensureInitialized(): void {
    if (!this.initialized) throw new Error('Provider not initialized. Call initialize() first.')
  }

  private calculateCost(model: string, usage: { promptTokens: number; completionTokens: number }): number {
    // 各模型定价
    const pricing: Record<string, { input: number; output: number }> = {
      'my-model': { input: 0.001, output: 0.002 },
    }
    const p = pricing[model] ?? { input: 0, output: 0 }
    return (usage.promptTokens / 1000) * p.input + (usage.completionTokens / 1000) * p.output
  }
}

注册自定义 Provider

import { registerProvider } from '@agentbench/core'

const myProvider = new MyCustomProvider()
await myProvider.initialize({ apiKey: 'sk-...' })

registerProvider(myProvider)

// 之后就可以在其他地方使用
import { getProvider } from '@agentbench/core'
const provider = getProvider('my-provider')
const result = await provider.createChatCompletion({
  model: 'my-model',
  messages: [{ role: 'user', content: 'Hello' }],
})

Provider 自动发现

如果自定义 Provider 在 node_modules 中并遵循 @agentbench/* 命名约定，CLI 会自动发现：

agentbench run --provider my-provider --model my-model

---

所有 SDK 的输出都可以直接传入断言 DSL：

import { expect, buildContextFromRun } from '@agentbench/core'

// 从 SDK 输出手动构建上下文
const context = {
  output: result.content,
  toolCalls: result.tool_calls?.map(tc => ({
    name: tc.function.name,
    arguments: JSON.parse(tc.function.arguments),
  })) ?? [],
  metrics: {
    totalTokens: result.usage.total_tokens,
    totalCost: result.cost,
    totalLatency: result.duration,
    stepCount: 1,
    llmCallCount: 1,
    toolCallCount: result.tool_calls?.length ?? 0,
  },
  scores: [],
  status: 'passed',
}

const assertionResult = expect(context)
  .tool('search_docs').toBeCalled()
  .tokens().toBeLessThan(4096)
  .output().toContain('退款')
  .run()

console.log(assertionResult.allPassed) // true/false

---

→ 返回文档中心