feat(sdk): support local OpenAI-compatible providers via providerOptions (closes #678)

[vraj00222]

Adds the ability to route an agent's LLM calls directly to a local or self-hosted OpenAI-compatible endpoint (Ollama, LM Studio, etc.) instead of the Codebuff backend. Resolves #678.

Followed @jahooma's review guidance from the issue thread:

• ✅ Reuses the existing providerOptions field (no new localProvider)
• ✅ Env var renamed: CODEBUFF_BASE_URL (dropped "LOCAL" — baseUrl / apiKey are also useful for non-local cloud endpoints)
• ✅ CodebuffClient constructor accepts providerBaseUrl / providerApiKey

Zero new runtime dependencies. Fully backward-compatible.

---

How a user uses it

Simplest — env var (every agent goes local)

export CODEBUFF_BASE_URL=http://localhost:11434/v1
codebuff

Inside the CLI — new /local slash command

/local on llama3.1:8b              # enable; sets URL default + model override
/local on http://x:1234/v1 llama-7b   # custom URL + model
/local list                        # query Ollama for available models
/local model qwen2.5-coder:7b      # swap models mid-session
/local status                      # show current config
/local off                         # disable, back to Codebuff cloud

Per-agent (the cost-split pattern — see "Recommended usage" below)

// .agents/my-local-file-picker.ts
const definition: AgentDefinition = {
  id: 'my-local-file-picker',
  model: 'llama3.1:8b',
  providerOptions: {
    baseUrl: 'http://localhost:11434/v1',
    apiKey: 'ollama',
  },
  // ...
}

SDK consumers

new CodebuffClient({
  apiKey: 'cb-...',
  providerBaseUrl: 'http://localhost:11434/v1',
})

Precedence

Per LLM call: agent's providerOptions.baseUrl > CodebuffClient.providerBaseUrl > CODEBUFF_BASE_URL env var > Codebuff backend (unchanged default). apiKey resolves on the same ladder, paired with whichever URL wins.

---

Recommended usage (today)

Cost-split pattern — exactly what #678 called out:

"cheap tasks (file picking) locally, complex reasoning on cloud"

Keep the default orchestrator agents on Codebuff cloud (best at planning, tool routing, complex reasoning). Set providerOptions.baseUrl per-agent for cheap sub-agents like file-picker and code-searcher. The orchestrator stays smart; the cheap tasks become free. The PR enables this exactly.

What's NOT recommended yet: replacing the default orchestrator with a small local model end-to-end. Codebuff's default agents have ~5000-token system prompts (with extensive tool-calling instructions, sub-agent definitions, mode hints, etc.) designed for cloud-class models. Small local models (≤8B) struggle to follow that scaffolding cleanly and often produce confusing output ("Please provide the task you would like me to perform") or empty turns. This is not a bug in the plumbing — the request reaches the local model correctly; the model just can't drive the orchestrator role.

A follow-up PR will add a compactPrompts flag on AgentDefinition plus /mode <custom-agent> so small-model end-to-end workflows work cleanly.

---

Demo

---

When it fails (Ollama down, wrong URL, model not pulled)

Users get an actionable error, not raw fetch failed:

Cannot reach LLM provider at http://localhost:11434/v1.

Check:
  • Is the provider running? (e.g. `ollama serve` or LM Studio's Local Server)
  • Is the URL correct? Currently configured: http://localhost:11434/v1
  • Is the model 'llama3.1:8b' loaded? (e.g. `ollama list`)

Error patterns cover both Node-style (ECONNREFUSED, ENOTFOUND, ETIMEDOUT) and Bun-style (ConnectionRefused, "Unable to connect...") variants — caught during local testing against Bun's fetch implementation.

No silent fallback to the Codebuff backend if the local endpoint is unreachable. Falling back would leak prompts the user wanted to keep local and incur billing they expected to avoid — violates user intent.

---

Implementation notes

Mirrors the existing ChatGPT-OAuth-direct pattern in sdk/src/impl/model-provider.ts — adds a third branch to getModelForRequest() parallel to ChatGPT-OAuth and Codebuff-backend defaults. Same OpenAICompatibleChatLanguageModel primitive, zero new dependencies.

Key code paths:

• Type surface — three mirrored AgentDefinition files (.agents/, agents/, common/src/templates/initial-agents-dir/) get optional baseUrl / apiKey on providerOptions. Backend OpenRouterProviderRoutingOptions extended likewise. Zod schema validates baseUrl as a URL.
• Dispatch — getModelForRequest() returns the new branch when customProvider.baseUrl is set. promptAiSdkStream() resolves precedence and skips Codebuff metadata / OpenRouter routing keys on direct-route requests (same as ChatGPT-OAuth path).
• Model override — CODEBUFF_LOCAL_MODEL env var (set by /local on <model>) substitutes the agent's declared cloud model id with a local one, so the request body sent to Ollama contains the right model name. Override is logged at INFO for transparency.
• maxRetries: 1 on direct routes — one retry handles brief model-load stalls without long waits when the provider is genuinely down.
• Lazy env reading — getCustomProviderBaseUrlFromEnv() reads process.env on every request, so /local on//local off mid-session takes effect immediately without rebuilding the cached CodebuffClient.

---

System notes discovered during integration

A few things worth documenting for future contributors:

1. The agent runtime auto-injects scaffolding around every agent's prompts (run-agent-step.ts:232-253): STEP_PROMPT user messages every iteration, INSTRUCTIONS_PROMPT user message once, additional system prompts (file tree, mode hints, codebuff meta-info). Even minimal custom agents inherit this scaffolding — which is fine for cloud models but is the root cause of small-model confusion. Addressed in the follow-up PR.

2. Orchestrator runs many LLM calls per turn. A single user prompt in DEFAULT mode triggers the orchestrator + sub-agents (file pickers, thinkers, editors, code-reviewers). Each makes its own LLM call. On cloud this is fast; on local 8B models it's 30-90 seconds per turn.

3. Even with local inference, the agent runtime still hits the Codebuff backend for run setup (/api/v1/agent-runs) and token counting (/api/v1/token-count). Only inference goes local. True air-gapped operation is a larger scope and is out of this PR (documented as known limitation).

4. OpenAICompatibleChatLanguageModel already exists in the codebase — it's the same primitive used by the ChatGPT-OAuth-direct path. No new SDK was needed.

---

Verification

Automated

• 6 unit tests in sdk/src/impl/__tests__/model-provider-custom.test.ts covering the custom-provider branch, trailing-slash tolerance, precedence over ChatGPT-OAuth eligibility, missing-apiKey defaulting.
• 40 unit tests in cli/src/commands/__tests__/local-provider.test.ts covering /local parser shapes, URL/model validation, idempotent disable, full toggle cycle.
• All existing tests pass: SDK 454/454, agent-runtime 426/426, CLI 2354/2354. All workspace typechecks clean.

Live (against Ollama 0.20.5)

Verified end-to-end during development:

• ✅ customProvider routes to local; llama3.1:8b streams a real response.
• ✅ Trailing slashes on baseUrl tolerated.
• ✅ Unreachable endpoint produces the wrapped error with hostname + model.
• ✅ CODEBUFF_BASE_URL env-var fallback works.
• ✅ CODEBUFF_LOCAL_MODEL override substitutes the cloud model id in the outbound request body (verified by sending 'anthropic/claude-opus-4-7' in params + env override → Ollama responded with 'llama3.1:8b' content).

---

Backward compatibility

Fully backward-compatible. All new fields and the env vars are optional. Existing agents with providerOptions: { order: [...] } (OpenRouter routing config) continue to work unchanged. The Codebuff-backend default path is untouched when no custom provider is configured. No breaking changes to the SDK API.

---

Known limitations (intentional — addressed in follow-ups)

These were considered and explicitly left out to keep this PR focused on the plumbing the maintainer signed off on:

• compactPrompts flag on AgentDefinition — for opting an agent out of Codebuff's auto-injected scaffolding (STEP_PROMPT, INSTRUCTIONS_PROMPT, additional system prompts). Makes small-model agents actually usable. (Follow-up PR)
• /mode <custom-agent> — letting users set a custom agent as the active orchestrator. (Follow-up PR)
• Auto-detection of local providers on startup (probing :11434, :1234). Adds UI surface and security considerations — separate concern.
• True air-gapped mode — agent runtime still hits Codebuff backend for run setup and token counting. Bigger scope.
• Model capability awareness — refuse / warn when pairing tiny models with heavy agents.
• fallbackToCodebuff opt-in — for users who explicitly want fallback on local failure.

---

Files changed

17 files, +1221 / −8.

TYPES (mirrored agent-facing × 3)
  .agents/types/agent-definition.ts
  agents/types/agent-definition.ts
  common/src/templates/initial-agents-dir/types/agent-definition.ts

TYPES (backend + contract + schema)
  common/src/types/agent-template.ts
  common/src/types/contracts/llm.ts
  common/src/types/dynamic-agent-template.ts

CONSTANTS
  common/src/constants/custom-provider.ts             (new)

SDK
  sdk/src/env.ts
  sdk/src/run.ts
  sdk/src/impl/llm.ts
  sdk/src/impl/model-provider.ts
  sdk/src/impl/agent-runtime.ts
  sdk/src/impl/__tests__/model-provider-custom.test.ts   (new)

CLI (the /local command)
  cli/src/commands/local-provider.ts                  (new)
  cli/src/commands/command-registry.ts
  cli/src/data/slash-commands.ts
  cli/src/commands/__tests__/local-provider.test.ts   (new)