Custom OpenAI-Compatible Provider Endpoints (BYO Endpoint for Local & Remote LLMs)

[tbitcs]

Pre-submit Checks

• I have searched Warp feature requests and there are no duplicates
• I have searched Warp docs and my feature is not there

Describe the solution you'd like?

Add support for configuring custom OpenAI-compatible API endpoints, allowing Warp to connect to local or self-hosted LLM providers.

This would enable users to specify:

• A base URL (e.g. http://localhost:8000/v1)
• A model name
• An optional API key

The configured endpoint would be used for Warp AI features such as command suggestions, agent interactions, and inline completions.

The implementation should support standard OpenAI-compatible APIs, including chat completions and streaming responses, so that it works with tools like vLLM, Ollama (via compatibility layer), and other OpenAI-compatible servers.

From a user perspective, this would extend the existing “Bring Your Own Key” model into a “Bring Your Own Endpoint” workflow, where users can choose between built-in providers and a custom endpoint.

Is your feature request related to a problem? Please describe.

Yes.

Currently, Warp AI only supports a fixed set of providers (e.g. OpenAI, Anthropic, Google) via API keys. There is no way to connect Warp to local or self-hosted models.

This creates several limitations:

• Users cannot run Warp AI in privacy-sensitive or offline environments
• Teams with existing LLM infrastructure cannot integrate it
• Developers cannot use open-source models running locally
• All prompts must be sent to external providers, even when a local option is available

Many modern LLM tools (such as vLLM and Ollama) expose OpenAI-compatible APIs, but Warp cannot currently take advantage of this without native support for custom endpoints.

Additional context

This request consolidates several related feature requests:

• Suggestion for Adding Support for Custom AI Model Configuration #6026 – Custom AI model configuration
• Custom AI API (Eg. Azure Open AI) or LiteLLM #8708 – Custom AI API endpoints
• Make Warp work with Local Language Models (like Ollama models) #4339 – Local LLM support
• Local AI #5735 – Privacy / local AI usage
• Allow users to specify the LLM endpoint (APIs) for other models #4687 – Custom endpoint (closed as duplicate)
• Using LLMs other than OpenAI for AI command search #3779 – Provider routing limitations

The proposed approach is to extend Warp’s existing provider model to allow any OpenAI-compatible endpoint, rather than introducing provider-specific integrations for each tool.

This would allow immediate compatibility with:

• vLLM (OpenAI-compatible server)
• Ollama (local runtime with compatibility layer)
• LiteLLM and similar proxy systems

Example configuration:

{
"provider": "custom",
"base_url": "http://localhost:8000/v1",
"model": "meta-llama/Llama-3-8b-instruct",
"api_key": "optional"
}

This is a provider-level feature and is expected to be cross-platform across all Warp-supported operating systems.

Operating system (OS)

Windows

How important is this feature to you?

5 (Can't work without it!)

Warp Internal (ignore) - linear-label:39cc6478-1249-4ee7-950b-c428edfeecd1

None

[warp-for-oss]

@Manoj7ar

Reviewed the thread and the current client code before further changes. I’d keep this grouped with the existing custom/local LLM endpoint cluster rather than treating it as ready for implementation; the useful distinction is provider abstraction parity for the full agent loop, but maintainers need to choose the architecture boundary first.

Code inspection confirms there is no generic OpenAI-compatible base_url / custom-provider path in the client today, and the full agent loop is currently mediated by Warp’s multi-agent backend: the client sends model IDs, BYO keys, context, and supported tool capabilities, then executes tool calls streamed back from the server.

Suggested maintainers to tag for review: @zachbai for Agent Mode/tool-loop behavior, @ianhodge for server API / GraphQL / provider-contract implications, and @lucieleblanc if this moves toward AI settings or BYOK UI changes.

Reasoning

Response mode selected because this run was triggered by an @oz-agent mention on an already-triaged issue with duplicate, triaged, repro:high, area:agent, and area:settings-keybindings labels. Per the workflow, this should answer the specific follow-up and not alter labels.

Duplicate / related-issue assessment: #9303 materially overlaps the existing custom/local LLM endpoint cluster. The strongest overlaps found via gh issue view are #9368 (open, same generic OpenAI-compatible base_url request for Oz/Warp UI), #4687 (open, specify LLM endpoint/APIs), #4339 (open, local LLM providers such as Ollama), and #6026 (open, custom AI model configuration). Closed/redirected related requests include #8708, #7182, #5735, and #3779. The newer thread’s stronger framing is provider abstraction parity / full agent-loop compatibility, but that is a product/architecture clarification over the same capability area rather than a separate implemented bug.

Code findings:

• app/src/ai/llms.rs:88 defines LLMProvider as OpenAI, Anthropic, Google, Xai, or Unknown; no generic/custom provider variant is present.
• app/src/ai/llms.rs:107 defines LLMModelHost as DirectApi, AwsBedrock, or Unknown; no arbitrary endpoint host exists.
• app/src/server/server_api/ai.rs:2138 maps GraphQL providers into those fixed client provider variants.
• crates/ai/src/api_keys.rs:23 stores BYO keys for named providers (google, anthropic, openai, open_router) and does not store custom base URLs or model endpoint config.
• app/src/settings_view/ai_page.rs:6456 renders API key inputs for OpenAI, Anthropic, and Google only.
• app/src/ai/agent/api.rs:222 builds the agent request using ApiKeyManager::api_keys_for_request(...); the request params carry API keys, model IDs, context, and permissions rather than a custom endpoint contract.
• app/src/ai/agent/api/impl.rs:45 sends a warp_multi_agent_api::Request to server_api.generate_multi_agent_output; app/src/ai/agent/api/impl.rs:67 includes model config, BYO keys, supported tools, autonomy, and orchestration flags for the backend.
• app/src/ai/agent/api/convert_from.rs:621 maps backend ToolCall messages into client-executed actions such as shell commands, file reads, diffs, MCP calls, subagents, and questions. There is no observed ExecuteLLMInference-style delegated inference action in the client-side conversion path.

Maintainer routing from .github/STAKEHOLDERS: app/src/ai/agent/ maps to @zachbai; app/src/server/server_api/ and crates/graphql/ map to @ianhodge; app/src/settings_view/ maps to @lucieleblanc. If this becomes a spec, the review should likely include both Agent/backend API ownership and Settings/BYOK UI ownership.

This is my automated analysis and may be incorrect. A maintainer will verify the details.

Powered by Oz

[tbitcs]

This is intended to go beyond the existing requests by focusing on how OpenAI-compatible endpoints integrate with Warp’s full agent system, not just basic chat completion.

From the discussion here, it looks like current approaches (e.g. local adapters) can return text responses but do not support:

• Warp tool calls
• command execution loops
• structured agent workflows

The goal of this proposal is to define a provider abstraction that allows OpenAI-compatible endpoints to participate in the full agentic loop, rather than acting as a limited chat backend.

This would unify:

• custom endpoints (Allow users to specify the LLM endpoint (APIs) for other models #4687)
• local models (Make Warp work with Local Language Models (like Ollama models) #4339)
• custom APIs (Custom AI API (Eg. Azure Open AI) or LiteLLM #8708)

into a single capability: first-class provider support for OpenAI-compatible backends within Warp’s agent architecture.

Happy to reflect this explicitly in the spec if this direction makes sense.

[shivaylamba]

I have implemented a similar design to add support for Nebius AI Token Factory models

https://github.com/shivaylamba/warp/tree/nebius-ai-provider

[shivaylamba]

@tbitcs
The current Nebius support is a local OpenAI-compatible chat adapter, not a full Warp model backend.

Main limitations:

• It can answer text in Agent, but it cannot call Warp tools yet.
• It does not run the full terminal automation loop: command execution, file edits, structured
  tool calls, approvals, etc.

Need to get to support the full agentic terminal loop next -

Do you know if OpenAI-compatible endpoints would automatically get support for the full agentic experience or do we need make it manually?

[Aeromix]

I already forked Warp and adding LM Studio local model support, but I am vibecoding it, and cannot imagine it becoming official code

[tbitcs]

This is intended to go beyond the existing requests by focusing on how OpenAI-compatible endpoints integrate with Warp’s full agent system, not just basic chat completion.

From the discussion here, it looks like current approaches (e.g. local adapters) can return text responses but do not support:

• Warp tool calls
• command execution loops
• structured agent workflows

The goal of this proposal is to define a provider abstraction that allows OpenAI-compatible endpoints to participate in the full agentic loop, rather than acting as a limited chat backend.

This would unify:

• custom endpoints (Allow users to specify the LLM endpoint (APIs) for other models #4687)
• local models (Make Warp work with Local Language Models (like Ollama models) #4339)
• custom APIs (Custom AI API (Eg. Azure Open AI) or LiteLLM #8708)

into a single capability: first-class provider support for OpenAI-compatible backends within Warp’s agent architecture

Happy to reflect this explicitly in the spec if this direction makes sense.

[Aeromix]

I am wiring up full Warp use, as that's it's value proposition, not to just use it as a chat terminal. I will report if it works