Skip to content

Custom OpenAI Endpoint

Jump to bottom
rookiemann edited this page Apr 23, 2026 · 1 revision

Custom OpenAI-Compatible Endpoints

Use this guide if your AI model is served by something other than OpenRouter or LM Studio — for example vLLM, Ollama, llama.cpp, TGI, a corporate OpenAI proxy, or any private server that speaks the OpenAI Chat Completions API.

You'll need three things from the server's owner (or your own setup):

Field Example
Base URL http://10.170.33.43:6397/v1
API Key xxx (or any string if the server doesn't check)
Model name Qwen3.5-35B-A3B-FP8 (must match what the server reports)

The base URL must end in /v1 for OpenAI SDK compatibility. If your server doesn't require auth, use any non-empty string for the key — Hermes will substitute no-key-required automatically for local URLs.

Option A: Windows App (GUI)

The API Key Setup dialog only exposes API keys, not base URLs — so for a custom endpoint, use the LM Studio panel as a generic endpoint switcher:

  1. Open Tools → LM Studio
  2. In the Endpoint field at the top, replace http://localhost:1234 with your server, e.g. http://10.170.33.43:6397
  3. Wait for the green status dot — your model (e.g. Qwen3.5-35B-A3B-FP8) will appear in the model list
  4. Select the model and click Use for Chat
  5. Status bar at the bottom should now show the model name

The panel works with any server that responds to GET /v1/models — vLLM, Ollama (use http://localhost:11434), llama.cpp server, TGI, etc.

Option B: CLI

Pick one of the following — config file or environment variables.

Config file (recommended — persists across runs)

Edit ~/.hermes/config.yaml (or cli-config.yaml in the repo root):

model:
  default: "Qwen3.5-35B-A3B-FP8"
  provider: "custom"
  base_url: "http://10.170.33.43:6397/v1"
  api_key: "xxx"

provider: "custom" accepts aliases — lmstudio, ollama, vllm, llamacpp all map to the same code path.

Environment variables (.env)

In ~/.hermes/.env:

OPENAI_BASE_URL=http://10.170.33.43:6397/v1
OPENAI_API_KEY=xxx
LLM_MODEL=Qwen3.5-35B-A3B-FP8
HERMES_INFERENCE_PROVIDER=custom

Don't use OPENROUTER_API_KEY for custom endpoints — Hermes deliberately keeps that key out of non-OpenRouter requests so it doesn't leak to third-party servers.

Multiple custom endpoints

If you switch between two or more servers, define them as named providers in ~/.hermes/config.yaml:

custom_providers:
  - name: "qwen-server"
    base_url: "http://10.170.33.43:6397/v1"
    api_key: "xxx"
  - name: "local-vllm"
    base_url: "http://localhost:8000/v1"
    api_key: "no-key-required"

Then select one with --provider custom:qwen-server or set provider: "custom:qwen-server" in the model: section.

Troubleshooting

"Connection refused" / "Cannot reach endpoint":

  • Verify the URL in a browser or with curl http://YOUR_HOST:PORT/v1/models — you should get a JSON list of models
  • Check firewall rules between your machine and the server
  • For remote servers, make sure the host is bound to 0.0.0.0 (not 127.0.0.1) and the port is open

"Model not found":

  • The model name must match exactly what /v1/models returns — case-sensitive
  • Some servers return Qwen/Qwen3.5-35B-A3B-FP8 (with publisher prefix); others return just Qwen3.5-35B-A3B-FP8

"401 Unauthorized" or "Invalid API key":

  • Some servers don't validate the key but still require the header — set any non-empty string
  • If the server uses a real key (e.g. corporate gateway), make sure you're not pulling from OPENROUTER_API_KEY — Hermes prefers OPENAI_API_KEY for non-OpenRouter URLs

GUI shows "OpenRouter" in the status bar even after editing .env:

  • The cloud-mode path in the GUI ignores OPENAI_BASE_URL to avoid confusion. Use the LM Studio panel (Option A) or the CLI for custom endpoints.

See also

Clone this wiki locally