llm-proxy

An LLM gateway that fronts Claude Code (Anthropic API) and OpenAI-compatible clients, routing each request to Anthropic, to Google Gemini, or to local and remote OpenAI-compatible backends (LM Studio, llama.cpp, vLLM, SGLang, Ollama) via a configurable rules engine.

Why it exists: running a coding agent against a mix of hosted and self-hosted models means every client needs to know which model lives where, speak the right wire protocol, and hold the right credential. This proxy makes that one endpoint — clients point at it, rules decide where each request goes, and translation between the OpenAI and Anthropic formats happens in the middle.

Related tools

asi is a companion CLI built to work with this proxy. Its asi gateway command wires supported AI coding tools (Claude Code, Codex, Antigravity, GitHub Copilot) to a gateway endpoint in one step — flipping the right environment variables and settings per tool, and installing Claude Code's model-cache SessionStart hook. Its default gateway URL is http://127.0.0.1:1235, this proxy's default port.

asi gateway on  --targets claude      # route Claude Code through the proxy
asi gateway status                    # inspect current wiring
asi gateway off --targets claude      # revert

Neither tool requires the other: asi targets any gateway that speaks these APIs, and every wiring step it performs is documented here as a manual alternative. asi also does a separate, unrelated job — rendering one canonical catalog of AI agents and skills into each tool's native format.

Integrating a client? Start at docs/consumer/index.md — getting started per client family (Claude Code / OpenAI / Codex) and the endpoint reference with auth semantics and API divergences. This README stays the maintainer-depth reference.

The repository is built entirely on ASP.NET Core (.NET 10) in dotnet/src/LLMProxy/.

Source layout (dotnet/src/LLMProxy/): Program.cs is the entry point; GatewayApp.cs implements the main pipeline and router; Routing/ contains the prefix-driven routing engine; Translation/ maps requests, responses, and streams between OpenAI and Anthropic formats; Configuration/ manages config.json and env parsing; Filtering/ implements prompt rules; and Grants/ handles auth credential refreshes.

Quick start

Requires the .NET 10 SDK (pinned in global.json), or Docker.

cp .env.example .env                        # every value is optional

dotnet run --project dotnet/src/LLMProxy    # runs on port 1235
# or
docker compose up -d --build                # builds + runs on port 1235

curl -fsS http://localhost:1235/admin/health

The shipped config.json ends in a deny-unmatched rule, so a request that no rule matches is rejected rather than silently forwarded to a paid upstream. That is deliberate — edit the rules (or set routing.default) before expecting traffic to flow. See Configuration and docs/examples/.

Point a client at it:

# Claude Code
ANTHROPIC_BASE_URL=http://127.0.0.1:1235 claude

# any OpenAI-compatible client
export OPENAI_BASE_URL=http://127.0.0.1:1235/openai/v1

Tests:

dotnet test dotnet/LLMProxy.slnx -c Release

Antigravity passthrough

Antigravity CLI can point ANTIGRAVITY_BASE_URL and CLOUD_CODE_URL at this proxy. Every /v1internal:* request is forwarded unchanged to https://cloudcode-pa.googleapis.com by default. Caller-owned OAuth bearer tokens and Google API keys are preserved; configured machine-profile keys are stripped before egress.

The passthrough does not parse or rewrite payloads, refresh credentials, retry, fall back to another provider, or translate responses. Override the server-side upstream with ANTIGRAVITY_UPSTREAM_BASE_URL; do not set it to the client-facing proxy URL.

To wire a client, point both endpoint variables at this proxy:

export ANTIGRAVITY_BASE_URL=http://127.0.0.1:1235
export CLOUD_CODE_URL=http://127.0.0.1:1235

Or, with asi, which sets both persistently and can roll them back:

asi gateway on --targets antigravity

On Windows these are user environment variables, so start the client from a new terminal after setting them. To roll back, remove those two overrides only — leave Google OAuth tokens and API keys alone.

Features

• Rules-based routing — match on model id, requested role, or an inferred task type, with ordered fallback across backends and a fail-closed deny-unmatched default.
• OpenAI ⇄ Anthropic translation, streaming included, so an OpenAI-compatible client can talk to Anthropic and vice versa.
• Google Gemini support, both native and through the Anthropic/OpenAI translation layers, plus an Antigravity straight-pipe passthrough.
• Machine profiles restrict backend visibility and dispatch to clients that present a profile key — enforced at discovery and dispatch.
• Stateful egress grants on /grants authorize short-lived, scoped, single-use upstream materialization for write-class operations.
• Prompt filtering strips known high-noise context blocks and caps oversized system/message content before a small-context model sees them.
• Effort and sampling-parameter normalization so a client that always sends effort: "high" does not 400 against a model whose template rejects it.
• Observability: GET /admin/health, host-stamped structured access logs with request/failover/token/grant fields and a per-request routing_decision_id, System.Diagnostics.Metrics, and OTLP export when OTEL_EXPORTER_OTLP_ENDPOINT is set.
• Claude Code model-picker integration via a SessionStart hook that mirrors GET /admin/model-cache verbatim — see "Consumers" below.

Endpoints

Path	Shape	Notes
GET /v1/models	Anthropic	Claude Code gateway model discovery
POST /v1/messages	Anthropic	routed; streaming supported
POST /v1/count_tokens, POST /v1/messages/count_tokens	Anthropic	local counted on-proxy; Claude forwarded to Anthropic
GET /openai/v1/models	OpenAI	backends only (incl. embeddings)
POST /openai/v1/chat/completions	OpenAI	routed; streaming supported. Alias: /v1/chat/completions
POST /v1/responses, POST /openai/v1/responses	OpenAI Responses	local backends only; translates Responses → Chat Completions
GET /codex/v1/models	Codex/OpenAI	unified Codex provider model discovery; local backends always, OpenAI models when caller bearer auth is present
POST /codex/v1/responses	Codex/OpenAI Responses	backend-prefixed models route locally; non-backend model IDs pass through to OpenAI with caller bearer auth
POST /openai/v1/embeddings	OpenAI	Alias: /v1/embeddings
POST /openai/v1/completions	OpenAI	Alias: /v1/completions
POST /openai/v1/audio/transcriptions, /v1/audio/transcriptions	OpenAI	Audio transcription; requires an audio-capable local backend (e.g. whisper); 400 if no matching backend visible to this machine
POST /openai/v1/audio/translations, /v1/audio/translations	OpenAI	Audio translation; same backend requirement
GET /admin/model-cache	cache	Authoritative SessionStart cache document (unauthenticated)
GET /admin/backends	—	List base + dynamic backends (requires ADMIN_KEY)
POST /admin/backends	—	Register or replace a dynamic backend at runtime
DELETE /admin/backends/:name	—	Remove a dynamic backend
GET /admin/config	—	Non-secret runtime config snapshot
GET /grants, POST /grants, GET /grants/:id, POST /grants/:id/activate, POST /grants/:id/revoke	grants	.NET runtime only; bearer-auth gated by LLMP_GRANTS_BEARER_TOKEN
anything else	—	passed through to Anthropic

Headers

Header	Description
X-Task-Type	Optional hint for task-based routing (e.g. coding, summarization)
X-Model-Role	Optional logical role (e.g. fast, summarize, rerank, classify)
X-Admin-Key	Required for all /admin/* requests when ADMIN_KEY is configured
X-LLMP-Grant	Optional active grant id for scoped .NET egress authorization
X-Routing-Decision-Id	Response + upstream header. The proxy mints a UUIDv7 routing decision id per inbound request; any client-supplied value is ignored

Routing decision id

Every inbound request is stamped at ingress with a freshly minted UUIDv7 routing_decision_id. That single id is:

• returned on every response as the x-routing-decision-id header — for streamed (SSE) responses it is written before the first chunk (a header, never a trailer);
• forwarded upstream as x-routing-decision-id (alongside x-correlation-id / x-session-id) so downstream services log the same id;
• stamped as the routing_decision_id field on every request-scoped structured log event (the access event, and the advisor-escalation log).

It correlates all telemetry for one request across the proxy and its upstreams. Because the proxy owns the routing decision, a client-supplied x-routing-decision-id is always overwritten.

The access log row also reserves three nullable fields — escalated, advisor_confidence, threshold — for advisor-escalation adoption. They follow the row's WhenWritingNull convention: absent today, emitted once populated, with no shape change required of consumers.

OpenAI endpoints are namespaced under /openai/v1/* so they don't collide with the Anthropic-shaped GET /v1/models Claude Code depends on.

Per-machine availability: which endpoints are effectively available depends on which backends are reachable from a given machine. Backends with allowedProfiles are only discoverable and routable from clients that authenticate with a matching machine profile key. Audio, Responses, and openaiCompat endpoints silently return 400/502 when no matching backend is visible. Additionally, specific endpoint categories can be completely toggled off globally via config.endpoints.

Configuration (config.json)

{
  "port": 1235,
  "maxBodyBytes": 26214400,        // 413 above this (Claude Code prompts get big)
  "upstreamTimeoutMs": 120000,     // abort a stalled backend; 504 or fail over
  "modelDiscoveryTimeoutMs": 3000, // bound each GET /v1/models backend fetch
  "anthropic": { "baseUrl": "https://api.anthropic.com", "apiKey": "" },
  "machineProfiles": {
    "restricted": { "clientKey": "${RESTRICTED_PROFILE_KEY}" }
  },
  "backends": [
    { "name": "local",  "baseUrl": "http://host.docker.internal:1234", "apiKey": "${LOCAL_BACKEND_KEY}", "modelsPath": "/api/v1/models" },
    { "name": "remote", "baseUrl": "http://inference.example.internal:1234", "apiKey": "${REMOTE_BACKEND_KEY}", "enabled": "${REMOTE_BACKEND_ENABLED}" },
    { "name": "vendor", "baseUrl": "https://api.example.com/anthropic", "apiKey": "${VENDOR_API_KEY}", "modelsPath": "/v1/models", "allowedProfiles": ["restricted"] }
  ],
  "routing": {
    "translateOpenAIToAnthropic": false,
    "autoClassify": { "enabled": true, "scanWindowChars": 16384 },
    "rules": [
      { "name": "role-fast",           "match": { "role": "fast" },           "target": "local",   "model": "a-small-local-model" },
      { "name": "coding-to-claude",    "match": { "taskType": "coding" },     "target": "anthropic",  "model": "claude-sonnet-5" },
      { "name": "summarization-local", "match": { "taskType": "summarization" }, "target": "local", "fallback": ["remote", "anthropic"], "anthropicModel": "claude-haiku-4-5-20251001" }
    ],
    "default": null
  },
  "effort": {
    "capableModels": ["claude-opus-4-8*", "claude-sonnet-4-6*", "claude-sonnet-5*", "claude-fable-5*"]
  },
  "samplingParams": {
    "restrictedModels": ["claude-sonnet-4-6*", "claude-sonnet-5*"],
    "overrides": [
      { "pattern": "qwen3.6-27b*", "temperature": 0.6, "topP": 0.95,
        "topK": 20, "minP": 0.0, "presencePenalty": 0.0, "repetitionPenalty": 1.0 },
      { "pattern": "qwen3.6-35b*", "temperature": 0.6, "topP": 0.95,
        "topK": 20, "minP": 0.0, "presencePenalty": 0.0, "repetitionPenalty": 1.0 }
    ]
  },
  "modelCache": {
    "baseUrl": "http://127.0.0.1:1235"  // emitted verbatim in GET /admin/model-cache; see below
  },
  "endpoints": {
    "audio": true,
    "responses": true
  }
}

${VAR} placeholders resolve from the environment (see .env.example). The shipped config.json is a deliberately minimal starting point; richer worked examples live in docs/examples/.

Config-only backend routing

A new backend never requires a proxy source change. Copy one of the example configs, edit it, and point the proxy at it:

CONFIG_PATH=./docs/examples/config.multi-backend.json dotnet run --project dotnet/src/LLMProxy

Example	What it shows
config.multi-backend.json	Several backends, role-based rules, failover, an anthropicNative upstream
config.sampling-overrides.json	Pinning per-model decode parameters clients do not send
config.prompt-filtering.json	Trimming oversized prompts and tool lists for small-context models

modelDiscoveryTimeoutMs (env MODEL_DISCOVERY_TIMEOUT_MS, default 3000) bounds each backend fetch behind GET /v1/models. Model discovery already tolerates a per-backend failure (a dead backend degrades to no models), but without this bound an enabled-but-dead backend that accepts the socket and never replies would stall the discovery Promise.all on the OS TCP timeout (~35s). With it, /v1/models returns promptly with the reachable backends' models.

Model-cache baseUrl (config.modelCache.baseUrl / MODEL_CACHE_BASE_URL)

GET /admin/model-cache (see Endpoints) stamps a baseUrl field into the cache document that Claude Code's SessionStart hook writes verbatim to ~/.claude/cache/gateway-models.json, then exact-matches against ANTHROPIC_BASE_URL. Resolution order: MODEL_CACHE_BASE_URL env var → config.modelCache.baseUrl → http://127.0.0.1:{port} (default). Set this explicitly whenever the proxy's advertised host differs from 127.0.0.1 — e.g. a remote/lightweight deployment where clients reach it by hostname — so it matches whatever ANTHROPIC_BASE_URL those clients configure. A mismatch here makes Claude Code reject the entire cache and fall back to Anthropic.

Backend fields

Field	Type	Default	Description
name	string	required	Unique id; used in {name}-{model} proxy ids (the LM Studio local backend is conventionally named lm, giving lm-{model} ids in the Claude Code picker)
baseUrl	string	required	Upstream base URL
apiKey	string	""	Injected as x-api-key and Authorization: Bearer; supports ${VAR}
modelsPath	string	/api/v1/models	Path appended to baseUrl for model discovery
enabled	boolean or string	true	${VAR} → env; "1"/"true"/"yes"/"on" → enabled
openaiCompat	boolean	false	Backend speaks OpenAI Chat Completions; proxy translates Anthropic→OpenAI in, OpenAI→Anthropic out (see openaiCompat backends)
openaiCompatModels	string[]	null	*-glob model ids that use the OpenAI Chat Completions bridge on this backend. Use this instead of openaiCompat when only selected models need translation.
supportsStreaming	boolean	true	openaiCompat only. Set false when the backend rejects stream:true/stream_options or returns a non-SSE body. The proxy then requests a buffered completion upstream and re-streams the translated message back to the client, so a streaming client still gets a well-formed Anthropic SSE response.
allowedProfiles	string[]	null	Restrict this backend to these machine profile names — enforced on discovery and dispatch (rules, {backend}-{id} prefixed ids, /v1/responses, audio). See Machine profiles.
thinkingMode	string	"intensity"	Normalization mode for thinking/reasoning fields on assistant turns ("binary" or "intensity"). "binary" maps truthy values to "on" and falsy values to "off".
requireApiKey	boolean	false	Refuse to start when the backend is enabled but apiKey resolves empty, instead of substituting the "lm-studio" placeholder. Set on paid third-party backends so a missing env var is a loud boot error, not a silent 401 upstream.
anthropicNative	boolean	false	Backend speaks the Anthropic Messages API natively (e.g. a vendor's /apps/anthropic). Skips the local-backend system-message hoist so mid-array system messages and cache_control blocks reach the upstream intact.

Discovery reads each backend model's context window from loaded_instances[].config.context_length / max_context_length (LM Studio's shape) or max_model_len (the vLLM/SGLang convention), whichever is present. The value flows into context_length/max_input_tokens in the model cache document, which is what Claude Code sizes its auto-compaction from — a backend that advertises neither leaves Claude Code assuming its default window, overrunning the backend's real limit on long sessions instead of compacting.

Endpoint toggles (config.endpoints)

Optional block to enable or disable specific proxy endpoints globally:

• audio (boolean, default true): If false, audio translation/transcription endpoints return 404.
• responses (boolean, default true): If false, Responses → Chat Completions translation endpoints return 404.

Model class aliases (config.modelAliases)

A class alias lets a client request a model family — "model": "qwen" — and have the proxy pick the concrete model for it, so agents no longer pin exact ids that break when a different family member is loaded:

"modelAliases": {
  "qwen": ["local-gateway-qwen*", "lm-qwen/*"]   // glob patterns against the backend-prefixed proxy id
}

Resolution (per request, against the short-TTL model-list cache):

1. Loaded wins. For each pattern in order, the first currently loaded matching model is chosen. LM Studio reports loadedness via loaded_instances; backends that don't report it (SGLang/vLLM/llama.cpp) only list what they serve, so listed = loaded.
2. Pattern order is preference order when several class members are loaded.
3. Fallback to an unloaded match when nothing of the class is loaded (LM Studio JIT-loads it on first request).
4. 404 with an alias-specific message when nothing matches at all.

The resolved id then routes through the normal precedence below — alias resolution happens before step 1 and simply rewrites the model id. The alias key is matched case-insensitively. Aliases are advertised on every discovery surface (/v1/models, /openai/v1/models, /admin/model-cache) as synthetic entries whose display name shows the current target ([alias] qwen → local-gateway-qwen3.6-35b) and whose context limits are copied from it, so Claude Code sizes auto-compact correctly. The access log carries both names (alias + resolved_model fields).

Keep patterns tight: a *qwen* catch-all would also match unrelated finetunes whose ids merely contain "qwen" (e.g. an unrelated lm-somefinetune-qwen3…).

Routing rules

config.routing.rules are evaluated in order; the first whose match matches wins (an empty match is a catch-all). Precedence overall:

1. Explicit backend prefix — a model ID matching {backend}-{id} (or legacy {localPrefix}{backend}-{id}) routes straight to that backend. A backend with allowedProfiles is skipped here unless the request resolves to a listed profile.
2. Claude passthrough — native claude-* model IDs go straight to Anthropic when credentials exist (hasAuth: true).
3. routing.rules — first match wins (evaluated for non-Claude IDs, or Claude IDs when hasAuth is false).
4. Explicit role: alias — a model id of the form role:{name} that matched no rule returns 404.
5. No-auth local fallback — if a claude-* model ID is requested offline/no-auth and matched no rule, it falls back to the lm backend.
6. routing.default — fallback rule.
7. Unmatched non-Claude model — returns 404 (never falls through to Anthropic).

match conditions: model (exact or *-glob / alias), modelPrefix, role (X-Model-Role header or role: model prefix), taskType (X-Task-Type header or body metadata.task_type), minContextTokens / maxContextTokens, path, allowedProfiles (array of profile names — request must resolve to one of these machine profiles).

A rule routes to target, which can be:

• Backend name (string): "lm", "remote", etc. — looks up the backend from config.backends.
• Inline endpoint (object): { baseUrl, apiKey?, modelsPath?, name? } — routes directly to that endpoint without pre-declaring it in backends. apiKey supports ${VAR} placeholders and defaults to "lm-studio" when unset; modelsPath defaults to /api/v1/models.
• "anthropic" — routes to Anthropic (special value).

A rule optionally overrides the upstream model with model (and anthropicModel for the Anthropic attempt). fallback is an ordered list of further targets (backend names, inline endpoints, or "anthropic") tried on a transport failure (connection refused/reset or timeout) before any response byte is sent — HTTP 4xx/5xx are passed through, never retried. localFirst: true is shorthand for target lm + fallback anthropic. OpenAI-family requests drop anthropic targets unless routing.translateOpenAIToAnthropic is enabled (see below); with it off (the default) an all-anthropic chain on the OpenAI surface returns 502.

Auto-classification

routing.autoClassify derives a taskType for requests that carry neither an X-Task-Type header nor metadata.task_type, by keyword-matching the prompt. It recognizes three task types — classify, summarization, extraction — each with a built-in keyword list that keywords extends:

"autoClassify": {
  "enabled": true,
  "scanWindowChars": 16384,          // default; how much text is scanned
  "keywords": {
    "extraction": ["harvest the fields"]   // merged with the built-ins
  }
}

Keywords under a task name outside those three are ignored. Matching is case-insensitive and the first task with any keyword present wins, in the order above.

Only the tail of the request is scanned. scanWindowChars caps the scan at the last N chars of the conversation and, separately, the last N chars of the system / prompt / input text — a keyword buried a hundred turns back does not classify the request, because classification is meant to key off the current instruction. Any request whose text fits inside the window classifies exactly as a full scan would. Raising the window costs a proportional scan on every classified request; it is clamped to 256…1048576, and 0 or negative restores the default.

Machine profiles

Machine profiles are a code-level feature for restricting backend visibility to specific clients. A client identifies itself by sending its profile's clientKey as the x-api-key header; the gateway resolves the profile name, uses it for backend filtering, and strips the key before forwarding upstream.

"machineProfiles": {
  "work": { "clientKey": "${WORK_MACHINE_CLIENT_KEY}" }
}

Backends with "allowedProfiles": ["work"] are only visible to requests that resolve to the work profile — enforced at discovery (/v1/models, /openai/v1/models, /admin/model-cache) and dispatch (routing rules, {backend}-{id} prefixed ids, /v1/responses, audio). A caller who knows a prefixed id but lacks the profile gets a 404/deny, not a route. Routing rules can additionally gate on allowedProfiles as a match condition.

Using profiles with Claude Code under OAuth: set ANTHROPIC_CUSTOM_HEADERS="x-api-key: <clientKey>" in Claude Code's environment. The Authorization bearer stays OAuth (Anthropic passthrough is unaffected); the proxy resolves the profile from the extra x-api-key header and strips it before forwarding to Anthropic, exactly as it does for key-authenticated profile clients. The SessionStart model-cache refresh hook must send the same header when fetching /admin/model-cache, or the cached model list will omit profile-gated backends. (Setting ANTHROPIC_API_KEY on the client instead also works, but drops OAuth entirely — avoid it.)

Per-machine backend isolation can still be done with per-machine config.json — deploy a separate proxy per host and declare only the backends relevant to that machine (one proxy per machine) — but profile gating is the right tool when one machine needs a backend the others must never touch.

Anthropic-native backends

Some upstreams expose an Anthropic-compatible surface rather than an OpenAI-compatible one. Declaring such a backend with anthropicNative: true forwards request structure intact instead of translating it. A hardened example:

{
  "name": "vendor",
  "baseUrl": "https://api.example.com/apps/anthropic",
  "apiKey": "${VENDOR_TOKEN}",
  "requireApiKey": true,
  "enabled": "${VENDOR_ENABLED}",
  "anthropicNative": true,
  "allowedProfiles": ["restricted"],
  "modelsPath": "",
  "models": [
    { "id": "example-large", "contextLength": 1000000, "maxOutputTokens": 131072 }
  ]
}

The pieces compose deliberately:

• anthropicNative: true — mid-array system messages and cache_control blocks reach the upstream intact (they are what earn explicit-cache discounts). The proxy still stamps the model id, clamps max_tokens to the declared per-model limit, and swaps in the backend credential.
• modelsPath: "" + static models — for surfaces with no /v1/models endpoint. Declare ids and limits explicitly; keep them in sync with the provider's catalog.
• requireApiKey: true — enabling the backend without its token is a startup error rather than a silent placeholder credential sent upstream. Hosts without the token leave enabled falsy and boot cleanly.
• enabled: "${VENDOR_ENABLED}" — one config file can serve several hosts, only some of which have the credential.
• allowedProfiles — restrict discovery and dispatch to clients that present a machine-profile key. Useful when a provider's terms limit which tools may use the key.
• Exact-id routing rules, never globs — one rule per declared model, so a model id you have not vetted can never leave your network by accident.

Supply the credential through the environment; the proxy injects it when forwarding (BackendAuthHeaders). Do not set ANTHROPIC_AUTH_TOKEN on the client — it would replace Claude Code's OAuth bearer for real Anthropic traffic.

openaiCompat backends (Anthropic → OpenAI translation)

Backends with "openaiCompat": true, or models matching a backend's "openaiCompatModels", speak OpenAI Chat Completions. When a /v1/messages request routes to one, the gateway translates on both sides:

• Request — Anthropic /v1/messages body → OpenAI chat/completions body: system field → system message; messages preserved; tools → function tools; tool_use/tool_result blocks → tool_calls/tool messages; max_tokens defaulted to config.maxTokens or 4096. Backend credentials are injected; the caller's Anthropic or profile credentials are never forwarded.
• Response (non-streaming) — OpenAI chat.completion → Anthropic message shape; SGLang/Qwen reasoning_content becomes a leading Anthropic thinking block.
• Response (streaming) — OpenAI SSE → Anthropic SSE (same event structure Claude Code expects), including reasoning_content → thinking_delta before the answer's text_delta block.

Model-scoped sampling overrides

samplingParams.overrides forces request-level sampling fields after protocol translation and immediately before an OpenAI-compatible backend call. Rules are evaluated in order; the first pattern glob matching the backend's unprefixed model id wins. Omitted fields remain client-controlled. This is useful when a client such as Claude Code cannot send SGLang extensions like top_k, min_p, or repetition_penalty itself.

For the Qwen3.6 27B and 35B precise-coding routes in thinking mode, the published baseline is temperature=0.6, top_p=0.95, top_k=20, min_p=0, presence_penalty=0, and repetition_penalty=1. These values affect sampling inside both the current reasoning trace and final answer; they do not enable thinking or preserve historical thinking. SGLang's reasoning parser controls the former, while Qwen's chat_template_kwargs.preserve_thinking controls the latter.

Use case: enterprise AI providers that expose an OpenAI-compatible API but are reached over the Anthropic surface from Claude Code. Models from an openaiCompat backend appear as lm-{name}-{model} in the Claude Code model picker and route through the translation layer transparently.

Worker-Advisor escalation

When a routing attempt targets a local backend (a worker) and the attempt chain has a subsequent non-local target (an advisor), the gateway injects a call_advisor tool into every non-streaming request to the worker. If the local model's response contains a call_advisor tool-use block, the gateway transparently re-issues the original request to the advisor target and returns that response to the client.

Claude Code → proxy → local worker (qwen, llama, …)
                          ↓ uncertain? calls call_advisor
                       proxy → advisor (claude-sonnet-4-6, …)

Configure a local-first rule with a frontier fallback:

{
  "name": "local-with-advisor",
  "match": { "role": "fast" },
  "target": "lm",
  "model": "qwen-2.5-7b",
  "fallback": ["anthropic"],
  "anthropicModel": "claude-sonnet-4-6"
}

Escalation only applies to non-streaming worker attempts that have an advisor target later in the chain. Streaming worker responses are forwarded directly (buffering for call detection would break the stream). All routing decisions — whether escalation fired or not — are emitted as structured routing_decision log events including worker_tokens, advisor_tokens, and confidence_score.

Admin API

The admin API enables runtime backend registration without restarting the proxy. It is disabled (all /admin/* paths return 404) unless ADMIN_KEY is set. Every request to these paths must carry a matching X-Admin-Key header.

Config-file backends (those declared in config.backends) are read-only at runtime — the admin API cannot overwrite or delete them.

Method	Path	Description
GET	/admin/config	Non-secret runtime snapshot: port, prefix, backends list, configured profiles
GET	/admin/backends	List all backends (base + dynamic), with dynamic: true/false flag
POST	/admin/backends	Register or replace a dynamic backend. Body: { name, baseUrl, apiKey?, modelsPath?, openaiCompat?, allowedProfiles? }
DELETE	/admin/backends/:name	Remove a dynamic backend by name

Stateful egress grants (.NET)

The ASP.NET Core runtime exposes /grants when LLMP_GRANTS_BEARER_TOKEN is set or config.grants.enabled is true. All grant management calls require Authorization: Bearer <LLMP_GRANTS_BEARER_TOKEN>.

Grant flow:

1. POST /grants creates a pending grant with { session_id, scope, ttl_seconds?, max_uses? }. Scope requires upstream, surface, and side_effect; resource is optional.
2. POST /grants/{id}/activate activates the grant after approval.
3. A proxied request can include X-LLMP-Grant: <id>. The runtime validates upstream/surface scope, records grant fields in the access log, and consumes the grant when its use limit is reached.
4. POST /grants/{id}/revoke revokes a pending or active grant.

A use is reserved before the first upstream call, not after it returns. Scope resolution is a read, so two requests holding the same max_uses=1 grant can both pass it; only one can claim the use, and the other is refused 403 grant_not_active without reaching an upstream. This is what keeps a single-use grant bounding the side-effecting calls it authorizes, at the cost of uses briefly reading as spent while a request is in flight — an unspent reservation is handed back if every attempt fails, so the grant returns to Active.

Every attempt in a failover chain is checked against the grant's scope, including the non-backend ones (anthropic, google_passthrough). A grant scoped to a specific backend cannot follow a 429/5xx failover onto an upstream it does not cover; that request is refused 403 grant_scope_mismatch. Failover and advisor escalation run on the reservation the request already holds, so a multi-attempt request still spends exactly one use.

Write-class side effects listed in config.grants.writeSideEffects are forced to single-use even if a larger max_uses is requested.

Prompt filtering (.NET)

config.promptFiltering.enabled defaults to false. When enabled, the proxy drops configured sections (dropContains needles) and caps oversized system/message content on prompts bound for matching local backends.

How aggressively to filter depends on how the backend is used. For a scratch or evaluation model, stripping generated workspace context wholesale (agent and skill inventories, the claudeMd system-reminder, git status) saves tokens that would confuse it. For a model serving as a real Claude Code backend, most of that context is load-bearing — project instructions, git state, and skill inventories are what the model works from — so the profile should drop only genuinely low-value-per-token blocks (e.g. the subagent catalog, which a directly-driven backend never uses) and keep the char caps as safety valves. Total context budget is better governed by advertising the backend's real window (see context-window discovery above) so Claude Code's own auto-compaction does the trimming.

Matching is per-section, not per-block

Claude Code sends its system prompt as roughly two blocks: a tiny identity block and one giant block containing everything else. Matching whole blocks therefore fails badly — any marker that appears anywhere in the giant block drops the entire thing, including the instructions you wanted to keep.

Blocks are instead split into sections at:

• <system-reminder>…</system-reminder> fences — each fence is its own section, inclusive of the tags;
• markdown H1/H2 headings (# , ## ) at line start, outside fenced code blocks. Code fences are tracked so a shell comment like # run tests is not mistaken for a heading.

A dropContains match removes only the matching section. Survivors are reassembled in order; splitting and rejoining an unfiltered block reproduces the input byte-for-byte, CRLF included. A block whose sections all drop is removed entirely, so whole-block dropping still falls out as a special case.

This applies everywhere text appears: system as a bare string, system as a block array, and message content in either shape — the last is what catches the claudeMd <system-reminder> that Claude Code embeds in the first user message.

maxSystemChars / maxMessageChars still apply as a final cap, after section filtering.

Filtering runs before translation

The filter is applied to the client-shaped body, ahead of the Anthropic→OpenAI / Gemini→OpenAI translators. This matters on openaiCompat backends: translation merges every system block into a single string, and a noise block with no headings then becomes one indivisible section — one marker match would zero the whole system prompt to "". Filtering first preserves the block boundaries so only the noise block is removed.

Scoping: rules are fail-closed

Resolution order for a given upstream model:

1. the first modelRules entry whose pattern glob matches, else
2. defaultProfile, if set and present in profiles, else
3. no filtering at all.

Step 3 is the important one. Set defaultProfile: null and filtering applies only to models you name explicitly — turning it on cannot silently reshape every backend-routed request.

Patterns match the upstream model id, after the <backend>- tag has been stripped. A client asking for local-gateway-qwen3.6-27b resolves to backend local-gateway and model qwen3.6-27b, so that bare id is what the pattern must match:

"promptFiltering": {
  "enabled": true,
  "defaultProfile": null,
  "logSizes": true,
  "profiles": {
    "sglang-27b": {
      "enabled": true,
      "dropContains": ["Available skills", "Scratchpad Directory", "# claudeMd"],
      "maxSystemChars": 12000,
      "maxMessageChars": 24000
    }
  },
  "modelRules": [
    { "pattern": "qwen3.6-27b", "profile": "sglang-27b" }
  ]
}

With logSizes: true each filtered request emits prompt_filter backend=… model=… original_chars=… filtered_chars=…. No line means no filtering was applied — a quick way to confirm scoping.

Anthropic is never filtered

PromptFilter.Apply is called only inside the backend-kind branch of RoutedDispatcher. Anthropic-routed attempts, raw Anthropic passthrough, and google_passthrough never reach it, regardless of configuration. A gateway regression test asserts an Anthropic-routed request arrives upstream carrying markers that would otherwise have been dropped.

Effort-level normalization

Claude Code's effortLevel setting (or the CLAUDE_CODE_EFFORT_LEVEL env var) makes it attach output_config.effort to every /v1/messages request when the level is anything other than off. The effort param is only accepted by models with Adaptive Thinking; sending it to a model without it — Claude Haiku 4.5, any local LM Studio model reached on a local/failover route, or an unrecognized id — returns a 400. Hardcoding per-model overrides in settings.json is not portable across machines, so the gateway normalizes the param dynamically against the actual upstream model instead.

On the Anthropic-shaped /v1/messages path, for each routing attempt the gateway checks the upstream model id (attempt.model, after any rule rewrite) against a configurable allowlist of effort-capable models. If the model is not in the allowlist, output_config.effort is stripped (and output_config dropped if that left it empty); any other output_config keys are preserved. If it is in the allowlist, the request is forwarded verbatim. Requests with no output_config.effort are untouched. The OpenAI surface and openaiCompat translation paths never carry output_config, so they are unaffected.

The allowlist is resolved once per process, in precedence order:

1. EFFORT_CAPABLE_MODELS env var — comma-separated *-glob patterns (e.g. claude-opus-4-8*,claude-sonnet-4-6*). Wins over everything.
2. config.effort.capableModels — an array of *-glob patterns in config.json. An explicit empty array [] means "no model is effort-capable" (suppress everywhere).
3. Built-in default — the models Anthropic documents as effort-capable: claude-opus-4-8*, claude-opus-4-7*, claude-opus-4-6*, claude-opus-4-5*, claude-sonnet-4-6*, claude-fable-5*, claude-mythos-5*, claude-mythos-preview*.

Patterns use the same *-glob matching as the routing engine. Unknown models default to suppression (fail-safe: a model the proxy has never heard of is assumed not to support effort).

Sampling-param normalization (Sonnet-class 400s)

Some Sonnet-class Claude builds reject temperature / top_p / top_k outright (400) when set to a non-default value, while other Claude families — Haiku, older Opus/Sonnet builds outside that constraint — still accept them. Blanket-stripping these params for every Claude-bound request would silently degrade callers that never had a problem, so the gateway strips them only when the resolved upstream model matches a configurable denylist, mirroring the effort allowlist's glob/precedence mechanism above (just inverted: a denylist of models known to reject the params, not an allowlist of models known to support a feature).

This only runs on the two translation paths where the caller and the upstream speak different wire protocols and the upstream might be a real Claude model:

• OpenAI → Anthropic (openaiToAnthropicRequest, only reachable with routing.translateOpenAIToAnthropic: true) — applied to the translated body right after attempt.model is stamped on it, since that's the actual Anthropic /v1/messages model about to be forwarded.
• Anthropic → OpenAI (anthropicToOpenAIRequest, the openaiCompat backend bridge) — applied the same way, in case the openaiCompat backend (e.g. a third-party gateway) is itself fronting a real Sonnet-class Claude model under its OpenAI-compat surface. Backend models that don't match the denylist (GPT-style, local, etc.) are left untouched.

Native Anthropic /v1/messages passthrough (no translation involved) is not touched by this normalization — Claude Code doesn't set these params by default, so that path isn't exposed to the same risk today; this is scoped to the translation bridges where a caller might set them explicitly.

The denylist is resolved once per process, in precedence order:

1. SAMPLING_PARAMS_RESTRICTED_MODELS env var — comma-separated *-glob patterns (e.g. claude-sonnet-4-6*,claude-sonnet-5*). Wins over everything.
2. config.samplingParams.restrictedModels — an array of *-glob patterns in config.json. An explicit empty array [] means "no model is restricted" (never strip).
3. Built-in default — the Sonnet-class ids currently constrained: claude-sonnet-4-6*, claude-sonnet-5*.

A model id that doesn't match any pattern (Haiku, older Opus/Sonnet, non-Claude backend models) is left untouched. The one fail-closed exception: a model id the proxy can't resolve at all (non-string/missing) is treated as restricted and stripped anyway — losing sampling control is a much smaller failure than a 400.

OpenAI ⇄ Anthropic translation

By default the OpenAI surface (/openai/v1/chat/completions) can only be served by LM Studio backends — Anthropic targets are dropped from the chain and an all-anthropic route returns 502. Set

"routing": { "translateOpenAIToAnthropic": true, ... }

to let OpenAI chat requests fail over to Claude. When a chat request routes (or fails over) to an anthropic target with the flag on, the gateway:

1. Request — maps the OpenAI Chat-Completions body to an Anthropic /v1/messages body (system hoisted, messages preserved, tools → Anthropic tool schema, tool_calls/tool messages → tool_use/tool_result blocks, max_tokens defaulted to config.maxTokens or 4096) and forwards it to …/v1/messages with the gateway's Anthropic key (injected only when the caller sent no x-api-key/authorization).
2. Response (non-streaming) — buffers the Anthropic JSON and maps it back to a chat.completion object (text + tool_calls, stop_reason → finish_reason, usage → prompt_tokens/completion_tokens).
3. Response (streaming) — pipes the Anthropic SSE through a transform that emits OpenAI chat.completion.chunk frames terminated by data: [DONE]. usage is attached to the final chunk only when the request sets stream_options.include_usage: true.

The translated response always echoes the model id the OpenAI client requested (the Anthropic upstream model is an internal routing detail). Failover semantics are unchanged: a transport error before the first response byte still falls through to the next attempt, so a local-first rule with an anthropic fallback transparently fails over to Claude and translates.

Use case — OpenAI-client failover to Claude: point an OpenAI-compatible client at a routing alias whose rule is target: lm, fallback: ["anthropic"]. When LM Studio is down (connection refused), the request is translated and served by Claude with no client-side model change.

Prompt caching and v1 gaps

Native Anthropic requests (/v1/messages) are passed through with the original body and anthropic-beta header, so Claude Code prompt-caching controls remain available on that path.

Translation is scoped to chat completions only. The following stay backend-only and return 502 on the OpenAI surface rather than being mis-routed or silently degraded:

• /openai/v1/completions (legacy text completions) — no chat/messages mapping.
• /openai/v1/embeddings — Anthropic has no embeddings endpoint.
• Image / multimodal content blocks — only text/tool content is mapped.
• Extended-thinking blocks on OpenAI→Anthropic requests — not translated. The opposite openaiCompat response direction does translate SGLang/Qwen reasoning_content into Anthropic thinking blocks.
• Prompt-caching controls (cache_control) — not translated; chat translation returns 400 unsupported_translation when these controls appear in an OpenAI request that would route to Anthropic.

These remain available to LM Studio backends that support them; only the OpenAI→Anthropic bridge omits them.

Responses → Chat Completions translation

Codex custom model providers use OpenAI's Responses API, while LM Studio exposes OpenAI-compatible Chat Completions. The proxy bridges that gap for local backend models on POST /v1/responses, POST /openai/v1/responses, and POST /codex/v1/responses.

/v1/responses and /openai/v1/responses remain local-backend-only. The request model must use a configured backend prefix, e.g. lm-qwen/qwen3.5-9b; non-local models return 400 unsupported_target and are never relayed to Anthropic or another cloud provider.

/codex/v1/* is the unified Codex custom provider surface. Configure Codex with:

model_provider = "lmproxy"

[model_providers.lmproxy]
name = "llm-proxy"
base_url = "http://localhost:1235/codex/v1"
wire_api = "responses"
requires_openai_auth = true

On GET /codex/v1/models, backend models are always listed and OpenAI models are appended only when Codex sends its normal bearer auth. On POST /codex/v1/responses, backend-prefixed model IDs such as lm-qwen-coder are translated to local Chat Completions with proxy-managed backend credentials, while non-backend model IDs such as gpt-5.5 pass through to OpenAI /responses with the caller's bearer auth. The caller's authorization and x-api-key headers are stripped before any local backend call.

The mapper converts Responses request fields into a Chat Completions request:

• instructions and developer items become system messages.
• message text parts become chat user / assistant messages.
• function_call and function_call_output items become chat tool call / tool result messages.
• flat Responses tools become Chat Completions function tools.
• max_output_tokens becomes max_tokens; when absent, the proxy defaults to config.maxTokens or 4096.

Non-streaming Chat Completions responses are mapped back to Responses-shaped objects with output_text or function_call output items. Streaming Chat Completions SSE is mapped to Responses events and terminates with response.completed; it does not emit data: [DONE].

Consumers

• Claude Code — a SessionStart hook keeps ~/.claude/cache/gateway-models.json fresh so backend models appear in the model picker. The hook scripts live in scripts/claude-hook/ in this repo, versioned with the proxy so the hook and the cache contract it reads never drift apart:

  • refresh-gateway-cache.{ps1,sh} — GET /admin/model-cache and write the response verbatim to ~/.claude/cache/gateway-models.json. Zero format knowledge: no re-derivation, no hardcoded model-id prefixes, no client-side baseUrl rewriting (the proxy resolves its own correct baseUrl — see "Model-cache baseUrl" above). Skips instantly unless ANTHROPIC_BASE_URL points at the proxy, and never blocks startup on failure.
  • install-gateway-hook.{ps1,sh} — idempotently register or remove the SessionStart hook entry in a Claude Code settings.json.

  Install it with:

  ./scripts/claude-hook/install-gateway-hook.sh

  Then point Claude Code at the proxy with ANTHROPIC_BASE_URL=http://127.0.0.1:1235. Port 1235 is a fixed contract.

  asi does both steps at once (asi gateway on --targets claude) and handles the equivalent wiring for Codex, Antigravity, and Copilot — see "Related tools" above. The scripts here are the dependency-free path and remain the source of truth for the cache contract.

• OpenAI-compatible clients — set the client's base URL to http://<proxy-host>:1235/openai/v1 and choose a proxied backend model or a routing alias.

See docs/consumer/ for per-client getting-started guides and the full endpoint reference.

CI

.github/workflows/ci.yml restores, builds, and tests dotnet/LLMProxy.slnx on every pull request and push to main, and runs a NuGet vulnerability audit. The SDK version is pinned in global.json.

Releases are cut by tagging: scripts/release/bump-version.sh updates VERSION, and the container image is built from the Dockerfile at the repository root.

Contributing

See CONTRIBUTING.md. Bug reports and pull requests are welcome. For security issues, please follow SECURITY.md rather than opening a public issue.

License

Apache License 2.0 — see LICENSE.