LLM Suite Local API Server

Local API Server

The Developer page (Ctrl+4) runs an HTTP server that exposes your models to other programs through OpenAI-compatible and Anthropic-compatible APIs. Anything that can talk to those APIs (scripts, editors, other apps) can use the models in your library, with no cloud and no separate engine.

Quick start

1. Open the Developer page and choose a port. The default is 1234, which LM Studio also uses; if that is taken, the page tells you, and you can pick another one.
2. Click Start server. The status line shows the address, for example 127.0.0.1:1234.
3. Call it:

curl http://127.0.0.1:1234/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"YOUR-MODEL-NAME","messages":[{"role":"user","content":"Hello!"}],"stream":true}'

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:1234/v1", api_key="not-needed")
r = client.chat.completions.create(model="YOUR-MODEL-NAME", messages=[{"role": "user", "content": "Hello!"}])
print(r.choices[0].message.content)

The page generates examples with a model from your library filled in, including the API key if you set one.

Models

The model field is the name shown on My Models (the file name without .gguf). The server also accepts the model's path in the library, its file name, or any part of the name that matches exactly one model. If a name matches several models, use the full path (GET /v1/models shows each model's id).

Load on request

If a request names a model that isn't loaded, the server loads it with that model's saved settings and then answers. The request log shows a load state while that happens, and several simultaneous requests for the same model share one load. If loading fails, the client gets a clear error (503).

Loading for a request follows the app's Unload the previous model when loading another setting (Settings → Models, on by default). With it on, a request for a different model unloads whatever is loaded first, including models you loaded yourself, so clients that alternate between two models will keep reloading them. To serve several models at once, turn that setting off (and make sure they fit in memory together).

Idle unload

Models that the server loaded are unloaded after they have been idle for the Unload after idle time (default 10 minutes; 0 means never). The idle timer never unloads a model you loaded yourself from the app, and ejecting a model by hand removes it from this tracking. The Developer page marks each loaded model as loaded by the server or by the app.

No model in the request

If a request has no model field, the server uses the single loaded model if there is exactly one, otherwise the Default model from the page, otherwise it returns an error.

A model you don't have

Some programs always ask for a model by their own name. Claude Code, for example, asks for claude-sonnet-…, and for a Haiku model for small background jobs. With Answer any model name on (the default since 0.7.0, under Model loading), a request for a model that isn't in your library gets the same model as a request with no model: the single loaded model, otherwise the Default model. If several models are loaded and no default model is set, it still returns an error that says so.

The request log shows "(asked for …)" next to the model when this happens, so a mistyped model name doesn't go unnoticed. Turn the switch off to get an error for any name you don't have.

Endpoints

Endpoint	Notes
GET /v1/models, GET /v1/models/{id}	Your whole library, with extra fields: path, arch, quantization, max_context_length, state (loaded or not-loaded) and loaded_context_length. /api/v0/models returns the same list.
POST /v1/chat/completions	OpenAI chat, streaming or not. Tools and structured output work as in llama.cpp.
POST /v1/completions	OpenAI text completion.
POST /v1/embeddings	Embeddings, if the loaded model supports them.
POST /v1/responses	OpenAI Responses API.
POST /v1/messages, POST /v1/messages/count_tokens	Anthropic Messages API.
POST /tokenize, /detokenize, /apply-template, /infill, /reranking	llama.cpp's own helper endpoints.
POST /v1/models/{id}/unload	Unloads every running copy of that model, including one you loaded from the app, and replies with the model's entry plus unloaded (how many it stopped). (0.8.0+)
GET /v1/models/{id}/load-config, POST the same	Read or change a model's saved load settings. See below. (0.3.5+)
GET /props, /slots, /metrics	llama.cpp status endpoints; pass ?model= to say which model.
GET /health	Returns {"status":"ok"}.

Requests are forwarded to that model's llama-server, so behaviour, parameters and response formats are llama.cpp's. If a client disconnects mid-stream, generation stops.

Anthropic example:

curl http://127.0.0.1:1234/v1/messages \
  -H "Content-Type: application/json" -H "anthropic-version: 2023-06-01" \
  -d '{"model":"YOUR-MODEL-NAME","max_tokens":256,"messages":[{"role":"user","content":"Hello!"}]}'

Load settings from other apps

Since 0.3.5, other programs can read and change a model's saved load settings (the same defaults that Remember settings for this model saves; see Loading Models). RulZ Coding Agent's Settings → Model settings uses this.

curl http://127.0.0.1:1234/v1/models/YOUR-MODEL-NAME/load-config

The reply has config (the settings below), custom (false while the model still uses Settings → Default load settings), max_context_length, state and, when loaded, loaded_config and loaded_context_length. Since 0.3.7 it also has gpus (index, name, used and total MiB), has_draft (whether the model has a draft head) and estimate (the same memory estimate as the load panel, including the per-GPU breakdown). Since 0.3.9 it also has default_draft (what automatic uses: name, kind, built_in, or null) and draft_choices, the drafts that can be picked. Each has an id ("builtin" for the model's own MTP layers, otherwise the file's path relative to the models folder), name, folder, kind and built_in.

To change them, POST the fields you want; the rest stay as they are:

curl http://127.0.0.1:1234/v1/models/YOUR-MODEL-NAME/load-config -H "Content-Type: application/json" -d '{"config":{"contextLength":32768,"gpuLayers":"all","cacheTypeK":"q8_0","cacheTypeV":"q8_0"},"reload":true}'

Field	Values
contextLength	512 or more, or null for Auto (fit)
gpuLayers	"auto", "all" or a number
flashAttention	"auto", "on", "off"
cacheTypeK, cacheTypeV	f16, bf16, f32, q8_0, q5_1, q5_0, q4_1, q4_0, iq4_nl
parallel	1 to 64
useDraft	true / false
splitMode (0.3.7+)	"auto", "single", "custom"
mainGpu (0.3.7+)	GPU index used with "single"
tensorSplit (0.3.7+)	proportions per GPU for "custom", numbers and commas only, e.g. "60,40"
draftTokens, draftMinTokens (0.3.7+)	1 to 64 and 0 to 64, or null for llama.cpp's default
draftPMin (0.3.7+)	0 to 1, or null
draftSampling (0.3.7+)	"auto", "greedy", "probabilistic"
draftGpu (0.3.7+)	"auto", "gpu", "cpu"
draftFile (0.3.9+)	null for automatic, "builtin", or the id of one of draft_choices. Anything else is refused, so a client can't point llama-server at an arbitrary file.
draftCacheTypeK, draftCacheTypeV (0.3.7+)	the KV cache types above
sampling (0.9.0+)	The model's sampling defaults: any of temperature (0 to 5), topK (0 to 1000), topP, minP (0 to 1), repeatPenalty (0 to 5), presencePenalty (-2 to 2), each null to clear it, and override (true/false). Only the values you send change.

• If the model is loaded, it restarts with the new settings. Send "reload": false to only save them for next time. While it restarts, requests for it wait for the load as usual.
• "dry_run": true (0.3.7+) saves nothing and restarts nothing: the reply shows the changed settings with their memory estimate, so a form can show whether they fit while you edit.
• Bad values and any other fields are refused with 400. File paths (chat template) and extra llama-server arguments can only be changed in the app, because the server may be reachable from your network.
• The API key setting applies here like everywhere else.

Using Claude Code with your local models

Claude Code can send its requests to LLM Suite instead of to Anthropic. How you set it up depends on which Claude Code you use:

• Command line (claude in a terminal) or the VS Code extension: follow the steps below.
• The Claude desktop app's Code tab ignores ANTHROPIC_BASE_URL in settings files and in Windows environment variables, so the folder and .bat methods below don't affect it. Use its own setting instead: Help → Troubleshooting → Enable Developer Mode, then Developer → Configure Third-Party Inference, and enter http://127.0.0.1:1234. This switches the whole app, not one folder, so turn it off again in the same place to go back to Claude. While it's on, cloud sessions and Remote Control aren't available. See Claude Code's gateway docs.

Command line and VS Code:

1. In LLM Suite, load the model you want, or pick a Default model on the Developer page, and start the server. Turning on Start the server when the app starts saves remembering.
2. Choose a model that is good at using tools (a coding or agent model), and load it with a large context; 64k tokens or more is a sensible floor, because Claude Code's instructions and tools alone are tens of thousands of tokens.
3. Tell Claude Code where the server is. To keep your normal Claude models too, do this only where you want local models, using one of these:
   • A separate command. Save this as claude-local.bat and run claude-local instead of claude. The settings last only for that window.

     @echo off
     set ANTHROPIC_BASE_URL=http://127.0.0.1:1234
     set ANTHROPIC_AUTH_TOKEN=your-api-key-or-anything
     claude %*

   • A local-only project folder. Put this in that folder's .claude\settings.local.json. Sessions opened in that folder use LLM Suite, and every other folder uses Claude.

     { "env": { "ANTHROPIC_BASE_URL": "http://127.0.0.1:1234", "ANTHROPIC_AUTH_TOKEN": "your-api-key-or-anything" } }

   In VS Code, put the two variables in the claudeCode.environmentVariables setting instead. Put the same env block in C:\Users\<you>\.claude\settings.json only if you want every Claude Code session to use your local models.

Inside Claude Code, /status shows which address it is using. To switch to a different local model, load it in LLM Suite; Claude Code uses whatever is there. Claude Code sends all its requests to one address, so you can't send only the Haiku jobs to LLM Suite and the rest to Anthropic. Expect a local model to be slower and less capable than Claude at long, multi-step work.

Claude Code also sends part of its instructions as a system message partway through the conversation, which some chat templates (Qwen 3.5's, for example) refuse with System message must be at the beginning. Since 0.9.1 the server moves those messages into the request's system prompt on /v1/messages, so those models work too.

Claude Code sends its own temperature with every request. To use the model's recommended settings instead, set its sampling defaults and turn on Override what programs send.

Access and security

Setting	Meaning
Reachable from	This computer only (default) binds to 127.0.0.1. Local network binds to all interfaces so other devices can connect. Windows Firewall may ask to allow it.
API key	Empty means no authentication. If set, requests must send it as Authorization: Bearer <key> or x-api-key: <key>, or they get 401. Generate makes a random one.
Allow browser apps (CORS)	Lets web pages on any origin call the API.
Start the server when the app starts	Off by default.

If you choose Local network, set an API key. Anyone who can reach the port could otherwise use your GPU and your models. Port and binding changes apply after you stop and start the server.

Request log

The right-hand panel lists each request with its time, status, endpoint, model, generated tokens, tokens per second and duration; click one for details. The log keeps the last 300 requests, lives in memory only, and is cleared when the app closes.