Local API Server

Local API Server

The Developer page (Ctrl+4) runs an HTTP server that exposes your models to other programs through OpenAI-compatible and Anthropic-compatible APIs. Anything that can talk to those APIs (scripts, editors, other apps) can use the models in your library, with no cloud and no separate engine.

Quick start

1. Open the Developer page and choose a port. The default is 1234, which LM Studio also uses; if that is taken, the page tells you, and you can pick another one.
2. Click Start server. The status line shows the address, for example 127.0.0.1:1234.
3. Call it:

curl http://127.0.0.1:1234/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"YOUR-MODEL-NAME","messages":[{"role":"user","content":"Hello!"}],"stream":true}'

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:1234/v1", api_key="not-needed")
r = client.chat.completions.create(model="YOUR-MODEL-NAME", messages=[{"role": "user", "content": "Hello!"}])
print(r.choices[0].message.content)

The page generates examples with a model from your library filled in, including the API key if you set one.

Models

The model field is the name shown on My Models (the file name without .gguf). The server also accepts the model's path in the library, its file name, or any part of the name that matches exactly one model. If a name matches several models, use the full path (GET /v1/models shows each model's id).

Load on request

If a request names a model that isn't loaded, the server loads it with that model's saved settings and then answers. The request log shows a load state while that happens, and several simultaneous requests for the same model share one load. If loading fails, the client gets a clear error (503).

Loading for a request follows the app's Unload the previous model when loading another setting (Settings → Models, on by default). With it on, a request for a different model unloads whatever is loaded first, including models you loaded yourself, so clients that alternate between two models will keep reloading them. To serve several models at once, turn that setting off (and make sure they fit in memory together).

Idle unload

Models that the server loaded are unloaded after they have been idle for the Unload after idle time (default 10 minutes; 0 means never). The idle timer never unloads a model you loaded yourself from the app, and ejecting a model by hand removes it from this tracking. The Developer page marks each loaded model as loaded by the server or by the app.

No model in the request

If a request has no model field, the server uses the single loaded model if there is exactly one, otherwise the Default model from the page, otherwise it returns an error.

Endpoints

Endpoint	Notes
GET /v1/models, GET /v1/models/{id}	Your whole library, with extra fields: path, arch, quantization, max_context_length, state (loaded or not-loaded) and loaded_context_length. /api/v0/models returns the same list.
POST /v1/chat/completions	OpenAI chat, streaming or not. Tools and structured output work as in llama.cpp.
POST /v1/completions	OpenAI text completion.
POST /v1/embeddings	Embeddings, if the loaded model supports them.
POST /v1/responses	OpenAI Responses API.
POST /v1/messages, POST /v1/messages/count_tokens	Anthropic Messages API.
POST /tokenize, /detokenize, /apply-template, /infill, /reranking	llama.cpp's own helper endpoints.
GET /v1/models/{id}/load-config, POST the same	Read or change a model's saved load settings. See below. (0.3.5+)
GET /props, /slots, /metrics	llama.cpp status endpoints; pass ?model= to say which model.
GET /health	Returns {"status":"ok"}.

Requests are forwarded to that model's llama-server, so behaviour, parameters and response formats are llama.cpp's. If a client disconnects mid-stream, generation stops.

Anthropic example:

curl http://127.0.0.1:1234/v1/messages \
  -H "Content-Type: application/json" -H "anthropic-version: 2023-06-01" \
  -d '{"model":"YOUR-MODEL-NAME","max_tokens":256,"messages":[{"role":"user","content":"Hello!"}]}'

Load settings from other apps

Since 0.3.5, other programs can read and change a model's saved load settings (the same defaults that Remember settings for this model saves; see Loading Models). RulZ Coding Agent's Settings → Model settings uses this.

curl http://127.0.0.1:1234/v1/models/YOUR-MODEL-NAME/load-config

The reply has config (the settings below), custom (false while the model still uses Settings → Default load settings), max_context_length, state and, when loaded, loaded_config and loaded_context_length. Since 0.3.7 it also has gpus (index, name, used and total MiB), has_draft (whether the model has a draft head) and estimate (the same memory estimate as the load panel, including the per-GPU breakdown). Since 0.3.9 it also has default_draft (what automatic uses: name, kind, built_in, or null) and draft_choices, the drafts that can be picked. Each has an id ("builtin" for the model's own MTP layers, otherwise the file's path relative to the models folder), name, folder, kind and built_in.

To change them, POST the fields you want; the rest stay as they are:

curl http://127.0.0.1:1234/v1/models/YOUR-MODEL-NAME/load-config -H "Content-Type: application/json" -d '{"config":{"contextLength":32768,"gpuLayers":"all","cacheTypeK":"q8_0","cacheTypeV":"q8_0"},"reload":true}'

Field	Values
contextLength	512 or more, or null for Auto (fit)
gpuLayers	"auto", "all" or a number
flashAttention	"auto", "on", "off"
cacheTypeK, cacheTypeV	f16, bf16, f32, q8_0, q5_1, q5_0, q4_1, q4_0, iq4_nl
parallel	1 to 64
useDraft	true / false
splitMode (0.3.7+)	"auto", "single", "custom"
mainGpu (0.3.7+)	GPU index used with "single"
tensorSplit (0.3.7+)	proportions per GPU for "custom", numbers and commas only, e.g. "60,40"
draftTokens, draftMinTokens (0.3.7+)	1 to 64 and 0 to 64, or null for llama.cpp's default
draftPMin (0.3.7+)	0 to 1, or null
draftSampling (0.3.7+)	"auto", "greedy", "probabilistic"
draftGpu (0.3.7+)	"auto", "gpu", "cpu"
draftFile (0.3.9+)	null for automatic, "builtin", or the id of one of draft_choices. Anything else is refused, so a client can't point llama-server at an arbitrary file.
draftCacheTypeK, draftCacheTypeV (0.3.7+)	the KV cache types above

• If the model is loaded, it restarts with the new settings. Send "reload": false to only save them for next time. While it restarts, requests for it wait for the load as usual.
• "dry_run": true (0.3.7+) saves nothing and restarts nothing: the reply shows the changed settings with their memory estimate, so a form can show whether they fit while you edit.
• Bad values and any other fields are refused with 400. File paths (chat template) and extra llama-server arguments can only be changed in the app, because the server may be reachable from your network.
• The API key setting applies here like everywhere else.

Access and security

Setting	Meaning
Reachable from	This computer only (default) binds to 127.0.0.1. Local network binds to all interfaces so other devices can connect. Windows Firewall may ask to allow it.
API key	Empty means no authentication. If set, requests must send it as Authorization: Bearer <key> or x-api-key: <key>, or they get 401. Generate makes a random one.
Allow browser apps (CORS)	Lets web pages on any origin call the API.
Start the server when the app starts	Off by default.

If you choose Local network, set an API key. Anyone who can reach the port could otherwise use your GPU and your models. Port and binding changes apply after you stop and start the server.

Request log

The right-hand panel lists each request with its time, status, endpoint, model, generated tokens, tokens per second and duration; click one for details. The log keeps the last 300 requests, lives in memory only, and is cleared when the app closes.