Repository navigation
v1.3.0 — Local OpenAI/Anthropic-compatible model API
Added
- Local model API: the GGUF models ThreadShelf manages
are now served over the OpenAI- and Anthropic-compatible HTTP API that LM
Studio and Ollama use, athttp://localhost:3000/v1. It covers
GET /v1/models,POST /v1/chat/completions,/v1/completions,
/v1/responses,/v1/messagesand/v1/messages/count_tokens, all with
streaming. The requested model loads on demand. Requests are relayed
unchanged to the managedllama-server, so tools, sampling parameters and
structured output behave as llama.cpp implements them. Errors come back in
each SDK's own format. The API accepts requests only from this machine, and
API calls are never archived. Settings shows the base URL, model ids and
copyable examples. See docs/LOCAL_API.md. - Local model API access settings. The defaults match LM Studio and Ollama
(on, this computer only, no key). Optional: an API key, required as
Authorization: Bearerorx-api-keyonce set (alsoTHREADSHELF_API_KEY);
Serve on the local network, a separate port (3001 by default) that serves
only/v1to other devices, with Host/Origin checks when no key is set; and
an off switch. - Model memory controls.
POST /v1/models/unloadfrees the loaded model,
GET /v1/modelsreports which model isloaded, and Unload model from
memory (LLAMA_CPP_IDLE_UNLOAD_MINUTES) unloads an unused model after an
idle time, like Ollama'skeep_alive. The default stays "never". - In-app guide for the local model API: what is on right now, step-by-step
setup for Claude Code, Codex CLI and DeepSeek Harness, and a plain-language
explanation of every generation setting. - docs/PERFORMANCE.md: measured speeds on an
RTX 3090 Ti (short and long context, Q8 vs Q4 KV cache, DFlash drafting),
recommended settings, and how to benchmark your own machine.
Changed
- Conversation generation is no longer marked Experimental Beta. The
labels, the chat's "alpha" note and theexperimentalAlphaconfig field are
gone;docs/GENERATION_BETA.mdis now
docs/GENERATION.md. Context sizes above 64K are no
longer called experimental: the note now states the real cost (slower
generation as the context fills, more VRAM).
Fixed
- Switching models now waits for a
llama-serverthat had to be force-killed
to exit before the next model starts, so its VRAM is free first. ingest --watchon a missing folder now exits with an error on Node 24,
where recursivefs.watchno longer throws for it. This failed the 1.2.3
publish run, so 1.2.3 never reached npm.- API/MCP end-to-end tests now isolate the model download directory
(THREADSHELF_MODELS_PATH), so models a developer has downloaded no longer
leak into test servers.
Install: npx threadshelf@1.3.0