Skip to content

Local Models

Jump to bottom
Vinh Nguyen edited this page Aug 5, 2026 · 1 revision

Local Models

VT Code can run models entirely on your machine via three local inference backends, all managed from the TUI.

Status: experimental. Local inference is less reliable than remote APIs. Remote providers are recommended for production work; use local models for privacy, offline use, or cost-free experimentation.

Backends

Backend Start command Notes
Ollama ollama serve Best-supported; auto-loads pulled models
LM Studio lms server start OpenAI-compatible /v1; select the loaded model in the picker
llama.cpp llama-server -m model.gguf Most automated; auto-starts via LLAMACPP_MODEL_PATH

Managing Local Servers

/local                 # interactive local server manager
/local status          # check server state
/local start ollama    # start a specific backend
/local troubleshoot    # guided diagnostics

Local vs Remote

Dimension Local Remote API
Privacy Data never leaves your machine Sent to a third party
Cost Free after hardware Per-token billing
Latency Limited by your CPU/GPU Typically faster
Model quality Smaller open-weight models Frontier models
Tool use / reasoning Works, but less consistent Most reliable
Reliability Server/load state must be correct Generally "just works"

Recommendation: use remote APIs by default; switch local for offline access, privacy, or avoiding API costs.

Hardware & Model Sizing

Tier VRAM / RAM Example models
Tiny 8–16 GB gemma-4-e4b (llama.cpp), llama-3.1-8b (LM Studio)
Mid 16–32 GB gpt-oss:20b, gemma-3-12b
High 32–64 GB gemma-4-26b-a4b, step-3.5-flash
Max 64 GB+ Multiple / larger MoEs

Quantized GGUF models run with far less VRAM — prefer Q4/Q5 quantizations for the best speed/quality trade-off.

Getting Reliable Results

Local generation fails most often because the server is stopped or the model isn't loaded. VT Code detects both before generating and prints the exact recovery command (e.g. ollama pull gpt-oss:20b).

  1. Start the server — /local status to check, /local start <provider> to start.
  2. Make sure the model is available — ollama pull <model>, lms load <model>, or pass the GGUF to llama-server.
  3. Pick a model that actually exists — the picker shows currently loaded models.
  4. Use /local troubleshoot for guided diagnostics.

Known Limitations

  • Ollama — no parallel tool calls or some tool_choice modes; requests fall back gracefully.
  • LM Studio — only one model active at a time; some models reject parameters OpenAI accepts (e.g. parallel_tool_calls).
  • llama.cpp — feature support (reasoning, tools, structured output) varies by build and model.

Related

Clone this wiki locally