Skip to content

AI Development Run LLMs Locally

Jump to bottom
Spinning Idea edited this page Jul 29, 2026 · 4 revisions

Below is a comprehensive guide to running Machine Learning (ML) and Large Language Models (LLMs) locally as of July 2026, evaluated specifically for coding, autonomous agentic software development, and IDE integration (VS Code, OpenCode, Aider) on consumer hardware (16GB VRAM NVIDIA GPU + 64GB System RAM).

Hardware Optimization (16GB VRAM GPU + 64GB System RAM)

For local AI coding, a system with 16GB VRAM (e.g., RTX 4080 / 5080 / 4070 Ti Super / 3090) and 64GB System RAM is an ideal workstation tier. Memory allocation and performance vary by model architecture:

  • Fully In-VRAM Models (8B – 16B Parameters at Q4/Q8): Models like Qwen 2.5/3.0 Coder 14B, DeepSeek-Coder-V2 Lite 16B MoE, Qwythos-9B, Gemma 4 12B, and LFM 2.5-8B occupy ~6–10GB VRAM. This leaves 6GB+ of VRAM dedicated strictly to the KV Cache, enabling huge context windows (32k–64k tokens) and ultra-fast generation (>60–120+ tokens/sec) for real-time autocomplete and refactoring.
  • Dense 22B – 24B Parameters: Models like Devstral Small 2 (24B) (~14.5GB at Q4_K_M) or Codestral 22B (~12.5GB at Q4_K_M) utilize almost the full 16GB VRAM buffer. Keep context limits conservative (2k–8k tokens) to prevent VRAM overflow into system memory.
  • MoE & Quantized Offload Models (27B – 33B Parameters): Mixture-of-Experts (MoE) architectures such as Laguna XS 2.1 (33B total / 3B active) or Qwen 3.6 27B A3B Coder only execute 2.4B–3B active parameters per token. They keep active weights in VRAM while leveraging 64GB system RAM for layer offloading and large context buffers without catastrophic speed drops.
  • Dual-Model Strategy (Recommended):
    • Agent / Chat Model: Laguna XS 2.1 (33B MoE) or Devstral Small 2 (24B) for deep architectural planning and issue resolution.
    • Inline Autocomplete Model (FIM): Fast 1.5B–8B model (e.g., LFM 2.5 8B or Qwen 2.5-Coder 1.5B) assigned exclusively to tab-autocompletion for sub-50ms latency.

Goat

Best

Top Local Runtimes & Inference Engines

  • Ollama - De facto CLI & background engine for local LLMs; native REST API & OpenAI-compatible server (http://localhost:11434/v1).
  • LM Studio - Top desktop interface & server (http://localhost:1234/v1) for GGUF model discovery, visual context monitoring, and hardware offload configuration.
  • Unsloth Studio & Engine (GitHub) - Advanced local AI environment featuring dynamic 4-bit quants, hardware acceleration, and the unsloth start [agent] CLI to connect local LLMs directly to coding agents.
  • Jan AI (GitHub) - Fully offline, open-source (Tauri-based) desktop ChatGPT alternative with local OpenAI-compatible API server (http://localhost:1337/v1) and Model Context Protocol (MCP) support.
  • OpenWebUI (GitHub) - Feature-rich web interface with deep Ollama integration, multi-model chat, RAG, and web search.
  • llama.cpp Server - Raw C++ inference engine (llama-server) providing maximum hardware control, custom layer offloading, and lowest memory overhead.
  • Text-Generation-WebUI - Flexible engine supporting GGUF, EXL2, HF transformers, and extensive plugin ecosystems.
  • LibreChat - Multi-model chat interface supporting custom local backend endpoints and agent plugins.

Top Local Coding Models (July 2026 Benchmarks)

Evaluated based on SWE-bench rankings, agentic terminal capability, Fill-In-the-Middle (FIM) speed, and tool calling:

  • Laguna XS 2.1 (33B MoE / 3B Active)
    • Model: Laguna XS 2.1 Q4_K_M
    • Details: 33B total parameter MoE model with only 3B activated parameters per token. Delivers +5.4% higher scores on SWE-bench Multilingual and superior performance on long-horizon agent tasks.
    • Run: ollama run laguna-xs-2.1:q4_K_M
  • Devstral Small 2 (24B by Mistral AI & All-Hands AI)
    • Details: Purpose-built for autonomous software engineering. Scores ~68% on SWE-bench Verified (Apache 2.0 license, 128k context). Fits 16GB VRAM at Q4_K_M (~14GB).
    • Run: ollama run devstral-small-2
  • Qwen 3.6 27B A3B Coder
    • Details: SOTA local coding model optimized for complex code synthesis, multi-file refactoring, and agentic workflows.
    • Run Commands:
      • ollama run mannix/qwen3.6-27b-a3b-coder:CD-IQ4_K_M
      • ollama run mannix/qwen3.6-27b-a3b-coder:CD-Q3_K_L
  • Qwen 2.5 Coder 14B
    • Details: Benchmark standard for 16GB VRAM setups. Occupies ~9GB at Q4_K_M, leaving 6GB+ for massive KV cache and instant completion speed (~35–40 tok/sec).
    • Run: ollama run qwen2.5-coder:14b
  • Qwythos-9B v2 / Claude-Mythos 5 (1M Context GGUF)
  • Gemma 4 12B (Google DeepMind)
    • Details: Encoder-free multimodal model (text, vision, audio) with 256k context window and Multi-Token Prediction (MTP) acceleration in Ollama (~90% speedup).
    • Run: ollama run gemma4:12b
  • LFM 2.5-8B-A1B
    • Model: LFM 2.5 8B Q8_0
    • Details: ~9GB memory footprint. Specialized edge model for fast, deterministic tool calling and function execution.
    • Run: ollama run lfm2.5:8b-a1b-q8_0
  • Codestral 22B (Mistral AI)
    • Details: Top-tier Fill-in-the-Middle (FIM) code completion model (~12.5GB VRAM at Q4_K_M).
    • Run: ollama run codestral:22b
  • DeepSeek-Coder-V2 Lite (16B MoE / 2.4B Active)
    • Details: Lightweight MoE model running at ~45 tok/sec; excellent low-latency choice for interactive agent loops.
    • Run: ollama run deepseek-coder-v2:16b

Community Benchmarks & Reviews

Better

  • Hugging Face Chat UI - Clean web UI with search integration
  • LocalAI - Developer-focused drop-in OpenAI API replacement for text, audio, and vision
  • AnythingLLM / UseAnything - All-in-one desktop application with document RAG and local model support
  • GPT4All - Cross-platform local LLM app with LocalDocs RAG
  • MindMac - Native desktop app supporting local endpoints (Ollama / LM Studio / Jan)
  • TypingMind - Advanced web interface for local OpenAI-compatible endpoints
  • NVIDIA ChatRTX - Hardware-accelerated local RAG for RTX GPUs
  • LocalGPT - Document chat powered by local models
  • Streamlit & Gradio - Prototyping frameworks for custom AI applications
  • Secret Llama - Browser-based private local LLM chatbot

Good

  • KoboldCpp - Lightweight single-file GGUF execution server
  • LobeChat - Modern chat UI with extension and MCP support
  • Lollms-WebUI - Multi-modal web UI with PDF and web search capabilities
  • H2O GPT - Document ingestion and query framework
  • SillyTavern - Persona and character interaction interface
  • ChainFury - Prompt chaining and web search framework
  • PrivateGPT - Offline document RAG stack
  • ExUI - VRAM-efficient GUI for EXL2 models
  • AI-Messenger - Desktop interface supporting EXLv2 and LLaVA

Mac Only

  • FreeChat - Native macOS chat app
  • Sanctum - Private local AI workspace for macOS

Enabling Local LLMs in OpenCode, VS Code & CLI Assistants

Connect local models running in Ollama, LM Studio, Jan AI, or Unsloth to developer tools.

1. OpenCode Setup

OpenCode connects to local models via OpenAI-compatible REST endpoints.

Option A: Automatic Launch via Unsloth CLI

If using Unsloth, launch OpenCode with pre-configured endpoints in a single command:

unsloth start opencode

Option B: Ollama & Manual Configuration

  1. Set Ollama Context Window:

    cat > Modelfile <<EOF
FROM laguna-xs-2.1:q4_K_M
PARAMETER num_ctx 32768
EOF
ollama create my-laguna-32k -f Modelfile

  2. Configure OpenCode Config File (~/.config/opencode/opencode.json):

    {
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "ollama": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "Ollama Local",
      "options": {
        "baseURL": "http://localhost:11434/v1"
      },
      "models": {
        "my-laguna-32k": { "name": "Laguna XS 2.1 (33B MoE 32k)" },
        "devstral-small-2": { "name": "Devstral Small 2 (24B)" },
        "mannix/qwen3.6-27b-a3b-coder:CD-IQ4_K_M": { "name": "Qwen 3.6 27B Coder" }
      }
    },
    "lmstudio": {
      "name": "LM Studio Local",
      "options": {
        "baseURL": "http://localhost:1234/v1"
      },
      "models": {
        "local-model": { "name": "LM Studio Model" }
      }
    }
  }
}

  3. Launch OpenCode:

    opencode
# Or launch directly with Ollama:
ollama launch opencode --model devstral-small-2

2. VS Code Integration

Local models can be connected to VS Code through native features or popular extensions:

Method A: VS Code Native Copilot BYOK (Bring Your Own Key/Model)

VS Code supports native local model connections without cloud subscriptions:

  1. Open VS Code Settings (Ctrl+, / Cmd+,).
  2. Navigate to GitHub Copilot / Language Models.
  3. Set Custom Model Provider Base URL to http://localhost:11434/v1 (Ollama) or http://localhost:1234/v1 (LM Studio).
  4. Select your local model directly inside the VS Code Copilot Chat panel.

Method B: Cline Extension (Recommended Agentic Tool)

(Note: Roo Code was archived in May 2026; Cline is the active community standard).

  1. Install Cline from the VS Code Marketplace.
  2. Open Cline settings (gear icon):
    • API Provider: Select Ollama or OpenAI Compatible.
    • Base URL: Set http://localhost:11434 (Ollama), http://localhost:1234/v1 (LM Studio), or http://localhost:1337/v1 (Jan AI).
    • Model ID: Enter laguna-xs-2.1:q4_K_M or devstral-small-2.
    • Enable Use Compact Prompt to preserve context budget.

Method C: Continue Extension

  1. Install Continue from the VS Code Marketplace.
  2. Configure ~/.continue/config.yaml (or config.json): 
    schema: v1
models:
  - name: Laguna XS 2.1 (Ollama)
    provider: ollama
    model: laguna-xs-2.1:q4_K_M
    apiBase: http://localhost:11434
    roles: [chat, edit, apply]
  - name: Devstral Small 2 (Ollama)
    provider: ollama
    model: devstral-small-2
    apiBase: http://localhost:11434
    roles: [chat, edit]
  - name: LM Studio Local Server
    provider: openai
    model: local-model
    apiBase: http://localhost:1234/v1
tabAutocompleteModel:
  name: LFM 2.5 Fast Autocomplete
  provider: ollama
  model: lfm2.5:8b-a1b-q8_0
  apiBase: http://localhost:11434

Method D: Official Microsoft / Ollama Extension

Install the official Ollama Extension from the VS Code Marketplace for direct one-click sidebar integration.

3. Aider CLI Pair Programmer Setup

Aider is a git-native CLI pair programmer that auto-commits code edits:

  1. Install Aider: 
    uv tool install aider-chat
  2. Configure Local Endpoint: 
    # Linux / macOS
export OLLAMA_API_BASE=http://127.0.0.1:11434

# Windows PowerShell
$env:OLLAMA_API_BASE="http://127.0.0.1:11434"
  3. Run Aider with Ollama Chat Prefix: 
    aider --model ollama_chat/devstral-small-2

Clone this wiki locally