Skip to content

LM Studio Guide

Jump to bottom
rookiemann edited this page Mar 15, 2026 · 1 revision

Part 5: LM Studio — Local AI Models

What is LM Studio?

LM Studio is a free application that lets you run AI models directly on your computer using your NVIDIA GPU. Models run entirely locally — no data is sent to the internet.

Why Use Local Models?

  • Privacy — conversations stay on your computer
  • No cost — no per-message fees after downloading the model
  • Offline — works without internet
  • Speed — can be faster than cloud for small models on a good GPU
  • Experimentation — try different models, quantizations, and settings

The LM Studio Panel

Open via Tools > LM Studio in the menu bar. The panel shows:

  • Status indicator — green dot = LM Studio is running
  • Model browser — list of all downloaded models
  • GPU selector — choose which GPU to load the model on
  • Context length slider — how much text the model can remember (512 to 131,072 tokens)
  • Load/Unload buttons — manage what's in GPU memory
  • Use for Chat — switch Hermes to use this model

Step-by-Step: Loading Your First Local Model

  1. Start LM Studio — open LM Studio on your computer
  2. Download a model if you haven't already:
    • In LM Studio, go to the Discover tab
    • Search for "Qwen 2.5 7B" (good starter model)
    • Click Download on a Q4 or Q6 quantization
  3. Open the LM Studio panel in Hermes (Tools > LM Studio)
  4. Wait for status — the green dot should appear showing LM Studio is connected
  5. Select your model from the model browser list
  6. Choose your GPU — if you have multiple GPUs, pick the one with more VRAM
  7. Set context length — start with 4096 (uses less memory) or 8192 (remembers more)
  8. Click Load — the model loads into your GPU memory
  9. Click Use for Chat — Hermes switches to using this model
  10. Start chatting! — your messages now go to the local model

Understanding VRAM (GPU Memory)

Your GPU has a fixed amount of VRAM. Models need to fit in VRAM to run:

Model Parameters Q4 VRAM Q6 VRAM Q8 VRAM
1-3 Billion ~2 GB ~2.5 GB ~3 GB
7-8 Billion ~4 GB ~6 GB ~8 GB
13-14 Billion ~7 GB ~10 GB ~13 GB
27-34 Billion ~18 GB ~24 GB ~34 GB
70 Billion ~40 GB ~55 GB ~70 GB

Context length also uses VRAM. Higher context = more VRAM used.

Example: You have an RTX 3060 with 12 GB VRAM.

  • Qwen 2.5 7B Q6 = ~6 GB + ~1 GB for context = fits easily
  • Qwen 2.5 14B Q4 = ~9 GB + ~1 GB for context = fits tightly
  • Llama 3.1 70B = too large, won't fit

GPU Isolation (Multi-GPU Systems)

If you have multiple GPUs, Hermes loads models on a single GPU to prevent "model splitting" — where a model is spread across GPUs, which hurts performance.

When you select a GPU in the LM Studio panel or use the lm_studio_load tool, Hermes automatically disables all other GPUs for that load to force single-GPU loading.

LM Studio Tools (Available in Chat)

When LM Studio is running, these tools become available:

Tool What to Say What It Does
lm_studio_status "Check LM Studio" Shows if it's running, loaded models, GPU memory
lm_studio_models "List my local models" Shows all downloaded models
lm_studio_load "Load Qwen 7B on my 3090" Loads a model onto a specific GPU
lm_studio_unload "Unload the current model" Frees GPU memory
lm_studio_search "Search for Llama models" Searches HuggingFace for downloadable models
lm_studio_download "Download that model" Downloads a model from HuggingFace
lm_studio_model_info "What model is loaded?" Shows loaded model details (context, config)
lm_studio_tokenize "How many tokens in this text?" Counts tokens in text
lm_studio_embed "Generate embeddings for this text" Creates vector embeddings
lm_studio_chat "Ask the local model: what is 2+2?" Sends a prompt directly to a loaded model

Switching Between Cloud and Local

You can switch at any time:

To local model: Say: "Switch to LM Studio" or "Use my local model" Hermes will set OPENAI_BASE_URL to your LM Studio server.

To cloud (OpenRouter): Say: "Switch to OpenRouter" or "Use cloud model" Hermes will set OPENAI_BASE_URL back to OpenRouter.

Or use File > Settings to change the model and provider manually.

Troubleshooting LM Studio

"LM Studio not running" or "Connection refused":

  • Make sure LM Studio is open and the server is started (Developer tab)
  • Check the port number — it should match what Hermes expects
  • The port is auto-detected from your OPENAI_BASE_URL in .env

"Model won't load — out of memory":

  • Try a smaller quantization (Q4 instead of Q8)
  • Reduce context length (4096 instead of 8192)
  • Close other GPU-intensive applications
  • Check GPU memory with the gpu_info tool

"Responses are very slow":

  • Reduce context length (smaller = faster)
  • Use a smaller model
  • Make sure no other models are loaded (check with lm_studio_status)

"Model gives bad/random responses":

  • Try a different model — quality varies significantly between models
  • Increase context length if conversations are being cut off
  • Make sure you're using an "instruct" or "chat" variant, not a base model

Clone this wiki locally