Repository navigation
Llama cpp mode
llama.cpp mode talks to llama-server, the local server that comes with llama.cpp. Dive can run that server for you: it downloads GGUF models from Hugging Face, keeps memory settings for each model, and loads and unloads models from Settings, so you don't need the terminal. If you already run your own llama-server, Dive can use that instead.
-
llama-server. On a Mac with Homebrew:brew install llama.cpp
Dive looks for
llama-serveron your PATH. If it lives somewhere else, enter its full path in Settings > MODELS > LLAMA-SERVER BINARY. -
A GGUF model. You can download one from inside Dive in step 2.
llama.cpp mode is switched on from the start, and it is the first icon in the top-left mode switcher. If you don't see it, open Settings, go to the MODES tab and tick it under ENABLED MODES.
- Open Settings and choose the MODELS tab. The tab only appears in llama.cpp mode.
- Under DOWNLOAD MODELS FROM HUGGING FACE, type a model name, for example
Qwen2.5 3B Instruct, and press SEARCH. - Pick a quantization and download it. Q4_K_M is the usual balance between quality and size.
The file is saved in your models folder, ~/models unless you change MODELS FOLDER. GGUF files you already have work too: put them in that folder.
Go to the MAIN tab. MODEL LIBRARY (MANAGED SERVER) lists the models in your folder. Press LOAD next to one.
Dive starts llama-server with that model on port 8130 and points llama.cpp mode at it. A large model can take a while; Dive waits up to five minutes for it to load.
What happens by default:
- Only one chat model uses memory at a time. Loading another one unloads the first (EVICT LOADED MODEL WHEN A NEW MODEL IS LOADED, on the MODELS tab).
- Embedding models, which the library uses for semantic search, run in a second server on the next port (8131). Loading a chat model never unloads them.
- A model gets a context window of 20,480 tokens, or less if it was trained for less, and all its layers go to the GPU. Change both per model on the MODELS tab with CONTEXT and GPU LAYERS: 99 puts everything on the GPU, 0 runs on the CPU only. Changes apply the next time you load the model.
Close Settings and type a message. The MODEL dropdown in the side panel lists what the server reports at /v1/models. Reasoning models show their thinking in a collapsible panel.
Ask something that needs a lookup:
What time is it in Tokyo right now?
The model calls the time_and_date tool and answers from the result. To run a tool yourself, use a slash command:
/calc 12 * 44
Most native tools start switched on. shell_command, macos_control and propose_plugin start switched off; turn them on in Settings > TOOLS. browse_act also starts off, but in 5.0.8 Settings has no switch for it or for browse_read. Tools that run code or act on a web page ask for your permission every time.
If you have skills in ~/dive/skills or ~/.agents/skills, type /skill: followed by the skill's name and your text, for example /skill:proofreader Please check this paragraph. Create a skill shows how to write one.
| Setting | What it does | Default |
|---|---|---|
| MODEL LIBRARY (MANAGED SERVER) | Your models, with LOAD and UNLOAD. LOAD uses a llama.cpp router already listening on the port if there is one, and otherwise starts llama-server inside Dive |
|
| MODEL | The model to chat with, from the server's /v1/models. Refresh models reloads the list |
|
| NATIVE TOOL CALLING (OPENAI SCHEMA) | Sends tools as structured function schemas. The server must run with --jinja, which Dive adds to servers it starts. If the server refuses the tools, Dive falls back to the text prompt |
On |
| AGENT MODE | Plan-first prompting and a larger tool budget. Tool calls appear as a step timeline in the chat | Off |
| AGENT TOOL BUDGET | Tool calls per request in Agent Mode, from 1 to 50. Without Agent Mode the limit is 6 | 25 |
| COLOR PALETTE, FONT FAMILY, FONT SIZE | How this mode looks | |
| SAMPLING PARAMETERS | Sent with every request |
temperature 0.2, top_p 0.5, top_k 30, min_p 0.05, repeat_penalty 1.1, presence_penalty 0, frequency_penalty 0, max_tokens -1, seed -1 |
| Setting | What it does | Default |
|---|---|---|
| EVICT LOADED MODEL WHEN A NEW MODEL IS LOADED | Unloads the running chat model before loading another. When off, a load is refused until you stop the running model | On |
| LOAD LAST MODEL WHEN DIVE STARTS | Reloads the last chat model when the Dive server starts | Off |
| MODEL LIBRARY | For each model: CONTEXT, GPU LAYERS, EMBED (marks it as an embedding model) and × (deletes the file). The advanced options add THREADS, BATCH, FLASH ATTENTION, KEEP IN RAM (MLOCK), KV CACHE K, KV CACHE V and MERGE SYSTEM PROMPTS | CONTEXT 20,480, GPU LAYERS 99 |
| DOWNLOAD MODELS FROM HUGGING FACE | Searches GGUF repositories and downloads the quantization you pick | |
| MODELS FOLDER | The folder scanned for .gguf files. Downloads go here |
~/models |
| SERVER PORT | The managed chat server's port. The embedding server uses the next one | 8130 |
| LLAMA-SERVER BINARY | The full path to llama-server. Empty means search the PATH |
Empty |
| EXTRA ARGUMENTS | Flags added to every launch, for example --no-mmap
|
Empty |
| SERVER URL (ADVANCED) | The address llama.cpp mode chats with. Dive sets it when you LOAD a model; change it only for a server you run yourself | |
| ROUTER PRESET SYNC (ADVANCED) | Keeps a router's --models-preset files in step with your models folder |
Only runs when a router is found |
Picking a quantized KV cache (anything other than f16) turns flash attention on automatically, because llama.cpp needs it for that.
- Start it with
--jinjaif you want native tool calling. - On the MODELS tab, set SERVER URL (ADVANCED) to its address, for example
http://127.0.0.1:8080/v1. If you leave outhttp://or/v1, Dive adds them. - If your server is a llama.cpp router started with
--models-presetand listening on the SERVER PORT, LOAD asks the router to load the model instead of starting another server, and UNLOAD frees the router's memory. ROUTER PRESET SYNC then keeps the router's.inipreset files in step with your downloads and deletions. Dive reads the preset path from the router's own command line, and restarts the router by ending its process, which assumes launchd brings it back. - The README section llama.cpp Ideal Settings walks through running routers like that as macOS LaunchAgents.
For developers and agents:
-
routes/llamacpp.jsruns two server slots:chaton the configured port andembeddingon the port plus one. Its settings live in~/dive/llamacpp.json. - A managed launch runs
llama-server -m <file> -c <context> -ngl <gpu layers> --host 127.0.0.1 --port <port>. It adds--jinjafor a chat model or--embeddingfor an embedding model,--mmprojwhen a matching vision projector is found,-t,-b,--mlock,-fa on,-ctkand-ctvfrom the model's advanced options, and then EXTRA ARGUMENTS. - Vision projectors are matched by download provenance and by the
general.namein the GGUF headers, never by filename. When nothing identifies a single projector, none is attached. Seeroutes/llamacpp-projector.js. - Chat goes through
/api/llamacpp/streamandhandleLocalModeStream()inroutes/chat.js, the same code LM Studio mode uses.routes/llamacpp-derived.jsis the only code allowed to write the derived copies of "which model is loaded, on which port" intolocal-model-settings.jsonandlibrary-config.json. - Changes are announced on
/api/events/globalasllamacpp_models_changedandllamacpp_selection_changed, so every open window updates.
| Message or symptom | What to do |
|---|---|
| "llama-server binary not found. Install it (brew install llama.cpp) or set its path in the llama.cpp settings." | Install llama.cpp, or fill in LLAMA-SERVER BINARY |
| "Port 8130 is already in use by another llama-server (not managed by this Dive instance)." | Stop the other server, or change SERVER PORT |
| A load is refused while another model is running | EVICT LOADED MODEL WHEN A NEW MODEL IS LOADED is off. Stop the running model, or switch the setting on |
| The model fails with "System message must be at the beginning" | Tick MERGE SYSTEM PROMPTS in that model's advanced options |
| A vision model refuses images | Put the model's mmproj file in the models folder. If Dive can't tell which projector belongs to the model, it attaches none |
| The model never calls tools | Check NATIVE TOOL CALLING. Some models follow the text prompt better, so try switching it off |
These pages describe Dive 5.0.8. Source code: DuckTapeKiller/dive. Spotted a mistake? Open an issue.
Get started
Tools and skills
Build your own
Share and stay safe
How Dive works
Help