Running models locally

Running models locally

Use malac with local models through Ollama or vLLM, including local embeddings for memory.

malac talks to local model servers through their OpenAI-compatible API. No key is needed; requests never leave your machine.

What matters most for a coding agent is tool calling: pick a model that supports it, and a context window large enough for real files (32k tokens or more).

Ollama

Install Ollama, pull a model that supports tool calling, and add Ollama as a provider:

ollama pull qwen3-coder
malac provider add local --base-url http://localhost:11434/v1 --model qwen3-coder --set-default

No --api-key-* flag is needed. Check that malac sees the models Ollama serves:

malac provider models local

Then run as usual:

malac explain the build setup in this repo

Context length

Ollama's default context length is much smaller than what a coding agent needs, and the OpenAI-compatible API cannot change it per request. Raise it on the Ollama side (for example with the OLLAMA_CONTEXT_LENGTH environment variable of the Ollama server, or a Modelfile with num_ctx), then tell malac the same number so compaction starts in time:

[models."local/qwen3-coder"]
context_window = 32768
max_output_tokens = 8192

Images

Most local coding models cannot read images. Mark them, and let another model describe screenshots:

vision_model = "local/llava"

[models."local/qwen3-coder"]
vision = false

vLLM

vLLM serves an OpenAI-compatible API, by default on port 8000. Start it with tool calling enabled; the parser depends on the model family (see the vLLM docs):

vllm serve <model> --enable-auto-tool-choice --tool-call-parser <parser>

[providers.vllm]
kind = "openai"
base_url = "http://localhost:8000/v1"
models = ["<model>"]

[models."vllm/<model>"]
context_window = 65536

Quote the model id in the [models."…"] table name; ids with slashes are fine there.

Local embeddings for memory

Vector memory needs an embedding model. Ollama serves embeddings on the same endpoint:

ollama pull nomic-embed-text

[providers.local]
kind = "openai"
base_url = "http://localhost:11434/v1"

[memory.embeddings]
provider = "local"
model = "nomic-embed-text"

This works even when your chat model is remote. See Vector memory with Milvus.

Mixing local and remote

Providers combine freely. A common setup keeps a strong remote model as the default and uses a local one for cheap work:

model = "lora/smart"
small_model = "local/qwen3-coder"   # compaction summaries stay local

[profiles.offline]
model = "local/qwen3-coder"

malac --profile offline then runs entirely on your machine. Subagents can also be pointed at a local model with the model argument of agent_spawn, or with model: in an agent type.