Tutorial: Offline Agentic coding with llama-server

[rujialiu]

Motivation

There are excellent commercial LLMs for agentic coding. However, using llama-server is still a very attractive option because maybe:

• You want a 100% offline environment (e.g. to make sure no private data is sent through network).
• You already run out of budget
• You want to do some unpredictable experiments (like using openevolve to discover new algorithms/optimize some critical code) that you have no idea how many tokens you will need, and don't want to risk.

2026-01-07 Update

Lots of things have changed since the original post, but I don't have time to rewrite this, so just some random hopefully useful updates.

New small(-ish) models you can try:

• gpt-oss-120b and gpt-oss-20b
• Qwen3-Coder and Qwen3-Coder-Flash (my new favorite)
• DevStral-2-Small (attention to the license if you want to try the larger variant)
• Qwen3-Next
• IQuest-Coder (very new. I haven't tried it yet because currently only 40B is released. smaller models are coming)

OpenCode becomes my new favorite. It's quite like Claude Code, works on Windows without WSL (still with some unimportant bugs though), and has multi-modality support.

However, currently it's not very llama.cpp-friendly. You need to create an opencode.json manually, like this:

{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "llama.cpp": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "llama-server (local)",
      "options": {
        "baseURL": "http://127.0.0.1:8090/v1"
      },
      "models": {
        "glm-4.6v-flash": {
          "name": "GLM-4.6V-Flash (local)",
          "modalities": { "input": ["image", "text"], "output": ["text"] },
          "limit": {
            "context": 64000,
            "output": 65536
          }
        }
      }
    }
  }
}

Then in opencode it should pickup your local model automatically. You can also change it with /models command.

Vision models that are known to work (image understanding + code writing + tool calling):

• Qwen3-VL series
• GLM-4.6V-Flash (however, sometimes it "thinks too much and failed to stop")

If you have a powerful GPU or Mac with a lot of RAM, you can also try bigger models. Just name a few newer models:

• MiniMax-2.1 (230B)
• MiMo-V2-Flash (309B)
• GLM-4.7 (360B)

If you're a GPU-poor like me:

• If your tasks are not very challenging, you may also try Qwen3-4B.
• If you need image support, you can serve a dedicated coding model and a Qwen3VL-4B for image understanding / Design2Code tasks with CPU in the same machine
• Use -np 1 -kvu with maximum context size you can afford, and rely on llama.cpp's smart context management & prompt caching (it's really good and still evolving)

If you can afford ~5GB VRAM ands wants a bigger model and wants fast pp/tg speed and willing to sacrifice output quality a bit, here is my (partial) command line to run Qwen3-Coder-Flash (30B-A3B):

-c 128000 -fa on -ngl 99 -ot ".ffn_.*_exps.=CPU" -ctk q4_0 -ctv q4_0 -np 1 -kvu -t 2 --temp 0.7 --top-k 20 --top-p 0.8 --min-p 0.0 --repeat-penalty 1.05 --jinja

With Qwen3-Coder-30B-A3B-Instruct-UD-Q4_K_XL.gguf from Unsloth, you'll get the following speed on 4070 SUPER:

• Initially: pp: ~400 token/s, tg: ~30 token/s
• When ~100k context is used: pp: ~300 token/s, tg: ~15 token/s

Which is quite good for a GPU-poor. But pay attention to q4_0 kv quat. It's not good. If you want higher quality, you have to use q8_0 or remove kv quant, but you have to add -nkvo which makes everything very slow.

LLM (needs update)

My personal favorite one is Devstral-Small-2507, which works great in real-world scenario (not just optimized for benchmarks or small programming problems). Key information

• 24B parameters
• 128k context
• Q4_K_M can fit completely in a 24GB-VRAM GPU with 128k context with 4-bit kv cache.
• Q2_K_L can fit completely in a 12GB-VRAM GPU with -nkvo

So if you have a 24GB GPU like 4090 and don't need concurrent access, start server like this:

llama-server.exe -a Devstral-Small-2507 -m Devstral-Small-2507-Q4_K_M.gguf -c 131072 -fa -ngl 99 -ctk q4_0 -ctv q4_0 --jinja

If you have 12GB a GPU like 3070 Super like me, the best combination I found, is to keep kv cache in RAM (with -nkvo) and use Q2_K_L. At first, I was hesitate to use such a low bit quantization and 4-bit kv quantization, but the result is surprisingly good (compared to my expectiation):

llama-server.exe -a Devstral-Small-2507 -m mistralai_Devstral-Small-2507-Q2_K_L.gguf -c 524288 -fa -ngl 99 -ctk q4_0 -ctv q4_0 -nkvo -np 4 --jinja

Note that since I already kept kv cache in RAM, and I have 64GB RAM, I use -np 4 to allow 4 concurrent access, each having 128k context (thus -c 524288 in total). This setup uses ~24GB RAM (but a little bit >32GB peak RAM).

If you have a less powerful GPU / less RAM, just reduce some of the numbers above. For example, if you have a 6GB GPU, you can use -ngl 20 (there are 41 layers in total), then llama-server will only use ~5.2GB VRAM at startup. If you have 8GB GPU, -ngl 30 (~7.2GB) might be good.

The used VRAM will slowly increase (like 0.2GB per hour), but it doesn't really matter a lot because you can just restart the server and most agentic softwares will retry and continue to work.

Bottom line: you can even use pure CPU inference to get some "background jobs" done. It's slow, but acceptable.

Also, you can try to reduce context size, but it looks like it's quite easy to have 80k~100k tokens in context. You might be able to get it working with 64k by clearing/compacting context frequently, but 32k seems to be too small.

Edit: as of July 31st 2027, another pair of attractive model is Qwen3-30B-A3B-Instruct-2507 and Qwen3-30B-A3B-Thinking-2507 with native 256k context and extensible to 1M.

So my personal recommendation is:

• Devstral-Small-2507
• Qwen3-30B-A3B-Instruct-2507
• Qwen3-30B-A3B-Thinking-2507

Software

There are already a couple of agentic coding softwares that supports OpenAI-compatible servers, so you can use llama-server with them without much effort:

• Cline
• Roo Code
• Aider
• OpenHands
• goose

With claude-code-router you can actually use llama-server served LLM in Claude Code!

For gemini-cli, it's still not clear whehther it'll support custom LLMs eventually (there are a few PRs not merged), but if you want to try today, you can check a fork with multi-provider support LLxprt Code

Usually, if you only use cline or similar softwares on your own machine, you don't need concurrent access; but if you use Claude Code, it will sometimes spawn multiple agents and make concurrent access.

Example: Using llama-server in Cline and Roo Code

Cline rarely (if at all) make concurrent LLM calls or very different (i.e. very short common prefix) consecutive requests, so it's ok to start your server without parallel sequence processing.

In Cline, setup your model like this:

• API Provider: OpenAI Compatible
• Base URL: http://127.0.0.1:8080/v1
• API Key: anything
• Model: (if your server is working, click here to pick one)

Expand "MODEL CONFIGURATION" setting, make sure the context windows size to match your server setting and adjust temperature according to your needs.

Roo Code is very similar, but a bit more control. I tend to reduce "match precision" in advanced settings if it fails to edit the code frequently.

This seems to be a very common problem to all coding agents, e.g.

cline/cline#4384

A concrete example

For quickstart, I recommend Qwen3-30B-A3B-Instruct-2507. For example:

https://huggingface.co/bartowski/Qwen_Qwen3-30B-A3B-Instruct-2507-GGUF/blob/main/Qwen_Qwen3-30B-A3B-Instruct-2507-IQ4_XS.gguf

And start the following task in Act mode:

Create a complete tetris game in a single HTML file using JavaScript and HTML5 Canvas. Keep the code as concise as possible while maintaining all core game mechanics.

A CPU-pnly server (you can add -v at the end for verbose output):

llama-server.exe -a Qwen3-30B-A3B-Instruct-2507 -m Qwen_Qwen3-30B-A3B-Instruct-2507-IQ4_XS.gguf -c 131072 -fa -ngl 0 -nkvo --jinja

Here is the speed on my machine, with the prompt above and an empty working directory. Your own numbers can be a baseline when you want to tune parameters.

prompt eval time =   31151.90 ms /  9965 tokens (    3.13 ms per token,   319.88 tokens per second)
       eval time =  580926.69 ms /  2765 tokens (  210.10 ms per token,     4.76 tokens per second)
      total time =  612078.59 ms / 12730 tokens

If that works, you can try to use some GPU. But if you don't have enough VRAM to fit the whole model, you can adjust parameters to balance speed and VRAM usage. For example this setup uses ~6.8GB VRAM. I always use LLAMA_SET_ROWS=1 to test the high-throughput mode #14363. It's not neccesary, but I recommend you do this and report bug if you have trouble:

set LLAMA_SET_ROWS=1
llama-server.exe -a Qwen3-30B-A3B-Instruct-2507 -m Qwen_Qwen3-30B-A3B-Instruct-2507-IQ4_XS.gguf -c 131072 -fa -ngl 99 -ot ".ffn_(up|down)_exps.=CPU" -ctk q8_0 -ctv q8_0 -nkvo --jinja

Here is the speed on my machine:

prompt eval time =   21334.71 ms /  9967 tokens (    2.14 ms per token,   467.17 tokens per second)
       eval time =  211964.32 ms /  2337 tokens (   90.70 ms per token,    11.03 tokens per second)
      total time =  233299.02 ms / 12304 tokens

If you want to use a thinking mode, try Qwen3-30B-A3B-Thinking-2507. It will produce much more tokens, but if you need high-quality code. it's worth trying.

Here is the speed on my machine with exactly the same prompt (but this time I'm using Qwen3-30B-A3B-Thinking-2507-UD-Q5_K_XL.gguf from unsloth, so it's slower). Attention to the number of tokens generated:

prompt eval time =   27062.12 ms /  9968 tokens (    2.71 ms per token,   368.34 tokens per second)
       eval time =  756723.70 ms /  7556 tokens (  100.15 ms per token,     9.99 tokens per second)
      total time =  783785.82 ms / 17524 tokens

Example: Using llama-server in Claude Code

Install (Current Claude Code version: 1.0.51):

npm install -g @anthropic-ai/claude-code
npm install -g @musistudio/claude-code-router

You only need to configure claude-code-router, with ~/.claude-code-router/config.json. You can find detailed description of settings in its documentation, but here is a minimal working example (attention: logs are huge! you may want to turn it off once you're happy with the settings):

{
  "LOG":true,
  "Providers": [
    {
        "name": "llamacpp",
        "api_base_url": "http://127.0.0.1:8080/v1/chat/completions",
        "api_key": "llamacpp",
        "models": ["Devstral-Small-2507"]
    }
  ],
  "Router": {
    "default": "llamacpp,Devstral-Small-2507",
    "background": "llamacpp,Devstral-Small-2507",
    "think": "llamacpp,Devstral-Small-2507",
    "longContext": "llamacpp,Devstral-Small-2507"
  }
}

Then use ccr code to launch claude-code-router along with Claude Code. Ask it to do something like "write a pacman game with pygame. Plan first, then write code." You will quickly get error messages in red if your setup doesn't work. For example, if you forgot the --jinja switch, you'll be told that your LLM doesn't support tool calling.

If everything goes on well, you'll see Claude Code wrote a TODO list, write some code, execute it, fixes bugs and update the TODO etc, with a lot of log messages from llama-server. Isn't that exciting? :)

PS: I'll update this tutorial when I have experience with more LLMs and softwares. Feedbacks and discussions are welcome!

[rujialiu]

Thanks @ggerganov for inviting me to write this!

[sayap]

Regarding the small requests from Claude Code, I feel -np 4 or even -np 2 is not feasible for the GPU poor, as each slot has the same context size, as noted in #4130 (reply in thread). For this scenario, ideally we need a big slot and a small slot, instead of 2 medium slots with the same context size.

For now, fronting 2 different llama-server instances with litellm is probably a good enough workaround.

[ggerganov]

Yeah, I realized this limitation of llama-server as well. I think it would be quite important to improve this in some way - wondering what the best solution is.

[rujialiu]

How hard/expensive is "migrating" slots? I mean, when a small slot is full, copy everything to a larger empty slot and clear the original slot. If that's cheap, maybe something like some 4k + 4k + 8k + 16k + 32k + 64k + 128k = 256k. We have 7 slots but the total size is only 2x the largest supported context. Or we can even do some mathematical optimization given the access pattern.

[ggerganov]

Not sure this is enough. I am playing with CC and I notice that sometimes it can start 3 parallel requests (maybe even more in some cases) with full context. I don't yet understand the logic completely - I think some of the requests are with stream: false.

But this indicates that parallel requests are not guaranteed to be small as I initially thought. Depending on these, improving support for saving and restoring the KV cache of the slots from disk might be a better option than adding "small slots".

[rujialiu]

+1 for "saving and restoring the KV cache of the slots from disk". CC and others support "background tasks" that intends to be low-priority but can be very token-intensive.

[am17an]

Pretty cool! I was looking for something like this. I also think there is need for an open source equivalent of Claude code we can iterate on with llama server

[am17an]

Probably time for llama.cpp community to build a MIT licensed alternative!

[rmatif]

There is also opencode, which is basically the same as Claude Code but open-source, MIT-licensed, and appears to be under active development

[rujialiu]

I actually stared opencode but it lacks Windows support so I didn't try. But on other platform it's definitel worth trying.

[rmatif]

They have a pre-built Windows binary in their releases section

[rujialiu]

Oh, I didn't notice that. I just saw anomalyco/opencode#631 is still open :)

[rujialiu]

Just for discussion: currently I don't have any VLM to recommend. I don't normally write front-end codes but for some in-house tools, it's really handy to convert a screenshot to html/css. Currently the best open source LLM for this is probably https://github.com/THUDM/GLM-4.1V-Thinking but it's not supported by llama.cpp yet. The best supported VLM is probably Qwen2.5VL and InternVL3, but they're far behind GLM-4.1V-Thinking when doing Design2Code-like tasks.

[rujialiu]

I've just tried https://github.com/acoliver/llxprt-code/

Install:

npm install -g @vybestack/llxprt-code

Run:

llxprt

Then in llxprt, type:

/provider openai
/baseurl http://localhost:8080/v1/
/model

And manually select the model from the list. That's all. I tried "Write a full-featured pacman game in python with pygame. Write specs to files, and write code according to the specs."

Seems to have much better windows support than OpenHands and Claude Code (I'm getting path issues and invalid "bash command" issues in both). UI is extremely similar, but some details are better than Claude Code. Console output is also much more helpful (I suspect it's also due to better Windows support).

Very good first impression!

[rujialiu]

Biggest disapointment: llxprt-code and gemini-cli doesn't support Claude Code's custom slash command. We only have GEMINI.md (and LLXPRT.md in case of llxprt). Luckily, there is a tracking issue on it google-gemini/gemini-cli#3789

[engrtipusultan]

Can you tell me how do you pass system prompt in llama.cpp server?

[engrtipusultan]

Yes thank you. there is a knob for manually giving jinja template. There is no knob for system prompt in server. I believe there is one for llama.cpp cli.
Do you know link to documentation about writing valid jinja template?

[rujialiu]

I think system prompt should be sent by the client, because it's very application-dependent (and different agent in the same app often use different system prompt). I never tried to write my own jinja template for an LLM but if you want to adapt an existing one, try to understand that existing template should be enough (along with official jinja project docs) @engrtipusultan

[engrtipusultan]

Conversation: valid input to the model is a conversation, which consists of a list of messages. Each message contains the following fields.

role (required): specifies the source of each message. As described in [Instructions and levels of authority](https://model-spec.openai.com/2025-02-12.html#levels_of_authority) and [The chain of command](https://model-spec.openai.com/2025-02-12.html#chain_of_command), roles determine the authority of instructions in the case of conflicts.
    system: messages added by OpenAI
    developer: from the application developer (possibly also OpenAI)
    user: input from end users, or a catch-all for data we want to provide to the model
    assistant: sampled from the language model
    tool: generated by some program, such as code execution or an API call

This is from openai. From this I inferred that system prompt is the one that you set while setting up the server. This is something client request cannot override.
(official jinja project docs) what do you mean by that? documentation for jinja or jinja acceptable by lama.cpp. Initially I looked at ollama and it does not accept all kinds of jinja template. If there is documentation on jinja acceptable by llama.cpp kindly point me towards it.

[rujialiu]

This is from openai. From this I inferred that system prompt is the one that you set while setting up the server. This is something client request cannot override.

The document was true long ago, but now it's misleading.

Let me explain it in detail. You already know that may models come with built-in jinja template. But what's exactly inside? Let's take a look. For example, devstral-small's chat template begins with:

{%- set default_system_message = 'You are Devstral, a helpful agentic model trained by Mistral AI and using the OpenHands scaffold. You can interact with a computer to solve tasks.\\n\\n<ROLE>\\nYour primary role is to assist users by executing commands, modifying code, and solving technical problems effectively. You should be thorough, methodical, and prioritize quality over speed.\\n* If the user asks a question, like \\"why is X happening\\", don\'t try to fix the problem. Just give an answer to the question.\\n</ROLE>\\n\\n<EFFICIENCY>\\n* Each action you take is somewhat expensive. Wherever possible, combine multiple actions into a single action, e.g. combine multiple bash commands into one, using sed and grep to edit/view multiple files at once.\\n* When exploring the codebase, use efficient tools like find, grep, and git commands with appropriate filters to minimize unnecessary operations.\\n</EFFICIENCY>\\n\\n<FILE_SYSTEM_GUIDELINES>\\n* When a user provides a file path, do NOT assume it\'s relative to the current working directory. First explore the file system to locate the file before working on it.\\n* If asked to edit a file, edit the file directly, rather than creating a new file with a different filename.\\n* For global search-and-replace operations, consider using `sed` instead of opening file editors multiple times.\\n</FILE_SYSTEM_GUIDELINES>\\n\\n<CODE_QUALITY>\\n* Write clean, efficient code with minimal comments. Avoid redundancy in comments: Do not repeat information that can be easily inferred from the code itself.\\n* When implementing solutions, focus on making the minimal changes needed to solve the problem.\\n* Before implementing any changes, first thoroughly understand the codebase through exploration.\\n* If you are adding a lot of code to a function or file, consider splitting the function or file into smaller pieces when appropriate.\\n</CODE_QUALITY>\\n\\n<VERSION_CONTROL>\\n* When configuring git credentials, use \\"openhands\\" as the user.name and \\"openhands@all-hands.dev\\" as the user.email by default, unless explicitly instructed otherwise.\\n* Exercise caution with git operations. Do NOT make potentially dangerous changes (e.g., pushing to main, deleting repositories) unless explicitly asked to do so.\\n* When committing changes, use `git status` to see all modified files, and stage all files necessary for the commit. Use `git commit -a` whenever possible.\\n* Do NOT commit files that typically shouldn\'t go into version control (e.g., node_modules/, .env files, build directories, cache files, large binaries) unless explicitly instructed by the user.\\n* If unsure about committing certain files, check for the presence of .gitignore files or ask the user for clarification.\\n</VERSION_CONTROL>\\n\\n<PULL_REQUESTS>\\n* When creating pull requests, create only ONE per session/issue unless explicitly instructed otherwise.\\n* When working with an existing PR, update it with new commits rather than creating additional PRs for the same issue.\\n* When updating a PR, preserve the original PR title and purpose, updating description only when necessary.\\n</PULL_REQUESTS>\\n\\n<PROBLEM_SOLVING_WORKFLOW>\\n1. EXPLORATION: Thoroughly explore relevant files and understand the context before proposing solutions\\n2. ANALYSIS: Consider multiple approaches and select the most promising one\\n3. TESTING:\\n   * For bug fixes: Create tests to verify issues before implementing fixes\\n   * For new features: Consider test-driven development when appropriate\\n   * If the repository lacks testing infrastructure and implementing tests would require extensive setup, consult with the user before investing time in building testing infrastructure\\n   * If the environment is not set up to run tests, consult with the user first before investing time to install all dependencies\\n4. IMPLEMENTATION: Make focused, minimal changes to address the problem\\n5. VERIFICATION: If the environment is set up to run tests, test your implementation thoroughly, including edge cases. If the environment is not set up to run tests, consult with the user first before investing time to run tests.\\n</PROBLEM_SOLVING_WORKFLOW>\\n\\n<SECURITY>\\n* Only use GITHUB_TOKEN and other credentials in ways the user has explicitly requested and would expect.\\n* Use APIs to work with GitHub or other platforms, unless the user asks otherwise or your task requires browsing.\\n</SECURITY>\\n\\n<ENVIRONMENT_SETUP>\\n* When user asks you to run an application, don\'t stop if the application is not installed. Instead, please install the application and run the command again.\\n* If you encounter missing dependencies:\\n  1. First, look around in the repository for existing dependency files (requirements.txt, pyproject.toml, package.json, Gemfile, etc.)\\n  2. If dependency files exist, use them to install all dependencies at once (e.g., `pip install -r requirements.txt`, `npm install`, etc.)\\n  3. Only install individual packages directly if no dependency files are found or if only specific packages are needed\\n* Similarly, if you encounter missing dependencies for essential tools requested by the user, install them when possible.\\n</ENVIRONMENT_SETUP>\\n\\n<TROUBLESHOOTING>\\n* If you\'ve made repeated attempts to solve a problem but tests still fail or the user reports it\'s still broken:\\n  1. Step back and reflect on 5-7 different possible sources of the problem\\n  2. Assess the likelihood of each possible cause\\n  3. Methodically address the most likely causes, starting with the highest probability\\n  4. Document your reasoning process\\n* When you run into any major issue while executing a plan from the user, please don\'t try to directly work around it. Instead, propose a new plan and confirm with the user before proceeding.\\n</TROUBLESHOOTING>\\n' %}
{{- bos_token }}
{%- if messages[0]['role'] == 'system' %}
    {%- if messages[0]['content'] is string %}
        {%- set system_message = messages[0]['content'] %}
    {%- else %}
        {%- set system_message = messages[0]['content'][0]['text'] %}
    {%- endif %}
    {%- set loop_messages = messages[1:] %}
{%- else %}
    {%- set system_message = default_system_message %}
    {%- set loop_messages = messages %}
{%- endif %}
    {{- '[SYSTEM_PROMPT]' + system_message + '[/SYSTEM_PROMPT]' }}

You can see that:

• it defines a default_system_message within the template
• client can override system message with a message whose role is system (the if messages[0]['role'] == 'system' part)

For a real example of how a complex client system message can be built, here is how agno (a popular agent framework) does it:

https://github.com/agno-agi/agno/blob/554a8d31972289888554c83d6cf697c208f43bde/libs/agno/agno/agent/agent.py#L4348

(official jinja project docs) what do you mean by that? documentation for jinja or jinja acceptable by lama.cpp.

Jinja is a popular python templating engine that existed long before LLM. People (including me) used Jinja to generate html files etc.
I don't think you need to learn "jinja acceptable by lama.cpp". Just look carefully into the template like I show you above and reason about it. That should be enough.

[engrtipusultan]

Thank you

[am17an]

@rujialiu I was following your tutorial, namely using claude code router + llama server with Devstral-Small-2507-Q4_K_M.gguf, I might've misconfigured something but when I ask it a simple question like "explain the speculative example in this repo" inside llama.cpp, it will do something like

● Search(pattern: "*speculative*", path: "examples")
  ⎿  Found 0 files

● {
    "tool_calls": [
      {
        "name": "Glob",
        "arguments": {
          "pattern": "speculative"
        }
      }
    ]
  }

But it won't actually make the tool call and just stop at this point. Did you face this issue?

[rujialiu]

I changed kv cache to Q8_0 from Q4_0 following your suggestion and observed some improvements when doing multi-file tasks (but no scientific numbers). But due to lack of RAM, I had to reduce -np 4 to -np 2. Will use this setup some more time. And since you have a mac with lot of memory, maybe DeepSeek-R1-0528 and/or its R1T2 variant is worth trying out. Also, I've heard that R2 is coming. @ggerganov

[rujialiu]

Today I've upgrade CC to 1.0.56. The windows support is greatly enhanced so no more path-related issues are obesrved. I don't know whether you're using windows, too, but you can try to upgrade and see whether we're still having trouble with Glob tool @am17an

[ggerganov]

And since you have a mac with lot of memory, maybe DeepSeek-R1-0528 and/or its R1T2 variant is worth trying out.

I don't think such big models make sense for local use at the context sizes needed here. I'm interested in more lightweight solutions that are practical.

[rujialiu]

I don't think such big models make sense for local use at the context sizes needed here. I'm interested in more lightweight solutions that are practical.

Ok, I understand. It's not practical for daily use yes, but I'm doing this in my office PC sometimes, let it write the initial version of some complex code for the whole night, then iterate on that (with lightweight models) when I'm back the next morning. Sometimes it works great (higher quality than lightweight model's output), sometimes it's too slow to finish its work before I come back. It's still an interesting use case I think.

[sayap]

{
  "tool_calls": [
    {
      "name": "Glob",
      "arguments": {
        "pattern": "speculative"
      }
    }
  ]
}

The suspect from #15186 (comment) is that Minja polyfills wrongly formats the tool calls and tool responses in this way, and after several back-and-forth, the models will then learn from the context to respond in this way as well.

Don't think the issue is caused by KV cache quantization.

[rujialiu]

I've just tried opencode v0.3.46 (thanks to @rmatif for telling me windows is already supported). While it frequently crashed (OOM?) with somewhat large projects (maybe not because of code size, but other big binary files that it should actually ignore), it works well with smaller projects.

Just download the binary, make sure ripgrep is on path (otherwise opencode will download it from github) and write an opencode.json in your project directory:

{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "llamacpp": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "llama.cpp (local)",
      "options": {
        "baseURL": "http://127.0.0.1:8080/v1"
      },
      "models": {
        "Devstral-Small-2507": {
          "name": "Devstral-Small-2507 (local)"
        }
      }
    }
  },
  "model": "llamacpp/Devstral-Small-2507"
}

I just tried OpenHands's repo, tell it to "summarize this repo", but I'm surprised that it changed README.md without asking me. But anyway, I will spend some more time and try to make best use of it.

BTW: I like its C/S design and TUI written in go.

[rujialiu]

Ok, I think it changed README.md because by default it's in BUILD mode. I switched to PLAN mode and it doesn't change anything now. However, it did Fetch https://github.com/sst/opencode (format=markdown) but the repository is OpenHands. Strange.

[rujialiu]

I'm been experimenting with Kimi-K2 and Qwen3-Coder for a few days, both quite good: K2 is better at enginnering and Qwen3-Coder seems to be a little bit better at producing complex but one-off scripts.

Since Qwen3-Coder's blog said "more sizes are coming", there's a good chance there will be some model similar in size to 30B-A3B, perfect for local use, and also some tiny models that can be used as draft models, so maybe we can experiment with speculative decoding after they're published?

I've heard that EAGLE-3 is worth trying (and supported by vllm and sglang), but I don't know which speculative decoding algorithm llama.cpp implements, and whether EAGLE-3 is planned or not. @ggerganov

[roykim98]

These folks look like they're using EAGLE-3 for speculative decoding here:

• https://www.baseten.co/blog/sota-performance-for-gpt-oss-120b-on-nvidia-gpus/#next-steps-in-performance-optimization

[rujialiu]

You can see section Example: Using llama-server in Cline and Roo Code, that's newly written for you. @marco-trovato

[rujialiu]

I hope I'll have some time to experiment with AST-based tools like https://github.com/zilliztech/code-context and/or https://github.com/ast-grep/ast-grep-mcp
For example, when I do some large-scale but conceptually simple refactoring, even Claude models missed lots of places, but actually a simple ast-grep should do a better job.

[lexasub]

https://github.com/lexasub/raged

[Eggwardhan]

have you tried qwen3-coder in claude-code-router?, I found it was not work
my launch llama-server code:
./build/bin/llama-server -m unsloth/Qwen3-Coder-480B-A35B-Instruct-GGUF/Q6_K/Q6_K/Qwen3-Coder-480B-A35B-Instruct-Q6_K-00001-of-00009.gguf --port 8000
--ctx-size 16384
--n-gpu-layers 99
--temp 0.7
-ot ".ffn_.*_exps.=CPU"
--min-p 0.0
--top-p 0.8
--top-k 20
--repeat-penalty 1.05
--jinja -a qwen-coder

my config is :

{
  "Providers": [
    {
      "name": "llamacpp",
      "api_base_url": "http://0.0.0.0:8000/v1/chat/completions",
      "api_key": "EMPTY",
      "models": ["qwen-coder"],

    }
  ],
  "Router": {
    "default": "llamacpp,qwen-coder",
    "background": "llamacpp,qwen-coder",
    "think": "llamacpp,qwen-coder",
    "longContext": "llamacpp,qwen-coder",
  },
  "LOG": true
}

~
the ccr occurs :
Error response from http://0.0.0.0:8000/v1/chat/completions: {"error":{"code":500,"message":"Unknown method: keys at row 57, column 46:\n {%- set handled_keys = ['type', 'description', 'enum', 'required'] %}\n {%- for json_key in param_fields.keys() | reject(\"in\", handled_keys) %}\n ^\n {%- set normed_json_key = json_key | replace(\"-\", \"_\") | replace(\" \", \"_\") | replace(\"$\", \"\") %}\n at row 57, column 81:\n {%- set handled_keys = ['type', 'description', 'enum', 'required'] %}\n {%- for json_key in param_fields.keys() | reject(\"in\", handled_keys) %}\n ^\n {%- set normed_json_key = json_key | replace(\"-\", \"_\") | replace(\" \", \"_\") | replace(\"$\", \"\") %}\n at row 57, column 13:\n {%- set handled_keys = ['type', 'description', 'enum', 'required'] %}\n {%- for json_key in param_fields.keys() | reject(\"in\", handled_keys) %}\n ^\n {%- set normed_json_key = json_key | replace(\"-\", \"_\") | replace(\" \", \"_\") | replace(\"$\", \"\") %}\n at row 46, column 80:\n {{- '\\n<parameters>' }}\n {%- for param_name, param_fields in tool.parameters.properties|items %}\n ^\n {{- '\\n<parameter>' }}\n at row 46, column 9:\n {{- '\\n<parameters>' }}\n {%- for param_name, param_fields in tool.parameters.properties|items %}\n ^\n {{- '\\n<parameter>' }}\n at row 39, column 29:\n {{- \"<tools>\" }}\n {%- for tool in tools %}\n ^\n {%- if tool.function is defined %}\n at row 39, column 5:\n {{- \"<tools>\" }}\n {%- for tool in tools %}\n ^\n {%- if tool.function is defined %}\n at row 36, column 51:\n{%- endif %}\n{%- if tools is iterable and tools | length > 0 %}\n ^\n {{- \"\\n\\nYou have access to the following functions:\\n\\n\" }}\n at row 36, column 1:\n{%- endif %}\n{%- if tools is iterable and tools | length > 0 %}\n^\n {{- \"\\n\\nYou have access to the following functions:\\n\\n\" }}\n at row 1, column 1:\n{% macro render_item_list(item_list, tag_name='required') %}\n^\n {%- if item_list is defined and item_list is iterable and item_list | length > 0 %}\n","type":"server_error"}}

[Eggwardhan]

My message only send "hello", so it may not related to context. And, when I use --ctx-size 256000, it still fails, the same error as above. I guess it may comes from the jinja template anylyze, or jinja not support keys() methods?

[rujialiu]

You only said "hello", but ccr added lots of system message before that, so it used lots of tokens already. But anyway you already enlarged contex size. Looking closer, it looks like llama.cpp's jinja engine doesn't support keys(). Looking at the implementation, I confirmed it doesn't support it (though items() is supported). Maybe you can open an issue? Though it looks not hard to implement it. See minja.hpp, search for this line "if (method->get_name() == "items") {".

[rujialiu]

You can also rewrite the template slightly

[rujialiu]

I'm waiting for a more lightweight version of qwen3-coder. When it's released (and I'm happy with its size), I'll start working with it. If the problem is still not solved, I'll do it and make a PR. @Eggwardhan

[rujialiu]

Ok, now Qwen3-Coder-Flash is released, 30B-A3B, non-thinking, as expected

[lexasub]

new opencode alternative - https://github.com/charmbracelet/crush (go)

[rujialiu]

Yes! Some one else also recommend this to me. It's on my TODO list. Thanks!

[rujialiu]

Crush is good overall, but I want to wait until charmbracelet/crush#453 is completed

[ggerganov]

The sst/opencode is doing a lot of things right, so probably deserves the top spot.

The claude code is now much simpler to use with the built-in anthropic base url setting.

@rujialiu Let me know if you plan to update this post. I can also create a new one if you don't have the time.

[rujialiu]

Yes, I'm planning to look into this, once I confirmed my current setup is able to finish some non-trivial project with minimal intenvention (I tried a lot of models and parameters so I even didn't finish a single project with it)

[rujialiu]

I modified llama-server to write the raw LLM input into files (with completion_id as key), and discovered following:
System prompt contains the following:

Here is some useful information about the environment you are running in:
<env>
  Working directory: E:\workspace\ai_workspace\testllamacpp_codingagent
  Is directory a git repo: yes
  Platform: win32
  Today's date: Sat Jan 03 2026
</env>
<files>
...
</files>

After the coding agent writes a new file, this part changed, so the common prefix is capped here. @ggerganov

[rujialiu]

hmmm... I would add that even if the agent didn't touch anything, that part can still change. The file system summary is unstable. For example:

				package-lock.json
				package.json
				[2 truncated]

vs

				[4 truncated]

I don't think we have much to do... maybe I should open an issue in opencode repository?

[rujialiu]

Ok, I see an issue anomalyco/opencode#5416 and a PR: anomalyco/opencode#5422

[rujialiu]

I've updated the tutorial a bit, new models to try, how to config coding models and image models in opencode and how to get maximum speed (and acceptable quality) with ~5GB VRAM. It's not very complete, but better than nothing 😄 @ggerganov

[pebaryan]

thanks! i finally tested today and uses vibe-kanban with claude code router. tried gpt4o-oss-20b, latest ministral 3 2512 and devstral 2 small, even Qwen3 4B Instruct seems quite ok. i am using only --parallel 2 and --cont-batching. i use multiple llama.cpp instance on separate gpu for smaller models

[pebaryan]

also, i tried the new opencode desktop app (beta) however i don't see any option for llama.cpp when i open connect provider dialog all available are cloud providers so i uninstall it

[rujialiu]

I haven't used ccr on windows for a while... But I'm working with opencode on windows without WSL everyday recently. You need to manually edit opencode.json in your project directory like this (it's manually truncated from my own json, tell me if it's not working):

{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "llama.cpp": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "llama-server (local)",
      "options": {
        "baseURL": "http://127.0.0.1:8080/v1"
      },
      "models": {
        "qwen3-coder-flash": {
          "name": "Qwen3-Coder-Flash (local)",
          "limit": {
            "context": 128000,
            "output": 65536
          }
        }
      }
    }
  }
}

Then in opencode you should be able to select that model in /model command.
Note that I'm not using desktop version. I'm installing opencode with bun 1.3.5

[pebaryan]

thanks! i finally made opencode working with llama.cpp . the opencode documentation is not quite explicit for important things (run opencode after install, config should be created manually)

[bssrdf]

thanks! i finally made opencode working with llama.cpp . the opencode documentation is not quite explicit for important things (run opencode after install, config should be created manually)

Can you share the config file for opencode? Which model do you use in llama.cpp? Thanks.

[rujialiu]

I've updated the post a little bit. You can see a usable config (with image support) and some suggested models you could try. @bssrdf

[jacekpoplawski]

I use OpenCode and Aider. I also recommend trying the Mistral Vibe CLI, it’s new but growing fast.

[rujialiu]

I think you mean this?
https://github.com/mistralai/mistral-vibe
I haven't tried it yet but since I'm a minimalist, I'm quite interested. It's only ~15000 loc (reported by scc) 😄

[lexasub]

any ideas about gastown (yet another orchestrator sdlc)?

[rujialiu]

Thanks! I've read an interview with steveyegge and heard of his beads (but never tried). It looks like gastown is built upon his own beads and it looks interesting!
Unfortunately, I'm quite busy recently like every Chinese else (in order to have a good Chinese New Year) so I think I'll try it 1 month later 😄

[pebaryan]

it would be nice i think to list a table of different coding tools and the length of the system prompt token and the list of models with maximum context length and different parameters/custom templates to make it work. for example, I have been using opencode since it only uses 11K token initially compared to claude which uses almost 32K. I tried small models and found that at least 7-8B are able to handle tool calls provided by the agentic tools for simple commands like (list files in current directory) before handling for more complex tool call and code reasoning.

[rujialiu]

Yes, that would be helpful, but personally I don't feel like doing this manually because they're all moving targets... I prefer 30B-A3B (BTW: there is a new released model, GLM-4.7-Flash you can try if you haven't) to 7B-8B because it's strong and faster when you have enough memory.

[am17an]

Qwen3.5 is quite usable at long contexts with OpenCode. I'm able to use 262k context using --n-cpu-moe 10 with 24 GB VRAM with the q4_k_m quant. Performance is not great but not terrible since it uses so much less kv-cache! @ggerganov I think we should try to enlist llamacpp-server as a provider to opencode, so people don't have to go through the hassle of adding the json config

[am17an]

Actually a simpler way would be to create a script which can catalog user's model from a model dir, and automatically create the json config. We already have some INI support if I'm not wrong for the llama-server, I think it should be easy to extend? cc @ngxson

[ngxson]

@am17an I think llama-cli --launch opencode will be a better UX, as it will be shipped with the builds we already had on brew/winget/etc

The --launch option will ideally install and launch opencode at the same time, but for simplicity, we can only install and not launching anything for now (so it will be the same with your python script)

This option will allow to extend llama-cli to "support" any local tools in the future

[rujialiu]

Are you using the original 397B model? I'm having good experience with 122B-A10B variant (which can run on much worse hardware) with harder tasks that previous Qwen3 model could never solve. And yes, it uses so much les kv-cache 😄

[am17an]

I was just using the 35B model, it is also pretty strong IMO, can run on my potato PC

[rujialiu]

Yes, 35B is definitely much better than Qwen3-30B and uses less kv-cache (no surprise). I just happily discovered 122B-A10B IQ2_M is useful for me with some harder tasks that 35B is unable to solve

[jacekpoplawski]

OpenCode has some serious issues, at least when working with llama.cpp anomalyco/opencode#23595
I use gemma 26B and with OpenCode I was able to make progress with my project but it's kind of hard

Now I use https://github.com/badlogic/pi-mono

some differences between OpenCode and pi:

• skills work (lots of problems with same repo in OpenCode)
• there is no long prompt preprocessing (like in OpenCode)
• it feels faster (shorter prompts?)
• I don't see problems with file edits

[emansom]

Working on better skills integration and Gemma 4 support here:

https://github.com/emansom/opencode/tree/gemma4-support
https://github.com/emansom/llama.cpp/tree/gemma4-fixes-rebase
https://github.com/emansom/llama.cpp/tree/llguidance-compat-rebased

It boils down to llama.cpp having a craptastic implementation for model output parsing (and where is the input validation??), and not guiding the model output itself properly. As well as some issues with the template that I still have to debug and resolve. It's a headache wrapping one's brain around the giant clusterfuck of code that is the jinja template handling, and the model specific human-error prone parser implementations.

All improvements will be submitted as a PR eventually by me, unless I get hit by a bus; if that happens, take my pushed code in the fork and upstream it.

OpenCode also needs to guide the tool parser a bit more, by submitting its own Lark grammar for specific tool call and tool response expected grammar. If that is in place, Gemma 4 models should perform much much better.

https://github.com/guidance-ai/llguidance/blob/main/docs/toolcalls.md

Not entirely sure if upstream developers agree, as it seems token per second is more important to them than actual useful implementation and behavior (see all the llama-bench results in PRs, instead of actual integration and e2e test results).

[rujialiu]

OpenCode has some serious issues, at least when working with llama.cpp anomalyco/opencode#23595 I use gemma 26B and with OpenCode I was able to make progress with my project but it's kind of hard

I hate Gemma4's templates so personally I use Qwen3.6-35B-A3B as coding agent much more (and use Gemma4 as a chatbot because it can sometimes give much better answers). Skills work, and file edits work (sometimes it fails the first attempt but later it will still succeed). But yes, pi-mono is a great alternative.

[jacekpoplawski]

If your local model is struggling with edits, I recommend trying the hashline edit idea. Originally, it was meant to use fewer tokens with cloud models, but I found that it also helps small local models edit files with fewer issues.