How to actually run Qwen 3.8 with Claude Code?

[mankeli]

After trying some unsloth qwen gguf and not understanding what was missing, I tried to follow #27080

I used cmdline llama serve --host 0.0.0.0 \ -hf ggml-org/Qwen3.8-27B-GGUF:Q4_K_M \ -hfd ggml-org/Qwen3.8-27B-GGUF:Q4_0 \ --spec-default \ --spec-type draft-mtp \ --reasoning-preserve

But this doesn't work either. llama just prints

34.27.081.920 W srv operator(): got exception: {"error":{"code":500,"message":"\n------------\nWhile executing CallExpression at line 106, column 32 in source:\n...first %}↵ {{- raise_exception('System message must be at the beginnin...\n ^\nError: Jinja Exception: System message must be at the beginning.","type":"server_error"}}

Now, after actually testing more, this seems to be related to Claude Code, since the web interface/cli works. I've been using 3.6 with some qwen3.6_chat_template.jinja I downloaded somewhere, but I didn't realize that the template needs to be harness specific?

Does anyone have good reference on explaining how these are supposed to work? I'm also very confused which models are supposed to be run with -fa for example.

[bogdanr]

The exception comes from the model chat template, not from llama.cpp. The template accepts only one system message, and that message must be first.

The rule in the template

chat_template.jinja of Qwen3.8-27B, lines 102-107:

{%- for message in messages %}
    {%- set content = render_content(message.content, true)|trim %}
    {%- if message.role == "system" %}
        {%- if not loop.first %}
            {{- raise_exception('System message must be at the beginning.') }}
        {%- endif %}

The header block (lines 57-87) writes messages[0] if it is a system message. The main loop above then ignores that first message. Any other system message stops the template.

Why Claude Code is different from the web UI

Claude Code uses the Anthropic endpoint /v1/messages. llama-server changes that request into the OpenAI format in server_chat_convert_anthropic_to_oai(), in tools/server/server-chat.cpp.

First it puts the top-level system parameter at index 0:

auto system_param = json_value(body, "system", json());
if (!system_param.is_null()) {
    ...
    oai_messages.push_back({
        {"role", "system"},
        {"content", system_content}
    });
}

Then it copies each item of messages. It keeps the role of each item, and it does not examine the role:

if (content.is_string()) {
    oai_messages.push_back(msg);
    continue;
}

Thus, if a request contains the top-level system parameter and a system message in messages, the result has two system messages. The second one is not first, and the template stops.

The web UI does not fail, because it sends only one system message.

How to confirm

Start the server with more log detail:

llama-server ... -lv 4

Then examine the messages of the failed request. Count the items that contain "role": "system". Two such items are the cause.

How to correct it

Give the server a patched template. Copy the template from the model, then change lines 104-107. Write the later system message instead of stopping:

{%- if message.role == "system" %}
    {%- if not loop.first %}
        {{- '<|im_start|>system\n' + content + '<|im_end|>' + '\n' }}
    {%- endif %}

Start the server with the new file:

llama-server ... --jinja --chat-template-file ./qwen3.8-patched.jinja

Your other two questions

Is a template specific to the harness? No. The template is a property of the model, and llama.cpp reads it from the GGUF metadata. But each template has different rules. This template is strict, thus a harness that sends two system messages fails.

Which models need -fa? None of them need it. The default is auto:

-fa, --flash-attn [on|off|auto]   set Flash Attention use ('on', 'off', or 'auto', default: 'auto')

Keep the default, and llama.cpp makes the decision for the model and the backend.

[mankeli]

If the template enforces a single system message, doesn't that mean that the model might not work properly if the template is hacked not enforce this?

[DanoPTT]

I use modified template too (same problem), and it works with claude code flawlesly.

[mankeli]

Thanks. Is it same template as with 3.6? I'm trying to extract the template to fix it myself but the extractor seems to want to connect huggingface instead of extracting the template from the .gguf

[mankeli]

But so, if the need of flash attention and other params are things that's stored in the .gguf, why does the example specify all this on cmdline? ./llama-server -hf Qwen/Qwen3-8B-GGUF:Q8_0 --jinja --reasoning-format deepseek -ngl 99 -fa -sm row --temp 0.6 --top-k 20 --top-p 0.95 --min-p 0 -c 40960 -n 32768 --no-context-shift (source https://github.com/QwenLM/Qwen3/blob/main/docs/source/run_locally/llama.cpp.md )

[bogdanr]

Is it safe to remove the guard?

Yes, but you lose content. In the main loop the system branch writes nothing (lines 104-107), because the header block at lines 77-86 already wrote messages[0]. If you delete only the raise_exception, the second system message is discarded in silence. That is why a patched template works — nothing malformed goes to the model.

To keep the content, write the block instead:

{%- if message.role == "system" %}
    {%- if not loop.first %}
        {{- '<|im_start|>system\n' + content + '<|im_end|>' + '\n' }}
    {%- endif %}

How to get the template out of the GGUF

Start the server, then read GET /props. The chat_template field is the original template of the model. This does not use HuggingFace.

curl -s localhost:8080/props | jq -r .chat_template > tpl.jinja

Then start the server with --chat-template-file tpl.jinja.

Why the example sets sampling parameters

llama.cpp does read general.sampling.* from the GGUF (common/common.cpp:1212-1270). A command-line flag overrides the file value. Without the flag, the file value applies. But many GGUF conversions do not include these keys, thus the Qwen example sets them.

-fa is no longer necessary. The default is auto. -ngl, -c, -sm and -n are decisions about your hardware, not properties of the model.

To see which keys your file has:

python gguf-py/gguf/scripts/gguf_dump.py --no-tensors model.gguf | grep sampling

[AmmarkoV]

Managed to run the current master (ee4c505) with the latest Claude Code ( v2.1.237 ) without any problems using this patch :
anthropic-system-merge.patch

using

./build/bin/llama-server \
      -hf unsloth/Qwen3.8-27B-GGUF:UD-Q6_K_XL \
      --alias "Qwen" \
      --host 0.0.0.0 --port 7860 \
      --flash-attn on \
      --cache-type-k q4_0 \
      --cache-type-v q4_0 \
      --ctx-size 200000 \
      --batch-size 1024 \
      --ubatch-size 1024 \
      --jinja

[mankeli]

thanks, this works!

[DanoPTT]

I analyze patch anthropic-system-merge.patch against a real Claude Code session before adopting it, and it
turns out to be a significant regression for prefix-cache reuse. Sharing the numbers
since they weren't obvious to me up front.

Setup: logging proxy in front of llama-server, real Claude Code session,
Qwen3.8-27B (Q5_K_XL) on ROCm, ctx 180224. 55 requests captured, largest
conversation 50 turns. Baseline is a patched chat template that renders a
non-leading system message in place instead of raising.

How Claude Code actually sends system turns:

• 50 of 55 requests carried a system turn inside messages — not an edge case.
• The count grows monotonically with the session: 1, 2, 3 ... 40 system turns,
  at indices 1, 4, 7, 10, ... — one appended per user turn, always at the tail.
• History is never rewritten: the sha of the first 11 system blocks is identical
  at turn 11 and at turn 50.
• In 34 of 50 increments the appended block was exactly 49 chars,
  <total_tokens>N tokens left</total_tokens>. N changes each turn, so every new
  block is a distinct string.
• Simulating this patch, the merged leading system message would differ from the
  previous request in 39 of 51 turns (76%).

Cost: the patch inserts changing content at offset 0, so the common prefix
collapses and the whole context is re-prefilled. Over those same 55 requests:

	actual (in-place template)	simulated (this patch)
prefill tokens	380k	~2.6M
avg prefill/request	6.9k	~67.7k (full prompt)

Full prompt averaged 67.7k tokens (max 82.5k), so ~90% of the context is reused
today. At the prefill rate measured at that depth (~420–460 t/s: 8795 tok/19.1 s,
3673 tok/8.8 s), the patch works out to roughly +150 s TTFT on each of those 39
turns — about 7x the prefill work for one session.

Suggestion: the merge itself is the right idea — doing it in the converter is
much cleaner than patching every template, and it fixes the OAI path users hit too.
But merging into the last system position (or leaving each system turn where it
is and letting the template emit it there) would preserve the prefix while still
handing templates a shape they accept. Hoisting to the front is what costs the cache,
and the clients that trigger this bug are exactly the ones that append system turns
every turn.

[mankeli]

I wonder if the messages-api requests can be translated into completions-api requests fully?

[realugbun]

This is my current working configuration on V100 32GB.

I updated my chat templates to the one in this repo which fixes several issues. https://github.com/jschmied/Qwen-Fixed-Chat-Templates

It runs behind my LiteLLM proxy with the qwen3.8-dense alias and I launch Claude Code with the command below.

Claude Code launch command.

env -u CLAUDECODE -u CLAUDE_CODE_ENTRYPOINT \                                                                                                                   
    ANTHROPIC_BASE_URL=https://litellm.example.com \                                                                                                                
    ANTHROPIC_AUTH_TOKEN="$LITELLM_API_KEY" \                                                                                                              
    ANTHROPIC_API_KEY= \                                                                                                                                          
    ANTHROPIC_SMALL_FAST_MODEL=qwen3.8-dense \                                                                                                                    
    claude --model qwen3.8-dense

lamma.cpp launch command

GGML_CUDA_DISABLE_GRAPHS=1 \                                                                                                                                                  
  /opt/llama.cpp/build/bin/llama-server \                                                                                                                                       
    -m /opt/models/Qwen3.8-27B-UD-Q4_K_XL.gguf \                                                                                                                                
    --alias Qwen3.8-27B-UD-Q4_K_XL.gguf \                                                                                                                                       
    --mmproj /opt/models/mmproj-Qwen3.8-F16.gguf \                                                                                                                              
    --jinja \                                                                                                                                                                   
    --image-min-tokens 1024 \                                                                                                                                                   
    --spec-type draft-mtp --spec-draft-n-max 2 \                                                                                                                                
    -ngl 99 -fit off -fa on \                                                                                                                                                   
    -ctk q8_0 -ctv q8_0 \                                                                                                                                                       
    -c 294912 --parallel 2 \                                                                                                                                                    
    -b 2048 -ub 512 \                                                                                                                                                           
    --cache-prompt --cache-reuse 256 \                                                                                                                                          
    --metrics \                                                                                                                                                                 
    --host 0.0.0.0 --port 8080

[mankeli]

GGML_CUDA_DISABLE_GRAPHS=1 \                                                                                                                                                  
  /opt/llama.cpp/build/bin/llama-server \                                                                                                                                       
    -m /opt/models/Qwen3.8-27B-UD-Q4_K_XL.gguf \                                                                                                                                
    --alias Qwen3.8-27B-UD-Q4_K_XL.gguf \                                                                                                                                       
    --mmproj /opt/models/mmproj-Qwen3.8-F16.gguf \                                                                                                                              
    --jinja \                                                                                                                                                                   
    --image-min-tokens 1024 \                                                                                                                                                   
    --spec-type draft-mtp --spec-draft-n-max 2 \                                                                                                                                
    -ngl 99 -fit off -fa on \                                                                                                                                                   
    -ctk q8_0 -ctv q8_0 \                                                                                                                                                       
    -c 294912 --parallel 2 \                                                                                                                                                    
    -b 2048 -ub 512 \                                                                                                                                                           
    --cache-prompt --cache-reuse 256 \                                                                                                                                          
    --metrics \                                                                                                                                                                 
    --host 0.0.0.0 --port 8080

Shouldn't lot of these options be set automatically from the .gguf metadata?

[realugbun]

Yes, several of them are set but the gguf metadata but my setup diverges in several places due to how old the V100 is and my desire to squeeze as much as I can out of it.

Also, I manage my deployments with infrastructure as code so I prefer to be explicit about exactly what is running.

[mankeli]

that makes sense. thanks for sharing!

[AmmarkoV]

Thanks @realugbun
Maybe try this:

anthropic-system-merge-2.patch

against the current master (ee4c505)
Looking forward to your feedback!

[mankeli]

I wonder what are the pros/cons of a fix like this, or modified template like https://github.com/jschmied/Qwen-Fixed-Chat-Templates ?

The patch itself seems to work fine with claude code harness.

[AmmarkoV]

¯_ (ツ)_/¯ I suppose our friend @realugbun could maybe try both and get some benchmarks ?

[bogdanr]

@mankeli The template is prefix-safe. Patch v2 is nearly prefix-safe, but not for Claude Code.

jschmied/Qwen-Fixed-Chat-Templates does two separate things (chat_template.jinja):

• Line 47: merges only a leading consecutive run of system/developer turns. The condition is sysns.count == loop.index0, so it stops at the first non-system message.
• Line 151: renders a later system turn in place instead of raising.

Nothing moves to offset 0, so the prefix stays stable. This is the "in-place" baseline @DanoPTT measured as good.

anthropic-system-merge-2.patch folds a non-leading system turn into the next message, which preserves the prefix. But a trailing system turn has no next message, so it falls back to merging into the leading system message. @DanoPTT measured that Claude Code appends its system turn at the tail every turn. So that fallback runs on nearly every request, and v1's full re-prefill comes back.

@AmmarkoV fold the leftover into the last message, not the first. For an append-at-tail client both are new that turn, so the prefix survives either way.

Trade-off: the patch needs no template maintenance and also fixes the OAI path. The template needs no rebuild.

[realugbun]

The Qwen-Fixed-Chat-Templates also sets the default reasoning to medium instead of xhigh. This can prevent surprises like sending minimal thinking in the request and getting xhigh.

[DanoPTT]

My observation regarding the "quality" of the produced C# code is that switching thinking off yields fewer errors and a faster response.

[mankeli]

I think the quality highly depends on the task. But why isn't it enough that agent harness sets the reasoning effort?