Eval bug: Gemma4 26B A4B repetition at high context

[aldehir]

Name and Version

$ ./llama-server --version
ggml_cuda_init: found 1 CUDA devices (Total VRAM: 81079 MiB):
  Device 0: NVIDIA H100 80GB HBM3, compute capability 9.0, VMM: yes, VRAM: 81079 MiB
version: 8763 (ff5ef8278)
built with GNU 13.3.0 for Linux x86_64

$ nvidia-smi
Sun Apr 12 08:56:27 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.126.09             Driver Version: 580.126.09     CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA H100 80GB HBM3          On  |   00000000:5D:00.0 Off |                  Off |
| N/A   29C    P0            117W /  700W |    1115MiB /  81559MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A           51217      C   ./build/bin/llama-server               1106MiB |
+-----------------------------------------------------------------------------------------+

Operating systems

Linux

GGML backends

CUDA

Hardware

Device 0: NVIDIA H100 80GB HBM3, compute capability 9.0, VMM: yes, VRAM: 81079 MiB

Models

ggml-org/gemma-4-26B-A4B-it-GGUF:Q8_0
bartowski/google_gemma-4-26B-A4B-it-GGUF:Q4_K_M
bartowski/google_gemma-4-26B-A4B-it-GGUF:F16
unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_M

Problem description & steps to reproduce

At high context, I noticed there is still a chance of repetition. Not sure if it's a model problem, but I was able to reproduce it with ggml-org's Q8_0, bartowski's Q4_K_M & F16, and Unsloth's UD-Q4_K_M.

I have seen this occur at high context with the recommended sampling params (top-k = 64, temp = 1.0, top-p = 0.95). Although usually it's a repetition of multiple lines. I caught this with top-k = 1, temp = 0.0 in case it's a useful data point for reproduction.

I could not reproduce this on Windows, only Linux.

Run server with:

./build/bin/llama-server -m /workspace/models/gemma-4-26B-A4B-it-Q8_0.gguf --chat-template-file models/templates/google-gemma-4-31B-it.jinja -c 256000 -np 1 --verbose --top-k 1 --temperature 0 -cram 0 -dio --no-cache-prompt

repro: https://github.com/aldehir/gemma4-26b-repros

Reproduce via /completions with the prompt already generated from the updated template by Google

$ OPENAI_BASE_URL=http://localhost:7000/v1 ./run-completions.py request-02-raw.json
<|channel>thought
Task 12 is complete.

**Task 13: Parser & Engine � `addcol`**

I've already implemented `addcol` in `parser.py` and `engine.py`.
Let's verify with a test.
I'll add a test to `test_engine.py`.

Wait, I already have `test_execute_addcol` in `test_engine.py`? No, I don't.
I'll add it.

Wait, I'll add it to `test_engine.py`.
Actually, I'll add it to `test_expr.py`? No, `test_engine.py` is better.

Wait, I'll add it to `test_engine.py`.
Actually, I'll add it to `test_engine.py`.
Wait, I'll add it to `test_engine.py`.
Actually, I'll add it to `test_engine.py`.
Actually, I'll add it to `test_engine.py`.
Actually, I'll add it to `test_engine.py`.
Actually, I'll add it to `test_engine.py`.
...

Reproduce via /chat/completions, use models/templates/google-gemma-4-31B-it.jinja template

$ OPENAI_BASE_URL=http://localhost:7000/v1 ./run.py request-02.json
Task 12 is complete.

**Task 13: Parser & Engine � `addcol`**

I've already implemented `addcol` in `parser.py` and `engine.py`.
Let's verify with a test.
I'll add a test to `test_engine.py`.

Wait, I already have `test_execute_addcol` in `test_engine.py`? No, I don't.
I'll add it.

Wait, I'll add it to `test_engine.py`.
Actually, I'll add it to `test_expr.py`? No, `test_engine.py` is better.

Wait, I'll add it to `test_engine.py`.
Actually, I'll add it to `test_engine.py`.
Wait, I'll add it to `test_engine.py`.
Actually, I'll add it to `test_engine.py`.
Actually, I'll add it to `test_engine.py`.
Actually, I'll add it to `test_engine.py`.
Actually, I'll add it to `test_engine.py`.
Actually, I'll add it to `test_engine.py`.
...

First Bad Commit

No response

Relevant log output

Server log: https://github.com/aldehir/gemma4-26b-repros/blob/main/request-02-server.log

Prompt (generated from updated template): https://github.com/aldehir/gemma4-26b-repros/blob/main/request-02-prompt.txt

[Dampfinchen]

Is there a possibility the updated chat template provided by Google might be broken in some way?

[am17an]

Does it go away with GGML_CUDA_DISABLE_FUSION=1? I tried with the Gemma-31B q8 ggml quant and I could not reproduce. This was my command

./llama-server -hf ggml-org/gemma-4-31B-it-GGUF:Q8_0 --chat-template-file models/templates/google-gemma-4-31B-it.jinja -c 256000 -np 1 --verbose --top-k 1 --temperature 0 -cram 0 -dio --no-cache-prompt

This is what I get

(.venv) aman@gx10-8a07:~/experiments/gemma4-26b-repros$ OPENAI_BASE_URL=http://localhost:8080/v1 ./run-completions.py request-02-raw-prefilled.json
 â `addcol`**

I need to wire `addcol` to the expression evaluator.

I already have `_parse_op_args` for `addcol` in `parser.py`.
I need to implement `execute_addcol` in `engine.py` and update `run()`.

Let's implement `execute_addcol` in `engine.py`.
<channel|><|tool_call>call:edit{newText:<|"|>from csvq.expr import evaluate

def execute_addcol(rows: List[Dict[str, str]], name: str, expr_str: str) -> List[Dict[str, str]]:
    new_rows = []
    for row in rows:
        new_row = row.copy()
        new_row[name] = evaluate(expr_str, row)
        new_rows.append(new_row)
    return new_rows
<|"|>,oldText:<|"|># No old text, I'll just add it before execute_select or something.
<|"|>,path:<|"|>csvq/src/csvq/engine.py<|"|>}<tool_call|>

[En3Tho]

I wonder if this one and #21769 are somehow related?

Issue I've experienced is that model keeps focusing on initial instructions as if it has lost somw of the context and on top of that it keeps responding with same repeated messages and/or tool calls.

I'm running Vulkan on Windows, 2x 9070XT

[aldehir]

./llama-server -hf ggml-org/gemma-4-31B-it-GGUF:Q8_0 --chat-template-file models/templates/google-gemma-4-31B-it.jinja -c 256000 -np 1 --verbose --top-k 1 --temperature 0 -cram 0 -dio --no-cache-prompt

@am17an, I haven't had this issue with the 31B variant, only the 26B-A4B one. The 31B variant is working spectacularly.

I'll try disabling CUDA fusion and report back.

I wonder if this one and #21769 are somehow related?

OP is using SYCL, not sure how much we can derive from that.

Is there a possibility the updated chat template provided by Google might be broken in some way?

I checked the prompt and it looks well-formed.

[the-phobos]

It does not loop when the newline after the <|think|> token is omitted, as in the previous template: "{"prompt":"<|turn>system\n<|think|>You are an expert coding…"

llama-server -m google_gemma-4-26B-A4B-it-Q8_0.gguf -np 1 --cache-reuse 0 -c 256000 --top-k 1 --temperature 0 -cram 0 -dio --no-cache-prompt --chat-template-file llama.cpp/models/templates/google-gemma-4-31B-it.jinja

$ set -x OPENAI_BASE_URL http://localhost:8080/v1; ./run-completions.py request-02-raw-prefilled.json
 � `addcol`**

I've already implemented `addcol` in `parser.py` and `engine.py`.
Let's verify with a test.
I'll add a test to `test_engine.py`.

Wait, I already have `test_execute_addcol` in `test_engine.py`? No, I don't.
I'll add it.

Wait, I'll check if I have any `addcol` tests in `test_parser.py`.
I don't.

Let's add them.

<channel|><|tool_call>call:edit{newText:<|"|>def test_execute_addcol(people_data):
    from csvq.types import Operation
    ops = [Operation(kind="addcol", args={"name": "bonus", "expr": "salary * 0.1"})]
    result = run(people_data, ops)
    assert len(result) == 5
    assert all("bonus" in r for r in result)
    assert all(float(r["bonus"]) == float(r["salary"]) * 0.1 for r in result)
<|"|>,oldText:<|"|>def test_execute_groupby_agg(people_data):
    result = execute_groupby_agg(people_data, ["dept"], [("count", "*"), ("avg", "salary")])
    assert len(result) == 3
    # Find Engineering
    eng = next(r for r in result if r["dept"] == "Engineering")
    assert eng["count(*)"] == "2"
    assert float(eng["avg(salary)"]) == 100000.0

def test_run_chained(people_data):
<|"|>,path:<|"|>csvq/tests/test_engine.py<|"|>}<tool_call|>

[aldehir]

GGML_CUDA_DISABLE_FUSION=1 did not improve the situation.

$ GGML_CUDA_DISABLE_FUSION=1 build/bin/llama-server -hf ggml-org/gemma-4-26B-A4B-it-GGUF:Q8_0  --chat-template-file models/templates/google-gemma-4-31B-it.jinja -c 256000 -np 1 --verbose --top-k 1 --temperature 0 -cram 0 -dio --no-cache-prompt

It does not loop when the newline after the <|think|> token is omitted, as in the previous template: "{"prompt":"<|turn>system\n<|think|>You are an expert coding…"

It doesn't have to be the \n. If you remove anything from the text, e.g. harness from coding agent harness, it does not reproduce. Same thing if I add something.

[Rotatingxenomorph]

--min-p 0.0 is the recommended sampler, not 0.05 (default). Not sure if this is what's causing it though.

[aldehir]

I've been testing various things, and honestly, I have no idea what I'm doing. So I'm going to close this. Best I can tell is, it's not great a high context. Sometimes I'll get a weird generation like Let'<|"|> or Let'\t\t\t\t\t. Almost always on that one word.