Strix Halo on Windows 11 with stock llama.cpp Vulkan: Qwen3.6-35B-A3B at 55-62 tok/s, Qwen3.5-122B-A10B at 23-24 tok/s, both loaded at once (field report)

[IanCrossland]

TL;DR: Two GMKtec EVO-X3 mini PCs (Ryzen AI Max+ 395, Radeon 8060S, 128 GB) running the stock llama.cpp b11401 Windows Vulkan release. No ROCm, no Linux, nothing tuned beyond a 16 GB graphics reservation. Every Strix Halo LLM benchmark we found was run on Linux, so here are Windows numbers.

• Qwen3.6-35B-A3B (UD-Q4_K_XL): 55 to 62 tok/s generation, 0.15 to 0.17 s warm first token.
• Qwen3.5-122B-A10B (UD-IQ4_XS): 23 to 24 tok/s, 0.4 to 0.5 s warm first token.
• Both models fit in memory at once with 27 GiB to spare, on only a 16 GB graphics reservation.
• -ctxcp 0 cut a fixed 10 s first-token delay on the 122B to 1.5 to 2.6 s. -cram 0 made the 35B faster at every prompt length.
• Long prompts are the weak point: a 19K-token prompt waits about 19 s on the 35B (with -cram 0) and 55 s on the 122B for its first token.
• whisper.cpp on the same iGPU transcribes 2.6 times faster than on the CPU, with no accuracy loss, and slows the LLM by about 9%.
• Cancelling during prompt processing does not stop it. The server finishes the prompt first.

Setup

Component	What we ran
Computer	GMKtec EVO-X3, both machines (called A and B below)
CPU and GPU	Ryzen AI Max+ 395 (16 cores, 32 threads), Radeon 8060S iGPU
Memory	128 GB at 8000 MT/s. 16 GB reserved for graphics; Windows sees 111.6 GB
What Vulkan reports	Vulkan 1.4.329. Device-local heap 70.21 GiB (budget 66.70 GiB), host heap 35.11 GiB, KHR_coopmat present
OS	Windows 11 Pro 10.0.26300
GPU driver	A: 32.0.22029.1019, then 32.0.31041.1004 for the two-machine test. B: 32.0.22029.1019, then re-tested on 32.0.32015.2008
LLM runtime	llama.cpp b11401 (commit a7fb71f), the release's prebuilt Windows x64 Vulkan zip, unmodified
Speech runtime	whisper.cpp b5130, built from source with -DGGML_VULKAN=ON (Vulkan SDK 1.4.363.0, MSVC 14.44)

Both machines had the same runtime files and the same model files (checked by sha256).

Model	Quant	Files	Size	Peak server memory
Qwen3.6-35B-A3B (3B active)	Unsloth UD-Q4_K_XL	1	22.4 GB	23 GB
Qwen3.5-122B-A10B (10B active)	Unsloth UD-IQ4_XS	3 shards	60.2 GB (56.1 GiB)	62 to 64 GB

We picked IQ4_XS for the 122B because the usual 4-bit files (UD-Q4_K_XL at 71.74 GiB, Q4_K_M at 71.28 GiB) exceed the 66.70 GiB budget Vulkan reports. That budget later proved not to be a hard wall (see "Both models loaded at once"), so the larger quants may load too. We did not test them.

Generation speed and first-token latency

Measure	35B, A	35B, B	122B, A	122B, B
Generation, median (tok/s)	55.3	61.8	23.1	23.6
Warm first token, median (ms)	171	145	470	425
Warm first token, p95 (ms)	1,465	2,311	4,249	1,745
First token, 22.5K-token prompt (s)	28.8	27.5	68.3	69.6
Lowest free memory during the run (GB)	72.5	88.6	31.7	50.0

Warm first token is 30 repeats of a short chat request to a loaded server, prompt cache on, 32K context, one slot. Machine A was also running a voice assistant and its speech recognizer; Machine B ran nothing else. That fits B's faster 35B and its extra free memory.

Updating Machine B's driver to 32.0.32015.2008 left speed unchanged (35B 60.6, 122B 23.8 tok/s). It cut the 35B's warm p95 from 2.3 s to 0.7 s and its 22.5K-token first token from 27.5 s to 24.7 s.

On realistic chat and tool prompts (a few hundred tokens, 85 test cases), the median first token was 0.32 s on the 35B and 1.1 s on the 122B. With thinking on, counting thinking tokens as output, it rose to 1.6 s and 5.1 s.

Long prompts

Seconds to first token. Median of 3 prompts per size, Machine A, 32K context, -ctxcp 0.

Prompt tokens	35B, cache on	35B, -cram 0	122B, cache on
~740	2.3	0.7	5.8
~3,300	4.7	2.8	11.6
~5,600	7.2	4.8	17.3
~13,200	15.2	12.1	36.9
~19,300	23.2	19.3	55.4

The wait grows roughly in a straight line with prompt length: about 1 s per 1,000 tokens on the 35B and about 3 s per 1,000 on the 122B. For prompts of 3K tokens and up, the 35B processed about 1,000 to 1,190 tok/s with -cram 0 (700 to 870 with the cache on), and the 122B 290 to 360 tok/s with the cache on. One earlier 35B run with the cache on stalled for 65 s at 13K tokens and 117 s at 19K.

Both models loaded at once

The 122B and the 35B fit together with 27 GiB of Windows memory to spare. Keeping both loaded costs nothing, but generating on both at once halves the 35B.

Generation (tok/s)	Alone	Other model loaded, idle	Both generating
122B	21.4	21.2	17.1
35B	47.9	49.6	24.0

• Memory. Windows had 91.9 GiB free with nothing loaded, 48.3 GiB with the 122B, and 27.0 GiB (minimum) with both.
• GPU memory past the budget. The two together used 80.5 GiB of GPU memory (15.8 dedicated plus 64.7 shared). That is past the 66.7 GiB budget Vulkan reports, and both still loaded and ran on the 16 GB reservation.
• Load times, with model files already in the Windows file cache: 122B 29.3 s; 35B 18.2 s with the 122B loaded.
• 35B first token while the 122B generates: median 0.37 s, worst 1.5 s, over 5 short requests.

One trial of 3.6 minutes: -c 8192 -ctxcp 0 on both, thinking off, temperature 0, 256-token replies, models loaded one after the other.

Server flags

For these two Qwen models, set -ctxcp 0 and -cram 0 on llama-server. Leave flash attention and the micro-batch at their defaults.

Flag	What it turns off or sets	Measured effect	Verdict
-ctxcp 0	Context checkpoints, which snapshot these hybrid models' recurrent state	122B, 6-turn chats with a 300-token system prompt: first token 10 to 11 s before, 1.5 to 2.6 s after	Use
-cram 0	The RAM prompt cache	35B: warm first-token p95 1,465 ms before, 159 ms after; faster at every prompt length. Stalls of 56 to 106 s before 11K to 22K-token prompts were seen only with the cache on	Use
-fa	Flash attention on or off	122B, llama-bench, 512-token prompt: 296 vs 299 tok/s at micro-batch 512, 202 vs 195 at 128. No consistent gain	Default
-ub	Micro-batch size	122B, 512-token prompt: 296 tok/s at 512, 202 at 128. No difference for prompts of 64 tokens or fewer	Default (512)

• Why -ctxcp 0 works is our inference from timings, not confirmed in llama.cpp's source. Each checkpoint seemed to cost about 3.5 s, and every request made at least one. llama-bench, with no server, showed no fixed cost: 4 prompt tokens took 0.08 s and 512 took 1.7 s.
• What -ctxcp 0 costs: no prefix reuse on these models, so every turn re-processes the whole prompt. -ctxcp 1 is untested.
• -cram 0 alone did not fix the 122B's 10 s delay; -ctxcp 0 did.

Whisper on the same GPU

whisper.cpp small.en on the Radeon 8060S transcribes in 291 ms against 755 ms on the CPU, with no accuracy loss. Sharing the iGPU costs the 35B about 9% of its generation speed.

Setup (small.en, beam 5)	Median (ms)	p90 (ms)	Word drift
CPU, q8_0, 16 threads	755	837	1.6%
Vulkan, q8_0, one process per clip	375	615	3.5%
Vulkan, f16, one process per clip	358	547	0.0%
Vulkan, f16, warm whisper-server	291	494	0.0%

Time runs from the end of the audio to the transcript, over 32 clips of one speaker, 4 to 9 s each (317 words). Drift is the word edit distance against CPU small.en f16 at beam 5. The Vulkan q8_0 row comes from an earlier run the same day. Its GPU kernels produce slightly different text, so use f16 on the GPU.

Worst case, with whisper looping nonstop while the 35B generates nonstop:

• The LLM went from 54.3 to 49.2 tok/s (-9.4%). Its median first token did not change (167 vs 166 ms), and the longest gap between streamed chunks was 83 ms.
• Whisper on the GPU still took 433 ms median, against 1,030 ms on the CPU under the same load.
• CPU whisper is not gentler on the LLM. It cost 5.6% of generation speed and 68% on first-token p90.
• Cost of a warm server: about 0.9 GB of GPU memory and 1.4 GB of committed RAM, flat over the run. The first request after a driver install takes about 2.6 s, once, while shaders compile.

Cancelling a request

On llama-server b11401, a cancel that arrives during prompt processing waits for the prompt to finish. Once tokens are streaming, closing the connection stops the server within one decode step.

• During prompt processing: a cancel sent right after the request started kept the server busy for a median of 0.8 to 1.0 s on the 35B and 2.9 to 3.5 s on the 122B, for the same chat prompt. The cost grows with prompt length.
• After the first token: the server went idle within one decode step of the client closing the connection, about 30 ms on the 35B and 60 ms on the 122B.
• The client has to close the connection. Our first client kept reading the abandoned stream, and the server stayed busy for up to 2 s longer.
• Size the stop timeout per model. A fixed 1 s wait for the server to confirm the stop failed on the 122B in 15 of 25 cancels. We now wait up to 5 s.
• No resumes: in 140 trials, no server task that had ended ever ran again.

We detected idle by polling the server's /slots endpoint every 20 ms, with 5 trials per case per model.

Method

Most figures are medians of 30 warm requests to one model on one machine, measured in October 2026.

• Server: llama-server on loopback, one slot, all layers on the GPU (-ngl 99), thinking off unless stated. Context was 8K for chat tests and 32K for the prompt-size sweep.
• Warm runs: one cold request first, then 30 timed requests. First token is the time from sending the request to the first streamed token.
• Prompt-size sweep: five target sizes, with median real lengths of 737 to 19,347 tokens, plus one 22.5K-token prompt.
• Isolation: a lock kept other heavy jobs off the machine during each run. In the two-machine test, Machine A still served its voice apps.
• Whisper: runs interleaved, so every clip went through every setup in turn, at below-normal process priority.
• Memory: Windows performance counters for available memory and for dedicated and shared GPU memory.
• Run-to-run noise: repeating the 85-case chat and tool set moved scores by about 1 case in 20, so treat small gaps as noise.

Not tested

Only the Vulkan backend on Windows with a 16 GB graphics reservation. We did not measure:

• ROCm or HIP on Windows, Linux on the same machines, or LM Studio and Ollama.
• A larger graphics reservation (AMD documents up to 96 GB on 128 GB systems).
• The 122B at UD-Q4_K_XL, with or without expert offload (--n-cpu-moe).
• -ctxcp 1, or prefix reuse of any kind on these models.
• Prompt-size sweeps with -cram 0 on the 122B, contexts over 32K, or more than one slot.
• Speculative decoding.
• A true cold boot, hours of continuous load, power draw or thermals.
• Fine-tuning. A bf16 LoRA on the 122B is ruled out by memory: its bf16 weights alone are 227.5 GiB.

Reproduce

The exact commands behind the numbers, run in PowerShell from the unzipped llama.cpp b11401 Windows x64 Vulkan release.

Serve a model (for the 122B, point -m at the first of its three shards, Qwen3.5-122B-A10B-UD-IQ4_XS-00001-of-00003.gguf):

.\llama-server.exe -m Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf -c 8192 -np 1 -ngl 99 --host 127.0.0.1 --port 18081 --jinja --no-webui --fit off -ctxcp 0 -cram 0

Prompt-processing bench (flash attention and micro-batch, no generation):

.\llama-bench.exe -m Qwen3.5-122B-A10B-UD-IQ4_XS-00001-of-00003.gguf -ngl 99 -p 1,4,16,64,512 -n 0 -ub 128,512 -fa on,off -r 2 -o md

whisper.cpp b5130 with Vulkan, built with the Vulkan SDK and the VS 2022 Build Tools, then served warm with small.en f16:

cmake -B build -DGGML_VULKAN=ON -DGGML_NATIVE=ON -DBUILD_SHARED_LIBS=ON
cmake --build build --config Release
.\whisper-server.exe -m ggml-small.en.bin -t 8 -bs 5 -bo 5 -nth 0.6 -sns --host 127.0.0.1

Write-up drafted with Claude from our benchmark logs; every number is measured. Questions welcome.