RTX 5060 Ti 16GB: Qwen3.8-27B IQ4_XS + MTP-2 achieves 50.2 tok/s, 37/37 ReAct

[SoundByGAIA]

Sharing results from an internal ORDIS evaluation of Qwen3.8-27B on an NVIDIA RTX 5060 Ti 16GB, using stock llama.cpp, Unsloth UD-IQ4_XS quantization, and an external MTP head.

The goal was to exceed 25 tokens/s while maintaining task performance. MTP-2 reached 50.2 tok/s on the short-prompt decode test, with unchanged aggregate evaluation scores.

Correction: VRAM units

My original post incorrectly divided the raw MiB readings by 1000 and labelled the results “GB”. The corrected table below uses GiB (1 GiB = 1024 MiB).

VRAM figures are the highest whole-GPU nvidia-smi memory.used readings sampled once per second, not isolated server-process allocations. The card reports 16,311 MiB (15.9 GiB) total.

Performance results

All configurations used UD-IQ4_XS, an 8192-token context, and full GPU offload.

Configuration	Decode: 30-token prompt	Decode: ~6.6K-token prompt	MTP acceptance	Sampled peak VRAM	Evaluation suite wall time
MTP disabled	27.6 tok/s	26.5 tok/s	—	13.2 GiB	259 s
MTP-1	45.8 tok/s	42.1 tok/s	88.5%	14.6 GiB	178 s
MTP-2	50.2 tok/s	48.1 tok/s	77.2%	14.8 GiB (15,141 MiB)	156 s

Decode speeds are the mean of two timed requests per prompt: three requests were sent, and the first was discarded. Each configuration completed one full evaluation pass. There are no repeated full runs or variance estimates.

Suite wall time covers the ReAct, general, and multi-step evaluations at temperature 0, seed 42. Decode throughput is measured separately and is not end-to-end task throughput.

Compared with MTP disabled, MTP-2 increased short-prompt decode throughput by approximately 81.9% and reduced suite wall time by approximately 39.8%.

Evaluation results

Configuration	ReAct tool-format suite	General suite	Multi-step suite
MTP disabled	37/37	42/44	7/8
MTP-1	37/37	42/44	7/8
MTP-2	37/37	42/44	7/8

These are internal ORDIS test sets, not standardized public benchmarks. Unchanged aggregate scores do not establish identical generated outputs or equivalent quality on other workloads.

The two general-suite failures occurred in all three configurations:

• A Python word-counting solution used [a-z]+, losing Turkish characters and splitting words such as dünya.
• A science question returned “Karbon” instead of the expected “karbondioksit”.

The multi-step tasks use a simulated tool world. The harness executes tools in a sandbox and supplies observations to the model. A pass requires the expected value in the final answer within a six-step budget. Across all three configurations, there were zero model-written observations and zero multiple-action steps.

The failed multi-step task asked for “the port value in config.json” without specifying a path. The sandbox file existed only at /app/config.json, and no search tool resolved a bare filename. The model attempted reads and a shell call, exhausted the six-step budget, and produced no final answer; it did not fabricate a value.

A clarified version with an explicit path was not evaluated. This case is confounded by task ambiguity and the six-step budget; without a clarified rerun, we cannot isolate the cause of failure. It remains scored as a failure in the reported 7/8 result.

Reproducibility: source and artifacts

Item	Recorded value
Engine source commit	23b0202a189c44a54625aadcb37a946dd1d6278d
Source tarball SHA-256	1fb4e5f71f50526ec7cae25bb1e9279d75d899e089386a82b71e42b951ea5d00
Binary version output	version: 0.6.0-dev (build 0, commit unknown)
llama-server SHA-256	b2c18fb08ee2d44e00ef21c1e2eac8e84044914cad377ccd52722099a2b59cc1
Model repository	unsloth/Qwen3.8-27B-GGUF
Model repository revision	4ca720788d1e01f1bff70c033e0d0028fd02e502
Model file	Qwen3.8-27B-UD-IQ4_XS.gguf
Model size	14,252,845,984 bytes
Model SHA-256	40fac4050e940397dbf13087afd50f4734a11805bf9d65ef8ddd7483470e6199
External MTP head	MTP/mtp-Qwen3.8-27B-Q4_0.gguf, from the same repository revision
MTP head size	1,369,590,656 bytes
MTP head SHA-256	50d9ce5a6da381bbcfb31061cf73df94a90e6faf8efeddee379a9cb8f1501c6e

The source commit was verified through the source tarball: git get-tar-commit-id returns the recorded SHA, the build directory’s CMAKE_HOME_DIRECTORY points to the extracted source tree, and diff -r against a fresh extraction shows the source tree unchanged.

The binary was built from a tarball without .git, so it reports commit unknown. The commit attribution comes from the source and build records, not from the binary’s version output. Shared libggml*.so files were not hashed.

Both GGUF files matched their Hugging Face LFS hashes after download and again after copying to the benchmark host.

Build environment

• Ubuntu 26.04.1
• NVIDIA driver 595.99.02
• nvcc 13.2.86
• GCC 15.2.0
• CMake Release build
• GGML_CUDA=ON
• GGML_CUDA_FA_QUANTS=all
• CMAKE_CUDA_ARCHITECTURES=120a-real
• LLAMA_CURL=OFF

Reconstructed launch command

The following MTP-2 command was reconstructed from the launcher script and per-run metadata. It is not a captured /proc command line.

Replace <MODELS_DIR> and <PORT> with your local values. The MTP head path below assumes it is stored directly in <MODELS_DIR>.

llama-server \
  -m <MODELS_DIR>/Qwen3.8-27B-UD-IQ4_XS.gguf \
  -ngl 99 -c 8192 -np 1 --flash-attn on -ctk q8_0 -ctv q8_0 -t 6 \
  --host 127.0.0.1 --port <PORT> --no-webui \
  --spec-type draft-mtp --spec-draft-n-max 2 \
  -md <MODELS_DIR>/mtp-Qwen3.8-27B-Q4_0.gguf --spec-draft-ngl all

[SoundByGAIA]

Supplementary limits (from the same run records; no new measurements)

Memory pressure. The host has 15,285 MiB (14.9 GiB) of system RAM. Minimum "available" RAM sampled during the runs was 3,529 / 4,483 / 4,867 MiB (3.4 / 4.4 / 4.8 GiB) for MTP off / MTP-1 / MTP-2. System-wide swap used peaked at 2,047 / 2,048 / 4,095 MiB, i.e. the whole 4 GiB swap file was in use during the MTP-2 pass. Swap is system-wide, part of it was already in use before the runs, and it was not isolated to the llama-server process, so it should not be attributed to MTP-2 itself. Treat 16 GB of system RAM as the practical floor for this setup.

Single machine. All numbers come from one host and one GPU (RTX 5060 Ti 16 GB). No cross-machine or run-to-run repetition.

How the launch parameters are known.

• Confirmed directly by the server log: one slot, n_ctx_slot = 8192, 6 threads, n_gpu_layers set to 99, the external draft model file loaded, UI disabled, listening on 127.0.0.1.
• Recorded in the per-run metadata written at launch: --spec-type draft-mtp, --spec-draft-n-max, -md , --spec-draft-ngl all, -ngl 99. The draft-acceptance lines in the log (mean accepted length up to 2.00 for MTP-1 and 3.00 for MTP-2) are consistent with these, but they are not a flag echo.
• Taken only from the launcher script text, not logged: --flash-attn on, -ctk q8_0, -ctv q8_0, and the port.

MTP head compatibility. I did not verify that the Unsloth MTP/ head is officially specified for this exact UD-IQ4_XS file. Compatibility was established only empirically: the head loaded, and drafts were accepted at 88.5 % (MTP-1) and 77.2 % (MTP-2).

Not covered: other prompt mixes, contexts longer than 8K, concurrent slots, other hardware or other quantizations.

[siimtolk]

Fyi, a data point from a similar setup: Qwen3.8-27B IQ4_XS on RTX 5060 Ti 16 GB, ⁠ --spec-type draft-mtp ⁠ (default 3 draft tokens), b11707.

•⁠ ⁠One card, 64k ctx, q4_0 KV (stock UD-IQ4_XS): MTP took decode from ~25 to ~41 tok/s for ~1.2 GB extra VRAM. Adding ⁠ ngram-simple ⁠ on top made it 15–20% slower: MTP already accepts ~97% of draft tokens on tool-call JSON.
•⁠ ⁠Two cards (layer split), production (RVN build, 2 slots × 128k): 47 tok/s prose, 65 tok/s tool JSON (91% draft acceptance), 1,220 tok/s on a 30k prompt, ~69 tok/s total with two requests at once. Splitting costs no decode speed and makes prompt processing ~35% faster.
•⁠ ⁠vLLM on the same two cards (TP2, NVFP4, MTP-3): faster decode (59–61 tok/s), but follow-up time-to-first-token ~3 s vs ~1.3 s, so llama.cpp stays.

Full settings and results in my write-up: https://siimtolk.github.io/articles/llm-testing-overnight/

[SoundByGAIA]

Thanks for sharing these results, Siim! Really interesting data, especially the MTP acceptance rates and the multi-GPU performance.

Our 50.2 tok/s result was measured at 8K context with MTP-2, so it's not a direct apples-to-apples comparison.

Two things I'm curious about:

• Was the ~97% MTP acceptance rate specific to structured tool-call JSON, or did you observe similar rates across multi-step agentic workloads?
• For the 64K context benchmark, how much of the context was actually populated during decoding?

We're particularly interested in end-to-end agentic performance rather than raw token throughput, so your findings are useful.

Thanks again for contributing!😍

[siimtolk]

Good questions, I checked the logs:

1. The ~97% was a single 10-tool-call JSON test. Across the benchmark, prose gets ~56% and tool-call JSON ~91%. For real agentic traffic, the production server's metrics over the last 7 days (a Hermes agent: tool calls, multi-step tasks, briefings, chat) show 77% overall: 87% / 76% / 67% for draft positions 1 / 2 / 3, with 3 drafts. So agentic work lands between prose and pure JSON.

⁠2. Honestly, very little: the decode numbers come from short prompts, so the 64K was allocated rather than filled. I only measured prompt processing at depth (~1,220 tok/s on a 30K prompt, two cards). In real use the slots reach ~100K tokens, but I don't have clean decode-at-depth numbers. That's a good gap to add to the suite.

For end-to-end agentic time, the bigger win for me was two context slots (a follow-up starts in ~1.3 s instead of re-reading 8–22K tokens) and keeping thinking short, rather than raw tok/s.