Llama.cpp merged MTP - now the most reliable for single 3090 for Qwen3.6-27B with high context + vision #152
Replies: 8 comments 34 replies
-
|
This is a genuinely important data point — thank you for the careful dual-stack comparison with per-category quality scores, not just headline TPS. Given your Cliff 1 / Cliff 2 reproduction history, a controlled report from your rig carries weight. The headline for us: full MTP landing in llama.cpp mainline is the upstream-merge re-test trigger we parked back on 2026-05-05. When we benched the pre-merge MTP fork (am17an's PR) we measured ~+34% TPS but explicitly did not ship it, for three reasons: it was unmerged, it regressed context hard (q8_0 KV ceiling vs the 262K the cliff-immune What happens next, honestly and without over-promising a date: we don't ship or recommend a path on bench numbers alone — we'll run this through our own validation gate first. That means boot + This is squarely the "single-card robustness" route the stack already champions, so an MTP speed uplift to the llama.cpp path is exactly the kind of thing we want to get right here. We'll make use of your findings appropriately as we plan that validation. Thanks again for sharing the full commands and the q-scored matrix — that's what makes a report like this actionable. |
Beta Was this translation helpful? Give feedback.
-
Sounds good - and I agree, do the full testing before launching. I only tested it for my use-cases I have also been testing ik_llama with IQ4_KS quants and am able to reach 70-75tps based on information found in this thread: I lowered ctx size to 128k since it seems that large context pushes (60k+) causes cuda crashes (sound familiar?) - still testing my ik_llama commands: Benchmarks:🚀 TEST: Prefill Heavy (~8000 Context Tokens) 📤 Generated Tokens : 11 🚀 TEST: Decode Heavy (Pure Generation) 📤 Generated Tokens : 1500 AGG mean_tps=66.7 max_tps=89.5 mean_q=0.97 passed=6/6 |
Beta Was this translation helpful? Give feedback.
-
|
Update from the validation gate, @ampersandru — and there's a direct answer to your Cliff 2 question in here too. We just shipped a llama.cpp single-3090 MTP profile through that gate. PR #166 adds
Direct answer on the "60K+ cuda crashes" you flagged: at this config, the cliff goes away. Verify-stress's long-context rungs include independent secrets recalled at 58K and 91K, both clean, on a single 3090. The "Cliff 2 single-prompt at 50–60K is architectural" framing was too strong — it's config-driven, not architectural. The per-pass activation peak at On your ik_llama numbers vs mainline: I went through your full config carefully. Decomposing where the gap actually comes from:
Calibration note on the headline rates: your "94 t/s decode" on the 8K-prompt-heavy bench is a short-output (11 decoded tokens) spec-decode-acceptance artifact — first accepted multi-token draft dominates the rate. Your sustained-decode-only run was 45 t/s at 1500 tokens generated, and your LLM:instruct essay was 52.6 TPS (close to our 51). The apples-to-apples gap on long-form decode is much smaller than the headline implies; most of the speed-difference perception is the n=2 → n=4 MTP retune, not the engine fork. Net: thanks for the careful matrix and the cross-rig data point — directly fed into a shipped compose family. The Cliff 2 walks-past finding wouldn't have happened without your gauntlet pushing us to retest at higher ctx + smaller |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for releasing the new profile for single 3090s @noonghunna . I would love to get your input and tests on ik_llama since it supports IQX_K quants, which is the new hybrid - Uses non-linear quantization instead of codebooks. Same quality as i-quants, but much faster, another point of speedup using ik_llama (I was testing ubergram's IQ4_KS). ikawrakow measured IQ4_KS as very close to, or better than, UD_Q4_XL. Unsloth UD-Q4_K_XL needs about 2.8 GiB more to land in the same neighborhood. Now that we've nailed down decent performance with 27B dense, its time to also increase the quality imo Key Tradeoffs (summarized by AI):
I-Quants Cons:
K-Quants Pros:
K-Quants Cons:
IQX_K Pros (Best of Both):
IQX_K Cons:
|
Beta Was this translation helpful? Give feedback.
-
|
Thanks @ampersandru — quality angle is the right pivot. We've been TPS-anchored; if IQX_K is actually landing UD-Q4_K_XL quality at -2.8 GB, that's the more interesting axis. Two honest concerns before any ship discussion:
Best path forward — we'd love to vet your eval against the v0.8.3 baseline. Rather than spinning up an ik_llama track on our side (queue is full through v0.8.x), if you can run our standard chain on your ik_llama + IQ4_KS rig we get apples-to-apples vs git clone --branch v0.8.3 https://github.com/noonghunna/club-3090 ~/club-3090
cd ~/club-3090
URL=http://localhost:<your_port> MODEL=<your-model-name> \
bash scripts/rebench-full.shThat's the single orchestrator we use ourselves — chains The 8-pack quality + soak pair is what actually decides ship — TPS alone isn't enough. Per-category quality tells us if IQX_K really preserves reasoning / tool / JSON, soak catches Cliff 2b under sustained ctx, and aider-polyglot-30 closes the agentic-code loop. Ship gate. If your numbers come back with (a) quality 8-pack ≥ v0.8.3 If you hit script bugs or unclear pointers, comment here and I'll fix as they surface. |
Beta Was this translation helpful? Give feedback.
-
|
it does seem to want to launch gemma4, but since qwen3.6 is loaded, llama-swap keeps failing to load it into memory even when I set the model specifically. I think the code somewhere is using the models endpoint and attempting to load the first model that shows up. I can fix it sometimes by running |
Beta Was this translation helpful? Give feedback.
-
thanks, Ill try that. Since my main usage is Frigate image analyzing, I want it as fast as possible, but have been tempted to just run Vision on Gemma4-e4b since I have that loaded on my 5060ti |
Beta Was this translation helpful? Give feedback.
-
|
Here are results from a full run on a non-heretic IQ4_KS without --parallel-tool-calls I will retest with --parallel-tool-calls and no PL, but will not run Soak and Aider tests 
🔍 Key Takeaways from the Final Run Raw results |
Beta Was this translation helpful? Give feedback.
-
|
@ampersandru — that model-mismatch is a real bug on our side, and you pinpointed it exactly. Fixed in #177: when you set bash scripts/rebench-full.sh --url http://HOST:PORT \
--model 'Qwen3.6-27B MTP ik_llama:instruct' --engine llama-cppThat also clears up the model-attribution question on your earlier quality run — with @laurimyllari — thanks, those are genuinely useful, especially @ampersandru — if you do try If either of you re-run after #177 lands, the |
Beta Was this translation helpful? Give feedback.
-
|
This is the leg that settles the most important question — thank you for running it. The non-heretic run exonerates the fine-tune.
And the failure pattern points straight at your next test. Every toolcall miss is multi-tool sequencing — the model fires the first tool and drops the rest: That's exactly what @laurimyllari's Two things this run also confirmed on our side:
One gap for a complete 150-pack comparison: Net so far: ~30% faster code (more at equal power), tied-or-better on most packs, math slightly better, one regression (multi-tool sequencing) that's quant/engine-attributable and very possibly |
Beta Was this translation helpful? Give feedback.
-
|
Here is another bench with 
🔍 AI Performance Analysis Uncorking the power limit to 370W resulted in an absolute explosion in raw throughput. 💡 Suggested Power Limit: The 300W Sweet Spot While 92 t/s is incredibly fast, running at 370W pushes the 3090 core to 78°C. This is safe, but it is riding close to the ~83°C thermal throttle point. When managing a dense Unraid infrastructure, dumping 370 watts continuously from a single component can strain the chassis thermals and significantly increase fan noise. The RTX 30-series architecture scales terribly at the top end of its power curve. You are paying a massive 48% increase in power (from 250W to 370W) for only a 19% increase in speed. |
Beta Was this translation helpful? Give feedback.
-
|
This is so good, well done you! Now i'm intrigued. |
Beta Was this translation helpful? Give feedback.
-
|
FYI,I forgot that I do have a memory overclock (+1000MHz), which also likely contributes to the fast tps My startup script that undervolts, oc, and sets PL: |
Beta Was this translation helpful? Give feedback.
-
|
Unfortunately, --parallel-tool-calls didnt change anything, unless Im doing something incorrect. almost 70tps narrative and over 90tps for code is pretty crazy for a dense model, I will have to play around with PLs to find a sweet spot, leaving it at 370W is a bit much. commands for ik_llama: |
Beta Was this translation helpful? Give feedback.
-
|
I wonder if you also need the fixed template for parallel tool calls. I'm getting this: My settings are similar to yours, but I use the recommended qwen3.6 sampler settings: And for the tool calls: I'm missing |
Beta Was this translation helpful? Give feedback.
-
|
i've been patching this one in recent composes |
Beta Was this translation helpful? Give feedback.
-
|
Thanks, I'll give the chat template a shot |
Beta Was this translation helpful? Give feedback.
-
|
chat template fixed it! I think 300W is the sweet spot for me in terms of narrative performance and power usage. IQ4_KS doesnt seem to score as high as Q4_K_M in quality, so it seems there is a slight trade off between speed, context window, and quality here. It depends on each persons workflow, I think the trade off is worth it for me Here are the latest benches with bugfind, 300W PL and chat template with the metric model. Ignore the middle column, it was my last run without bugfind working. I will try testing with non-heretic tomorrow to see if it does better on bugfind as with @laurimyllari results 
|
Beta Was this translation helpful? Give feedback.
-
|
FYI, I was getting full thought responses back with that chat template in Frigate, so for my use case, I had to remove that template. I reported the bug: I was testing 320W last night and it started looping during quality tests, (0 tokens in/out, 0tps spam in logs) - not sure if that had to do with the chat template. Will try to rerun a 270W test without chat template today |
Beta Was this translation helpful? Give feedback.
-
|
Summary of ik_llama performance from baseline vs 250W up to 370W. I would ignore the toolcall % because of all the chat template and bugreport differences. Sweet spot seems to be 300W for good narrative and code TPS. Whether you choose to use a chat template or heretic model is up to you 
|
Beta Was this translation helpful? Give feedback.
-
|
Updated with 270W results. |
Beta Was this translation helpful? Give feedback.
-
|
I need this now!! Can't wait for the final release ;) |
Beta Was this translation helpful? Give feedback.
-
|
Huge thanks to @ampersandru and everyone in this thread — your cross-engine → What chasing single-GPU efficiency with ik_llama taught us about our own llama.cpp stack The honest headline: at a verified, matched 370 W on the same card, ik_llama IQ4_KS ties mainline llama.cpp on decode TPS, quality, and context — the throughput edge we (and our early numbers) expected turned out to be a power-state artifact on our rig, not an IQK win. ik's one real advantage here is a ~0.5 GB leaner VRAM footprint (genuinely handy when you're tight on memory). We couldn't reproduce the ~70/92 t/s from this rig either — closer to ~50/58 decode at matched power, likely rig/methodology differences — but chasing it was well worth it: it exposed that our own mainline composes were shipping 131K when the full 262K was free (a batch-size lever), and it sharpened how we benchmark (verify power before every run, decode-TPS ≠ agentic-coding score, harness fidelity). It's all in the post. Thanks again for the nudge — this is exactly the kind of cross-rig report that makes the whole project better. 🙏 |
Beta Was this translation helpful? Give feedback.
-
|
Extremely interesting there was no speed up for you! I originally tested ik_llama because of reports of faster performance with 27B and MTP (from this thread). Likely possible that its because of my undervolt and memory overclock, and that ik_llama is more beneficial from it than mainline llamacpp? If I get time, I will try to retest both llamacpp and ik_llama on stock gpu settings |
Beta Was this translation helpful? Give feedback.
-
|
yup precisely that! |
Beta Was this translation helpful? Give feedback.
-
|
you do get 500MB worth of more headroom in kv cache with IQK quant though. This could be handy for people starving for kv cache on windows as WSL add 1.3GB of overheads. |
Beta Was this translation helpful? Give feedback.
-
|
another day, another claim, want to test this @noonghunna 😆 https://www.reddit.com/r/LocalLLaMA/comments/1tkpz2y/beellama_v020_major_dflash_update_single_rtx_3090/ |
Beta Was this translation helpful? Give feedback.
-
|
Haha bring it on! Dflash comes with its own drafter unlike MTP which is inbuilt, will fly on 2x3090s. Not worth testing on single gpu. Tom has synced llama MTP into his turboquant repo, I'm more inclined towards testing that next. Turboquant could really help a lot of folks with less than 24GB nvram. Carnice, Qwopus, supergemma are also worth exploring. |
Beta Was this translation helpful? Give feedback.
-
|
@ampersandru — following up, and I owe you a correction of our correction. Your ~70/92 t/s finding was right. Earlier in this thread I wrote that we "couldn't reproduce the ~70/92 … closer to ~50/58 at matched power" and chalked your throughput edge up to "a power-state artifact on our rig." That was wrong — and the mistake was ours, not your rig's. We re-ran it this week as a proper power-cap A/B, both engines at a set-and-read-back 370 W (the bug last time: we recorded power without enforcing it, and this rig boots to a 230 W systemd cap). That surfaced a second, worse error: our "matched-370 W ik" bench had almost certainly been hitting a stale mainline container still answering on Corrected result — 5 independent ik runs + the 4-cell A/B:
ik_llama IQ4_KS is ~18–20% faster than mainline at matched power — quality-tied (8-pack 101 vs 100), ~0.5 GB leaner. Faster and leaner, exactly the direction you reported. (We don't reach your absolute ~92 on code — rig/build/quant differences — but the engine ranking and the ~70-class code rate are confirmed.) Full corrected writeup + the power-cap table: discussions/184. Your launch flags from this thread ( Sorry for the detour casting doubt on your report — and thank you for the original signal. It's now a shipped, faster single-card path because you flagged it. |
Beta Was this translation helpful? Give feedback.
-
|
glad you got the performance increase! Ive been using ik and iq4_KS the past few days and am happy with the performance, quality, and context window |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
I have been using a mix of the commands found here with PIECEWISE for my 3090 but full MTP support was merged into Llama.cpp yesterday, giving it a nice boost while retaining low resource usage for high context window. On most prompts, you will average around 60tps, which is a little slower than vLLM + PIECEWISE (can hit an upwards of 70-80tps). But, if you hunger for high context window (I can get around 145k + vision), vision, and tool compatibility with things like Opencode, VS Code, hermes, etc, this is the way to do it. If you drop to q4 cache and drop vision, Im sure you can hit 256k
My recommendations for single 3090:
Speed + vision, 65k context: vLLM + Piecewise + MTP: 70-80tps
High context + vision, 145k context: latest llama.cpp with MTP compatible gguf: 50-60tps
my llama.cpp commands:
llamacpp Benchmarks:
AGG mean_tps=53.1 max_tps=62.7 mean_q=0.98 passed=6/6
JSON_RESULT={"label": "LLM:instruct", "mean_tps": 53.1, "max_tps": 62.7, "mean_q": 0.9833333333333334, "passed": 6, "total": 6, "by_cat": [{"cat": "list", "tps": 45.3, "q": 0.9}, {"cat": "code", "tps": 60.3, "q": 1.0}, {"cat": "json", "tps": 62.7, "q": 1.0}, {"cat": "reason", "tps": 59.1, "q": 1.0}, {"cat": "tool", "tps": 44.6, "q": 1.0}, {"cat": "essay", "tps": 46.6, "q": 1.0}]}
🚀 TEST: Prefill Heavy (~8000 Context Tokens)
⏱️ Time to First Token (TTFT) : 7.446 seconds
📥 Prompt Tokens : 8020
⚡ PREFILL SPEED : 1077.0 tokens/s
📤 Generated Tokens : 11
⚡ DECODE SPEED : 59.0 tokens/s
⏳ Total Request Time : 7.633 seconds
🚀 TEST: Decode Heavy (Pure Generation)
⏱️ Time to First Token (TTFT) : 0.311 seconds
📥 Prompt Tokens : 35
⚡ PREFILL SPEED : 112.5 tokens/s
📤 Generated Tokens : 1500
⚡ DECODE SPEED : 43.5 tokens/s
⏳ Total Request Time : 34.808 seconds
vLLM + PIECEWISE:
Docker-compose:
Benchmarks:
AGG mean_tps=80.3 max_tps=103.9 mean_q=0.97 passed=6/6
JSON_RESULT={"label": "LLM:instruct", "mean_tps": 80.3, "max_tps": 103.9, "mean_q": 0.9666666666666667, "passed": 6, "total": 6, "by_cat": [{"cat": "list", "tps": 44.4, "q": 0.8}, {"cat": "code", "tps": 95.4, "q": 1.0}, {"cat": "json", "tps": 103.9, "q": 1.0}, {"cat": "reason", "tps": 95.3, "q": 1.0}, {"cat": "tool", "tps": 78.2, "q": 1.0}, {"cat": "essay", "tps": 64.6, "q": 1.0}]}
🚀 TEST: Prefill Heavy (~8000 Context Tokens)
⏱️ Time to First Token (TTFT) : 6.788 seconds
📥 Prompt Tokens : 8020
⚡ PREFILL SPEED : 1181.5 tokens/s
📤 Generated Tokens : 11
⚡ DECODE SPEED : 99.8 tokens/s
⏳ Total Request Time : 6.898 seconds
🚀 TEST: Decode Heavy (Pure Generation)
⏱️ Time to First Token (TTFT) : 0.116 seconds
📥 Prompt Tokens : 35
⚡ PREFILL SPEED : 301.2 tokens/s
📤 Generated Tokens : 1500
⚡ DECODE SPEED : 62.3 tokens/s
⏳ Total Request Time : 24.182 seconds
========== NARRATIVE (prompt=65 chars, max_tokens=1000) ==========
=== warmups (3) ===
warm-1 wall= 13.33s ttft= 114ms toks=1000 wall_TPS= 75.01 decode_TPS= 75.66
warm-2 wall= 13.69s ttft= 90ms toks=1000 wall_TPS= 73.07 decode_TPS= 73.55
warm-3 wall= 13.40s ttft= 95ms toks=1000 wall_TPS= 74.62 decode_TPS= 75.15
=== measured (5) ===
run-1 wall= 13.76s ttft= 89ms toks=1000 wall_TPS= 72.68 decode_TPS= 73.15
run-2 wall= 14.27s ttft= 90ms toks= 991 wall_TPS= 69.44 decode_TPS= 69.88
run-3 wall= 14.16s ttft= 89ms toks= 965 wall_TPS= 68.13 decode_TPS= 68.56
run-4 wall= 13.85s ttft= 87ms toks=1000 wall_TPS= 72.22 decode_TPS= 72.67
run-5 wall= 13.71s ttft= 89ms toks=1000 wall_TPS= 72.94 decode_TPS= 73.42
=== summary [narrative] (n=5) ===
wall_TPS mean= 71.08 std= 2.16 CV= 3.0% min=68.13 max=72.94
decode_TPS mean= 71.54 std= 2.18 CV= 3.0% min=68.56 max=73.42
TTFT mean= 89ms std= 1ms min=87ms max=90ms
========== CODE (prompt=78 chars, max_tokens=800) ==========
=== warmups (3) ===
warm-1 wall= 4.84s ttft= 90ms toks= 455 wall_TPS= 93.97 decode_TPS= 95.75
warm-2 wall= 8.61s ttft= 90ms toks= 800 wall_TPS= 92.86 decode_TPS= 93.84
warm-3 wall= 4.73s ttft= 92ms toks= 454 wall_TPS= 96.00 decode_TPS= 97.90
=== measured (5) ===
run-1 wall= 8.85s ttft= 91ms toks= 800 wall_TPS= 90.44 decode_TPS= 91.39
run-2 wall= 8.23s ttft= 89ms toks= 772 wall_TPS= 93.81 decode_TPS= 94.84
run-3 wall= 6.69s ttft= 93ms toks= 629 wall_TPS= 94.08 decode_TPS= 95.40
run-4 wall= 7.60s ttft= 91ms toks= 675 wall_TPS= 88.80 decode_TPS= 89.87
run-5 wall= 6.96s ttft= 91ms toks= 650 wall_TPS= 93.35 decode_TPS= 94.59
=== summary [code] (n=5) ===
wall_TPS mean= 92.10 std= 2.35 CV= 2.5% min=88.80 max=94.08
decode_TPS mean= 93.22 std= 2.44 CV= 2.6% min=89.87 max=95.40
TTFT mean= 91ms std= 1ms min=89ms max=93ms
=== GPU state ===
0, 90 %, 24014 MiB, 24576 MiB, 249.43 W, 65
1, 0 %, 12841 MiB, 16311 MiB, 5.36 W, 29
=== Last 3 SpecDecoding metrics ===
(APIServer pid=1) INFO 04-30 21:39:33 [metrics.py:101] SpecDecoding metrics: Mean acceptance length: 3.57, Accepted throughput: 67.80 tokens/s, Drafted throughput: 79.20 tokens/s, Accepted: 678 tokens, Drafted: 792 tokens, Per-position acceptance rate: 0.955, 0.871, 0.742, Avg Draft acceptance rate: 85.6%
(APIServer pid=1) INFO 04-30 21:39:43 [metrics.py:101] SpecDecoding metrics: Mean acceptance length: 3.50, Accepted throughput: 66.40 tokens/s, Drafted throughput: 79.80 tokens/s, Accepted: 664 tokens, Drafted: 798 tokens, Per-position acceptance rate: 0.951, 0.838, 0.707, Avg Draft acceptance rate: 83.2%
(APIServer pid=1) INFO 04-30 21:39:53 [metrics.py:101] SpecDecoding metrics: Mean acceptance length: 3.45, Accepted throughput: 64.59 tokens/s, Drafted throughput: 79.19 tokens/s, Accepted: 646 tokens, Drafted: 792 tokens, Per-position acceptance rate: 0.920, 0.826, 0.701, Avg Draft acceptance rate: 81.6%
Verify Script
[1/10] Server reachable on /v1/models ...
�[32m✓�[0m server is serving
[2/10] Genesis patches applied ...
�[33m⊘�[0m no Genesis marker in logs (skipped)
[3/10] Basic completion — capital of France ...
�[32m✓�[0m reply contains 'Paris'
[4/10] Tool calling ...
�[32m✓�[0m tool_calls[] populated with get_weather
[5/10] Streaming (SSE) ...
�[32m✓�[0m streamed 10 chunks, 67 chars: Code breaks, logic fails, One missing semicolon, Sleep is far away. ...
[6/10] Thinking / reasoning mode ...
�[32m✓�[0m reasoning 638 chars, content 3 chars (finish=stop)
�[2mreasoning:�[0m Here's a thinking process: 1. Analyze User Input: -...
�[2mcontent: �[0m 4...
[7/10] Long-context needle (ladder: 10K / 30K / 60K / 90K) ...
�[32m✓�[0m 9819 tokens: recalled 'golden iguana 42' (got: golden iguana 42 )
�[32m✓�[0m 29319 tokens: recalled 'crimson falcon 32' (got: crimson falcon 32 )
�[32m✓�[0m 58571 tokens: recalled 'violet axolotl 76' (got: violet axolotl 76 )
�[33m⊘�[0m scale=1400: HTTP 400 (exceeds --max-model-len, expected — clean rejection)
�[32m✓�[0m all in-budget long-ctx depths recalled secret (above-budget depths cleanly rejected by engine pre-check)
[8/10] Tool response prefill OOM (~15K-token mock tool response) ...
�[32m✓�[0m tool prefill OK — text response (695 chars, finish=stop)
[9/10] Output quality / cascade detection (2K-token completion) ...
�[32m✓�[0m output OK — 9456 chars, variety=0.645, max_line_repeat=0, finish=length
[10/10] MTP acceptance length threshold ...
�[32m✓�[0m MTP acceptance length = 2.72 (>=2.0 — spec-decode contributing)
�[32mAll checks passed.�[0m Stack is ready for full-functionality use.
Beta Was this translation helpful? Give feedback.
All reactions