New model: Tess-4-27B — llama.cpp dual, external MTP, 262K (agentic-leaning alternative to the fast Qwen dual)

[noonghunna]

llamacpp/tess-dual-mtp · migtissera's Tess-4-27B (Qwen3.5-based dense 27B, Q4_K_M GGUF) · llama.cpp (mainline server-cuda-b9246) · dual 3090 layer-split (-ts 1,1) · external MTP spec-decode · 262K ctx. Weights + MTP draft by migtissera; first external-MTP compose in the catalog.

Headline

Tess trades ~½ the throughput of the Qwen3.6-27B dual-max reference for a behavioral-quality tie-to-edge — and it leads the agentic packs (hermes 15/20 vs 9, cli-40 25/40 vs 20) on a smaller VRAM footprint, with a vision-capable base in reserve.

---

🎴 Results Card — 2× RTX 3090, llama.cpp, temp 0.6

① Serving

Config	Spec-dec	KV / ctx	decode TPS (narr / code)	TTFT	VRAM
llamacpp/tess-dual-mtp · Q4_K_M	external MTP n=2 (draft-mtp)	q4_0 / 262K	52 / 68	233 ms	12.6 + 17.2 GB¹
ref: vllm/qwen-27b-dual-max · FP8	MTP n=3	e4m3 / 262K	~114 / 115@35K	—	~22 GB/card (TP=2)

verify-full 8/9² · verify-stress 8/8 (NIAH ladder clean to 240,634 tok = 91% of 262K, ~5.9 GB free at deepest fill) · soak-continuous PASS (0 err, 0/100 silent-empty, p50 decode 66.4, 96.3% retention). Engine-internal decode; 3 warm + 5 measured; temp 0.6 / top-p 0.95 / top-k 20; image ghcr.io/ggml-org/llama.cpp:server-cuda-b9246; reasoning-OFF (shipped default).

_{¹ layer-split -ts 1,1 — the external-draft context lands on card 1, so the split is uneven (both cards keep ample headroom). ² the one miss is streaming tool-calls + thinking-ON → finish=length (see Takeaways).}

② Quality — 8-pack /150 (benchlocal-cli --full, n=1) — thinking OFF / ON vs the dual-max reference

Pack	Tess-4-27B (OFF / ON)	qwen3.6-27b dual-max (OFF)
toolcall-15	14 / 13	14
instructfollow-15	13 / 15	13
structoutput-15	14 / 14	14
dataextract-15	12 / 13	13
reasonmath-15	11 / 12	12
bugfind-15	11 / 14	14
hermesagent-20	15 / 15	9
cli-40	25 / 22	20
TOTAL /150	115 / 118	109

_{Pack noise is ±5–7, so OFF (115 vs 109) is a tie and ON (118 vs 109) a slight edge — but the per-pack shape is the story: Tess leads the two agentic packs decisively (hermes +6, cli +5).}

③ Takeaways

• Ties-to-edges the fast Qwen dual on general behavior (115/118 vs 109) and wins the agentic packs — hermesagent 15/20 (vs 9) and cli-40 25/40 (vs 20). If your workload is tool-loops and agents, Tess is the stronger quality pick.
• The trade is throughput: ~52 narr / 68 code tok/s vs the Qwen dual's ~114 — roughly half. You buy quality/agentic parity + a vision-capable base (mmproj on disk; text-only ship for now) + a smaller footprint (~15 vs ~22 GB/card). Both hit 262K.
• Thinking helps analysis, hurts tool-calling: ON lifts bugfind (+20pp → 93%), instructfollow (→100%), reasonmath/dataextract; but nudges toolcall/cli down. Serve thinking-OFF for tool/agent work, ON for debugging/analysis.
• ⚠️ Caveat: streaming tool-calls with thinking-ON → finish=length (Tess is a heavy reasoner and exhausts the token budget before emitting the call). Non-streaming tool-calls are fine. Use thinking-OFF (the shipped default) or a larger budget for streamed tool-calling.
• tl;dr: a vision-base, agentic-leaning, 262K dual-card alternative to the fast Qwen dual — pick it when you want the quality/agentic edge and can spend the throughput. Ships ⚠️ Production w/ caveats.

---

Why it's in the catalog

The Qwen dual-max is the throughput king on this rig. Tess fills a different slot: same 262K context, but a quality/agentic edge and a vision-capable base, at a smaller VRAM cost — for users who value tool-loop accuracy over raw tok/s. It's also the catalog's first external-MTP compose (migtissera ships the nextn draft head as a separate GGUF, engaged via --spec-draft-model … --spec-type draft-mtp, vs Deckard's embedded head), which we live-validated on mainline llama.cpp b9246.

Getting it

# main Q4_K_M weights + the external MTP draft head (~18 GB total)
hf download migtissera/Tess-4-27B-GGUF \
  Tess-4-27B-Q4_K_M.gguf mtp-Tess-4-27B-Q4_K_M.gguf \
  --local-dir "$MODEL_DIR/tess-4-27b-gguf/migtissera-q4km"

(setup.sh / launch.sh auto-fetch both from the profile.)

Run it

scripts/switch.sh tess-4-27b        # dual-card default → llamacpp/tess-dual-mtp
# serves on :8020, model id "tess-4-27b"
curl -s localhost:8020/v1/models

Run the evals yourself (or add your rig to the matrix)

Got Tess running (above)? Reproduce these numbers — or contribute your rig's — with two independent, non-overlapping passes: behavioral quality (the 8-pack) and operational health (verify / stress / soak / bench / agentic).

One-time: install benchlocal-cli + build the sandboxes

The quality suite wraps benchlocal-cli. Three of the eight packs (bugfind, cli-40, hermesagent) run inside Docker sandboxes that build once:

pip install git+https://github.com/noonghunna/benchlocal-cli.git
git clone https://github.com/noonghunna/benchlocal-cli
bash benchlocal-cli/tools/build-sandboxes.sh     # ~30 GB free; `docker system prune` if tight

(Without the images the 5 deterministic packs still run; the 3 sandboxed ones skip with a warning.)

1. Behavioral quality — the 8-pack, both reasoning modes

bash scripts/quality-test.sh --full --no-thinking       # reasoning OFF
bash scripts/quality-test.sh --full --enable-thinking   # reasoning ON

⚠️ Tess is a thinking model — for the reasoning-ON run to score correctly, boot the compose with reasoning parsing on (REASONING=on) so <think> lands in reasoning_content, not the graded answer.

2. Operational health — verify / stress / soak / bench / agentic

bash scripts/report.sh --full        # ~43 min; redacted, paste-ready bundle

The two passes don't overlap — quality-test.sh is behavioral-only, report.sh --full is operational-only — so together they cover everything with nothing run twice.

Share your numbers: paste the report.sh bundle + your 8-pack totals into a comment here, or via the numbers-from-your-rig issue template.

What'd help

Cross-rig numbers, especially: (a) single-card behavior (the 16 GB Q4_K_M fits one 24 GB card — architecturally valid, not yet benched), and (b) a validated vision variant (mmproj is on disk; we ship text-only until it's gate-passed). Drop numbers via the numbers-from-your-rig issue template.

Credits

Model + GGUF + external MTP head: migtissera. Arch confirmed qwen35-dense (dense = non-MoE; hybrid attention — 48 linear + 16 full-attention layers — carried internally by the GGUF qwen35 arch; corrected 2026-07-11, was "standard GQA"). Validated on the reference rig (2× RTX 3090, PCIe, no NVLink), 2026-07-09.

[trinity-cloud]

Guys, Migel here. Fine-tuning the MTP head as I'm typing this. If all goes to plan, should have an update later today or tomorrow.

[tomByrer]

layer-split -ts 1,1 — the external-draft context lands on card 1, so the split is uneven (both cards keep ample headroom).

I wonder if I can get away with using my RTX3080 as the 2nd card for draft-context, with a smaller context / draft head?

[noonghunna]

you should be able to yes, plenty of headroom.

Might try bartowski/migtissera_Tess-4-27B-GGUF q3 too? I haven't tested it but apparently it has mtp inbuilt so you save on the extra drafter.

[trinity-cloud]

Hey! Here you go: https://huggingface.co/migtissera/Tess-4-27B-EAGLE3

SGLang: --speculative-algorithm EAGLE3 --speculative-draft-model-path migtissera/Tess-4-27B-EAGLE3
vLLM: --speculative-config '{"method":"eagle3","model":"migtissera/Tess-4-27B-EAGLE3","num_speculative_tokens":4}'

More info on the README: https://huggingface.co/migtissera/Tess-4-27B

[sudeposutemizligi]

yet another question : there's a huginnfork/Tess-4-27B-FP8 finetune which is fp8 (reminded me the dual max fp8 on vllm) it says it lacks mtp (lets say its worse with mtp) would this finetuned model and the eagle 3 separate speculator be better in vllm instead of llama cpp for 2 rtx3090s ?

[trinity-cloud]

Also, let me know where you see the biggest gap in the model. I can do a surgical training run and see whether it improves the gaps. Obviously won't be able to match frontier models, but let me know where you'd like to see improvements for the local setups.

[sudeposutemizligi]

congratulations on this good model, i haven't tested yet thoroughly. but heard many good things about it. but i wonder why you(and the 35b moe finetuners)prefer the 3.5 base instead of the newer 3.6?
Edit: now reading at hf it writes 3.6 , but at the beginning of this thread it reads :

llamacpp/tess-dual-mtp · migtissera's Tess-4-27B (Qwen3.5-based dense 27B, ima a bit lost though.

[lolren]

This model is quite the best I've tried from all the Qwen 27B finetunes.
@trinity-cloud is there a way for the models to be fined tuned on 2 specific issues:
Make the model hallucinate less (false positives in a bug hunt) and recheck after writing code or giving answers? Even if more time is spent on the final answer. As this makes a model good.

[trinity-cloud]

Yep, will take this feedback into account when I do the next release. Hallucination can be worked on with RL (DPO) but the recheck -- you sure you want that at the model level? Some folks might be annoyed with it. Probably a good idea to solve at the agent-harness layer.

[lolren]

The harness already has a system prompt for this.
The way I am asking is that, for example the latest Tencent model, HY3, free on open-router, is advertised as reducing false positives to a low level. I was using it, and I could observe that the model check a result at the model level (same harness that I use with Qwen), and after giving a result, it checks it and double checks it, resulting in a very low false positive or hallucinations.
I was therefore thinking that , while this is slower, it actually gives better answers. Slower is not an issue, when running locally.
Maybe I an wrong about this, but I've watched models a lot failing into the trap of just answering without checking a question. One such model, is base 3.6 27B.

[noonghunna]

Cross-rig result: single RTX 4090 matches/beats the dual-3090 reference — from @seanyourhighness in #665 (the single-card datapoint the "What'd help" section asked for):

Full 8-pack breakdown — dual-3090 reference (external MTP n=2, 262K) vs 1× 4090 (DFlash n=4, 96K, llama.cpp b9932 host build, Unsloth-corrected template):

Pack	dual ref OFF	dual ref ON	4090 OFF	4090 ON
toolcall-15	14	13	15	14
instructfollow-15	13	15	13	15
structoutput-15	14	14	14	14
dataextract-15	12	13	12	13
reasonmath-15	11	12	12	14
bugfind-15	11	14	13	14
hermesagent-20	15	15	15	17
cli-40	25	22	23	24
TOTAL /150	115	118	117	125

Serving: decode 78 / 118 narr/code tok/s (vs the dual ref's 52 / 68) at 96K ctx, 22.1 GB on the single card. Same per-pack shape on both rigs: thinking lifts the analysis packs (reasonmath, bugfind, instructfollow) and trims tool-calling — his run independently reproduced our "thinking-OFF for tool/agent work" guidance (toolcall 15→14 with reasoning on).

Notes worth carrying: the quality edge comes from his Unsloth-corrected chat template + a 16K thinking budget (--thinking-max-tokens 16384 killed every token_limit truncation, 2→0) — spec-dec is lossless at the packs' temp-0, so the drafter choice (DFlash vs MTP) affects throughput only. Pack noise is ±5–7, so the think-off totals (117 vs 115) are a tie; the think-on delta (125 vs 118) is the more-real one since the truncation fix is mechanical. Full report, config delta, and the bench-agentic bug he found (fixed in #667): #665. 🙏

[lolren]

So better quality on this model on a 4090? hmm. can this be applied to 3090s?

[noonghunna]

@lolren Yes — and that's the key takeaway: none of the gain is the 4090. Quality is hardware-independent (the same model + config scores the same on a 3090; the faster card only changes tok/s). What actually moved Sean's score, per his own ablations in #665, was two config knobs — both fully applicable on 3090s:

1. A corrected chat template — he pinned the Unsloth-fixed Qwen3.6 template (--chat-template-file) instead of the GGUF's embedded one.
2. A bigger thinking budget on reasoning-ON runs — --thinking-max-tokens 16384 eliminated every token_limit truncation (2 → 0); that's most of the think-on gain.

The DFlash drafter contributed zero quality — speculative decoding is lossless (the target model verifies every drafted token; at the packs' temp-0 the output is byte-identical with or without a drafter). It bought throughput only.

Honest caveats: single run, and the 8-pack noise band is ±5–7 — so read 115→117 (think-off) as "parity, likely small gain"; the think-on delta (118→125) is more real because the truncation fix is mechanical. We plan to re-run the dual-3090 reference with the corrected template + 16K budget to confirm, and if it holds, wire the template into the shipped compose (@trinity-cloud's new drops also ship a fresh chat_template.jinja, which may already fold the fix in).

Usable today on any 3090: bash scripts/quality-test.sh --full --enable-thinking --thinking-max-tokens 16384 — and for your audit-style workloads specifically, reasoning-ON is the right mode regardless: it's the axis where Tess gained most in our A/B (bugfind 73% → 93%).

[lolren]

I smell another big testing coming on :)

[noonghunna]

I got 5 runs in queue for tess + nvfp4. i hope i can make it :)

[noonghunna]

@trinity-cloud Thanks for the drop (EAGLE3 + NVFP4) — we ran the full forensics on the reference 2× RTX 3090 overnight. Three things worth your time:

1. Your EAGLE3 head: the data, and why your 1.76× and our numbers are both true.
Draft acceptance is ~40% at position 1 against both your NVFP4 trunk and an FP8 trunk — we ruled out quantization (4-bit ≡ 8-bit), chat template, and temperature. We also ruled out layer taps: your EAGLE3 config.json omits eagle_aux_hidden_state_layer_ids, so vLLM silently falls back to its own default trunk taps (2, 32, 61) while SpecForge defaults to (1, 32, 60) — we brute-forced all three plausible sets ((2,32,61) / (1,32,60) / (0,31,59)) and acceptance stayed 34–39% on all of them, so a tap mismatch isn't the cause either. Worth pinning the field in the config anyway so SGLang and vLLM users get identical behavior instead of engine-dependent defaults.

At ~40% acceptance, whether EAGLE3 pays is pure overhead economics: against our fast NVFP4 serve (62.4 tok/s) it's −27%; against a slower FP8 serve (41 tok/s) it's +8%; on an H100 with proportionally cheap drafting, your 1.76× is entirely consistent. So on consumer Ampere it currently doesn't pay. The control group that shows the ceiling: NVIDIA's NVFP4 exports keep the mtp.* head unquantized inside the checkpoint — that recipe accepts at 97–99% on the exact same Marlin path on our cards. Two routes that would unlock consumer rigs: (a) regraft/bake an MTP head into your checkpoints NVIDIA-style (llm-compressor drops it during quant — your NVFP4 ships 0 mtp tensors; you already regrafted the base repo once), or (b) a stronger EAGLE3 (acceptance ~70%+ makes it pay everywhere). We know the Tinker-LoRA constraint makes (a) non-trivial — sharing the data so the effort lands where it counts.

2. Biggest quality gap, per our verifier-backed suite: autonomous CLI-agent work. cli-40 is Tess's weakest pack (55–62% vs 87–100% on everything else) — multi-step terminal-agent loops (the Claude Code / opencode workload). A surgical run on agentic tool-loop traces (act → observe → recover) would move the needle most. Second: hallucination control under audit-style prompts (confident false claims — @lolren's ask), where a "verify before asserting" bias would help.

3. Small fix, big downstream effect: the NVFP4 repo's chat_template.jinja is the stock Qwen template (developer-role crash, agentic-loop crashes). vLLM auto-loads it from the model dir, so every vLLM user gets the broken one silently. Shipping the corrected template (unsloth's fix or equivalent) in the repo fixes it for everyone.

Good news to close: via the v0.24 Marlin fallback, Tess-NVFP4 serves on 2× 3090 at 62.4 tok/s — the fastest Tess we've measured on this hardware, no drafter needed. With a working head on top, it'd be the definitive config. Full methodology in this thread + #665.

[trinity-cloud]

Thank you for the detailed feedback @noonghunna. Appreciate it! I'll take these into account for the next training run.

On the point 3 -- do you mind opening a PR on HuggingFace? I can just accept it straightaway. If not, all good -- I'll get it done when I have the time.

[noonghunna]

@trinity-cloud Done — PR opened: https://huggingface.co/migtissera/Tess-4-27B-NVFP4/discussions/2

It's the Unsloth-corrected Qwen3.6 template, drop-in for Tess (~48-line delta): accepts the developer role (stock raise_exceptions on it — the crash every OpenAI-compat agent harness hits) and branches tool-call arguments on mapping vs string (stock crashes mid-agent-loop on replayed string arguments).

So you can accept without re-evaluating: we A/B'd this exact template against stock before filing — full verifier-backed 8-pack on the reference 2× 3090, both reasoning modes. Quality-neutral both modes (totals within the noise band); it changes nothing about how the model scores, it only stops the crashes. Data in #665.

One heads-up: your FP8 repo ships the same stock template — the identical file applies there if you want to mirror it.

[noonghunna]

@trinity-cloud Follow-up with new intel that makes the MTP-retrain path cheaper than it looks. @huginnfork published Tess-4-27B-NVFP4A16 and verified something useful along the way: Tess's 15 mtp.* tensors are byte-identical to the base Qwen3.6-27B head — the fine-tune inherited it untouched (consistent with the Tinker constraint you described).

That also explains the 0%-in-vLLM / 62%-in-llama.cpp split from our earlier report: vLLM's qwen3_5_mtp feeds the head the trunk's hidden state (via mtp.fc), and the fine-tune shifted that feature space — a base-trained head reads misaligned features → 0% accept. llama.cpp's external draft-mtp runs the head token-fed, never touching the drifted representations → 62% accept (Tess still agrees with base at the token level most of the time). His positive control matches ours: the identical head accepts 92.7% on the base trunk, same stack — so the head architecture is proven, only its alignment to your trunk is stale.

The practical consequence: you don't need to train a head from scratch. Initialize from the base head (known-good at 92.7%) and fine-tune only against Tess trunk hidden states — a good init like that typically converges far faster and with less data than from-scratch. And huginnfork's NVFP4A16 already ships the tensor slots regrafted in bf16, so a retrained head would be drop-in for quantized-checkpoint users too (his W4A16 recipe also sidesteps this family's activation-quant sensitivity — worth a look for your own NVFP4 export).

Stakes, to justify the effort: the NVFP4 trunk serves at 62.4 tok/s spec-off on 2× 3090; a head at llama.cpp-level acceptance (~60%+) would put vLLM-Tess in the ~85–95 tok/s range (estimate) — the single biggest speed lever left on this model, and it stacks on top of every quantized export.

[noonghunna]

Update: llama.cpp engine bump → think-ON quality improved. If you're running any of our llama.cpp composes, re-pull.

Chasing the 118-vs-125 gap from @seanyourhighness's 4090 run (#665) ended somewhere unexpected: the engine build itself. Our pin was server-cuda-b9246 (built 2026-05-20 — 721 builds behind current); Sean's host build was b9932. Identical Tess config, only the image changed:

leg	b9246 (old pin)	b9967 (current)
think-OFF (greedy)	115	115 — exact tie
think-ON	118 — pinned across 3 runs	122

Two things worth knowing beyond the headline:

• The lift is thinking-path-specific. Greedy generation ties exactly across 721 builds; the improvement lives entirely in the reasoning leg. Three scenarios we had never passed flipped on the new build — including the safety-refusal scenario (previously a hard 0) and the byte-exact archive-repack one. cli-40 under thinking hit 26/40, our highest measured.
• This corrects our earlier attribution (the cross-rig relay above): Sean's edge was not primarily the corrected template or the 16K thinking budget — we A/B'd both on this rig and they're quality-neutral at the total (the template is a real compat fix though, and the budget protects tool-calling under thinking: 13 → 15/15, so both ship anyway). ≈4 points of the gap was the engine build; the rest is think-ON sampling tail (temp 1.0 by pack contract — single ON runs are draws from a distribution; we now use --repeat 3 for any ON number that gates a decision).

Shipped: the pin is bumped fleet-wide across all llama.cpp composes (#680, server-cuda-b9967), and the Tess compose now defaults --reasoning-budget 16384. The 122 is a single run by our own policy note above — the shipped-config refresh lands in the compose's Quality line once it's run. Full forensics + the step-by-step elimination: #665.

[trinity-cloud]

Nice work! Makes sense for Tess to perform higher with reasoning. I put a lot of effort in to getting the thinking traces right. The shit that I did to bypass the Fable-5 safety classifier to obtain some legit thinking! Then used an ensemble (GPT-5.5, Opus-4.8 and GLM-5.2) to get the intelligence-density and the style approximation right, for all training traces.

The training data came from my real engineering and personal agentic usage of Fable-5, and I trained with 64K context length (not many people can do this).

I'm coming back after 2 year break man, took some time off to touch grass -- lol, but I'm back now. Chat with you all later!

[noonghunna]

And you are back with big bang! great going on training. There is something in your method that has genuinely worked really well. 9/10 fine tunes of qwen 27b are a wash, tess isn't.

I would like to run aider polygot on it at some point to see how it behaves on programming languages. But, if we can address alignment of spec decode drafters (mtp/eagle/dflash etc) , that would be awesome. Tess seems to be what qwen hasn't managed to release yet, qwen 3.7 27b =)

[trinity-cloud]

So with the spec decode drafters -- what I did yesterday isn't working? I did that work on a H100 and it looked like it was working.

[noonghunna]

@trinity-cloud No — your work is fine, and nothing new contradicts your H100 result. Two different claims ended up adjacent and read like one; untangling:

1. "Not the drafter" was about the quality score, not about drafters working. We were hunting a quality-points gap between two rigs. Speculative decoding is output-lossless by construction (the target model verifies every drafted token — greedy verify is exact-match, temp>0 verify is distribution-preserving), so a drafter can never be the cause of a quality-score difference. That line was us eliminating drafters as a suspect — it's a statement in spec-dec's favor, not against it. The gap turned out to be the llama.cpp engine build version (reasoning-path handling changed between builds), which is orthogonal to everything you did.

2. Your EAGLE3 works everywhere it runs — the only open question was ever economics on consumer cards, and that's unchanged from what we posted yesterday:

where	how the head is fed	outcome
your H100, vLLM	hidden-state (native EAGLE3 path)	✅ pays — your 1.76× is real and consistent with our data (drafting is proportionally cheap there)
our 2× 3090, vLLM	hidden-state	runs mechanically fine, but ~40% accept → net-negative vs a fast quantized trunk (−27%), +8% vs a slower one
Sean's 1× 4090, llama.cpp	token-fed external GGUF draft	✅ lossless AND pays — merged as a BENCHMARKS row today (#674)

That third row is new since yesterday and it's good news for you: Sean ran a Q4_K_M of the Tess EAGLE3 head as an external draft on mainline llama.cpp — quality-equivalent to his DFlash baseline, ~76 code tok/s at n=2 on a single 4090 (we've asked him to post the conversion recipe on #674 so it's reproducible). Token-fed drafting sidesteps the trunk-feature-drift issue entirely, so your head now has a proven consumer-GPU path it didn't have when we wrote the Ampere-economics comment. Your head's list of working configurations grew this week; nothing got subtracted.

The one thing that would flip the remaining red cell (consumer vLLM, hidden-state) is the head-alignment retrain we sketched upthread — initialize from the base head (huginnfork's positive control: 92.7% on the base trunk, same stack) and fine-tune only against the Tess trunk's hidden states. Everything else is already working.