Qwen3.8-27B-oQ4e on M3 Ultra / M4 Pro (omlx 0.6.1) #2811
Replies: 2 comments
|
Update, 20 August 2026: 64 GB mini vs 512 GB Ultra, named profiles, TurboQuant, ANE Same homemade High levelOne dense multimodal 27B, two honest recipes. RAM changes the KV story, not the weights.
The rest of this post is the measurements and two traps: ANE fraction 0.75 is worse on Ultra, and two omlx profile ids at once 409. Named profiles ( Copy-paste recipes512 GB Ultra (measured, four nodes):
64 GB M4 Pro (measured on two minis, omlx 0.6.2): One catalog id for the agent loop, titles, and compression. Named profiles: thinking vs instruct as catalog idsThis is the bit that made the pack easy to use. Any OpenAI client (Hermes, OpenClaw, OpenCode, a curl) picks thinking or interactive by model id. You do not stuff Not a new 0.6.2 trick. Per-model profiles were requested in #341 (March). The current README already describes What we are documenting is the Qwen3.8 recipe: two (or four) named ids with the card's official thinking vs instruct sets, ANE/MTP copied onto both so a flip does not drop prefill. What
|
| API id | Mode |
|---|---|
Qwen3.8-27B-oQ4e-mtp |
thinking / xhigh (unnamed default) |
Qwen3.8-27B-oQ4e-mtp:thinking |
same as default |
Qwen3.8-27B-oQ4e-mtp:instruct |
official instruct: think off, temp 0.7 / top_p 0.80 / presence 1.5 |
64 GB mini (unnamed default = thinking / medium, for the agent loop):
| API id | Mode |
|---|---|
Qwen3.8-27B-oQ4e-mtp |
thinking / medium |
:thinking-card-default |
xhigh, explicit /model only |
:thinking-medium |
same knobs as unnamed default |
:thinking-low |
brief CoT |
:instruct-non-thinking |
dump-text instead of the loop |
Create them (admin UI or API)
Admin: model → Profiles → save a bundle → tick Expose as model. api_name is the bit after the colon.
API (same body either host; change context / TQ / reasoning_effort to match the recipe above):
# thinking (Ultra xhigh; mini uses medium and api_name thinking-medium)
curl -sS -X POST http://127.0.0.1:8033/admin/api/models/Qwen3.8-27B-oQ4e-mtp/profiles \
-H 'Content-Type: application/json' \
-d '{
"name": "thinking",
"display_name": "Thinking (card default)",
"api_name": "thinking",
"expose_as_model": true,
"settings": {
"enable_thinking": true,
"preserve_thinking": true,
"chat_template_kwargs": {"enable_thinking": true, "reasoning_effort": "xhigh"},
"temperature": 1.0, "top_p": 0.95, "top_k": 20, "presence_penalty": 0.0,
"force_sampling": true,
"mtp_enabled": true, "mtp_num_draft_tokens": 3,
"qwen35_ane_prefill_enabled": true, "qwen35_ane_prefill_fraction": 0.53,
"qwen35_ane_prefill_dual_ane": true, "qwen35_ane_prefill_gdn": true,
"turboquant_kv_enabled": false,
"max_context_window": 262144, "max_tokens": 32768
}
}'
# instruct / interactive
curl -sS -X POST http://127.0.0.1:8033/admin/api/models/Qwen3.8-27B-oQ4e-mtp/profiles \
-H 'Content-Type: application/json' \
-d '{
"name": "instruct",
"display_name": "Instruct (non-thinking)",
"api_name": "instruct",
"expose_as_model": true,
"settings": {
"enable_thinking": false,
"chat_template_kwargs": {"enable_thinking": false},
"temperature": 0.7, "top_p": 0.80, "top_k": 20, "presence_penalty": 1.5,
"force_sampling": true,
"mtp_enabled": true, "mtp_num_draft_tokens": 3,
"qwen35_ane_prefill_enabled": true, "qwen35_ane_prefill_fraction": 0.53,
"qwen35_ane_prefill_dual_ane": true, "qwen35_ane_prefill_gdn": true,
"turboquant_kv_enabled": false,
"max_context_window": 262144, "max_tokens": 32768
}
}'On a 64 GB mini, same shapes with turboquant_kv_enabled: true, turboquant_kv_bits: 4, turboquant_skip_last: true, max_context_window: 131072, and reasoning_effort: "medium" on the unnamed default.
Thinking profiles must set both enable_thinking: true and reasoning_effort. If you only set effort, a previous instruct apply can leave think off and the "thinking" id never thinks.
Copy ANE / MTP / TQ / context onto both profiles. A flip that omits ANE silently reverts prefill.
Call them
# thinking
curl -sS http://127.0.0.1:8033/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{"model":"Qwen3.8-27B-oQ4e-mtp:thinking","messages":[{"role":"user","content":"17×19"}]}'
# interactive / instruct
curl -sS http://127.0.0.1:8033/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{"model":"Qwen3.8-27B-oQ4e-mtp:instruct","messages":[{"role":"user","content":"17×19"}]}'Hermes / OpenClaw / OpenCode: put that string in model. No extra body fields. Thinking ids emit reasoning_content.
GET /v1/models lists the exposed ids a moment before engine_pool accepts them. A throughput bench right after create can 404; retry, or apply onto the base id.
Trap: two variants at once 409
Sampler + chat template change the tokenised prompt, so KV from the other variant usually will not match. omlx keeps one runtime settings variant on the loaded engine. A second id in flight (titles on :instruct while the loop is on the base id) 409s in ~140 ms:
Model is busy; cannot reload runtime settings variant until active requests finish
If the client failovers, you get a cold 24k–75k prefill on another box and think the local model is slow.
Fix: unnamed default = the daily driver. Titles and compression use that same string. :instruct / xhigh replace the loop; they do not sit beside it.
PUT /admin/api/models/{id}/settings can clear active_profile_name (cosmetic; knobs stay). Re-apply the named profile if you care about the label. mtp_num_draft_tokens is not on ModelSettingsRequest; profile apply is the working path.
Why TurboQuant is off on Ultra and on on the mini
This is a dense hybrid, not MLA. KV geometry we used:
64 layers × 4 KV heads × 256 dim × fp16 (K+V) ≈ 256 KB/token.
| Window | fp16 KV | Fits 64 GB (~54.5 GB ceiling + 18 GB weights)? | Fits 512 GB? |
|---|---|---|---|
| 128k | ~32 GB | No | yes, trivially |
| 262k | ~16 GB (16 full-attn layers in this pack) | no | yes |
On 512 GB we left TQ off years ago for Qwen: ~9× prefill, ~40% decode. 262k KV is only ~16 GB. No reason to pay that tax.
On 64 GB, TQ-off 128k does not fit next to TTS/STT. 4-bit + skip_last brings 128k KV to ~8 GB. That is the opposite of the usual advice, and it is why 0.6.2 matters: #2808 (TQ+MTP _QuantizedStateProxy crash) is fixed. We still see 86–92% MTP accept with TQ on. Do not drop to 2/3-bit without an accuracy pass.
Kernel iogpu wired limit on these minis is 56 GB. 54.5 GB is the working custom guard. 52 GB throttled hard around 65–115k. Raising the ceiling to steal from speech is a false win.
Quality: oQ4e is enough
Same bf16 master, greedy, thinking off, MTP 3.
M4 Pro, n=512
| MMLU | GSM8K | ARC-C | avg | |
|---|---|---|---|---|
| oQ4e | 82.0 | 92.8 | 94.0 | 89.6 |
| oQ8e | 81.0 | 94.0 | 95.1 | 90.0 |
M3 Ultra, full sets (omlx intelligence upload)
| MMLU | GSM8K | ARC-C | avg | |
|---|---|---|---|---|
| oQ4e | 82.4 (11571/14042) | 91.4 (1206/1319) | 95.7 (1122/1172) | 89.8 |
| oQ8e | 81.5 | 92.0 | 96.3 | 89.9 |
Tied. Decode, same master, Ultra @8k: 65 tok/s oQ4e vs 43 oQ8e, ~15 GB less resident. Mini: 20.8 vs 16.6. Tool calls (think off, OpenAI tools, write/read/run_python, four tasks): 4/4 vs 4/4, 0 invalid JSON.
Lightning MTP: draft 3
Built-in throughput bench, 8k / tg 128, one depth per Ultra.
mtp_num_draft_tokens |
gen @8k | batch-4 tg |
|---|---|---|
| 1 | 50.0 | 135 |
| 2 | 60.7 | 141 |
| 3 | 75.0 | 143 |
| 4 | 65.6 | 142 |
Same shape as llama.cpp --spec-draft-n-max on a 5090. Tokens per verify step, not raw acceptance. PUT …/settings cannot set mtp_num_draft_tokens (on ModelSettings, not ModelSettingsRequest). Profile apply works.
Thinking vs instruct
Pick the catalog id (:thinking vs :instruct); do not hand-roll temperature. GSM8K n=64, greedy, M3 Ultra. The card is a thinker.
| mode | score | wall |
|---|---|---|
| thinking / xhigh | 62/64 (96.9%) | 521–556 s |
| instruct (think off) | 59/64 (92.2%) | 301 s |
About 5 points at ~1.8× time. That extra time is CoT decode, not slower prefill. Forced instruct often waffles (same 17×19: thinker finished 323; instruct wrote a methods essay).
On the mini agent loop we still do not want xhigh on every tool turn: CoT runs before the call and, with preserve_thinking, fills 128k. Medium is the daily driver. xhigh is an explicit /model switch. 512 GB unnamed default stays xhigh.
ANE prefill on 512 GB (do not raise the fraction)
#2874 is a Max 128 GB recipe. We A/B’d it on four Ultras (one permutation each, 8k/16k/32k, tg 128). MTP 3 and TQ off held.
| arm | 8k PP | 16k PP | 32k PP | 32k e2e |
|---|---|---|---|---|
| ANE off | 443 | 429 | 398 | 84.2 s |
| 0.53 + GDN | 496 | 501 | 468 | 72.0 s |
| 0.53, GDN off | 494 | 488 | 454 | 74.1 s |
| 0.75 + GDN | 460 | 456 | 424 | 79.3 s |
Decode ~68–70 tok/s on all arms. Gain is prefill / TTFT. 0.75 drops GPU util to ~88% and loses. Extra RAM does not want more ANE; the ANE saturates. GDN is ~+3% PP at 32k, near zero at 8k. Mini runs 0.53+GDN; we did not A/B ANE on 64 GB. Do not quote “2× prefill” as a mini number.
Context
512 GB, built-in context bench, no YaRN:
| target | applied | prefill tok/s | wall |
|---|---|---|---|
| 32k | yes | 397 | 97 s |
| 65k | yes | 345 | 204 s |
| 131k | yes | 274 | 494 s |
| 262k | yes | 193 | 1372 s |
Needle at ~80% depth: requested 80k, actually 123k prompt tokens, planted code returned. Gotcha: /admin/api/bench/context writes max_context_window to the target. Restore it after.
64 GB, TQ on: 115k needle, codes at 60% depth and at the end, both retrieved. Thinking still emits reasoning_content. Cold 115k prefill ~20 min (throttled, completes). Warm SSD prefix-cache re-prefill of a long agent thread ~43 s. That is the agent pattern: pay once, reuse the prefix.
Agent loop: one catalog id
Same trap as the profiles section, in agent clothes. Unnamed default on the mini = thinking/medium. Hermes titles and compression use that same string. :instruct / xhigh replace the loop; they do not sit beside it. The 409 + failover-to-another-box path is how a local 140 ms busy looks like a 24k-token remote prefill.
Throughput snapshot (Ultra, 0.6.1, ANE off at the time)
Built-in matrix, code_python, tg 128:
| 1k gen | 4k | 8k | batch-8 | |
|---|---|---|---|---|
| four Ultras | 62–72 | 61–63 | 67–79 | 259–274 |
| same box 0.6.0 | 60 | 60 | 65 | 263 |
| M4 Pro 0.6.0 | 25 | 21 | 21 | 120 |
Prefill then ~420–442 tok/s ANE off; ~470 @32k with 0.53+GDN. Mini decode is about 1/3 of Ultra. Fine for a local agent; the Ultra is the fleet driver.
What we did not measure
- Mini ANE on vs off (we run 0.53+GDN; Ultra A/B only).
- TQ on vs off quality at 128k on the mini (needle passed; no MMLU-with-TQ table).
- xhigh vs medium vs instruct as a Hermes tool-loop bake-off (GSM8K is not a tool loop).
- YaRN to 1M. We do not set it.
Happy to take questions. 0.6.2 on the mini is load-bearing for TQ+MTP. Draft 3 still wants a profile apply, not PUT …/settings.
|
Thanks for the excellent write-up — the 64 GB M4 Pro "Hermes agent" recipe was the direct kicker for re-testing the dense Qwen3.8-27B-oQ4e-mtp pack on the same hardware. We run oMLX 0.6.2 (open-source brew build) on an M4 Pro 64 GB Unified Memory, and reproduced your flags (oQ4e, MTP3, TQ4-bit + skip_last, model-specific ANE fields). Here is an independent measurement set with a clean A/B against the Qwen3.6-35B-A3B-oQ4e MoE default, in six phases plus a real multi-file code-fix. Our usecase is mainly pi agent-related work with text files, not coding-heavy tasks. What we verified1. Both models co-reside on 64 GB. 2. After MTP3 + TQ4, the dense pack is no longer "5× slower". The earlier "dense = 5× slower" result in the thread is largely a without-MT-no-without-TQ historic artifact. With your flags, our long-decode median was ~5.4 tok/s (dense) vs ~9.0 tok/s (MoE) — about 1.7×, not 5×. 3. Dense-instruct is the strong variant for short, structured, agentic tasks. With
4. MoE is still clearly better where context/reasoning depth dominates: long context warm-cache (3.7 s vs 10–15 s dense) and deep reasoning decode throughput (~1.7×). 5. ANE could not be exercised in our build — we flag this as a caveat, not a result. On oMLX 0.6.2 (open source) the enable field is present but the private backend is absent; the server logged per load:
So our numbers above were without ANE acceleration — we can only confirm the field wiring, not its effect. (Presumably requires a closed / non-brew build with the ANE SDK.) 6. Practical takeaway for the 64 GB mini
Thanks for the original thread — the 64 GB recipe + TQ4/skip_last + MTP3 combo is exactly what unblocked the dense model on this small box. Happy to share the full table or scripts on request. |
Uh oh!
There was an error while loading. Please reload this page.
Field report, 18 August 2026. Single-node ep-pool only. Distributed serving off.
Stack: omlx 0.6.1, mlx 0.32.0, macOS 26.6.2. Four Mac Studio M3 Ultra 512 GB as independent
:8033replicas. Quant ladder and tools on a Mac mini M4 Pro 64 GB (omlx 0.6.0). GPU reference: one RTX 5090, llama.cpp NVFP4+MTP (not omlx).Pack: homemade oQ from one bf16 master, native MTP kept. Live id
Qwen3.8-27B-oQ4e-mtp(qwen3_5, 16 GB, vision tower present). 0.6.1 is the right release for this model. This is a thank-you plus the settings that actually survived measurement.Full-set intelligence is on omlx.ai under the Ultra owner hash; n=512/128 on the mini.
Recommendations (the bit people will copy)
Same knobs on Apple silicon. Only the context cap changes with RAM.
Named profiles: unnamed default = thinking /
xhigh.:instruct= think off, temp 0.7 / top_p 0.80 / presence 1.5.-fa on. ~145 tok/s interactive, 236 @8-wide.oQ8e is tied on quality and loses on speed and RAM. We left it on disk, not hot.
Quant: oQ4e is enough
Deterministic greedy, thinking off, MTP 3. Same weights.
M4 Pro, n=512
M3 Ultra, full sets (omlx intelligence upload)
n=128 on oQ4e/5e/6e/8e sat in a 90.6–91.9 band; too small to rank. At 512 and full set they are tied.
Decode, same master, Ultra @8k: 65 tok/s oQ4e vs 43 oQ8e, ~15 GB less resident. Mini: 20.8 vs 16.6.
Tool calls (thinking off, OpenAI tools, write/read/run_python, four tasks): 4/4 vs 4/4, 0 invalid JSON. Built-in accuracy upload skipped the n=64 GSM8K run (
min_questions= 100). That is correct behaviour; flagging it so nobody thinks the upload path failed.Lightning MTP: draft 3 is the peak
Built-in throughput bench, 8k / tg 128, one depth per Ultra so they do not contend. Profile
settings.mtp_num_draft_tokensapplied, model reloaded.mtp_num_draft_tokensSame shape as llama.cpp
--spec-draft-n-maxon a 5090. Tokens per verify step, not raw acceptance.PUT /admin/api/models/{id}/settingscannot setmtp_num_draft_tokens(it is onModelSettingsbut not onModelSettingsRequest). Profile apply works. Worth a one-line on the request model.0.6.1 did not regress 0.6.0
Same built-in matrix (
code_python, pp 1k/4k/8k, tg 128, batch 2/4/8), four Ultras, after 26.6.2 + 0.6.1.Prefill ~420–442 tok/s. 8k gen is the noisy column. Mini 0.6.0: 24.9 / 20.5 / 20.8, batch-8 120. Ultra is about 3× decode, 4× prefill.
Context bench: 262k applies on 512 GB
No YaRN. Built-in
/admin/api/bench/context.Mini OOM around 168k is the memory guard, not the architecture. KV at 262k is ~16 GB (16 full-attn layers).
Needle at ~80% depth: requested 80k, actually 123,130 prompt tokens, planted code returned. Not a 262k needle.
Bug/gotcha: the context bench writes
max_context_windowto the target. After a four-node sweep we had 32k / 64k / 131k / 262k left on disk. Easy to miss. Restoring viaPUTworks. A bench that does not persist the cap, or that restores the previous value, would save someone a bad day.Thinking is the quality knob
Card default is thinking on,
xhigh. The intelligence tables above are thinking off / greedy, so they do not describe the live profile.GSM8K n=64,
enable_thinkingon the accuracy request, two replicas each way:~5 points for 1.8× time. Full-set thinking-off GSM8K was 91.4%. We leave thinking as the unnamed default and expose
:instruct.GPU reference (not omlx, for scale)
llama.cpp,
BLACKWELL_NATIVE_FP4=1,-fa on,Qwen3.8-27B-NVFP4-MTP-MEDIUM.gguf, MTP n-max 3.One stream: 5090 is ~2× Ultra, ~7× mini, and first token is not in the same sport. Many short streams: Ultra 8-wide is the same class as the 5090 (different client). DSPARK lost on that card (0.19 accept vs MTP 0.48–0.62). Accuracy we trust is still the Apple oQ tables.
Small omlx notes from the sweep
max_context_windowto the target (above).mtp_num_draft_tokensis not onModelSettingsRequest; profile apply + reload is the working path.expose_as_modelprofiles show in/v1/modelsbeforeengine_pool.get_entryaccepts them. Throughput bench 404s for a bit; retry or apply onto the base id.Happy to run a 262k needle or a vision pass on this pack if that helps 0.6.1 docs.
All reactions