Skip to content

Releases

v2.12.3-m1.18

Choose a tag to compare

@shunya1810 shunya1810 released this 18 Nov 10:17

Unofficial M1-family build of MTPLX, tuned for long contexts and coding agents on an M1 Max. Not affiliated with or endorsed by the MTPLX project. Developed with AI assistance (Claude) and measured on a single machine.

  • Base: MTPLX v2.12.3 (youssofal/MTPLX). Native extensions: unchanged since v2.12.2-m1.12 (mfa_attn, ane_ffn, qsa_kernels); coming from m1.12–m1.17 there is nothing to rebuild, coming from an older release run scripts/build-m1-native-extensions.sh.
  • Changes in detail (Japanese): docs/m1max-longctx/CHANGES.ja.md

What is new in v2.12.3-m1.18

Qwen3.8-Flash-Next (Unsloth UD-Q2_0) reads long prompts faster and now reaches 256K on a 64 GB Mac. Its QSA KV is stored in int8 by default. The other models' paths are unchanged.

  • int8 KV (Flash-Next default). One float16 scale per head and token, about 12 KB a token instead of 24 KB. --paged-kv-quantization auto (the default) and q8 turn it on; off or MTPLX_QSA_KV_Q8=0 keeps fp16. Decode, verify and prefill attention read the int8 values in place; the draft head's one layer stays fp16; session save/restore keeps its fp16 form; the memory plan prices int8. In m1.17 the Unsloth file (41.9 GiB of weights, fp16 KV) did not reliably fit past 128K tokens: 192K now completes, and 256K too.
  • Prefill attention on the rows-MMA kernel. The verify kernel that reads each row's ~2,050 selected tokens in place (m1.17) now also serves 4,096-row prefill chunks, in about half the time of the vendored Steel kernel per layer and chunk (32K / 64K / 128K: 103 / 109 / 111 ms → 53 / 55 / 57 ms; int8: 67 / 67 / 69 ms). On by default with int8 KV; for fp16 KV MTPLX_QSA_PREFILL_ROWS_MMA=1.
  • Selection scores in one kernel (qsa_indexer_prefill_scores_simd): relu and the head sum happen inside the simdgroup matrix product, no per-head fp32 planes. With 32K+ of history the selected blocks are the same (ISTA pack, 128K prefill 373 → 404 tok/s). MTPLX_QSA_PREFILL_SIMD_SCORES=0 restores the MLX expression.
  • Long prompts. While the prompt's KV is small the prefill runs wide (up to 4,096 rows), with a 4,096 / 2,048 / 1,024 / 512 width ladder; from 131,072 tokens the decode-only packs are dropped during prefill; from 200K interior recurrent boundaries are skipped and the QSA buffers are sized once.
  • Prefill experts. The un-sort, the weighted sum and silu×up are kernels, and the tiles come in two sizes (32 and 16 rows) to waste less on an expert's last tile. Large cold n-gram row batches are read ahead with F_RDADVISE (macOS); the n-gram row cache is 256 MB by default.
  • ISTA Q2_0 packs keep their experts as GGUF blocks (−3.3 to −3.6 GB, decode unchanged); Unsloth's file already did.
  • Decode and verify. The 1-row decode reads only the selected blocks on M1 Flash-Next by default; the hyper-connection read and the compiled decode take up to 8 rows.
  • Off by default (kept, no gain): MTPLX_QSA_DECODE_SCORES=1 and MTPLX_QSA_TOPK_KERNEL=1 (a fused decode/verify selection and a radix top-k: faster in 12-layer micro-benchmarks, 53.7 → 55.0 ms slower in a real 128K verify forward); the compiled fixed-bank verify (MTPLX_QWEN4_FIXED_M4_VERIFY=1, 50.2 → 54.4 ms at 8K); the DFlash 2 drafter module (mtplx/dflash2.py, not wired in: more tokens per round, offset by draft time).

Speed, m1.17 → m1.18 (Unsloth UD-Q2_0, M1 Max 64 GB, servers alternated for two rounds the same day, llm-bench lean code / Japanese prompts, temperature 0; each value is the two rounds)

m1.17 m1.18
prefill 8K, code / Japanese (tok/s) 373, 368 / 379, 379 395, 396 / 401, 398
prefill 128K, code (tok/s) 281, 281 388, 389
decode 128K, code (tok/s) 36.7, 36.8 40.5, 40.3
peak memory 128K (GB) 54.6, 54.5 53.0, 53.1
decode 8K, code / Japanese (tok/s) 48.5, 49.0 / 42.7, 42.8 46.5, 46.6 / 38.3, 38.3

The 8K decode rows are lower on these two texts because fewer tokens were accepted per round (3.11 → 3.00 and 2.75 → 2.49); a round takes the same 64 ms. On four 8K prompts (code / Japanese / English / agent log, greedy, 300 tokens) tokens per round were 2.73 / 2.17 / 2.33 / 2.63 with fp16 KV (mean 2.47) and 2.75 / 2.10 / 2.33 / 2.75 with the int8 default (mean 2.48): no systematic change, text-to-text variation.

Long contexts, m1.18 only (code prompt, one run each, --context-window 262144; the default window on a 64 GB Mac stays 65,536)

cold prompt read prefill decode peak memory
128K 338 s 388 tok/s 40.5 tok/s 53.0 GB
192K 524 s 375 tok/s 38.8 tok/s 54.0 GB
256K 700 s 374 tok/s 36.8 tok/s 53.4 GB

m1.17 completed 192K in 1 of 4 runs and refused 256K. With the same build and --paged-kv-quantization off, 192K stopped after about 377 s (memory guard) and 256K was refused. Follow-up turns and the other genres were not re-measured at these lengths.

Accuracy. Paired per-token comparisons: Unsloth m1.17 → m1.18 mean NLL difference +0.0003 / −0.0018 / +0.0009 / −0.0021 on English prose / Japanese prose / code / an agent log (all within 1.1 standard errors); int8 KV through 4-row verify forwards after 32K of history −0.00002 (SE 0.003) and −0.0016 (SE 0.005) on code and Japanese.

  • Tests: the failures are the 84 that also fail on m1.17 (same set). Tests were added for the int8 cache, the int8 and rows-MMA kernels, the fused score kernel and the radix top-k.

Everything in v2.12.3-m1.17 is included.

v2.12.3-m1.17

Choose a tag to compare

@shunya1810 shunya1810 released this 16 Nov 08:01

Unofficial M1-family build of MTPLX, tuned for long contexts and coding agents on an M1 Max. Not affiliated with or endorsed by the MTPLX project. Developed with AI assistance (Claude) and measured on a single machine.

  • Base: MTPLX v2.12.3 (youssofal/MTPLX). Native extensions: unchanged since v2.12.2-m1.12 (mfa_attn, ane_ffn, qsa_kernels); coming from m1.12–m1.16 there is nothing to rebuild, coming from an older release run scripts/build-m1-native-extensions.sh.
  • Changes in detail (Japanese): docs/m1max-longctx/CHANGES.ja.md

What is new in v2.12.3-m1.17

Qwen3.8-Flash-Next (Unsloth UD-Q2_0) decodes and reads prompts faster; Japanese decode is 1.4–1.6x with thinking on. Long anonymous chats no longer re-read their whole prompt on every turn. The other models' paths are unchanged apart from that session change.

  • Japanese drafting. Flash-Next's FR-Spec draft was pinned by upstream's qwen4 settings to a code-only 64K vocabulary, which covers 78–84% of Japanese text tokens: the first draft was accepted in 35% of Japanese rounds. The M1 family now drafts from the Japanese-aware list (qwen38-code64k-ja), and the K20 prescatter draft route admits its width.
  • QSA verify attention reads each row's token list in place (mtplx/kernels/qsa_rows_mma.py): 0.10–0.17 ms a layer instead of about 1 ms for 4 verify rows (8K–60K), no gathered copies, no full-context mask. MTPLX_QSA_ROWS_MMA=0 restores the stock routes.
  • Prefill experts on MLX's quantized-matmul tile (mtplx/kernels/moe_gguf_qmm.py): MLX's own qmm_t tile with a GGUF block loader. One layer at 2,048 tokens 53–55 → 42–47 ms; perplexity unchanged.
  • Smaller ones: the hyper-connection prefill kernels take fp16 (the M1 packs) and the decode read is faster (1 row 47 → 34 µs); the first ~2K tokens of a prompt use the fork's MMA prefill attention instead of an unfused head-dim-256 SDPA.
  • Long anonymous chats. A request without a session id or agent tools kept its state only as a copy, and a copy past the per-session cap was dropped: on a 64 GB Mac a 131K-token Flash-Next follow-up re-read its prompt for 423 s. Prompts of 32K tokens and more now keep the live cache (MTPLX_SESSIONBANK_LIVE_REF_MIN_PROMPT_TOKENS, 0 turns it off); with default settings the same 128K follow-up now starts in 0.6 s.
  • Correction: the v2.12.3-m1.16 notes said Flash-Next was measured with 8-bit KV. Its QSA attention has no quantized-cache path, so q8 falls back to fp16; every Flash-Next number (then and now) is fp16 KV.
  • Neural Engine (experimental, off): MTPLX_ANE_PREFILL_QWEN4_GDN=1 splits the GDN input projection between the GPU and the ANE: a 2,048-token chunk 5.57 → 5.26 s, but within noise on the server's 4,096-token chunks, so it stays off.

Speed, 2K–32K (M1 Max 64 GB, llm-bench b3 tasks, thinking on, fp16 KV, m1.16 and m1.17 servers alternated; m1.16 → m1.17)

cold prompt read 2K / 8K / 32K (s) decode with thinking 2K / 8K / 32K (tok/s)
code 5.81 / 21.9 / 93.9 → 5.45 / 20.6 / 83.0 45.3 / 45.6 / 41.3 → 47.8 / 48.1 / 44.2
English prose 5.73 / 21.8 / 87.7 → 5.36 / 20.5 / 83.1 41.2 / 37.5 / 36.5 → 42.0 / 40.7 / 41.6
Japanese prose 5.77 / 21.7 / 92.2 → 5.40 / 20.4 / 82.6 30.5 / 28.5 / 26.6 → 43.7 / 44.6 / 38.4
agent log 5.96 / 22.1 / 88.0 → 5.60 / 20.8 / 88.1 44.2 / 45.4 / 44.3 → 45.3 / 45.9 / 48.6

Facts buried in the prompt: 48/48 on both. Peak memory unchanged (about 50 GB at 2K, 53–54 GB at 32K).

Long contexts, m1.17 only (server started with --context-window 262144; the default window on a 64 GB Mac is 65,536)

cold prompt read follow-up read decode with thinking completed
128K, all four genres 431–448 s 0.5–1.0 s 34.8–42.8 tok/s 4 of 4
160K 645–657 s 0.9–1.1 s 32.9–33.0 tok/s 2 of 4
192K 738 s 1.2 s 40.5 tok/s 1 of 4
256K — — — does not fit
  • At 160K and 192K the Mac ran out of free memory near the end of the prompt in some runs and the memory guard stopped that request (the server stays up); whether it fits depends on what else is running. 256K is refused at admission (the machine fit is 200,704 tokens) and, admitted with --allow-swap, ran out of memory 826 s into the prompt.

  • The bench runner sends no session id, so these runs set MTPLX_SESSIONBANK_LIVE_REFS_FOR_IMPLICIT_SESSIONS=1 (what agent clients get); with this release prompts past 32K tokens get it without the setting.

  • Tried and dropped: five rewrites of the decode GLU kernel (tables in device memory, computed signs, three ways of grouping the verify tokens that share an expert), other prefill tile shapes, a half8 grid table in the prefill tile, split-K for the hyper-connection read, the fused QSA indexer (slower at verify), the ANE q-projection split and resident ANE weights.

  • Tests: the failures are the 84 that also fail on m1.16 (the full suite run on both, the failing test names are identical). 15 test cases were added.

Everything in v2.12.3-m1.16 is included.

v2.12.3-m1.16

Choose a tag to compare

@shunya1810 shunya1810 released this 15 Nov 19:12

Unofficial M1-family build of MTPLX, tuned for long contexts and coding agents on an M1 Max. Not affiliated with or endorsed by the MTPLX project. Developed with AI assistance (Claude) and measured on a single machine.

  • Base: MTPLX v2.12.3 (youssofal/MTPLX). Native extensions: unchanged since v2.12.2-m1.12 (mfa_attn, ane_ffn, qsa_kernels); coming from m1.12–m1.15 there is nothing to rebuild, coming from an older release run scripts/build-m1-native-extensions.sh.
  • Changes in detail (Japanese): docs/m1max-longctx/CHANGES.ja.md

What is new in v2.12.3-m1.16

Qwen3.8-Flash-Next runs on a 64 GB M1 Mac from Unsloth's UD-Q2_0 GGUF, the recommended file: far lower perplexity than ISTA's Q2_0 GGUF for a few percent of speed. The other models' paths are unchanged.

  • Setup: scripts/setup_flashnext_unsloth.py downloads the GGUF (3 splits, 70.5 GB, resumable) and the config, tokenizer and MTP head of Youssofal's Optimized-Speed pack, then converts them in a few minutes on the CPU. Needs about 78 GB of free disk. Steps: USAGE.md.
  • Routed experts read as GGUF blocks. Unsloth quantizes each layer's experts as IQ2_XXS or IQ2_S (gate, up) and Q2_0 or IQ4_NL (down), none of which is an MLX affine grid, so the blocks are kept as they are and read by new M1 kernels (mtplx/kernels/moe_gguf_iq.py): a decode/verify GLU with half8 grid and sign tables, a down pass that folds in the router weights and the shared expert, and an expert-grouped, software-pipelined simdgroup-matrix prefill. Tables and block layouts from ggml (MIT, credited in NOTICE).
  • Dense weights from Unsloth too. ISTA's GGUF keeps the dense layers at Q2_0–Q4_K (the shared expert is Q2_0); Unsloth's are Q5_K–Q8_0. Both are requantized to 8-bit, so memory does not grow.
  • Quality (perplexity, 4 texts × 4 places × 1,024 tokens; English prose / Japanese prose / code / an agent log): ISTA 7.30 / 34.6 / 2.77 / 2.65, Unsloth 6.77 / 20.2 / 2.47 / 1.94 (Qwen3.8-27B Optimized-Speed pack: 6.97 / 21.8 / 2.50 / 2.23). All four differences are well outside the 95% intervals.
  • Speed (llm-bench b3 tasks, thinking on, fp16 KV — corrected in v2.12.3-m1.17: this line first said 8-bit KV, but Flash-Next falls back to fp16 when q8 is requested — ISTA and Unsloth servers alternated on the same build; ISTA → Unsloth). Decode after a 2K prompt, code / English / Japanese / agent log: 42.6 / 39.9 / 29.9 / 42.8 → 40.9 / 37.3 / 28.1 / 40.9 tok/s; after 32K: 39.6 / 34.6 / 25.2 / 40.0 → 37.4 / 32.7 / 23.8 / 41.3. Cold prompt reading is 11–17% slower at 2K–8K and 5–8% at 32K (32K code: 97.9 → 102.5 s). Facts buried in the prompt: Unsloth answered all 48, ISTA missed 2 of 4 on the 32K agent log. Peak memory is the same (about 50 GB at 2K, 53 GB at 32K).
  • Flash-Next decode: mid-forward async evals every 2 layers instead of 6 (1–1.6% shorter rounds). Packs may now declare an MTP depth policy in mtplx_runtime.json (adaptive_policy_default); the converted Flash-Next packs keep static depth 3, which beat the streak policy with thinking on (agent 2K 42.8 vs 36.0 tok/s).
  • Tried and dropped: an 8-bit n-gram table built from the official BF16 weights (same perplexity as IQ4_NL), the official BF16 / 8-bit MTP heads (same acceptance), and several prefill tile layouts.
  • Tests: the failures are the 84 that also fail on m1.15 (the full suite run on both, the failing test names are identical). Eight test cases were added.

Everything in v2.12.3-m1.15 is included.

v2.12.3-m1.15

Choose a tag to compare

@shunya1810 shunya1810 released this 14 Nov 13:17

Unofficial M1-family build of MTPLX, tuned for long contexts and coding agents on an M1 Max. Not affiliated with or endorsed by the MTPLX project. Developed with AI assistance (Claude) and measured on a single machine.

  • Base: MTPLX v2.12.3 (youssofal/MTPLX). Native extensions: unchanged since v2.12.2-m1.12 (mfa_attn, ane_ffn, qsa_kernels); coming from m1.12–m1.14 there is nothing to rebuild, coming from an older release run scripts/build-m1-native-extensions.sh.
  • Changes in detail (Japanese): docs/m1max-longctx/CHANGES.ja.md

What is new in v2.12.3-m1.15

Qwen3.6-35B-A3B family models read short prompts and the short follow-up turns of an agent session faster. Decode and prefill of 29K tokens and longer are unchanged.

  • Prompt-tail restore points are recorded inside the last wide forward. A cold prompt records a few restore points (GDN boundaries) near its end, and the last chunk used to be cut at them: a 2,093-token prompt ran as 1,024 + 1,024 + 45 rows. On the A3B a narrow forward is slow (an MoE forward costs about 217 ms plus 0.345 ms per row), so 2,016 tokens took 2.14 s and 2,094 tokens 3.0 s. The gated-delta recurrence is sequential in the token, so running it in segments gives the state of a forward cut there; the layers now record it inside one wide forward. MTPLX_QWEN35_INFORWARD_BOUNDARY=0 restores the old cut.
  • The Neural Engine MoE pipeline serves chunks of 2,560 rows and more (MTPLX_ANE_PREFILL_MOE_MIN_ROWS); it splits a chunk in two, which made narrow chunks slower than the GPU alone.
  • Narrow MoE tiles step K by 64 (16-row tiles, groups of 64 or more; bit-identical to the stock op at 2, 4 and 8 bits). One 512-row forward: 810 → 651 ms.
  • Speed. Servers alternated with m1.14 (A3B Youssofal Optimized-Speed FP16, 8-bit KV, a different cold prompt per request). Cold prefill, tok/s, m1.14 → m1.15: 2.1K 691 → 1,049; 2.5K 774 → 1,078; 3.6K 889 → 1,177; 5.8K 1,007 → 1,084; 7.3K 939 → 1,088; 10.6K 937 → 1,030; 14.8K 925 → 1,003; 29K 864 → 867. Follow-up turns on a 3K cached prefix, time to read the new tokens: 336 tokens 667 → 486 ms, 560 tokens 808 → 738 ms. A 14-turn agent session on a 19K-token system prompt spent 44.5 / 46.3 s reading new tokens on m1.14 and 39.4 / 39.7 s on m1.15 (−11% to −14%).
  • Unchanged: decode (2K greedy 110.3 / 110.5 → 110.5 / 110.8 tok/s), peak memory (32.6 GB through 128K, 33.8 GB at 256K), restore behaviour (turns, retries, forks and an SSD-cache restart reuse the same number of tokens as m1.14). 64K prefill: 635 / 639 → 640 / 641 tok/s; 256K: 175 → 156 and 180 tok/s (run-to-run spread).
  • Answers differ from m1.14. Prompts are cut at other places, and chunks under 2,560 rows no longer go through the Neural Engine.
  • Opt-in, off by default: MTPLX_M1_MOE_SEG_GATHER_LOW_ROWS=384 lets the MoE tiles serve prefill chunks of 48–127 tokens (64 tokens 175 → 153 ms per chunk, 127 tokens 329 → 224 ms). It is not bit-identical to the stock op: on real hidden states 96.9–99.0% of next tokens agree.
  • Tried and dropped: fusing MoE stages 2 and 3 (already near the bandwidth limit in the graph), reading each shared expert once in stage 2 (bit-identical, 1–2% in isolation), resident depth 2 (no gain on average), merging the MTP history forward into the draft (no measurable cost to remove).
  • Tests: the failures are the 84 that also fail on m1.14 (the full suite run on both, the failing test names are identical). Five tests were added (restore-point recording, the K step of the narrow tiles, the low-row default).

Everything in v2.12.3-m1.14 is included.

v2.12.3-m1.14

Choose a tag to compare

@shunya1810 shunya1810 released this 07 Nov 12:00

Unofficial M1-family build of MTPLX, tuned for long contexts and coding agents on an M1 Max. Not affiliated with or endorsed by the MTPLX project. Developed with AI assistance (Claude) and measured on a single machine.

  • Base: MTPLX v2.12.3 (youssofal/MTPLX). Native extensions: unchanged since v2.12.2-m1.12 (mfa_attn, ane_ffn, qsa_kernels); coming from m1.12 or m1.13 there is nothing to rebuild, coming from an older release run scripts/build-m1-native-extensions.sh.
  • Changes in detail (Japanese): docs/m1max-longctx/CHANGES.ja.md

What is new in v2.12.3-m1.14

This release brings in upstream MTPLX 2.12.3. It has no fork-only changes.

  • Upstream 2.12.3. The full list is in upstream's CHANGELOG. Among them:

    • Long sessions from clients that send no session id keep their cache when the resent history differs inside a turn the model generated (youssofal#624). This rule runs before the fork's token-seam rule; a restore still reuses only the tokens verified to match.
    • Plain decoding (--generation-mode ar and friends) saves each prompt before its answer, so a follow-up resumes at the previous prompt's end.
    • The turn's final save keeps the restore points (GDN boundaries) recorded while its prompt was read, as far as the histories match. This replaces the fork's own version of the same fix.
    • Retries and regenerates resume from the saved prompt (youssofal#661); SSD-cache restore fixes (youssofal#608, youssofal#609, youssofal#655, youssofal#668); a server at Metal's 499,000-buffer limit keeps serving (youssofal#640); Claude Code, Codex and OpenCode 2.x fixes; --trust-network; dashboard improvements.
  • Conflicts. Four files. The fork's own paths are kept (retrying a request when macOS stops a long GPU command, releasing an entry restored at a checkpoint inside it, M1 MMA prefill and draft attention) and upstream's additions sit next to them.

  • One default differs from upstream. When an answer's entry replaces its prompt's entry, the fork accepts a restore point one token short (it avoids a 3.8 GB duplicate at 64K; MTPLX_SESSION_BANK_SUPERSEDE_TAIL_SLACK=0 matches upstream). A follow-up therefore resumes one token before the previous prompt's end.

  • Speed is the same as m1.13. Qwen3.8-27B and Qwen3.6-35B-A3B (Youssofal Optimized-Speed FP16 packs), servers alternated with m1.13 (8-bit KV, SSD session cache off, a fresh server and a cold prompt per length); cells that differed in the first round were measured again in the opposite order. tok/s, m1.13 → m1.14:

    A3B prefill A3B decode 27B prefill 27B decode
    8K 945 → 952 107.3 → 107.0 201 → 202 26.7 → 26.7
    32K 910 → 907 89.0 → 89.0 189 → 188 26.3 → 26.2
    64K 629 → 629 76.8 → 76.6 157 → 157 23.4 → 23.4
    128K 383 → 391 58.4 → 58.4 122 → 121 20.5 → 20.5
    short prompts — 116.2 → 116.4 — 28.9 → 28.9

    Every difference is 1% or less; peak memory is the same. The README tables were not re-measured.

  • Tests: the failures are the 84 that also fail on m1.13. Five fork tests were updated to the behaviour upstream changed on purpose (session identity, restore-point carry-over, entry replacement), and one upstream test runs with the tail slack above set to 0.

Everything in v2.12.2-m1.13 is included.

v2.12.2-m1.13

Choose a tag to compare

@shunya1810 shunya1810 released this 05 Nov 10:07

Unofficial M1-family build of MTPLX, tuned for long contexts and coding agents on an M1 Max. Not affiliated with or endorsed by the MTPLX project. Developed with AI assistance (Claude) and measured on a single machine.

  • Base: MTPLX v2.12.2 (youssofal/MTPLX). Native extensions: unchanged since v2.12.2-m1.12 (mfa_attn, ane_ffn, qsa_kernels); coming from m1.12 there is nothing to rebuild, coming from an older release run scripts/build-m1-native-extensions.sh.
  • Changes in detail (Japanese): docs/m1max-longctx/CHANGES.ja.md

What is new in v2.12.2-m1.13

  • Qwen3.6-35B-A3B family decode about 15–25% faster. Servers alternated with m1.12 twice (8-bit KV, SSD session cache off): greedy decode 100.3 → 125.7 tok/s at 2K (+25%), 83.3 → 96.8 at about 29K tokens (+16%), 58.9 → 66.5 at about 104K (+13%); temperature 1.0 at 2K 93.4 → 115.9 (+24%). Greedy output is the same as m1.12 (2K: two seeds four times each, 29K and 104K twice each, all identical).
    • Each GDN layer of the verify (30 layers) runs as one kernel: the conv, SiLU, recurrence and gated RMSNorm used to be eight dependent GPU dispatches. On M1 a dependent dispatch costs about 4.5 µs whatever its work, and these eight did almost none. One verify 15.1 → 14.0 ms. The output is bit-identical to the chain it replaces (checked on all 30 layers with real weights and states; the server re-checks it at every load).
    • The 4-bit projections of the 2-row verify compute both rows in one pass over the weights (the stock kernel runs once per row and reads the weights twice). Greedy verifies use arithmetic bit-identical to the stock kernel; with temperature above 0 a faster variant holds the inputs in half precision (relative error up to about 7e-4).
    • The MoE down projection is split by row, and the gate/up projections keep their inputs in 16-bit registers (both bit-identical). The residual add now runs inside the next RMSNorm's kernel (bit-identical).
    • With temperature above 0, the candidates are picked from per-chunk winners instead of sorting each 248K-wide row (about 0.5 → 0.09 ms of GPU per round; the same candidates).
    • With temperature above 0, output for a given seed differs from m1.12 (half-precision projection inputs); the sampling distribution is the same.
  • fp16 KV: a lighter attention kernel for verify windows of up to 16 rows (4 simdgroups instead of 8). With the CLI default (auto, fp16 KV below 128K tokens), Qwen3.6-35B-A3B greedy decode 78.8 → 80.3 tok/s at about 29K and 55.5 → 58.4 at about 104K. From 16K tokens the attention is split differently, so the output changes in the last bits. 8-bit KV (the app's default and the README tables) is unchanged.
  • Qwen3.8-Flash-Next from ISTA's Q2_0 GGUF (experimental). scripts/convert_flashnext_ista_gguf.py --gguf <both splits> --ref <dir> --out <dir> turns the GGUF Splash serves on 64 GB Macs (ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF, Q2_0, two splits) into a model folder. --ref is a folder with the config.json, mtplx_runtime.json, tokenizer and chat template of Youssofal/Qwen3.8-Flash-Next-MTPLX-Optimized-Speed. The 2-bit experts and the n-gram table stay in the GGUF and are read at load. The GGUF has no MTP head: put that pack's mtp.safetensors in the output folder. M1 additions: 2-bit expert kernels, blocked GDN prefill, sparse attention (QSA) from 8K. Against Splash 1.5.0-m1 on the M1 Max: thinking decode at 85–95% of Splash's speed (Japanese about 70%), the same time to first token at 2K and 8K, 87 s against 78 s at 32K, peak memory 53 GB against 41 GB.
  • Parallel tool results reach the template in tool-call order. Agent clients (Claude Code, Codex) return parallel tool results in completion order. The Qwen templates render each result as a bare <tool_response> without its call id, so a swapped pair made the model read each result as the other call's (the same bug as oMLX #4999). Each run of tool messages is now sorted by the preceding assistant's tool_calls order, for /v1/chat/completions, /v1/messages and /v1/responses.
  • README tables: the Qwen3.6-35B-A3B and Ornith-1.5-35B-A3B rows were re-measured on m1.13. Decode: short prompts 107.6 → 128.5 (Qwen3.6) and 110.5 → 135.2 tok/s (Ornith), 128K 60.4 → 63.5 and 63.8 → 92.8 (Ornith's 128K answer had 289 of its 512 tokens written by context-copy). Qwen3.6-35B-A3B's prefill came out below the old table at 32K–64K (32K: 1,028 → 931 tok/s) while Ornith-1.5-35B-A3B, with the same layout, matched it (1,034 → 1,026); on m1.10 the two were equal. m1.13 does not change prefill and was not slower than m1.12 on the same day (cold prompts of about 29K and 104K tokens: 910 and 513 tok/s against 697 and 444). The cause is being investigated.

Not re-measured in this release: the 27B models (this release does not change their path with the 8-bit KV cache). Everything in v2.12.2-m1.12 is included.

v2.12.2-m1.12

Choose a tag to compare

@shunya1810 shunya1810 released this 31 Oct 08:41

Unofficial M1-family build of MTPLX, tuned for long contexts and coding agents on an M1 Max. Not affiliated with or endorsed by the MTPLX project. Developed with AI assistance (Claude) and measured on a single machine.

  • Base: MTPLX v2.12.2 (youssofal/MTPLX). Native extensions: qsa_kernels joined the build; only the Qwen4-family path uses it. mfa_attn and ane_ffn are unchanged since v2.12.2-m1.11, so coming from m1.11, run scripts/build-m1-native-extensions.sh only if you want that path; coming from an older release, run it.
  • Changes in detail (Japanese): docs/m1max-longctx/CHANGES.ja.md

What is new in v2.12.2-m1.12

  • Dense packs with bf16 scales no longer end in NaN on prompts of 512 tokens or more. Present since v2.12.2-m1.1. On M1, prefill chunks of 512 rows and up hand part of the work to the Neural Engine, and the kernels that stage its weights read the quantization scales and biases as fp16. A bf16 pack's bits were read as fp16 values, and every logit came out NaN. With mlx-community/Qwen3.8-27B-4bit (bf16 scales), a 1,536-token prefill gave all-NaN logits (finite with MTPLX_ANE_PREFILL=0), and the server failed a 2,400-token prompt with HTTP 500 (non_finite_logits). The scales and biases bound for the Neural Engine are now cast to fp16 per call, and the same prompt is answered correctly.
    • The models in the README (Youssofal's FP16 packs and three packs converted with MTPLX forge) have fp16 scales, so they were never affected; the cast does nothing for them.
    • mlx-community's 4-bit conversions carry no MTP heads, so MTPLX serves them without speculative decoding (upstream too).
  • Groundwork for Qwen4 (not tested on a real Qwen4 model yet). Upstream's qwen4_exp path is written for the shape of Qwen3.8-Flash-Next (hidden size 2560, MoE, 24 query / 2 KV heads). In case a Qwen4 model uses this architecture at another size, the path now takes other shapes and is tuned for M1. Tested on synthetic packs with random weights at Qwen3.8-27B widths (hidden 5120, 64 layers, 24 query / 4 KV heads, dense MLP, fp16 and bf16).
    • Dense MLP layers. The load-time projection fusions assumed width 2560, so a 5120-wide pack did not load; they now read the model's width.
    • The QSA sparse-attention prefill kernel served only 24 query / 2 KV heads, so with any other head layout every attention layer fell back to dense attention past 32K. It now takes layouts with GQA 1–16, and M1 switches to it from 16K. For a 2,048-row chunk at 36K, attention went from 403 to 245 ms per layer and the chunk from 15.9 to 14.3 s.
    • The fork's M1 verify matmul kernels now serve this path (depth 3 at 2K: one verify 196 → 135 ms), and a hyper-connection read kernel for 1–4 rows takes it to 126 ms.
    • M1 GPUs have no bf16 arithmetic, so bf16 packs of this family run in fp16 by default: 8K prefill about 1.4× faster (169 tok/s), verify about 10% shorter. MTPLX_M1_FP16_CAST=0 turns it off; MTPLX_M1_FP16_CAST=1 turns it on for other families (no effect on fp16 packs).
    • Flash-Next-only fused kernels (2560 wide, opt-in through environment variables) are no longer taken by models of other widths.
    • Through the server, the synthetic fp16 pack prefills at 160 tok/s at 1.6K and 133 tok/s at 77K, with 120 ms per verify at 1.6K. With random weights the draft never matches, so decode speed is not measured yet.
  • No change for the models in the README. Against m1.11 with the same prompts, servers alternated twice with the SSD session cache off: Qwen3.8-27B prefill 183 → 185 tok/s at 1.6K and 192 → 192 at 25K, one verify 88.5 → 87.1 ms; Qwen3.6-35B-A3B prefill 1,018 → 1,020 tok/s at 1.6K and 1,097 → 1,103 at 6.3K, one verify 12.0 → 11.9 ms. Decode tok/s moves by about ±5% with draft acceptance, so the comparison uses the time of one verify round.

Everything in v2.12.2-m1.11 is included.

v2.12.2-m1.11

Choose a tag to compare

@shunya1810 shunya1810 released this 25 Oct 03:06

Unofficial M1-family build of MTPLX, tuned for long contexts and coding agents on an M1 Max. Not affiliated with or endorsed by the MTPLX project. Developed with AI assistance (Claude) and measured on a single machine.

  • Base: MTPLX v2.12.2 (youssofal/MTPLX). Native extensions: mfa_attn changed, so rebuild them: scripts/build-m1-native-extensions.sh.
  • Changes in detail (Japanese): docs/m1max-longctx/CHANGES.ja.md

What is new in v2.12.2-m1.11

  • Long prompts no longer end in NaN on the 27B. On M1, attention over a cached history of 8K tokens or more runs through a flash-attention kernel (head_dim 256, after metal-flash-attention). After every 128-key block it moves the running, unnormalized output into the output buffer and reads it back for the next block. That buffer was fp16, so on a long prompt the sum could pass 65,504: one row went inf, the merge with the causal tail turned it into NaN, and the request ended with all-NaN logits. In bench 3 of apple-silicon-llm-bench, Qwen3.8-27B hit it on 8 of 9 cold prompts at 192K–256K. Rows that stayed finite lost precision too: replaying the failing call, the largest error against an fp32 reference was 2.6 before and 0.037 after. The output is now fp32 in the kernel and in the extension, with kernel time unchanged within 1% (prefix 32K–245K).
    • After the fix, all six 27B cells at 192K–256K (agent, English, Japanese and code prompts) pass: prefill 78–90 tok/s, decode 17–21 tok/s.
    • At 224K, which passed before without NaN, the 27B's decode went from 10.9 to 20.6 tok/s with thinking on and from 9.0 to 18.0 with thinking off. The imprecise rows stayed in the KV cache and the draft's history; the draft acceptance rate was not logged, so that part is not confirmed.
    • From 32K on, answers on both the 27B and the A3B differ from before; at 8K they are identical. At 32K–128K decode moved by a few percent.
  • Switching thinking on or off on the same prompt keeps the cache. The cache identity carried the thinking flag, so a thinking-off request on the same 32K–256K prefix as a thinking-on turn found no saved state and re-read the whole prompt (upstream v2.12.2 does the same). Saved states are matched token for token, and anything thinking changes in the prompt already shows up as a token difference, so the flag left the identity. Time to answer after the switch: 27B at 128K 996 s → 1.8 s, A3B at 256K 1,036 s → 7.1 s. States saved before the update are re-read once.

Everything in v2.12.2-m1.10 is included.

v2.12.2-m1.10

Choose a tag to compare

@shunya1810 shunya1810 released this 20 Oct 11:40

Unofficial M1-family build of MTPLX, tuned for long contexts and coding agents on an M1 Max. Not affiliated with or endorsed by the MTPLX project. Developed with AI assistance (Claude) and measured on a single machine.

  • Base: MTPLX v2.12.2 (youssofal/MTPLX). Native extensions: unchanged since v2.12.2-m1.5; no rebuild needed.
  • Changes in detail (Japanese): docs/m1max-longctx/CHANGES.ja.md

What is new in v2.12.2-m1.10

  • Dense models (Qwen3.8-27B and the like): part of every prefill past 32K was computed wrong; fixed. Present since v2.12.2-m1.1. Past the ANE pipeline's 32K limit, a 4,096-row prefill chunk ran through the Neural Engine as two 2,048-row blocks that share one output buffer, and the second block overwrote the first block's output before it was read. The first 2,048 rows of every such chunk got the wrong values for the FFN channels the ANE computes (the model's next-token top-1 from those rows: 29–39%, against 87–96% with the ANE off). Those rows stay in the KV cache and in the draft's history, so answers to long prompts may have been affected. Each block is now read before the next one runs; the rows match the ANE-off computation, and draft acceptance matches it too (64K / 128K, 3 seeds: 2.82 / 2.86 tokens per round, ANE off 2.85 / 2.81). MoE models (Qwen3.6-35B-A3B and the like) never took this path.
  • No more re-reading the prompt when the next request arrives right after a turn. On the 27B, after a turn whose thinking was cut off, the next turn re-read 3,061 tokens at 32K (22 s to the first token) and the whole prompt at 246K (51 min). The restore point is now kept as a checkpoint, and a turn that released its restored entry still stores its prompt: the same sequence starts in 1.6 s at 32K, 4.1 s at 128K and 3.3 s at 246K.
  • The conversation state is no longer kept twice. The prompt-end entry and the answer-end entry (64 tokens apart, 3.8 GB each at 64K) both stayed in the session bank; the answer entry now replaces the prompt entry. Wired memory after a 64K turn: −2.9 GB; README peak at 128K: 39.9 → 35.5 GB.
  • Dense models read prompts over 192K with fp16 KV during the prefill and repack to 8-bit afterwards (MTPLX_M1_PAGED_PREFILL_MAX_CONTEXT). Writing straight into the 8-bit pages peaked at 51.3 GB on the 27B's 253K prompt and, with the fix above, hit the memory guard's floor (HTTP 507); fp16 peaks at 43.4 GB at the same speed. In the 3-turn bench at 246K, peak wired went from 42.0 to 44.4 GB. MoE models keep writing into the pages.
  • Experimental knobs for deeper MTP drafts (off by default): an 8-row simdgroup-MMA verify kernel (MTPLX_M1_MMA8=1), q8 verify attention for 5–8 rows split into two 4-row launches (on; only reached at depth 4+), a compiled verify window of depth + 1 for depths 6+, and MTPLX_EXPERIMENT_DEPTH_CEILING. On an M1 Max depth 3 stays fastest (depth 7: −5% at 2K, −15% at 32K), so the default depth is unchanged.

The README adds a 246K table from apple-silicon-llm-bench (245,637 tokens of non-repeating text, three thinking turns): the 27B prefills at 80 tok/s and decodes 18–21 tok/s (upstream stops out of memory), the A3B 253 tok/s and 46–49 tok/s (upstream 147 and 12–13). The README bench's own prompts repeat one 11K-token file, which the 27B cannot read well past the third repeat (upstream too), so they stay at 8K–128K.

Everything in v2.12.2-m1.9 is included.

v2.12.2-m1.9

Choose a tag to compare

@shunya1810 shunya1810 released this 18 Oct 02:08

Unofficial M1-family build of MTPLX, tuned for long contexts and coding agents on an M1 Max. Not affiliated with or endorsed by the MTPLX project. Developed with AI assistance (Claude) and measured on a single machine.

  • Base: MTPLX v2.12.2 (youssofal/MTPLX). Native extensions: unchanged since v2.12.2-m1.5; no rebuild needed.
  • Changes in detail (Japanese): docs/m1max-longctx/CHANGES.ja.md

What is new in v2.12.2-m1.9

  • A long prompt no longer fails with HTTP 507 after several large conversations on one server. On the 27B, after 8K, 32K and 64K conversations, a cold 128K prompt stopped mid-prefill as out of memory while 9–11 GB of idle conversation state was still in RAM. The memory guard gave conversation state back once per prefill, and only enough for the next chunk. It now keeps giving back, in the same check and on later chunks, while there is something left to free (at most 8 rounds per prefill). The same sequence now completes the 128K prompt (two rounds, 7.0 GB released).

Everything in v2.12.2-m1.8 is included.