Skip to content

TensorFold 1.2.0

Latest
Latest

Choose a tag to compare

View all tags
@ashhart ashhart released this 31 Oct 10:44
· 49 commits to main since this release
v1.2.0
a388876

TensorFold 1.2.0

1.2.0 makes Qwen3.8-27B faster on Macs. On an M3 Ultra with DFlash2 it decodes 14% faster than 1.1.9 at one stream
and 44% faster at 16; on an M5 Ultra, 6% faster at one stream and 25% at 16. On a GB10, none of the ten models we
read got slower. It adds a new family,
MiniMax-M3.1-Flash on one 256 GB M5 Ultra, and new checkpoints and hardware: NVIDIA's NVFP4 Qwen3.6-35B-A3B, dense
EXL3 Qwen3.8-27B on CUDA, and Gemma 4 E4B on Intel's Arc Pro B70. Every speedup keeps replies token-equal to 1.1.9.
One change is deliberate: pictures on the Qwen-VL models are now prepared as the checkpoints' own processor prepares
them, so replies to pictures can differ slightly from 1.1.9. Text-only replies are unchanged. The full list is in
the changelog.

Measured

Every row was measured with the same tokens as its base.

What Hardware Base Before After Change
Qwen3.8-27B MLX 4-bit, DFlash2, 1 stream M3 Ultra 1.1.9 80.0 tok/s 91.6 tok/s +14.4%
The same, 4 streams M3 Ultra 1.1.9 127.0 150.3 +18.3%
The same, 16 streams M3 Ultra 1.1.9 145.2 209.4 +44.2%
The same, prefill M3 Ultra 1.1.9 +17%
Qwen3.8-27B MLX 4-bit, DFlash2, 1 stream M5 Ultra 1.1.9 168.5 tok/s 178.3 tok/s +5.8%
The same, 4 streams M5 Ultra 1.1.9 298.0 334.6 +12.3%
The same, 16 streams M5 Ultra 1.1.9 378.5 474.0 +25.2%
Qwen3.8-27B, one stream, code / chat (#1726) M5 Ultra main before it +2.6% / +4.6%
Qwen3.8-27B, four greedy code streams (#1726) M5 Ultra main before it +18.6%
Flash Next, sampled code / chat at 5 streams, 5 x 131,072 (#1609) GB10 main before it +24% / +19%
The same at 4 streams GB10 main before it +15% / +19%
Qwen3.6-35B-A3B MTP, 2 / 4 / 8 users (#1562) GB10 streams drafting one at a time 116 / 138 / 159 tok/s 137 / 182 / 220
LFM2.5-24B-A2B, a 2,048-row prompt chunk (#1590) M1 Pro main before it 243 s 3.9 s
LFM2.5-24B-A2B, 16 greedy streams together (#1590) M1 Pro main before it 11 to 17 tok/s 116 to 123
Qwen3.8-27B MLX 4-bit, cold prefill at 2k (#1663) Ryzen AI Max+ 395 (gfx1151) main before it 8.60 s 5.83 s 32% faster
Qwen3.8-27B, cold prompt at 2k (#1648) gfx1151 main before it 8.64 s 7.60 s
GLM-5.3-Flash XSFP4 load at TP=2 (#1616, #1636) two GB10s before #1616 327 s 51 s
A 2,000-message DeepSeek-V4.1 chat's peak memory while its template renders (#1728) any main before it 154 MB 15 MB

New models and hardware

  • MiniMax-M3.1-Flash on Metal (new family): the minimax_m3 family serves MiniMax's 428B-A23B model on one 256 GB
    M5 Ultra. It supports tools, structured output, thinking on or off, images, the prompt cache, two streams, and
    DFlash2 or DSpark drafts that keep replies equal to plain decoding.
    • It reads two community checkpoints. Both are lossy conversions of MiniMax's NVFP4 release, labelled with their
      cost against it: mean KL 0.244 on mix34 and 0.295 on mix256k. Everything from those weights on is exact, with a
      bf16 key/value cache (147,456 tokens on mix256k, 32,768 on mix34).
    • With DFlash2, one stream decodes 44 to 58 tokens a second on mix256k.
    • MINIMAX_LOAD_LIMIT_GB sets the memory budget, and MINIMAX_DRAFTER picks the drafter (#1235, for #1519).
  • NVIDIA's NVFP4 Qwen3.6-35B-A3B on CUDA (nvidia/Qwen3.6-35B-A3B-NVFP4) serves as downloaded, and --drafter
    naming the same checkpoint drafts with its MTP layer. One GB10 decodes 66 tok/s without drafts and 112 to 125 with
    them. Drafted, concurrent and resumed replies equal their plain runs (#1662).
  • Dense EXL3 Qwen3.8-27B on CUDA: turboderp's dense EXL3 checkpoints now load instead of being refused, and
    DFlash2 drafts them. At 4.00 bpw on a GB10, decode runs at 45.8 tok/s for chat and 71.9 for code with DFlash2,
    against 15.3 and 15.5 without (#1465). The drafter reads only the two vocabulary spans it proposes from, about
    545 MB fewer reads a round at a 6-bit head (#1507).
  • ModelOpt Qwen3.8-27B on sm_80 to sm_89: NVFP4 and FP8 checkpoints now serve on cards such as a Jetson AGX Orin.
    This path hasn't run on such a card yet. Nothing changes on sm_90 and newer (#1470).
  • Gemma 4 E4B on Intel's Arc Pro B70 (Xe2): the XPU backend serves mlx_lm.convert 4-bit checkpoints, with an
    optional 8-bit attention, up to 8 streams with shared forwards, and the prompt cache. On the B70, 2,048-token prompts
    read at 6,772 tok/s, one stream decodes at 135.6 tok/s, and 4 streams reach 426 tok/s together, each reply equal to
    that stream's reply alone. The audio tower and the drafter aren't in yet (#1552, for #1222).
  • Gemma 4 structured output on CUDA: response_format and tool_choice now constrain its replies, with or without
    a drafter (#1386).
  • Three GB10s in a ring without a switch now connect at three ranks or more. An 8 KiB exchange takes 10.7 µs,
    against 26.3 µs for NCCL. A ring of four still falls back to NCCL (#1516).
  • gfx1100 with clang 21 (opt-in): -Dhip-relax-branches=true builds the dot2 prompt tiles with Ubuntu's clang 21.
    It hasn't run on a gfx1100 GPU yet (#1542).

Speed

Each of these keeps replies token-equal to main before it.

  • Metal:
    • Qwen3.8-27B:
      • The quantized tiles unroll their fixed loops, with bit-identical outputs, and a shared round whose drafts get
        no rows skips the drafter (#1686, for #1621).
      • On M5 Macs, one stream takes the lane loop by default (#1726, for #1621). On an M3 Ultra, the same change makes
        16 streams 3.5% faster and one stream 1.9%.
    • Nemotron 3.5:
      • On M3-class GPUs, the decode attention and the experts' 2-row tiles unroll their fixed loops. On an M3 Ultra,
        Lightning decodes 1.8% faster at one stream (#1704, for #1621).
      • On M1-family Macs, routed experts in windows of 16 rows or more take two members a pass. On an M1 Max the
        kernels take 14 to 20% less time at 64 to 128 rows (#1466).
    • LFM2.5-24B-A2B on M1 and M2 Macs: the prompt tiles store their matrix fragments whole, and run at 2.4 to
      2.7 TFLOPS on an M1 Pro. M3 and newer compile them as before. The M2 takes the M1's path but wasn't measured
      (#1590, fixes #1523).
  • CUDA:
    • Qwen3.6-35B-A3B: every stream of a shared round drafts together in the MTP layer (#1562).
    • Flash Next: a shared round's sampled streams draft in the batched MTP head pass. Greedy streams and time to
      first token are unchanged (#1609).
    • EXL3:
      • A context copy beside the dense 27B now needs 20 matching tokens instead of 5. On an RTX 3090 Ti, decode after
        long prompts is up 12.8% sampled and 18.5% greedy (#1477).
      • The dense projections zero their split-K counters only when the buffer grows. On a GB10 with the 27B 3.50 bpw
        pack, decode is 0.4% to 0.9% faster (#1475).
      • The trellis loads skip L1. On three GB10s at TP=3 with GLM-5.3-Flash EXL3 4 bpw, decode rose 2.3% on one stream
        (#1534).
    • GLM-5.3-Flash load: reader threads fetch each layer ahead (#1616), and 8 threads decode and pack each layer's
      routed experts (#1636). On three GB10s it now loads in 46 s, from 268 s.
    • Nemotron's router top-k runs on the shared MoE top-k: 2.74 µs a call instead of 4.37 on a GB10 (#1354).
  • HIP:
    • gfx11's matrix-core prompt tile no longer spills to scratch (#1663) and runs in bands of four row blocks
      (#1648). Cold prefill on a gfx1151 at 64k is 24.5% faster from #1663. #1648 cuts 365.7 s to 330.8 s.
    • RDNA4 short routed prompts: on an R9700, cold prefill takes 0.53 to 0.80 times as long on Qwen3.6-35B-A3B at
      33 to 128 tokens. Other GPUs are unchanged (#1607).

Opt-in

  • GLM --mtp-confidence: a draft chain stops where its chance falls under the floor. At TP=3 on GB10s, decode
    went from 51.6 to 54.1 tok/s. At TP=2, code was 3.7% slower with it (#1585).
  • GLM all-gathers on RoCE: TF_GLM_ROCE_MULTI=1, at three ranks or more. Decode at TP=3 went from 51.2 to
    52.6 tok/s with drafts. A cold 136k-token prompt is 1.2 to 1.6% slower, so NCCL stays the default (#1524).
  • Qwen3.8-27B lane windows on pre-M5 chips: TF_QWEN27_WINDOW=32 or 64, and TF_QWEN27_COPIES=1. Both stay
    off until qualified (#1686).
  • --decode-share X: while a long prompt fills beside running replies, the replies get X of each fill step's
    time, on the CUDA, Metal and HIP lane servers. Without the flag nothing changes (#1327, for #840).

Groundwork

None of these changes what a default build serves.

  • Paged KV:

    • Nemotron on Metal with TF_NEMOTRON_METAL_KV_PAGES=1, bit-equal to the window kernels. It saves no memory yet
      (#1537).
    • The Qwen prompt cache as GPU pool pages with TF_QWEN_KV_SHARE=n: a kept 4,096-token state takes about 68 MiB of
      pinned memory instead of 148 MiB (#1357).
    • Paged attention kernels that nothing launches yet (#1618).
    • A paged prompt store that can serve GLM-style rows (#1659).

  • Qwen3.8-27B on two Macs: the pair seam's kernels exchange whole output rows, so every sum stays on one Mac
    (#1432).

  • Ling 3.1: a host-only family package for both engines, with config, weight map and a float64 reference (#1541).

  • Vulkan: a first opt-in backend for Linux GPUs, built only with -Dvulkan=true. It has the runtime and one kernel
    under the kernel contract. No model runs on it yet (#1526).

  • LFM2.5-24B-A2B taps for an external drafter, through a new lfm2_tap kernel. Nothing uses them yet (#1390).

  • Shared code:

    Inputs and compiled code are the same as before.

  • MOSS-Image: the seeded starting latents are made in host code, byte for byte what x86 torch makes (#1668, fixes
    #1530).

Fixes

  • Resume after a tool reply: the next turn resumes at the right place with templates like Gemma 4's. On an Arc Pro
    B70 with Gemma 4 E4B, those turns reached their first token in 0.8 to 1.4 s instead of 1.7 to 2.7 s (#1551).
  • Chat templates: a prompt built by appending to a string no longer copies the whole prompt every turn, so memory
    grows with its length, not its square. Rendered prompts are byte-identical (#1728, for #1633).
  • Nemotron on CUDA keeps NaNs: squared ReLU no longer turns a NaN into 0, so a NaN reaches the output instead of
    vanishing. Finite values give the same bits (#1628).
  • DeepSeek-V4.1 (experimental):
    • After a sticky CUDA context fault, it fails fast instead of staying up and unable to answer (#1646).
    • The Flash-1017 EXL3 pack boots at TP=2 (#1643).
    • Boot hashes the pack once instead of twice (#1644).
    • EXL3 packs that declare the dsq-1 format are admitted (#1641).
  • Picture and video downloads (--vision-urls) stop at their deadline, however the remote server behaves (#1645).
  • Conversation states on disk:
    • A save that fits the staging permit waits for a write in flight instead of being skipped (#1548).
    • A state its model refuses is forgotten after one recall (#1543).
  • Cortheon: the task goal and planner queries stay valid UTF-8 (#1575, #1610).

Pictures on the Qwen-VL models

Qwen3.8-27B, Qwen3.6-35B-A3B and Qwen3.8 Flash Next, on Metal and CUDA, now prepare pictures as transformers'
Qwen2VLImageProcessorFast does (#1586). Before this, 0.1% of one photo's pixel values differed from it, by at most
2 levels. So replies to pictures can differ slightly from 1.1.9. Text-only replies are unchanged.

Known issues

  • DeepSeek-V4.1 at TP=2: a repeated long prompt can change the reply (#1395). Sending the same 32,768-token prompt
    again can give a reply that differs from its first run, with drafts on or off. We're tracing the cause.
  • DeepSeek-V4.1's prompt cache (#1625) resumes only at 2,048-token boundaries, and its server ignores
    --prompt-cache-gib. Multi-turn chats re-read more of their prompt than they need to.

Upgrade notes

  • Unknown TF_QWEN27_* switches are now refused by name (#1686). Check your environment for misspelled ones.
  • Qwen3.8-27B on M5 Macs: one stream now takes the lane loop. TF_QWEN27_LANES=0 keeps the serial loop (#1726).
  • Flash Next on CUDA: TF_FLASHNEXT_SAMPLED_HEADS=0 restores drafting sampled streams one at a time (#1609).
  • DeepSeek-V4.1: after a sticky CUDA context fault the server now exits instead of staying up, so run it under
    something that restarts it. TF_DSV41_STALL_EXIT_S (off by default) also exits when a round makes no progress
    (#1646).
  • Dory: with TENSORFOLD_DORY_KV_PAGES=1, a context must be whole 256-position pages (#1387).
  • GLM load: TF_GLM_LOAD_PACK (1 to 16) sets the packing threads (#1636).
  • Pictures on the Qwen-VL models can give slightly different replies, as described above (#1586).

Tests and tools

  • Kernel contract:
    • The CUDA runner loads fp8_experts' image and makes all 1,325 live-limit checks (#1579).
    • Nemotron's six pocket kernels on CUDA are checked against float64 (#1629).
    • Qwen4's dense activation check no longer fails by mistake near a BF16 midpoint (#1547).
  • tools/zig/cuda_sass_identity.py rebuilds a differing image and fails an unstable build (#1532).
  • Docs:
    • The Flash Next CUDA recipe documents TF_FLASHNEXT_CHUNK (#1553).
    • The runbook documents TENSORFOLD_PREFILL_SHARE and TENSORFOLD_PREFILL_BUDGET.

Release testing

Reads of the release tree against 1.1.9 found no regression, with tokens equal on both, alternating fresh servers.

  • M3 Ultra: Qwen3.8-27B MLX 4-bit with DFlash2 (the table above) and Qwen3.8 Flash Next MLX 4-bit.
  • M5 Ultra: Qwen3.8-27B MLX 4-bit with DFlash2 (the table above), Nemotron 3.5 Lightning (prefill within 0.1%),
    LFM2.5-24B-A2B and Qwen3.5-2B.
  • GB10: Qwen3.8-27B, Nemotron 3.5 Lightning and Bolt, Flash Next FP4 and MLX 4-bit, Kolibri-1, Gemma 4 31B and
    12B, Qwen3.6-35B, Ornith-122B and Spark-X2.6. No model was slower beyond run-to-run noise.

Thanks

@AbrahamCete, @akol1, @alexderz, @Arminova, @BadAd84, @BHCC2025, @Bizuayeu, @BlivionIaG, @BobClawblaw,
@diwakergupta, @fche, @gilby, @javierbits, @jayleaton, @joshuaswarren, @jrb-ai, @jschmied, @laudney, @Mirrdhyn,
@mrv777, @ppardus, @simon-lin88, @swbiggart, @ThinkOffApp, @ThomasHalwax, @valacroix and @wojo.

Contributors

diwakergupta, wojo, and 25 other contributors
Assets 11
Loading