TensorFold 1.1.9

TensorFold 1.1.9

1.1.9 speeds up prompts on DGX Spark and Flash Next on Macs: on a GB10, Qwen3.8-27B reads prompts 9% to 11% faster than 1.1.8 and Kolibri-1 3% to 8% faster, and on an M3 Ultra, Flash Next's copy-heavy edits run at 212 to 257 tok/s and sampled prose at up to 136 tok/s, from 122.5. On an M5 Ultra, Qwen3.8-27B reads prompts about 4% faster. Every model gives the same tokens as 1.1.8. The full list is in the changelog.

Speed

• Qwen3.8-27B prompts on CUDA: a prompt runs up to four of its projections in one launch, each keeping the shipped BF16 kernel's block order, and takes a 2x2-warp tile at large prompt widths (#1439, for #1398). Outputs are the same bytes. On a GB10 with the MLX 4-bit checkpoint, prefill is 10.8% faster at 2k tokens, 9.7% at 8k and 7.8% at 32k. Prompts on the default path (without TF_QWEN_KV_PAGES) also take whole-window attention kernels again, which wins back the long-prompt prefill 1.1.7 lost (#1458): 0.5% at 32k and 0.8% at 64k.
• Kolibri-1 on CUDA, and Drama 3 with prompt chunks of more than one row: the block-FP8 routed experts' prompt kernels copy weights into shared memory in stored order, so each 32-byte sector is read from L2 once instead of twice (#1460). Outputs are the same bytes. On a GB10, Kolibri-1's prompts ran 13% faster at 2k tokens and 3% to 4% faster from 8k to 64k.
• Flash Next copy drafts on Metal: when a reply repeats text from its prompt or from earlier in the reply (edits, rewrites, renames, quotes), its copied drafts verify in windows as wide as their measured acceptance pays for, about 11 rows where they stopped at about 4. Requests whose prompt plus max_tokens passes about 2,000 tokens, which includes every request at the default max_tokens of 4,096, now get the GPU gate that sets each round's width, as shorter requests already did. On an M3 Ultra, copy-heavy edits ran 212 to 257 tok/s and rewrites 222 to 300; at the default max_tokens, prose ran 138 to 142 and long documents 173 to 179.
• Sampled Flash Next replies on Metal: the checkpoint samples by default (temperature 1.0, top_k 20, top_p 0.95), and a sampled reply now gets the GPU gate, MTP drafts drawn under the reply's own keyed rule, and a faster draw: for top_k up to 64 the GPU keeps each of 32 slices' top_k and merges them in a second pass, about 29 µs where the whole-vocabulary draw took about 700 µs, with the same sums, cuts and tie rule. It picks the same token at every seed and position. On an M3 Ultra, sampled prose ran 122.5 to 136.0 tok/s, code 180 to 194 and hard prompts 133 to 146. Greedy replies are unchanged.
• Qwen3.8-27B prompt rows on M5: a request asks for 2,048 prompt rows when no other request is decoding, and caps each chunk at 512 beside a decoder (#1478). The larger frame is used only when the context, every stream and the full prompt scratch fit beside the weights and the existing 8 GiB margin, and startup reserves that scratch. Earlier chips keep 512 rows; TF_PROMPT_ROWS overrides the width. On an M5 Ultra, prefill is 3.7% to 4.5% faster from 2k to 64k tokens.
• Nemotron 3.5 on Metal: a GPU round prices its draft head's depth from the backend's window-cost table, and no longer steps back up to a level that has shown it doesn't land (#1115). With Nemotron 3.5 Lightning 30B-A3B (MLX 4-bit), decode after 2k to 64k-token prompts is 1.2% to 2.4% faster on an M3 Ultra and within 0.3% on an M5 Ultra.
• Qwen3.8-27B follow-up turns on Metal: a follow-up turn resumes from the state its conversation's last reply left in the server's first slot when the prompt cache holds nothing as far, so on a Mac whose cache can't keep a long conversation (a 32 GB Mac) the next turn reads only its new tokens instead of its whole prompt (#1197). Replies are the same tokens as a fresh prompt pass. TF_QWEN27_SLOT_RESUME=0 turns it off.
• Qwen3.5-2B conversations on Metal (opt-in): with --prompt-cache-gib N, a request that extends a conversation resumes from its deepest kept prompt state in RAM instead of prefilling it again (#505). On an M5 Ultra the next turn of a 28.5k-token chat resumed with 28,544 tokens cached, and on an M5 Ultra and an M3 Ultra every resumed reply equalled a fresh one. Without the flag nothing changes.
• Flash Next on CUDA with NVFP4 experts: at decode, the routed experts' work goes out in equal waves instead of leaving a short last wave (#1351). Each output keeps the same multiply-adds in the same order. On a GB10, served decode is 0.6% faster at one stream and 0.2% at two.
• GLM-5.3-Flash on CUDA:
  • At long positions a decode step's DSA select gives each row a 1,024-thread block where it used 128, with the same picks in the same order (#1424). On three GB10s at TP=3, decode after a 136k-token prompt ran 1.8% faster.
  • With two or more ranks, a prompt chunk's DSA indexer splits across the ranks from position 2,048 on, and an all-gather gives every rank every row's picks (#1404). Replies are the same tokens, and decode never splits. TF_GLM_INDEX_SPLIT sets the starting position, and 0 turns it off. On two GB10s at TP=2, prefill at 128k tokens was 1% to 2% faster.
  • With TENSORFOLD_PREFILL_SHARE below 1, a long prompt's step ends on the GPU before it returns when the lane host bills it to decoding streams, so its tail no longer runs in the decoders' next round (#1352). At the default of 1 nothing changes.

Server

• The serving line names its drafters: for example drafts: copies only, drafts: MTP + copies or drafts: DFlash2 (z-lab/Qwen3.8-27B-DFlash2) + copies, where it used to say drafts: on whether or not a drafter was loaded (#1515). Drafting itself doesn't change.
• Qwen3.8-27B profiling on Metal (opt-in): TF_Q27_PROFILE=N logs each kernel class's GPU time, the DFlash2 drafter's included, every N decode rounds, and adds gpu_seconds to replies (#1215). It is a diagnostic: on an M5 Ultra one stream decodes at about a ninth of its usual rate with it on. Without it replies are byte-identical to before.

Fixes

• --compact-at auto on small windows: on a context window smaller than its 16,384-token reserve, it reserves fifteen percent of the window instead, so a short request on --context 8192 is no longer compacted (#1536). Larger windows keep the 16,384-token reserve.
• Release builds: the release path check no longer refuses a clean binary whose address tables happen to hold bytes that read as \.o (#1533). Every other rule is unchanged.

HIP

• AMD RDNA4 cards (gfx1201) take a tuning table of their own, which keeps gfx1100's bounds for now, so no kernel choice changes; zig build hip-affine-tune -Dhip=true times the MLX affine prompt tiles on a card to fill it (#1527).

Tests and tools

• CUDA kernels take their shared helpers from one set of headers; every kernel compiles to the same code as before (#1440).
• kernel-contract-limits no longer reports Qwen's prompt partials as launched past their bounds on every SM.

Release testing

1.1.9 against 1.1.8 on the same machines and checkpoints, alternating the two builds (1.1.8, 1.1.9, 1.1.9, 1.1.8). The cells were cold prompts from 2k to 64k, and the stream sweep at 1 to 8 streams (16 for LFM2) with every stream checked against its run alone. Every change above was also measured against main on its own before it landed.

• Every model gives the same tokens on both releases in every cell.
• DGX Spark (GB10):
  • Qwen3.8-27B: prefill +11.1% at 2k, +10.4% at 8k and +8.8% at 32k. Decode and the sweep are within 0.2%.
  • Kolibri-1: prefill +8.1% at 2k, +3.7% at 8k, +3.4% at 32k and +3.0% at 64k. Decode after the prompt is within 2%, about as much as the 1.1.8 runs vary between themselves. Across twelve alternating runs, one stream is within 0.4%.
  • Qwen3.8 Flash Next FP4-Local: prefill +0.3% to +1.4%, decode +0.1% to +0.5%, and the sweep +0.8% to +0.9%.
  • Nemotron 3.5 Lightning: no difference beyond run-to-run spread.
• Two GB10s at TP=2, GLM-5.3-Flash NVFP4: no stream differs from its run alone (52 of 52), prefill is within 0.2%, and chat and code at one and two streams are within 0.4%.
• M5 Ultra:
  • Qwen3.8-27B: prefill +3.7% to +4.5% from 2k to 64k (#1478). The sweep is within 1%.
  • Nemotron 3.5 Lightning: prefill within 0.2%, the sweep within 0.1%.
  • LFM2.5-24B-A2B: prefill within 0.2%; the sweep is no slower at any level.
  • Qwen3.5-2B (default settings): prefill within 0.4%, the sweep within 0.3%.

Thanks

@jschmied, @yguanrong, @Bizuayeu, @jishnuvenugopal, @akol1, @RS91Dev, @grantoverton, @Deesha08, @laudney and @ppardus.