TensorFold 1.1.5

TensorFold 1.1.5

1.1.5 makes Qwen3.8 Flash Next faster on both engines with the same tokens: up to 28% faster prefill and decode on a GB10 with NVFP4 checkpoints, and 16% to 23% faster decode on Macs without tensor units. A follow-up question about a long document you already sent no longer prefills the document again, and GLM-5.3-Flash at TP=2 serves two streams 22% faster. It also adds Nemotron-Labs Dory on CUDA, NVFP4 and EXL3 exports for Qwen3.8-27B on CUDA, and Drama 3's pruned NVFP4 checkpoint on Metal. The full list is in the changelog.

Speed

• Flash Next on CUDA with NVFP4 checkpoints prefills with NVFP4 experts on prompt tiles, skips zero fills, and splits the hyper-connection and dense-mix work across more of the GPU (#1143, #1147, #1164, #1171). On a GB10 with Qwen3.8 Flash Next FP4-Local against 1.1.4, prefill goes from 1,306 to 1,583 tokens/s at 2k (+21%), 1,359 to 1,723 at 16k (+27%), 1,323 to 1,687 at 32k (+28%) and 1,174 to 1,373 at 64k (+17%). The served sweep gains 2%, 10%, 20% and 28% at 1, 2, 4 and 8 streams.
• Flash Next on Macs without tensor units decodes 16% to 23% faster: the hyper-connection kernels run for wider windows without staging, rounds are gated on the GPU, and wide windows use grouped experts. On an M3 Ultra with Qwen3.8 Flash Next MLX 4-bit MTP, one stream goes from 120.8 to 147.8 tokens/s and eight from 154.8 to 180.2. Prefill is unchanged.
• A follow-up about a long document (the same document again with a new question, in one user message) resumes from a kept prompt state instead of prefilling the document again, on Flash Next CUDA and Qwen3.8-27B on Metal (#1262). With a 35k-token document, the follow-up's first token comes after 2.8 s instead of 25.7 s with Flash Next FP4-Local on a GB10, and after 9.0 s instead of 125.9 s with Qwen3.8-27B on an M3 Ultra. The reply equals a fresh run's.
• GLM-5.3-Flash. At TP=2 on two GB10s with NVIDIA's NVFP4 checkpoint, two streams serve 22% more tokens/s than in 1.1.4 for greedy chat and code, and one stream and prefill are unchanged. On Metal, decode reads the weights once per decode window (#1130): on an M5 Mac, two streams gain 8%.
• Drama 3 on Metal reuses a catalog voice's reference prefill, so the first audio of a streamed reply comes 90% sooner (15.6 s to 1.5 s over four texts on an M3 Ultra) with byte-identical audio (#1237).
• A CUDA request cancelled during its prompt keeps the prefill it completed, so a retry resumes from there; its reply equals an uncancelled run's (#1228).
• Smaller changes: Flash Next CUDA's copy window widens after a copy that lands whole (#1252), tokenizer scratch starts on the stack (recovering a small one-stream loss in 1.1.4, #1248), and the lane host gives prompt steps their share of wall time beside decoding streams, so a long prompt no longer stalls them (#1188).

Models

• Nemotron-Labs Dory 1.0 4B on CUDA (new family, dory), at 1 to 4 loops from ModelOpt W4A16 NVFP4 checkpoints, with kept prompt states. On a GB10 at 2 loops it decodes 44 tokens/s on one stream and 299 in aggregate at eight (#1256 to #1261).
• Qwen3.8-27B on CUDA serves ModelOpt and compressed-tensors NVFP4 and FP8 exports such as nvidia/Qwen3.8-27B-NVFP4, with bf16 activations against the stored weights (#1145), and ExLlamaV3 (EXL3) Qwen3.5-MoE checkpoints such as Ornith-1.5-122B (#1150).
• GLM-5.3-Flash on CUDA loads nvidia/GLM-5.3-Flash-FP4-Local as stored (#1050), and Bizuayeu/GLM-5.3-Flash-XSFP4 through the same path, at TP=2 or TP=3.
• Drama 3 on Metal reads the pruned NVFP4 checkpoint as stored, and has opt-in classifier-free guidance (--cfg-gamma and related flags). The default and catalog voices' audio is byte-identical to 1.1.4 (#1268).
• Gemma 4 runs on Macs before the M5, with its dense projections on the row path.

Serving

• A tool call that max_tokens cuts now comes back in tool_calls with finish_reason: "length" in non-streamed replies too, as it already did when streamed (#1238, #1299).
• return_token_ids adds the reply's token ids to the response's tensorfold block (#1290).
• GLM-5.3's chat template receives clear_thinking from TF_GLM_CLEAR_THINKING or the request; the EXL3 checkpoint, whose template used to clear earlier reasoning, now keeps it, so earlier turns' prefill is reused (#1300).
• LFM2 tool calls take the schema's types, parse the JSON list form, stream each call as it ends, and work through a grammar when required or named (#1265).
• GLM-5.3-Flash on CUDA queues a request when the shared context pool is momentarily full instead of failing it (#1263), bootstraps /32 rings (#1178) and runs RoCE at 2 to 16 ranks (#1209).
• Drama voice catalogs can carry descriptions, listed by /v1/voices (#1270).
• Fixes: --prompt-precision int8 (opt-in) now falls back to bf16 for a chunk with a non-finite input, as intended; Gemma 4 on CUDA counts its vision workspace and drafter at admission and refuses a malformed DFlash config (#1245); a rare crash at shutdown after a tensor-parallel rank fails; CUDA builds on Windows with MSVC as the host compiler (#1296); the desktop app stops an agent run cleanly at a question's deadline.
• The CUDA kernel contract's gates pass on a GB10 again: all 338 registered kernels pass gates 1 to 4 and all 92 planted controls are caught. Its resource-limit check still reports 52 problems, all in Qwen3.5 lane kernels' records (stale or missing register counts, no failing launch); those records are being completed.

Release testing

1.1.5 against 1.1.4 on the same machines and checkpoints: cold prompts from 2k to 64k, and the stream sweep at 1 to 8 streams with every stream checked against its run alone. Every change above was also measured against main on its own before it landed.

• Every model gives the same tokens on both releases in every cell.
• DGX Spark (GB10), Qwen3.8 Flash Next FP4-Local: the speedups above.
• Two GB10s at TP=2, GLM-5.3-Flash NVFP4: 52 of 52 token checks equal; prefill within 0.4%; one stream within 0.5%; two streams +22%; resumed chats equal fresh ones with and without thinking.
• DGX Spark, Qwen3.8-27B and Nemotron 3.5 Lightning: prefill within 0.2%; the sweep from +0.3% to +1.4% (Nemotron at 1, 2 and 4 streams across six alternating runs).
• Mac Studio M3 Ultra, Qwen3.8-27B: prefill and the sweep within 0.2%. Nemotron 3.5 Lightning: prefill within 0.1%; across six alternating runs, greedy decode at one stream within 0.3% and two and four streams within 0.3%. Sampled one-stream decode averages 1.5% lower, inside its run-to-run spread (243 to 250 tokens/s on 1.1.4 alone).
• Mac Studio M3 Ultra, Qwen3.8 Flash Next MLX 4-bit MTP: the sweep +16% to +23%, prefill unchanged.
• M5 Mac, GLM-5.3-Flash: prefill within 0.1%; one stream +0.3%, two +8.4%.
• The release archive's Linux binary on a GB10 reports 1.1.5, serves Qwen3.8-27B, and answers plain chat and a JSON-schema request; host tests and the archive checks pass on Linux and macOS.

Thanks

@jschmied, @Bizuayeu, @BlivionIaG, @harrisonfriia, @plotarmordev, @jerryfane, @drowzeys, @MiaAI-Lab, @outcastofmusic, @neko-legends, @Sapphire-Rapids, @GumbiiDigital, @Boscoeuk, @BobClawblaw, @jvr0x, @vcruz305, @CanadaApollo6, and for reports @devopsfortunato, @laidick, @flashosophy and @ss-cong.