TensorFold 1.1.6

TensorFold 1.1.6

1.1.6 stops concurrent requests from each prefilling the same new system prompt: on a GB10, eight chats arriving together on a new 28k-token prompt get their last first token after 31 s instead of 201 s. Conversations pushed out of memory on a Mac come back from disk (--spill-gib), Qwen3.8-27B on Metal keeps decoding streams moving while a long prompt fills, and GLM-5.3-Flash image chats stop re-running the vision tower on later turns. It also adds Spark-X2.5/2.6-4B on CUDA and Metal, per-request sampling and dialogue for Drama speech, and tensorfold plan. The full list is in the changelog.

Speed

• Requests that arrive together on a new shared prefix no longer each prefill it when prompts fill a chunk a step. A request whose prefix another prompt is filling waits for that prompt's cut and resumes it (#1321, fixing #1310); TENSORFOLD_PREFIX_WAIT_MS bounds the wait (60000 by default, 0 off). On a GB10 with eight chat requests on a new 28k-token system prompt, the last first token comes after 31 s instead of 201 s with Qwen3.8-27B, and after 25 s instead of 146 s with Flash Next FP4. Every reply equals its run alone.
• Conversations come back from disk on Macs. --spill-gib N writes a conversation state the prompt cache pushes out of memory to disk (at most N GiB), and reads it back when the conversation returns; --checkpoint-slots N bounds how many stay in memory (#840, from Python #68). Off by default. With one slot and a second 16k-token conversation in between, the returning conversation's next turn starts after 2.6 s instead of 56 s with Qwen3.8-27B on an M3 Ultra (1.8 s instead of 31 s with GLM-5.3-Flash), with the same reply as a fresh pass.
• Qwen3.8-27B on Metal fills prompts a chunk at a time between the other streams' rounds, so streams already decoding keep going while a long prompt fills (#1303), and kept states reuse spare buffers (#1304). With an 8.7k-token prompt arriving while two streams decode on an M3 Ultra, the decoding streams' longest pause goes from 28.6 s to 0.47 s; the long prompt's first token comes 7.9% later.
• Gemma 4 on CUDA with a drafter keeps its prompt-cache mark after a stepped prompt chunk, so a chat's next turn resumes: a 30k-token chat's next turn starts after 3.6 s instead of 45 s with Gemma 4 31B and its DFlash drafter on a GB10 (#1286).
• GLM-5.3-Flash on CUDA takes KV pages from the core pool from two ranks, not only three (#1322, fixing #1313): at TP=2 on two GB10s with the NVFP4 checkpoint, two streams serve 14% more tokens/s, and prefill is flat to +17%.
• GLM-5.3-Flash images on Metal: later turns of an image chat no longer run the vision tower again for images the kept prompt state already holds: on an M3 Ultra a follow-up image turn's first token comes about 90 ms sooner.
• Host sampling with no rank cut (top_k: 0, top_p: 1) finds the sampled token by a keyed race instead of sorting the whole vocabulary (#1337). The draw is the same token for token.

Models

• Spark-X2.5/2.6-4B (new family, spark2_5) on CUDA from MLX affine checkpoints converted by tools/zig/spark25_convert.py (#1297), and on Metal from the bf16 release as published (#1320). On a GB10 at 8 bits it prefills 5,800 tokens/s at 2k and decodes 160 tokens/s on one stream; on an M3 Ultra, 2,840 tokens/s and 69 tokens/s, and its tokens match the CUDA engine's.
• GLM-5.3-Flash reads the vision tower of MLX conversions that name it vision_tower. with channels-last convolution kernels, which it used to refuse with NoVisionTower.
• DeepSeek-V4.1 (experimental native engine): zig build dsv41-serve installs runnable tensorfold-dsv41 and tf-dsv41 binaries (#1325), and packs that store the router bias or DSpark confidence head in F16 load them widened exactly to F32 (#1326).
• Dory on CUDA can keep its full-attention caches in pages of the core pool with TENSORFOLD_DORY_KV_PAGES=1 (#1284), and Qwen3.5 on Metal with TF_QWEN35_METAL_KV_PAGES=1 (#1294); both off by default, with the same tokens as the contiguous caches.

Serving

• Drama speech: POST /v1/audio/speech takes per-request temperature, top_p, top_k and an exact 64-bit seed, whole and streamed (#1277), and dialogue with speaker tags <|speaker:0|> to <|speaker:15|> (#1278). Requests without them give byte-identical audio.
• tensorfold plan MODEL says, before any download or weight load, the memory CUDA admission would grant and whether the checkpoint's weights fit (#1319).
• --name-priority ID=background gives a served name a background default (#1317).
• GLM-5.3: a tool call that max_tokens cuts comes back in tool_calls with finish_reason: "length" instead of leaking its markup into the reply's text (#1298). DeepSeek-V4.1 (experimental): a tool call in the model's spaced DSML spelling comes back in tool_calls instead of the reply's text, and a lone DSML opener no longer crashes the parser (#1324).
• Metal: Qwen3.5, Nemotron, LFM2 and Gemma 4 lower an unnamed --parallel to what fits under the load limit at startup, and refuse a named one that does not fit, instead of failing under load (#1302). The lane host's prompt scheduling around short prompt bursts is fairer, with tokens and speed unchanged (#1287).
• Fixes: on GB10s the start-up budget counts free pages the kernel parks on its per-CPU lists, so a start right after another process freed memory admits what fits (#1323, fixing #1311); tensorfold pull on macOS finishes a connection interrupted by a signal instead of crashing; tools/image_cache_turns.py measures multi-turn image chats' cache reuse against a served model.

Release testing

1.1.6 against 1.1.5 on the same machines and checkpoints, alternating the two builds: cold prompts from 2k to 64k, and the stream sweep at 1 to 8 streams with every stream checked against its run alone. Every change above was also measured against main on its own before it landed.

• Every model gives the same tokens on both releases in every cell.
• DGX Spark (GB10): Qwen3.8-27B prefill within 1% (2k -1.0%, 8k -0.8%, 16k-64k within 0.2%) and the sweep within 0.1%; Nemotron 3.5 Lightning prefill within 0.3% and the sweep from -1.7% to +10%; Qwen3.8 Flash Next FP4-Local prefill within 1.1% to 16k and the sweep +0.3% to +0.7%, while its 32k and 64k prefill varied between two levels on both builds alike across six alternating runs (a box effect, not a build difference).
• Two GB10s at TP=2, GLM-5.3-Flash NVFP4: no stream differs from its run alone; prefill and one- and two-stream decode within 0.4%; resumed chats hit the cache with and without thinking.
• Mac Studio M3 Ultra: Qwen3.8-27B prefill and the sweep within 0.3%; Nemotron 3.5 Lightning prefill within 0.1% and, across six alternating runs, the sweep from -1.8% to +3.0%; Qwen3.8 Flash Next MLX 4-bit MTP prefill within 0.1% and, across six alternating runs, the sweep within 1%.
• M5 Mac, GLM-5.3-Flash: prefill within 0.1%, one and two streams within 0.1%.

Thanks

@jschmied, @wojo, @jvr0x, @BobClawblaw, @krunkosaurus, @ecohash-co, @Maxrio9, @ajensenwaud, @akol1, @philip-pentatonic, @gilby, @BlivionIaG, and for reports @Boscoeuk.