TensorFold 1.1.4

TensorFold 1.1.4

1.1.4 makes Qwen3.8-27B's cold prompts on CUDA up to five times faster, brings Gemma 4 12B and 31B to the native CUDA engine with their drafters, lets several Flash Next streams verify their drafts in one pass on CUDA, and adds idle unload, a family of opt-in speech guards, and a server that reports a dead CUDA context instead of looking healthy. The full list is in the changelog.

Speed

• Qwen3.8-27B on CUDA has a new default prompt and decode recipe. On a GB10, cold prefill goes from 834 to 1,281 tokens/s at 2k (+54%), 612 to 1,266 at 8k (+107%), 326 to 1,165 at 32k (+258%) and 203 to 1,045 at 64k (+416%); decode gains about 2% at 1 to 8 streams. Quality equals the previous recipe's (27 of 40 exact cases on both). Replies can differ in bits from 1.1.3; every stream still equals its run alone and a resumed prompt equals a fresh one. --prompt-precision int8 is a further opt-in (#1079, #1080, #1168, #1176).
• Gemma 4 on CUDA (new): the 12B and 31B from MLX affine checkpoints. On a GB10 the 12B decodes 37 tokens/s on one stream, and its MTP assistant takes greedy chat from 30 to 67 tokens/s (2.2x); z-lab's DFlash drafter takes the 31B's greedy chat from 12.6 to 29.9 tokens/s (2.4x). Drafted, concurrent and resumed replies equal plain ones. Image input is in; its GB10 image cells have not run yet (#1174, #1181 to #1186).
• Flash Next on CUDA verifies several streams' drafted windows in one pass: the served sweep gains 35%, 75% and 116% at 2, 4 and 8 streams on a GB10, with tokens unchanged (#1069, #1155).
• Drama 3 on CUDA can prefill prompts in row chunks: with TF_DRAMA_PREFILL_ROWS=128, first audio comes after 0.76 s instead of 18.2 s on a GB10. It is opt-in; the default audio is byte-identical to 1.1.3 (#1120).
• Smaller gains: Flash Next's fused DeltaNet prompt block and pooled PLE gather, Qwen's attention partials in L2 and segmented split-K decode, RADIO's fused bias launches, GLM-5.3's Metal sparse-index scan (a 16,384-key scan for 16 rows in 0.14 ms instead of 1.59 ms), and a window-cost table that one noisy start can no longer set.

Serving

• --idle-unload SECONDS frees the model after that many idle seconds and reloads it on the next request (#1172).
• A sticky CUDA fault now turns /health to 503 instead of leaving a dead engine that looks ready, and a CUDA graph the driver refuses (error 800) runs eagerly instead of spinning (#1116, #1216).
• GLM's ring of draws no longer fills after 64 completed or cancelled cold requests (#1218), long chats and kept-alive connections hold less memory (#1229 to #1232), and Drama 3 on Metal refuses a load that does not fit (#1166).
• Speech: opt-in silence, onset, mid-utterance, short-reply and register guards with whole-reply redraws, a controls allowlist, --max-frames, and attested voice catalogs (#1152, #1157 to #1162, #1134, #1135, #1153).
• POST /v1/decisions on Flash Next CUDA (#1154), Hugging Face-named Qwen vision towers (#1151), tensorfold benchmark --kv-cache (#983), Windows builds with nvcc again (#1213), Nemotron-H on the native HIP backend behind -Dhip=true (#1082 to #1103), and app fixes for fake-IP proxies and refused checkpoints (#1037, #884).

Release testing

1.1.4 against 1.1.3 on the same machines and checkpoints: cold prompts from 2k to 64k, and the stream sweep at 1, 2, 4 and 8 streams with every stream checked against its run alone and against "draft": false. Every change above was also measured against main on its own before it landed.

• Every model gives the same tokens on both releases in every cell, and no request failed.
• Mac Studio M3 Ultra, Qwen3.8-27B and Nemotron 3.5 Lightning: prefill within 0.1%, every sweep cell within 1.6%.
• Mac Studio M3 Ultra, Qwen3.8 Flash Next MLX 4-bit MTP: prefill within 0.2%; across six alternating runs every stream count is within 1%.
• DGX Spark (GB10), Nemotron 3.5 Lightning MLX 4-bit: prefill within 1.2%, and across six alternating runs 2 streams are within 0.2%. One known change: greedy chat at 4 streams is 3.8% slower (338.7 to 325.8 tokens/s, every run); greedy code at 4 streams moves between about 390 and 510 tokens/s from run to run on both releases, the draft-schedule behaviour 1.1.3 noted.
• DGX Spark, Qwen3.8-27B: the 1.1.4 recipe against main as above.
• The release archive's Linux binary on a GB10 reports 1.1.4, serves Qwen3.8-27B, and answers plain chat and a JSON-schema request; host tests pass on Linux and macOS.

Thanks

@BobClawblaw, @jschmied, @Mirrdhyn, @plotarmordev, @wojo, @heitke, @BlivionIaG, @CanadaApollo6, @MiaAI-Lab, @Bizuayeu, @harrisonfriia, @jishnuvenugopal, @paragontasx, @ThinkOffApp, @wesley4546, and for reports @Chris9121, @flashosophy, @Gogo6969, @Idoo689, @johncarterlives, @mikog8tor, @samwang0041-star and @tonghuaxuan2016-dot.