TensorFold 1.1.8

TensorFold 1.1.8

1.1.8 starts Flash Next on CUDA much sooner and brings conversations back from disk without holding up other streams: a Flash Next server on a GB10 (Qwen3.8-Flash-Next-FP4-Local) starts in about 83 s instead of about 200 s, and with --spill-gib on an M5 Ultra, returning Qwen3.8-27B conversations came back in 0.6–1.0 s instead of up to 21 s. LFM2.5-24B-A2B on Macs serves 4 to 16 streams 20% to 34% faster in aggregate. All three give the same tokens as before. The full list is in the changelog.

Speed

• Flash Next on CUDA starts sooner: NVFP4 experts load on several threads straight from the memory-mapped checkpoint (#1368). On a GB10, Qwen3.8-Flash-Next-FP4-Local starts in about 83 s instead of about 200 s, with the same tokens and prompt and decode speed unchanged or slightly faster.
• LFM2.5-24B-A2B on Macs runs each layer's short convolution and attention once over all of a round's streams, bit-equal to the per-stream kernels (#1377). On an M5 Ultra with the 4-bit export, aggregate decode rose 20% at 4 streams, 34% at 8 and 31% at 16, and one stream about 2%, with the same tokens. The recipe has the author's own receipt, measured against an earlier main.
• Conversations on disk no longer stall other streams: with --spill-gib (off by default), evicted states are written to disk in the background (#1453), and a queued request's state is read back ahead of its turn (#1464; TENSORFOLD_RESTORE_PREFETCH=0 turns this off). On an M5 Ultra with Qwen3.8-27B, returning conversations came back in 0.6–1.0 s instead of up to 21 s, and decoded at 149 tok/s instead of 61.
• Restarts keep conversations on CUDA: with --learn-conversations, Qwen3.8-27B, Qwen3.6-35B-A3B and Flash Next keep conversation states on disk on CUDA too, as Nemotron does, so a turn after a restart reads its state back instead of prefilling it again (#1358, #1359). On a GB10, a 16k-token conversation's next turn after a restart took 3.9 s instead of 12.7 s with Qwen3.8-27B, and 0.4 s instead of 10 s with Flash Next, with the same tokens. Flash Next refuses the flag with --tp.
• --spill-gib on CUDA: Nemotron, the Qwen3.5 hybrids and Flash Next take --spill-gib and --checkpoint-slots on CUDA too, with the same store and disk cap as a Mac (#1413). On a GB10, a conversation sent to disk came back for its next turn in 0.9 s instead of 2.5 s with Nemotron 3.5 Lightning, 7.0 s instead of 16.8 s with Qwen3.8-27B and 2.0 s instead of 9.7 s with Flash Next, with the same tokens.
• Idle unloads keep conversations: with --spill-gib, conversations held only in RAM are written to disk when the server unloads for idleness, so the next turn resumes (#1471). On an M5 Ultra, a 32k-token Nemotron conversation woke in 1.3 s instead of 4.9 s.
• GLM-5.3-Flash on CUDA (opt-in): TF_GLM_INDEX_CUDA=1 runs a prompt chunk's DSA indexer in two kernels of its own, byte-equal to before (#1449, for #1430). A cold prompt's first token came 2% sooner at 131k tokens, 9% at 524k and 16% at 1M.
• Short prompts beside decoding streams (opt-in): TENSORFOLD_PREFILL_BUDGET=N caps the short-prompt rows filled before decoding streams get their next round, so a burst of short prompts holds them up for less time.
• The KV pool keeps its page counts up to date as pages change hands, so an admission check takes constant time instead of about 55 µs at 262k pages.
• The prompt cache finds a conversation's kept state through an index kept longest first instead of comparing the prompt with every kept state, with the same resumes and evictions.

Memory

• macOS memory-pressure governor (on by default) for engines served through the shared lane host (Flash Next with --parallel above one, Qwen3.8-27B and GLM): at warning, queued requests wait until no other stream runs and the prompt cache sheds kept states down to half its budget (TENSORFOLD_PRESSURE_CACHE_PERCENT), writing each to disk first with --spill-gib; at critical, new requests get 503 (#1454). Replies are the same tokens, and at normal pressure nothing changes. TENSORFOLD_MEMORY_PRESSURE=0 turns it off; see the runbook.
• Resident memory on Linux: /metrics reports tensorfold:process_resident_bytes, its anonymous, file and shared parts, and the kernel's lifetime peak (#1422). On a GB10, host memory counts against the same pool as the weights.
• Nemotron on CUDA counts checkpoint bytes for admission from the shard headers alone, so with memory mapping off it no longer reads every shard into memory. The estimate is unchanged.
• Qwen3.6-35B-A3B on CUDA with TF_QWEN_KV_PAGES keeps each stream's MTP and DFlash2 draft ring and state in that stream's pool pages, instead of a fixed allocation for every stream past the first (#1356). Replies are the same tokens at the same speed.

Models

• MOSS-Image on Metal (test tool): MOSS-Image-Base's 5B transformer runs natively on an M5's tensor units as tf-moss-image, from the released bf16 checkpoint as published, on cases prepared with the publisher's own pipeline, which still encodes the prompt and decodes the picture (#1457). On an M5 Ultra a 1024x1024 picture's 50 steps take 47.3 s, and the decoded picture is 31.9 dB PSNR from the publisher's torch pipeline. Nothing is served yet; see the recipe.

Server

• Key scopes (opt-in): a key in --api-key-file can be limited to model requests (app[inference]: KEY) or to listings and metrics (grafana[read]: KEY), and gets 403 elsewhere before its body is read. Keys without brackets, and keys from --api-key or TENSORFOLD_API_KEY, keep every route. See key scopes.
• Admission limits (off by default): --max-inflight N and --max-inflight-per-key N cap model requests in progress, overall or per key, and --requests-per-second N with --request-burst N limit how fast a key starts them; a refused request gets 429 with Retry-After. See admission.
• Grammar budget (off by default): "grammar_budget": true keeps a reply under a grammar or a required tool call able to finish within max_tokens, so a tool call is no longer cut off unparsed (#1473). A grammar it cannot follow gets HTTP 400; a new grammar costs about 60 ms once for one tool, 640 ms for twelve.
• --loop-guard-stop (opt-in, implies --loop-guard): when the reasoning-loop guard fires, the reply ends there, with finish_reason: "content_filter" and error code reasoning_loop, instead of answering, so a client running tools no longer receives calls that repeat finished work (#1463).
• Streamed tool replies now carry an unoffered or malformed tool call that the whole reply keeps as text (#1365, #1429). An answer that starts with whitespace is sent when the reply ends.
• CUDA peer-to-peer report (on by default): with two or more ranks, the server logs at startup each GPU's P2P status and PCIe paths, the driver, CUDA and NCCL versions, the P2P settings in effect and NCCL's chosen transport, and changes nothing. TENSORFOLD_P2P_DIAGNOSTICS=0 turns it off.

Fixes

• GLM-5.3-Flash on CUDA with the DFlash2 drafter: at one rank, or three or more, loading the drafter at 4 bits (the default) no longer writes about 9.4 MB past the end of its scratch (#1496). Two ranks were not affected.
• Qwen3.5 and 3.6 on HIP (Linux): when a shard is read into memory because it can't be memory-mapped (as on some network or FUSE mounts), releasing a layer no longer zeroes neighbouring tensors not yet uploaded (269 of 690 layer tensors on Qwen3.5-2B), which ran the model on wrong weights. A malformed shard header is refused by name.
• MLX affine checkpoints: a projection with malformed geometry is refused by name before it is loaded. On HIP, a MoE shared expert stored in a different dtype from its routed experts is now converted from its own dtype instead of being read as the wrong bytes.
• Speech: the speech lane is freed as soon as a request finishes, and before any error reply, so back-to-back requests no longer get 503 "speech output capacity is full" (#1438).
• Compaction reads text parts as the chat template does, so a turn sent as an array of text parts is no longer treated as empty (#1406).
• --cortheon reads a vision turn's text parts as the chat template does, so its checks see the turn's words and different vision turns no longer share a key (#1469).

Tests and tools

• CI runs the release path check on a built release server, on Linux for pull requests and on every platform for pushes to main, so a private path left in a binary fails CI instead of the release (#1472).
• tools/bench_cache.py takes --api-key-env NAME to reach a server started with --api-key, and never prints or stores the key (#1483).
• The CUDA kernel contract runs to completion in a ReleaseSafe build (#1428).
• The kernel contract checks Nemotron's routed-expert Metal pipelines from one row to 128 (#1494).
• Gemma 4's E-series per-layer inputs have a host reference equal to MLX 0.32.4 bit for bit at one row; the E-series stays refused (#1402).
• The LoRA symlink tests skip on Windows when the account may not create symlinks (#1405).

Release testing

1.1.8 against 1.1.7 on the same machines and checkpoints, alternating the two builds. The cells were cold prompts from 2k to 64k, and the stream sweep at 1 to 8 streams (16 for LFM2) with every stream checked against its run alone. Every change above was also measured against main on its own before it landed.

• Every model gives the same tokens on both releases in every cell.
• DGX Spark (GB10):
  • Qwen3.8-27B: across eight alternating runs, prefill at 2k and 8k is within 0.2%; one pair puts 32k within 0.3%. The sweep is within 0.4%. #1458 still tracks the 32k and 64k prefill lost in 1.1.7.
  • Nemotron 3.5 Lightning: prefill within 0.7%. Across eight alternating runs, the sweep is within 0.3% at 1, 4 and 8 streams. At 2 streams the sampled replies are within 0.5%, and the greedy ones vary by up to 10% between repeats in both builds.
  • Qwen3.8 Flash Next FP4-Local: prefill +0.3% to +3.4%, decode −0.6% to +2.6%, and the sweep +0.3% to +0.5%.
• Two GB10s at TP=2, GLM-5.3-Flash NVFP4: no stream differs from its run alone (52 of 52), prefill is within 0.1%, and chat and code at one and two streams are within 0.4%.
• M5 Ultra:
  • LFM2.5-24B-A2B: the sweep rises 1.9%, 18.4%, 33.6% and 31.4% at 1, 4, 8 and 16 streams, and prefill is within 0.2%.
  • Qwen3.8-27B: across eight alternating runs, prefill is within 0.1% at every size from 2k to 64k. The sweep rises 2.3% and 1.4% at 1 and 2 streams.
  • Nemotron 3.5 Lightning: prefill within 0.2%. Across eight alternating runs, the sweep is within 0.3%.

Thanks

@Bizuayeu, @plotarmordev, @nuk3s, @ppardus, @BobClawblaw, @Arminova, @wojo, @kesslerio, @Maxrio9, @BlivionIaG, @Mirrdhyn, @jschmied, @gilby, @BadAd84 and @paragontasx.