TensorFold 1.0.5

TensorFold 1.0.5

• Nemotron 3.5 Lightning keeps each conversation's prompt state between requests on Metal and on CUDA, so a later turn prefills only its new tokens, and every reply equals the same request with the prompt cache off. On an M3 Ultra the second turn of a 19.8k-token chat prefilled in 0.16 s instead of 8.1 s; on an M5 Max, later turns of a 7k-token chat reach their first token in 0.02 to 0.17 s instead of 1.6 to 2.1 s. On a GB10 a conversation resumes from the state kept at the last 2,048-token chunk boundary before its history, and a resumed turn of a 7,200-token chat reaches its first token in 0.19 to 0.21 s instead of 1.06 to 1.09 s. --prompt-cache-gib sizes the kept states and 0 turns them off (#497, #467). Thanks to @BobClawblaw and @AdrianBinDC, and to @SvangenStudios and @plotarmordev for the CUDA port's receipts.
• Editing, deleting or regenerating a conversation's last turn resumes from a prompt state kept just before the latest user message, on every native engine with a prompt cache, and the reply equals the cache-off reply. On an M3 Ultra, the Qwen3.8-27B's deleted, edited, regenerated and re-edited last turns of a 9,000-token chat took 1.7 s of prompt processing in all against 10.9 s (#571). Thanks to @otarkhan.
• GLM-5.3-Flash runs on one 256 GB M5 Ultra. 1.0.4 refused it because the 4-bit checkpoint with its MTP head, 180.6 GB, is over the one-Mac limit of 70% of RAM. The load now leaves the head on disk when only the head is over, and says so; copy drafts still work. On one M5 Ultra the full model reads 2k to 64k-token prompts at 1,859 to 1,990 tok/s, and decodes prose at 78 tok/s and code edits at 165.
• GLM prompts run more of the model on the M5 tensor units. On two M5 Ultras they read 2.1k, 4k, 8k and 32k tokens at 2,090, 2,079, 2,513 and 2,497 tok/s, against 1,417, 1,559, 1,803 and 1,662 in 1.0.4. The prompt matmuls sum each row in fp32, which also brings them closer to fp64: 1.1e-6 of the outputs' rms, from 3.8e-3. GLM decode runs attention and index scores on the tensor units too: on two M5 Ultras code edits decode at 262 tok/s, from 239, and plain decode at 93, from 91.
• GLM keeps lighter prompt states (141 MiB instead of 587 for a 26.6k-token chat), so the full model on one Mac keeps them, and it keeps states for prompts under 4,096 tokens too, so requests that share a shorter system prompt resume instead of reading it again (#584). GLM loads read the checkpoint around the page cache and wait while free memory is under 3% of RAM: a 160 GB load on a 256 GB M5 Ultra had filled the page cache and restarted the Mac. Thanks to @BobClawblaw.
• --learn keeps Nemotron's shared prompt prefixes, such as a system prompt and its tools, on disk, so a new server resumes them without reading them again; weights that --slide writes into the model's files start a new set (#499). Thanks to @BobClawblaw.
• --learn keeps a free-disk floor, --learn-min-free-gib (4 GiB by default): least recently used learned states make room when that is enough, otherwise learning pauses with a log line and serving goes on. Learned files carry a checksum, so a damaged or older file is refused and learned again. Both Macs of a GLM pair need the same release (#508). Thanks to @wojo.
• --learn keeps what it learned through a crash in the middle of a save. A record cut short at the end of the learned index is trimmed when the server starts, so states learned after that restart are read back at the next one, and a .part file left by an interrupted save is removed at startup instead of counting toward the --learn-gib cap. These were two of 1.0.3's known issues. Neither ever changed a reply; a lost state cost a cache miss.
• Flash Next keeps the weight packs it builds at startup under ~/.cache/tensorfold/packs, one folder per checkpoint (about 6.3 GB for the 6-bit checkpoint), instead of in a zig-pack folder inside the checkpoint, so the model folder can be read-only. A build is published only once every pack in it is complete. A complete zig-pack from an earlier release is still read and never rewritten. This was 1.0.3's third known issue.
• logprobs and top_logprobs (0 to 20) on nonstreamed /v1/chat/completions for Nemotron on CUDA. Each reply token's row is its log probability under the model's raw distribution, read on the GPU from the verified target logits, so drafted, "draft": false, concurrent and resumed requests give the same rows. With thinking on, the rows cover the answer only. Streaming, tools, stop strings, structured output and a thinking budget stay refused with logprobs, and so do the Metal engines and the Responses and completions routes. On a GB10 every row matched a float64 log-softmax of the same logits within 2e-6, and a 256-token reply with 20 alternatives decoded 7% slower than without (#572, answers #449). Thanks to @plotarmordev and @DakotaTexas.
• Nemotron's shared rounds hold four rows a lane, up to 128, so 32 and 64 sessions keep drafting, and its routed experts take three member rows a pass with the same bits: on an M3 Ultra, 64 sessions reach 409 tok/s in all, from 376. --parallel 32 and 64 use 3 GiB more memory (#518, #502). Thanks to @BobClawblaw.
• Living Weights learns from Japanese and other text written without spaces, and holds a fact told in another language against prompts in that language (#570). Its leak check no longer counts an everyday word as a leak: "If you'd like, I can help" used to undo a round that taught "I like blue." (#590). Thanks to @takanobuouchi-cell and @ZackSample.
• The Qwen3.8-27B loads on M5 Macs running macOS 27 again: under Metal 4.1 two of its kernels failed to compile, so the engine stopped with LibraryLoad (#573, fixes #579). Thanks to @jkuepker and @ApaApatu.
• A completion request whose nested prompt ids reach 2^32 or more no longer crashes the server (#594). A request keeps its own copy of its API key's label when the key file is reloaded under it, and the server frees its key store and request counters when it stops (#560, #513). Flash Next on two Macs bounds each wait for the other rank: a handoff that never arrives fails the request after 10 seconds instead of spinning forever (#512). A two-Mac server whose other rank never answers says what to check, macOS Local Network access included (#567). Thanks to @chaog992, @CerebralCoding, @321sssrt-bit and @appljd.
• The repository checks out on Windows: eight server golden files had ?, " or : in their names. They now carry those characters as %XX, and zig build test fails on any path Windows can't check out.
• /metrics reports pinned host memory on CUDA and counts prompt tokens restored from kept prompt states, repeated as vLLM's prefix_cache_hits_total and prefix_cache_queries_total (#541, #531). The --dashboard page shows a decode/prefill dial, memory, tokens-in-flight and MTP acceptance tiles, and a five-minute decode-rate chart (#561). The native server accepts --chat-template FILE (#551). The native Qwen3.5-2B serves checkpoints that ship their own lm_head (#493). Thanks to @GustavBlack, @juliankang4, @heitke and @akol1.
• The CUDA tensorfold command built from source runs models, info and pull without opening the driver (#576). Nemotron on CUDA can pause a long prompt while the machine runs hot (TF_HEAT_HIGH, TF_HEAT_LOW), and, experimentally on a GB10, keep its KV caches in the display carveout (TF_CUDA_CARVEOUT=1, #487). Thanks to @eleqtrizit, @Bizuayeu, @urtho and @plotarmordev.
• RUNBOOK.md describes today's CUDA qualification (GB10, and Ampere 8.6 cards) and drops the 1.0.0 gaps (#537). The CUDA and GLM recipes tell Python 0.6.6 features from native support (#534), the speed-up guide says its two-Mac numbers come from a build with a recorded dump (#567), and the CUDA recipe links a user's 1.0.4 receipt on a bare-metal RTX 3090 (#592). A flag or checkpoint the native engine does not serve is refused with the Python 0.6.6 install line. Thanks to @GustavBlack, @akol1, @appljd and @briepala.
• For contributors: a model-free block-FP8 projection on CUDA (#482), one home for CUDA quantized projections, cuda/qlinear.zig (#587), one shared routed-expert plan, cuda/grouped.zig (#558), an external-drafter interface in the lane core (#455), an opt-in HIP runtime for AMD GPUs with no model yet (#463), a backend-neutral EXL3 reference decoder (#496), token bytes from the tokenizer (#532), rank-five tensor shapes (#533), the HTTP server's unit tests on Linux (#543), logprob.cu on nvcc 13.3 (#539), and tf-moe-bench and tf-mm-bench for Metal kernels. Thanks to @jschmied, @ThinkOffApp with @BlivionIaG, @jkuepker and @millaguie, @akol1, @MiaAI-Lab and @GustavBlack.

Behaviour changes:

• tensorfold.seconds in a native reply is now its decode time, the span tokens_per_second divides, so a long prompt no longer reads as a slow decode; prefill_seconds and time_to_first_token carry the prompt (#588). Thanks to @eleqtrizit.
• A request with a temperature to an engine that decodes greedily only (GLM-5.3-Flash, Flash Next) is refused with a 400 before it is admitted, instead of failing after it waited its turn (#582). Thanks to @BobClawblaw.
• top_k: a request that sets none, on a checkpoint whose generation config sets none, keeps every token, on CUDA as on Metal, as in 1.0.4. Python 0.6.6 filtered CUDA requests to the top 20; --top-k 20 gives that back (#515).
• Nemotron servers now keep prompt states by default, inside the prompt-cache budget; --prompt-cache-gib 0 turns them off.
• GLM-5.3-Flash's greedy replies differ from 1.0.4's for the same request: its prompt rows now sum in fp32, closer to fp64, and attention runs on the tensor units. A reply read by tf-glm-run in one chunk can differ from the server's on a long prompt, because the server cuts prompts into chunks; served replies still equal fresh ones and drafted replies plain ones.
• GLM_LOAD_LIMIT_GB loads GLM-5.3-Flash past 70% of a Mac's RAM only when you set it, up to the GPU's recommended working set, and the server logs the wired-load panic risk at startup whenever the limit passes 70%, which stays the default (#498, #521, #580). Thanks to @heitke and @BobClawblaw, and to @Gogo6969 and @tpischke for the measurements.

Known issues:

• GLM-5.3-Flash on two M5 Ultras decodes prose about 4% slower than 1.0.4 with drafts: 106 tok/s against 111 on a 256-token story, and on five of six prose prompts measured. Plain decode is 0.6% faster, code is not slower, and prompts read 1.4 to 1.8x faster. 1.0.5 drafts more tokens a round on prose for about as many accepted.
• The Qwen3.8-27B drafts greedy requests only: a sampled request decodes plain, about 19 tok/s on an M5 Max or an M3 Ultra, as in 1.0.4.
• Qwen3.5-2B reads long prompts slowly: on an M5 Max 893 tok/s at 64k tokens and 218 at 256k, against 2,588 at 2k.

Homebrew: brew upgrade tensorfold.