TensorFold 1.0.6

TensorFold 1.0.6

• Structured output on the native server. Nemotron 3.5 Lightning enforces response_format (json_object, json_schema) and vLLM's guided_json, guided_regex, guided_choice, guided_grammar and structured_outputs on Metal and CUDA, and Qwen3.5-2B on Metal. Drafting stays on: the grammar cuts the drafts it rejects and drafts the text it forces itself (keys, closing markup), and each row's logits are masked before its draw, so every constrained reply equals its "draft": false reply token for token, alone or beside other streams, and replies without a grammar are unchanged. With thinking off, Nemotron decodes JSON-schema replies at 245 to 290 tok/s on an M3 Ultra and 645 to 765 on an RTX PRO 6000. xgrammar v0.2.8 loads at runtime from lib/libtfgrammar, which each release archive now ships; without it, a structured request gets a 400 and everything else serves (#596, #564). Thanks to @BobClawblaw.
• tool_choice: "required" and named functions work on the same engines: the answer is one or more calls in the chat template's own markup, each to an offered tool with its arguments inside the tool's schema, and parallel_tool_calls: false allows exactly one. Nemotron decodes required tool calls at 290 to 320 tok/s on an M3 Ultra and 790 to 955 on an RTX PRO 6000. Before, the native server refused them with a 400.
• RTX 50-series and RTX PRO 6000 Blackwell cards (compute capability 12.0) run the native engine from the Linux x86_64 archive, which now carries every CUDA kernel for sm_120 beside sm_86 and sm_121. Before, they stopped at kernel load with CUDA_ERROR_NO_BINARY_FOR_GPU. On an RTX PRO 6000, Nemotron MLX 4-bit decodes greedy at 874 tok/s on code and 510 to 607 on prose with drafts (#610). Thanks to @Nipale-ai for the report and the RTX 5090 receipt.
• NVIDIA's NVFP4 checkpoint of Nemotron 3.5 Lightning serves on the native CUDA engine (sm_90 or newer). Its FP8 Mamba projections, NVFP4 experts and head, and bf16 attention load as stored, while activations and the KV cache stay bf16, about 10x closer to a float64 reference than NVIDIA's own math on the FP8 layers. On a GB10 it reads prompts as fast as the MLX 4-bit checkpoint (6,131 to 7,281 tok/s from 2k to 32k tokens) and decodes at 96 to 182 tok/s drafted; drafted, resumed and concurrent replies equal plain ones. It builds on #595, #629 and #633. Thanks to @jschmied.
• GLM-5.3-Flash on two Macs times its draft windows when it first loads (about 6 s, kept per build) instead of pricing them from a fixed table. That fixes 1.0.5's known issue: on two M5 Ultras a 256-token story decodes at 112.5 tok/s against 106.6 in 1.0.5, and six prose prompts at a median of 115.2 against 108.7 in 1.0.5 and 112.7 in 1.0.4. Code, plain decode and prompts are unchanged.
• Qwen3.8-27B serves concurrent requests: streams share each decode round across --parallel runner slots (up to 8 by default, each slot past the first taking its memory when first used, all of it counted when the server loads), and the DFlash2 drafter drafts for one of them. Sampled requests draft too: on an M3 Ultra a sampled code reply decodes at 125 tok/s and a prose one at 51, where 1.0.5 decoded both at about 20, with the same tokens. Eight greedy streams decoding at once reach 1.7x the throughput of one reply at a time on an M3 Ultra (124 against 73 tok/s; 1.68x on an M5 Max), and every stream equals its run alone and "draft": false. An explicit --parallel N that doesn't fit refuses to start. A request sent right after a reply no longer waits behind that reply's background warm-up (#578). Thanks to @BobClawblaw.
• Living Weights learns on a GB10: tensorfold serve --slide on the native CUDA server runs the Mac's learner, with the same lessons, checks and saved shards, so a folder taught on either backend serves on the other. Every CUDA training kernel matched a float64 reference within 2.4e-6, and a lesson repeats run to run because the training step's split products add their slices in a fixed order. Without --slide, replies are the same tokens and decode is unchanged; with it, a decoded token on a GB10 costs about 1% more with no lessons, 6% with one kept lesson and 11% with four (#591, #632, closes #590). Thanks to @ZackSample.
• Living Weights keeps a lesson only when every fact came back. One that misses its held-out questions, or disturbs another answer, is taken out whole before anything is written, and learned says "recalled": false with the reason (#599, fixes #589). Thanks to @ThinkOffApp.
• GLM-5.3-Flash on one Mac shares one MLA cache across its streams, so --parallel 3 serves the full 1M-token window: with 8 of the 45 layers loaded, three streams took 48 GB against 85 GB, with every reply unchanged (#639). GLM loads drop the checkpoint's cached pages before reading it, only as much as free memory lacks: a load with the whole 169 GiB checkpoint cached served in 16 s, where 1.0.5 had not served after 5 minutes (#600, #640). A new GLM stream takes a free slot that holds no kept prefix before one that does, so while one is free an unrelated request leaves another conversation's kept prefix alone: after a prompt A, an unrelated B and A again, the third request resumes 3,165 of its 3,181 tokens on one Mac and on two, where 1.0.5 read them all again (#622). Thanks to @BobClawblaw.
• --learn keeps Flash Next's shared prompt prefixes on disk on one Mac, so a new server resumes them without reading them again: on an M5 Ultra a fresh server resumed 5,591 of a 6,122-token first prompt from disk (#519). Thanks to @BobClawblaw.
• Nemotron on CUDA captures CUDA graphs for each concurrent sequence the first time it runs a step and replays them after, instead of running other sequences' windows eagerly (#615, #597). Thanks to @BobClawblaw.
• Fixes: the JSON writer turns a UTF-8 sequence cut off at a string's end into U+FFFD instead of reading past the string (#634); a NaN or infinite value written the way Python's str() does no longer crashes the server (#604). Thanks to @jayleaton and @chaog992.
• The opt-in AMD HIP build accepts RDNA2 (gfx1030) and RDNA3 (gfx1100) cards, with a capabilities table from RDNA2 to RDNA4; no model runs on HIP yet (#607, #609). Thanks to @BlivionIaG.
• For contributors: weight formats behind one interface, MLX affine 2- to 8-bit and dense, with tensor-parallel cuts (#612); a kernel registry that picks a product's tile by tuned cost (#613); a prompt pass of one chunk a round between other streams' rounds (#623); serve flags a backend opts into (#624); a paged prompt cache, a radix tree over the backend's own KV pages (#625); what a model engine needs from the HIP runtime (#611); the CUDA affine-4 repack beside its format (#602) and grouped experts behind one call (#605); and zig fmt on three files (#616). Thanks to @BlivionIaG, @jschmied and @BobClawblaw.

Behaviour changes:

• Nested token ids in a completion prompt that fall outside the vocabulary, or are booleans, get a 400 as top-level ids do. Before, the server dropped them or read them as other ids (#619). Thanks to @chaog992.
• A structured-output request or tool_choice: "required" used to get a 400 from the native server; it now runs, and gets a 400 only when lib/libtfgrammar is missing. A request that combines a grammar with logprobs or with a required tool call is refused.
• GLM loads make a pass over the checkpoint's file cache first (about 2.5 s a load); a two-Mac GLM server spends about 6 s timing its draft windows on a build's first load.

Known issues:

• Qwen3.8-27B on an M3 Ultra: greedy code decodes about 4.5% slower than in 1.0.5 (138 to 140 tok/s against 146 on the release gate's code prompt; same tokens, one or two more draft rounds). Prose and every M5 Max cell are level. After drafted sampled requests on the same server, greedy code can trail 1.0.5 by up to 10% for a while, since their rounds share one table of round times. A fix is planned for 1.0.7.
• GLM-5.3-Flash on two Macs: once every free slot holds a kept prefix, an unrelated request can still land on the newest conversation's slot, because the choice counts kept prefixes rather than their age. One Mac's shared cache is not affected. 1.0.7 picks the least recently used.
• Qwen3.5-2B with a JSON schema: on an integer field it wants to write as a decimal, the 2B can keep adding digits until max_tokens (the grammar allows no decimal point there). Nemotron writes the same schema correctly.

Homebrew: brew upgrade tensorfold.