TensorFold 1.0.8

TensorFold 1.0.8

• GLM-5.3, the whole 744B model with its MTP layer, serves on two Macs from kindlingai's trellis-coded EXLR8 release. tensorfold pull REPO@REVISION --exlr8 PROFILE --node R/N --dir DIR fetches one Mac's share by range, 139.5 GiB of expert halves and 11 GB of whole files for the home profile. serve --speed-up gives each Mac half of every expert and attention head, 159.6 GB at --context 8192 --parallel 4, so it needs two 256 GB Macs. On two M5 Ultras over one cable, one stream decodes at 21 tok/s with "draft": false and 31 to 32 with the MTP layer's drafts, and reads prompts at about 200 tok/s. Drafted, resumed and concurrent replies equal "draft": false, fresh and solo ones, idle and under load. The trellis decode rounds two weights to E4M3 at a time and splits few, long jobs over sets of simdgroups, with the same bits on an M5 Max, an M3 Ultra, an RTX PRO 6000 and a GB10. GLM-5.3 on two Macs has the commands. The Metal kernels build on @akol1's EXL3 projection (#681). Thanks to @akol1.
• Living Weights learns facts about the user more reliably. Ten facts told in turn bring back 18 of their 20 questions on an M5 Ultra, against 12 in 1.0.7, and 13 to 16 on a GB10, against 10. A newer fact that changes an earlier one replaces it, so after "I like blue." and two updates to red the model says red, where 1.0.7 kept saying blue (#678). A fact told in another language gets that language's own questions, and a tester's two Polish facts came back 3 of 4 times, against none in 1.0.7. A lesson that makes the model answer a question about the user it was never told, such as a cat's name, is taken out (#657). Thanks to @ThinkOffApp.
• Living Weights learns facts about other people, things and companies. A chat fact about someone or something other than the user is learned as a document's fact is, and a person's role at an organisation both ways round. A tester's six facts about two made-up companies bring back 10 of 14 questions on an M5 Ultra and 9 on a GB10, against 1 and 3 in 1.0.7. A 20-fact handbook taught in four parts brings back 13 to 15 of its 20 questions on an M5 Ultra, against 8 in 1.0.7, and the GB10's range is under known issues. A subject is cut at its verb, so "Our company's warehouse robot is called Pebble." is learned (#687).
• Qwen3.8-27B on a Mac keeps its weights in memory once, so a served 27B with its DFlash2 drafter takes 17.6 GiB less. In the release gate at a 70,000-token context it was ready at 29.6 GiB against 1.0.7's 47.2 on an M5 Max, and 24.6 against 42.2 on an M3 Ultra, with the same tokens and speed. A round the GPU fails ends only its own requests, where every later request used to fail (#643). Thanks to @RS91Dev for the report and the analysis.
• Two-Mac speed-up mode stays exact and fast while a Mac is busy. With 36 busy threads on the first Mac, two races in Flash Next's exchange between the Macs made "draft": false replies differ from drafted ones on 43 of 48 requests. Now 70 of 70 gave the drafted reply, the same on both Macs. Every two-Mac server also sets MCDMA_FABRIC_QOS=1 and sends from user-interactive threads, so under that load Flash Next's "draft": false replies decoded at 104.5 tok/s, level with an idle pair and against 64.5 with MCDMA_FABRIC_QOS=0, and GLM-5.3-Flash's at 80.1 against 87.9 idle. The speed-up guide no longer has a dump step.
• Flash Next keeps thinking budgets. A request's budget, or --thinking-budget, now ends its thinking at the budget with the chat template's close, and the answer follows, as on the other engines. Flash Next used to ignore it. Drafting stays on, with replies equal to "draft": false. The repro in #701, a 32-token budget, now thinks for 33 tokens counting </think>, against 136. When an engine reports that it can't enforce a budget, the server now refuses one with a 400, and no engine in 1.0.8 does (#649). Thanks to @akol1 and @dinadur.
• Nemotron conversations on CUDA resume 1.7 to 2.7 times sooner. Prompt states keep and resume at any row instead of on a 2,048-row grid, since every prompt row gets the same bits wherever the prompt is cut, at 17 cut patterns on a GB10 and an RTX PRO 6000. On a GB10, turns 2 to 5 started in 0.09 to 0.11 s against 1.0.7's 0.15 to 0.30 s (#670). A kept state now holds only its prompt's rows, so the 16 GiB default holds 258 states instead of 30. Short prompts keep states too, so on an M5 Max turns 2 to 5 of chats under 2,800 tokens start in 0.19 to 0.31 s instead of 0.66 to 0.85 s (#669). Replies are unchanged. Thanks to @jschmied.
• logprobs on CUDA covers streamed chat completions, answer-only with thinking on, and /v1/completions in the legacy form, streamed or not (#645). Nonstreamed /v1/completions also takes vLLM's prompt_logprobs, with each prompt token's logprob, rank and up to 20 best tokens. It prefills the whole prompt, so 2,931 tokens with prompt_logprobs: 20 took 0.83 s against 0.44 s (#647). On a GB10 every row stayed within 2.1e-6 of a float64 reference, and other requests keep their tokens and speed (#449). Thanks to @plotarmordev, to @Bizuayeu for the spec in #399 and to @jetnet for asking in #298.
• GLM-5.3-Flash keeps other conversations' prompt states while it is busy. A kept state's prefixes are copied out of its slot before a new stream writes over them, on one Mac with GLM_POOL=0 (#644) and on two, and new streams take the least recently used slot. On two M5 Ultras after a busy session, a prompt A, an unrelated B and A again resumes 3,165 of A's 3,181 tokens, where 1.0.7 resumed none. That was the known issue of 1.0.6 and 1.0.7. Thanks to @BobClawblaw.
• GLM-5.3-Flash on one Mac lets short requests pass one that waits for room, so they answer in 0.2 to 0.3 s instead of 12.3 to 12.5 s, and it resumes a conversation longer than half of --context instead of reading it again. Qwen3.8-27B and GLM-5.3-Flash answer every turn that fits --context, where turns near its end got an HTTP 500.
• Qwen3.8-27B on an M1 to M4 Mac drafts as on a fresh server again after sampled requests, fixing 1.0.7's known issue. On an M3 Ultra, greedy code after two sampled prose replies decodes at 150.6 tok/s against 144.0. With the DFlash2 drafter it now starts at a small --context, where it refused with BadRoundPlan (#661), and its served checks test the host that ships (#659). Thanks to @BobClawblaw.
• Nemotron reads prompts 0.9 to 1.2% faster on an M3 Ultra and 2 to 5% faster on an M5 Max, with the same bits. On a GB10 its structured-output windows replay captured graphs, so decode under a grammar is 4.4% faster with "draft": false (#660). Thanks to @BobClawblaw.
• Flash Next with FZ_LANE=1 loads on M1 to M4 Macs, with projection outputs byte-equal to an M5 Max's (#679). The core lane projection's group-32 prepared sums are fixed, a path no shipped model takes, and zig build test-core-lane passes all 115 layouts (#698, fixes #675). Thanks to @cwschroeder.
• Whole chat completions, /v1/messages and /v1/responses replies keep the prose a model writes before its tool calls (#650). Chat templates can pass **mapping to macros, functions and filters, as Jinja2 does (#648). A streamed /v1/messages reply's message_start carries the prompt's size (#672). Tool calls inside a legacy DSML envelope stay inside it (#684). Thanks to @akol1 and @kky42.
• For contributors: the lane host fills a foreground prompt first, then the shortest (#651); CUDA's keyed draws live in cuda/sample.zig (#654); a backend sets the server's std options (#696); the cluster draw refuses non-finite logits (#658); and the opt-in HIP runtime has events and memory info (#630). Thanks to @BlivionIaG, @jschmied, @chaog992 and @jkuepker, and to @ThinkOffApp for HIP runs.

Behaviour changes:

• On a Mac, every model but Nemotron refuses --slide at startup, since Living Weights learns into Nemotron only. A CUDA server, and Qwen3.8-27B, Qwen3.5-2B and GLM-5.3 (EXLR8) on a Mac, refuse --learn, --learn-dir and --learn-gib. These servers used to take the flags and do nothing with them.
• Two-Mac servers set MCDMA_FABRIC_QOS=1 unless the variable is already set.
• The speed-up guide no longer makes a dump or sets TF_FLASHNEXT_DUMP, since the server loads Flash Next from its checkpoint, or FZ_LANE=1 FZ_GDN=2, which decode 2 to 5% slower than the default kernels. Leave them out of a setup that followed the old guide.
• A streamed reply that starts with {, [ or a code fence gets its tool calls whole at the end instead of as argument deltas (#684). XML calls such as Nemotron's stream as before.
• Nemotron keeps prompt states for prompts under 4,096 tokens, so short chats fill the prompt cache up to its budget and evict as long ones do, at 50 to 62 MiB a state (#669, #670).
• On a Mac, the prompt cache's default budget stops at 16 GiB for Nemotron, Flash Next and Qwen3.8-27B. It used to take what 70% of RAM leaves, 68.9 GiB for Nemotron on a 128 GB M5 Max.
• A chat template call that names one keyword twice now fails, as in Jinja2, where it used to take the last value (#648).

Known issues:

• GLM-5.3 (EXLR8) decodes slowly next to the other models, since a one-row round on two M5 Ultras takes about 45 ms. No CUDA engine serves it, and only the home profile on M5 Ultras has run. Replies past 2,048 tokens, where the sparse indexer starts picking keys, have not been compared with a whole-model reference.
• Living Weights: facts about the same subject can overwrite each other. In a 20-fact handbook some questions come back with another fact about the same company: on a GB10, "Who is the head of marketing at Quillo Labs?" was answered with other officers' names as facts were added. The handbook brings back 13 to 15 of 20 on a GB10 across three runs, against 10 for 1.0.7. Teach a few facts per subject.
• Living Weights on a GB10: a later fact about the user can disturb an earlier one's answers, now and then with a wrong value ("You live in Dublin." after "I live in Bristol." and a birthday). On both machines, yes-or-no questions about a changed value are unreliable ("Do I like blue?" after the change to red).
• Living Weights: learned facts cost decode speed on a Mac. On an M5 Ultra decode runs at 213 tok/s after ten facts and 205 after a hundred, against 248 before any. On a GB10 thirty facts cost 1 to 6%.
• Living Weights: facts pile up badly past a few dozen. On an M5 Ultra, 100 facts taught in turn kept 69 and brought back 82 of their 200 questions, as later lessons overwrote earlier ones. General answers held at 28 of 30 checks, as before any fact.
• Living Weights with thinking on brings back fewer answers. In the release's three-fact checks, 3 of 6 thinking wordings came back on an M5 Ultra and 4 to 6 of 6 on a GB10, against 6 of 6 with thinking off.
• Nemotron on CUDA: on a GB10 a cold first turn under 1,400 tokens takes 1 to 3 ms longer (1 to 2%), since it now keeps two or three states (#670). From 10,000 tokens it takes as long as before.
• Flash Next in speed-up mode: a drafted request right after a "draft": false one decodes slowly, 103 against 169 tok/s on a short story, and the next request is back to speed. Same tokens.
• Qwen3.8-27B on an M1 to M4 Mac: the drafter's landing counts last for the server's life, so after four prose replies greedy code drafts shallower, about 4% slower on an M3 Ultra (141.7 and 146.4 against about 150 tok/s). Same tokens.
• Qwen3.5-2B with a JSON schema: on an integer field it wants to write as a decimal, the 2B can keep adding digits until max_tokens, since the grammar allows no decimal point there. Nemotron writes the same schema correctly.

Homebrew: brew upgrade tensorfold.