TensorFold 1.0.9

TensorFold 1.0.9

• tensorfold serve MODEL --cortheon runs Cortheon's reasoning program and web research inside the server, for chat completions, /v1/responses and /v1/messages, so any OpenAI-compatible client gets it with no setup. It is off by default, and without the flag every reply is unchanged. The model gets Cortheon's stages for its task type and two tools the server runs itself, cortheon_search and cortheon_fetch, whose calls the client never sees. A question about current or version-specific facts, or a request to build with an outside library or API, is researched before the model starts: the model writes up to three queries and the server reads up to three of the best pages. With the flag on, the search queries the model writes from the conversation leave the machine, for DuckDuckGo, or for Brave's API when BRAVE_SEARCH_API_KEY is set, and pages are fetched from this machine, so each site sees its address. Web text loses sentences that read as orders to the model, and reads reach public addresses only, with every redirect checked and TLS verified. The filter is a heuristic, though: a page written to steer the model could make it put conversation text into the path of a URL it fetches, so use the flag where that risk is acceptable. Cortheon mode has the details.
• Nemotron on CUDA serves concurrent requests faster. A shared round drafts every stream's MTP head in one batch (#768), runs each Mamba layer's conv and scan and each attention layer's key write, chunks and merge as one launch for all its streams (#770), prices each stream's drafts by the head steps they take (#769), and keeps more of each decode step's loads in flight (#771). Every stream's rows equal its one-at-a-time rows to the bit. On a GB10 with distinct prompts and 256-token replies, aggregate decode rose 4.4 to 5.5% at 2 streams, 9.4 to 11.6% at 4 and 11.8 to 14.7% at 8, and a lone stream decodes 1.9 to 2.4% faster. In this release's gate on a GB10 with the MLX 4-bit checkpoint, against 1.0.8, one stream decoded 1.7 to 2.3% faster and eight streams 15.1 to 16.0% faster (258.5 against 224.5 tok/s greedy). Every reply equalled its solo run. Thanks to @plotarmordev.
• Nemotron on CUDA reads long prompts faster. A prompt runs in passes of up to 4,096 rows instead of 2,048-row chunks, so each pass reads the routed experts' weights once for twice as many rows. On a GB10 with the MLX 4-bit checkpoint, cold prompts of 8k, 16k, 32k and 64k tokens prefill 7.2%, 6.8%, 6.3% and 5.4% faster, and 2k as before. This release's gate measured 7.3%, 7.0% and 6.4% at 8k, 16k and 32k against 1.0.8, 2k level, and a 64k prompt answered in 11.7 s. A row's bits don't depend on where a pass is cut, so replies and kept prompt states are unchanged (#720). Thanks to @plotarmordev.
• Nemotron on CUDA keeps more requests on its lone driver, the fast path for a request served alone. It now takes requests with logprobs (#777), thinking budgets above 32 tokens and --loop-guard (#778), and structured output (#779), and it takes back the last live stream once the requests beside it finish (#780). On a GB10, a 400-token reply with logprobs costs 0.3%, 0.9% and 3.1% more than one without at 0, 5 and 20 alternatives, against 4.6%, 5.6% and 8.4% before. 700-token thinking replies with a 4,000-token budget finish 3.3 to 8.1% sooner, 500-token JSON replies 8% sooner under json_object and 13% under a schema, and required tool calls 14% sooner. prompt_logprobs reads its prompt rows through the head 256 at a time, so on an 8,741-token prompt its extra cost fell from 0.70 s to 0.13 s (#776). Every reply equals the lane core's and "draft": false. Thanks to @plotarmordev.
• Nemotron conversations on CUDA pick up where the last reply ended. With the prompt cache on, each reply is prefilled in the background for the next turn, as on the Macs (#723), once the engine has had no other work for 150 ms, so a request sent the moment a reply ends goes first and never waits for that pass. In chats with a second between turns, later turns resumed past each reply and started 2.0 to 2.2 times sooner than in 1.0.8 on a GB10; turns sent back to back resume what they resumed in 1.0.8, and a new chat's first turn sent right after another reply starts as soon as in 1.0.8. This release's gate on a GB10 sent each turn the moment the last reply ended: every turn resumed as many tokens as in 1.0.8 and gave 1.0.8's reply, later turns of chats with 9k to 16k-token histories started 1.17 to 1.26 times sooner, and a new chat's first turn started within 1.1 ms of 1.0.8's at 331 and 1,034 tokens and 6.5 to 7.2% sooner at 10k and 19k. The native server's tokenizer keeps the ids of long text pieces, so a chat's earlier messages aren't tokenized again on every turn: on a GB10 the time before the prompt pass fell from 12.6 to 1.6 ms at 12k tokens and from 41.5 to 2.3 ms at 35k, with the same ids from every family's tokenizer (#767). Thanks to @plotarmordev.
• Qwen3.8-27B on a Mac decodes faster, with the same tokens. A chain round no longer copies each DeltaNet layer's state, about 288 MiB read and written for every decoded token (#711). On an M5, a chain of decode rows attends to its own keys in the prefix kernel, 4 dispatches a layer instead of 7 (#712). The DFlash2 drafter starts sooner and waits less (#713). On an M5 Ultra, greedy decode went from 406.1 to 417.0 tok/s on code and from 100.6 to 103.6 on prose, and seeded sampling from 294.0 to 316.9 and from 86.2 to 89.0. On an M3 Ultra, greedy code went from 150.6 to 154.7 and prose from 53.3 to 54.8. In this release's gate on an M3 Ultra, against 1.0.8, prose decoded 2.2 to 2.8% faster and code level to 1.5% faster, with the same tokens. Thanks to @jkuepker.
• Qwen3.8-27B serves on Apple's GPU generation 18 (M6). A Mac mini M6 with 32 GB ran the 27B with its DFlash2 drafter, with drafted replies equal to "draft": false at 16k and 32k contexts, thinking budgets cut where asked and no swap. Later generations still refuse until someone runs them. On a Mac where the Neural Engine prompt share doesn't fit under the 70% load limit beside the model, as on a 32 GB Mac, the 27B now runs prompts on the GPU alone and says so at load; the share's 5,760 MiB used to push such a Mac into swap for about 5% faster prompts. With the 4-bit checkpoint, Macs with 48 GB or more keep the share (#643). Thanks to @RS91Dev.
• Short prompts and shared histories resume. The 27B and Flash Next on a Mac keep a prompt state inside prompts under 4,096 tokens, where they kept none, so a repeated first turn and a follow-up with thinking on now resume. On an M3 Ultra a 27B follow-up of about 2,900 tokens with thinking on starts in 0.4 to 0.5 s instead of 10.4 to 10.6 s, and on an M5 Ultra Flash Next's follow-ups of 1,000 to 3,000 tokens start in 0.08 to 0.10 s instead of 0.35 to 0.82 s. The prompt cache also keeps a state that several continuations resume, such as a regenerated reply or an agent that starts each step from the same history: with a small cache, a third fork of a 3,962-token conversation resumed 3,956 tokens where 1.0.8 read the history again (#735). Replies are byte for byte the same. Thanks to @BadAd84.
• More clients can call the server. /v1/responses accepts OpenAI's include values and ignores them, where it refused any include; an unknown value is still refused (#759). GLM tool calls the model wrote slightly wrong still parse: a key written without its <arg_key> gets it back, and a call the end token cut off inside its value is closed when the closed call then parses strictly (#761). The repairs and their cases come from @MiaAI-Lab's #285. Thanks to @Bizuayeu and @MiaAI-Lab.
• M1 Macs run GLM-5.3-Flash's sparse attention and Living Weights training. Their kernels now declare the 1,024 threads their callers launch; an M1 Max compiled them for 832 and 768, which left attention outputs at zero and could leave gradients unwritten, with no error (#704, #715). The Metal EXL3 projection reads each trellis state without a branch, with the same bits at every offset (#718). Thanks to @mrv777.
• On a Mac, a Nemotron layer that Living Weights taught runs its pocket as two dispatches beside other work, where it ran three after the shared expert, each behind a barrier. Every sum keeps its order. On an M3 Ultra, a folder taught two facts gave the same replies and decoded 8.2% faster greedy and 8.3% sampled (#775). Thanks to @ThinkOffApp.
• The fragment check Nemotron and Flash Next run at every load on a Mac now catches a transposed operand, which its old test values could not see (#744), and GLM-5.3-Flash runs the same check right after its kernels compile (#745). A GPU whose layout gives wrong sums stops the load with error.FragLayout instead of serving wrong tokens. Thanks to @harrisonfriia.
• tensorfold pull finishes into a relative cache root (a relative HF_HUB_CACHE or HF_HOME), and pull --exlr8 into a relative --dir, where the shipped binary stopped at the first finished file (#700). The Flash Next recipe names the checkpoint the native engine serves on Macs (#705). Flash Next loads in Debug and ReleaseSafe builds of the engine, where config.pleRef's multiply overflowed and panicked at load; release builds computed the same values before and after (#783). Thanks to @chaog992, @cwschroeder and @grantoverton.
• For contributors: the CUDA host forwards every entry a family's lane backend gives it, so the lane core can run a CUDA family's prompt a chunk at a time once a family does (#742); CUDA has shared RMS norm and token-row kernels with float64 references, and Nemotron's 4-bit token rows run on them with the same bits when it serves with our own kernels (#736); FP8 and NVFP4 values and their float64 references live in core/quant, and the host reads e4m3's two NaN codes as NaN (#730); Metal and CUDA samplers read one sampling rule built from the request (#731); tf-metal-counters times GPU work with the GPU's own counters, no model needed (#680, #676); tf-cuda-test prompt-cuts checks kept prompt states against one uncut pass (from #721 and #722) and tf-cuda-test glue checks the shared-window kernels byte for byte (#770, #771); Nemotron on CUDA times its draft-depth windows on the text a round verifies, not the MTP head's own looping chain. Thanks to @Bizuayeu, @jschmied, @akol1 and @plotarmordev.

Behaviour changes:

• Nemotron and Qwen3.5-2B on a Mac refuse --speed-up at startup, with the 27B's refusal now worded the same way, since only Flash Next, GLM-5.3-Flash and GLM-5.3 join two Macs; they used to take the flag and do nothing. A CUDA build refuses --speed-up, and the CUDA server refuses --prompt-cache-over-cap, and --drafter for a family that drafts with its own MTP head, instead of ignoring them (#733). Thanks to @jschmied.
• Nemotron on CUDA says "prompts in passes of up to 4096 rows" at startup. Staggered segments, layer dumps, lessons and a pass with 16 or more prompt-cache marks inside it keep 2,048-row chunks (#720).
• Nemotron on CUDA, with the prompt cache on, prefills a reply for the next turn in the background once the engine has been quiet for 150 ms. A request that arrives while that pass runs waits for its current chunk, at most 256 rows, and the pass keeps the state it reached (#723).
• The 27B and Flash Next on a Mac keep a prompt state for prompts under 4,096 tokens, where they kept none, so short chats now take space in the prompt cache.
• The prompt cache treats a state that is resumed again, after a longer prompt kept a state extending it, as a branch point and frees it only as the least recently used. A conversation that only grows still frees its old turns first (#735).
• The 27B on a Mac prints at load when it runs prompts on the GPU alone because the Neural Engine share doesn't fit under the 70% load limit (#643).
• GLM-5.3-Flash on a Mac stops its load with error.FragLayout when the fragment check fails, as Nemotron and Flash Next do (#745).

Known issues:

• GLM-5.3 (EXLR8) decodes slowly next to the other models, since a one-row round on two M5 Ultras takes about 45 ms. No CUDA engine serves it, and only the home profile on M5 Ultras has run. Replies past 2,048 tokens, where the sparse indexer starts picking keys, have not been compared with a whole-model reference.
• Living Weights: facts about the same subject can overwrite each other, a later fact about the user can disturb an earlier one's answers on a GB10, yes-or-no questions about a changed value are unreliable, facts pile up badly past a few dozen, and thinking on brings back fewer answers. 1.0.8's notes have the measurements; 1.0.9 doesn't change how facts are learned.
• Living Weights: learned facts cost decode speed on a Mac. #775 takes back about 8% on an M3 Ultra with two facts; 1.0.8's M5 Ultra figures (213 tok/s after ten facts, 248 before any) have not been measured again.
• Flash Next in speed-up mode: a drafted request right after a "draft": false one decodes slowly, and the next request is back to speed. Same tokens.
• Qwen3.8-27B on an M1 to M4 Mac: the drafter's landing counts last for the server's life, so after four prose replies greedy code drafts shallower, about 4% slower on an M3 Ultra. Same tokens.
• Qwen3.5-2B with a JSON schema: on an integer field it wants to write as a decimal, the 2B can keep adding digits until max_tokens, since the grammar allows no decimal point there. Nemotron writes the same schema correctly.

Not run for this release:

• No sm_120 GPU (RTX 50 series, RTX PRO 6000 Blackwell) ran 1.0.9. Of the 34 CUDA kernel files, 29 are byte for byte 1.0.8's, which passed on an RTX PRO 6000. The five that changed, Nemotron's attention, Mamba and norm glue and the new shared norm and token-row kernels (#736, #770, #771), are built for sm_120 but ran only on a GB10 (sm_121).
• Flash Next and GLM-5.3-Flash were not served on M5 Ultras, alone or as a pair, from the release build. Builds from earlier on release day, with 1.0.9's Metal kernels for both models, ran there: Flash Next 6-bit on an M5 Ultra loaded through #744's check and gave 1.0.8's tokens in 9 of 9 cases, drafted equal to "draft": false, and in 12 of 12 chat turns, resumed equal to a cache-off server's, and GLM-5.3-Flash on two M5 Ultras loaded through #744's and #745's checks on both Macs with every reply equal to 1.0.8's. The changes after those runs that reach these models are host-side: the short-prompt states (measured on M5 Ultras for Flash Next, above), the tokenizer memo (#767, checked with Flash Next's and GLM-5.3's own tokenizers), #783 (release builds compute the same values) and #761 (the tool-call parser). Those receipts and the contributors' own runs stand in.
• GLM's shared rounds under #769's draft pricing were not measured. The pricing lives in the lane core every family uses; it was measured with Nemotron on a GB10 and an M3 Ultra and with the 27B on an M3 Ultra, not with GLM-5.3-Flash or GLM-5.3 on two Macs.
• --cortheon was not part of this release's served gate, which ran every model without the flag. Its unit tests, the server's golden tests with the flag off, and live runs on the 27B with a coding harness stand in.

Homebrew: brew upgrade tensorfold.