TensorFold 1.2.4

TensorFold 1.2.4

1.2.4 brings Gemma 4 E4B on CUDA, Cloudflare's Clef-Flash on HIP, LFM3-1.7B's text decoder on Metal, and the HIP runtime on Windows. DeepSeek-V4.1 recalls a saved long conversation in seconds instead of minutes. Flash Next gets faster prompts and decode, sooner first tokens, kept prompt states with drafts at --tp 2, faster SSD experts, and much faster overlapping clients on M5 Macs. Gemma 4 and Qwen 3.5 load faster cold on CUDA. HIP decode and rerank are faster. GLM on two or more GB10s prefetches its next decode weights into L2. DeepSeek-V4 and V4.1 prompts follow the release's recorded encoder outputs. An engine's refusals now reach the client as clear 400s.

• New:
  • Gemma 4 E4B (NVIDIA Nemotron 3.5 Bolt E4B) on CUDA;
  • Clef-Flash's joint schema decisions on /v1/systemone on HIP;
  • DeepSeek-V4.x chat rendered with its own encoder;
  • LFM3-1.7B text on Metal, from the official MLX checkpoints;
  • the HIP runtime on Windows;
  • Clef-Flash's BF16 Qwen3.5 backbone on CUDA, as a decision backbone.
• Speed:
  • A 319K-token DeepSeek-V4.1 conversation restores from NVMe in 1.2 to 1.4 s instead of 155 to 178 s.
  • A Flash Next chat resumed at --tp 2 with drafts replies in 1.5 s instead of 21.2 s.
  • Flash Next prompts at --parallel 1 on one CUDA GPU are 15-23% faster, and 8 streams decode 3-6% faster.
  • A new short Flash Next prompt's first token on CUDA comes 19% sooner.
  • Two overlapping Flash Next clients on an M5 Ultra finish in 47 s instead of 127 s.
  • Gemma 4 and Qwen 3.5 on CUDA load 28-31% faster cold.
  • A lone "draft": false Flash Next reply on Metal is 17-18% faster.
  • Rerank on HIP is about 25% faster, and one-stream HIP decode up to 2.7% faster.
  • Qwen 3.5 first tokens on a GB10 come 1 to 3% sooner.
  • GLM on CUDA prefetches a decode window's next weights into L2 on two or more GB10s, with the same replies.
• Serving:
  • An engine's refusal reaches the client as a 400 before any header.
  • Flash Next on Metal enforces structured output at --parallel 1 too.
  • Unsupported penalties are refused instead of dropped.
  • A refused prompt-cache restore tries a shorter kept state.
  • DeepSeek-V4.1 tool calls stream sooner.
  • Tool calls made just before the thinking close are returned.
  • DeepSeek-V4 and V4.1 prompts match the release's recorded encoder outputs, and malformed tool requests are refused with a 400.
  • An opt-in queue-wait histogram for native metrics.
  • /v1/models lists what each model supports.
  • An opt-in drift gate for Living Weights lessons.
  • Malformed TF_, TENSORFOLD_ and TENSORFOLD_PREFILL_* settings stop the server at startup, by name.
  • A kept prompt state on disk is reused only by the same build and checkpoint files.

Where a speedup was checked for equal output, its line says so. Some changes alter scheduling, memory, loading, prompts or startup by default. They're listed under What can change for you.

Headline numbers

Each number names its machine. Where replies were checked against the base, the line says how.

Release reads, each machine with 1.2.3 and 1.2.4 in alternating fresh servers (base, new, new, base, after an unscored warm-up):

• Flash Next MLX 4-bit on a GB10, against 1.2.3: prompts and one stream level, with the same tokens; two to eight streams up to 5.1% faster (eight streams 1.7% to 5.1%). The 64K long-context answer is right on both.

• Qwen3.8-27B on an M5 Ultra, against 1.2.3: level, within 1.2% in every sweep and on every prompt and decode cell from 2K to 64K, with the same tokens.

• Qwen3.8-27B on a GB10, against 1.2.3: level, within 1.0% in every sweep and on every prompt and decode cell from 2K to 64K, with the same tokens.

• Qwen3.8-27B on an M3 Ultra, against 1.2.3: level on every prompt and decode cell from 2K to 64K, with the same tokens. At 64K one of the two 1.2.4 runs was up to 2.5% lower and the other matched both 1.2.3 runs. Sweeps were within 1.3%, against one 1.2.3 run.

• Nemotron 3.5 Lightning 30B-A3B on a GB10, against 1.2.3: level, with the same tokens; no cell slower than 1.5%, and decode after a 32K prompt 2.0% to 2.6% faster.

• DeepSeek-V4.1 on CUDA, on two GB10s at TP=2, with TF_DSV41_PREFILL=replay:

  • recalling a 319K-token conversation from the NVMe session tier takes 1.2 to 1.4 s instead of 155 to 178 s;
  • replies are byte-identical to a cold prefill (#2182).

• Flash Next on CUDA at --tp 2 with drafts, on two GB10s: a chat with about 28,500 tokens of history replies in 1.5 s resumed, against 21.2 s fresh, with the same replies (#2214).

• Flash Next on CUDA, on a GB10:

  • at --parallel 1, prompts run 15-23% faster, for about 3.2 GiB more memory (#2246);
  • 8 streams decode 3-6% faster with the same tokens (#1942);
  • a new short prompt's first token comes 19% sooner, with replies unchanged (#1866).

• Gemma 4 and Qwen 3.5 on CUDA, on a GB10: loading cold is 28-31% faster, with the same weights and replies, for about 0.6 GB more host memory (#2282).

• Flash Next on Metal, on an M5 Ultra with the MLX 4-bit checkpoint, at --parallel 2 or more: two overlapping clients finish in 47 s instead of 127 s, and a 64K prompt beside a 1,024-token reply in 21 s instead of 36-38 s, with the same tokens (#2394).

• Flash Next on Metal, on an M3 Ultra: a lone "draft": false reply at --parallel above one is 17-18% faster, with the same tokens (#2272).

• Qwen3-Reranker-8B on HIP, on a Radeon AI PRO R9700: 16 documents of 323 tokens, 87 of them shared, take:

  • 1.98 s instead of 2.63 s on the default dot2 tile;
  • 0.73 s instead of 0.99 s with the gfx12 WMMA tile (--policy matrix=on).
  • The responses are the same byte for byte on that card (#2262). On RDNA 3.5 (gfx1150 to gfx1153), a prompt row now gets the same bytes wherever its chunk starts (#2357).

• HIP decode, on a gfx1151: one stream decodes 2.7% faster on Qwen3.6-35B-A3B and 1.1% on the 27B, with the same output bytes (#2247).

• Qwen 3.5 on CUDA, on a GB10:

  • first tokens come 1 to 3% sooner, with the last row's logits bit-equal (#2229);
  • 8k and 32k prompts start 1.6 to 2.1% sooner with the same replies (#2113).

New models and backends

• Gemma 4 E4B on CUDA (NVIDIA Nemotron 3.5 Bolt E4B): served from mlx_lm.convert 4-bit checkpoints.
  • Per-layer inputs run in their own launch, and layers 24 to 41 read the caches of layers 22 and 23.
  • Replies are equal across streams and resumes (#2230, part of #1222).
• Cloudflare's Clef-Flash on HIP (Radeon AI PRO R9700): its BF16 Qwen3.5 backbone and joint schema head answer /v1/systemone.
  • The head runs on the GPU over each prompt piece in the shared rounds.
  • Validation errors are 422s with their location.
  • Chat and the other routes refuse the checkpoint, naming /v1/systemone.
  • capabilities --json lists BF16 for HIP Qwen3.5 under a new decisions key; the chat formats are unchanged.
  • Checkpoints whose tokenizer_config.json publishes a pretokenize_regex split words with it (#2048, #1360, #1952).
• DeepSeek-V4.x chat on the native server renders with its own encoder and schema prompt.
  • A served image part becomes the image token.
  • The startup line names the chat renderer.
  • Prompt token ids now equal the release's encoder goldens (#1698).
• LFM3-1.7B on Metal: the text decoder serves the official MLX 4-, 6- and 8-bit checkpoints, with per-stream convolution and KV state, verification windows, snapshots and the prompt cache. LFM3-VL checkpoints are refused by name until the vision tower lands (#1514, part of #1491).
• HIP on Windows: the HIP runtime builds and loads on Windows.
  • hipcc.exe is preferred, and hipcc.bat is found through -Dhipcc, PATH or HIP_PATH/ROCM_PATH and run through cmd /c call.
  • The runtime and the optional libraries load by their Windows names (amdhip64.dll and the versioned ones; TENSORFOLD_HIP overrides).
  • The admission budget comes from the driver where /proc/meminfo and sysfs don't exist.
  • check-native-hip-host compiles for the host OS, and the cooperative grid test runs plain where the driver reports no cooperative launch (ROCm/hip#3640) (#2178).
• Clef-Flash's backbone on CUDA: Qwen3.5 on CUDA loads a plain BF16 checkpoint as a decision backbone, not for chat.
  • Chat, and the flags the joint schema head isn't qualified with, are refused by name before the weights load.
  • Its norms scale by 1 + w in FP32.
  • A quantized Qwen3.5 checkpoint that ships joint_head_config.json is refused (QwenHeadNeedsBf16), as on HIP (#2383, part of #2324).

Speed

• DeepSeek-V4.1 on CUDA with TF_DSV41_PREFILL=replay: a conversation saved to the NVMe session tier now restores, instead of being refused as BadBlob and prefilled again. Numbers are above (#2182).
• Flash Next:
  • On CUDA at --tp 2 with drafts, prompt states are kept, and the server no longer needs --prompt-cache-gib 0 to start. Numbers are above (#2214, fixes #1829).
  • On CUDA at --parallel 1 on one GPU, a prompt pass takes up to 8,192 rows, so prompts run 15-23% faster on a GB10 for about 3.2 GiB more memory. TF_FLASHNEXT_CHUNK_SOLO still decides (#2246).
  • On CUDA, a shared round's DeltaNet steps run in one launch each across its lanes instead of one launch a lane. So 8 streams decode 3-6% faster on a GB10 with the same tokens; one stream keeps its own launches (#1942).
  • On CUDA, the shared expert's down projection writes straight into its output slot, so NVFP4 prompts run about 0.7% faster on a GB10 with the same tokens (#1792).
  • With --ssd-experts, a request no longer waits for a forecast read it doesn't need, and a forecast that never started keeps the expert it displaced. That's about 8% faster at 35 GiB of experts and 4% at 17 GiB (#2236).
  • On Metal, a lone "draft": false reply at --parallel above one runs as it does at --parallel 1, 17-18% faster on an M3 Ultra with the same tokens (#2272).
  • On CUDA, a prompt-cache mark inside a prompt chunk is kept on the way through the pass instead of cutting the pass there. A new short prompt's first token comes 19% sooner on a GB10, with replies unchanged. TF_FLASHNEXT_KEEP_INPASS=0 cuts the pass at the mark as before (#1866).
  • On Metal with tensor units (M5), at an explicit --parallel 2 or more, the shared host takes the same defaults as other Macs: replies take 0.8 of a filling prompt's time, and the prompt's steps grow to 512 rows. Numbers are above (#2394, refs #1893).
• HIP: the decode round's residual-add and norm tail issues each row's loads before its branches, and a MoE tail loads its slots' rows together. Output bytes are unchanged. Numbers are above, and tf_tail is now under the kernel contract (#2247).
• Qwen 3.5 and 3.6 on CUDA:
  • A prompt chunk's last layer runs q, attention, o and the MLP over the kept row's 128-row tile only. The cache is unchanged and the last row's logits stay bit-equal. A model served with an MTP head, int8 or images keeps the full last layer (#2229, closes #2165).
  • On an integrated GPU, prompt states stay in device memory instead of pinning a host copy at each save, still within the host cache budget (#2113, #2043).
• Loading on CUDA: Gemma 4 and Qwen 3.5 load with direct reads by default, 28-31% faster cold on a GB10 with the same weights and replies, for about 0.6 GB more host memory. TF_LOAD_DIRECT=0 keeps the old read-ahead (#2282).
• Qwen 3.5 and 3.6 on CUDA, opt-in pages: with TF_QWEN_KV_PAGES=256, dense and MoE alike, --parallel is a cap rather than a reservation: a request can wait for pages instead of every stream holding a full-context window. Paged prompt attention stages a chunk's page ids in shared memory, decode looks a page up once, and a paged prompt-cache save or restore copies one 2D block a page. Replies are the same tokens on pages and windows. Unset keeps a window each, as before (#2011, part of #1943).
• GLM-5.3-Flash on CUDA at TP=2: a prompt step takes two chunks while no other stream decodes, each one's exchanges under the other's, so prompts of 8K to 128K finish about 2% sooner, with replies unchanged. TF_GLM_PROMPT_PAIR=0 turns it off (#1423).
• GLM-5.3-Flash on CUDA, on a GB10 at two ranks or more: a decode window's next weights are read into L2 on a side stream while its gathers and chains run, inside a 12 MiB budget. Replies are identical, and every other GPU is unchanged. TF_GLM_L2PF=0 turns it off, and TF_GLM_L2PF=1 turns it on for any GPU (#2162, #1580).
• Qwen3.8-27B on Metal: a reply that would end well before a long prompt now goes first, and the prompt lends it at most a quarter of its own time (#2224).
• Prompt scheduling: a resumed turn that hasn't started its prompt pass is ranked by its uncached tail, not its whole prompt. A long cached history no longer waits behind a cold prefill (#2276, #2256).
• Qwen3-Reranker-8B on HIP: a rerank request's shared start (the system prompt, the instruction and the query) is computed once a pass instead of once a document. The responses are the same byte for byte on the R9700; RDNA 3.5's chunk starts are fixed in #2357. The numbers are above (#2262, part of #2245).

Serving and API

• DeepSeek-V4 and V4.1 prompts follow the release's recorded encoder outputs, except where those outputs drop a tool's namespace, which we keep (#2193, #2141).
  • Assistant thinking markers, blank-line text joins and tool schemas change. minimal keeps thinking on at budget 50, and xhigh uses 75. Developer instructions render as system.
  • Listings and tool-call history keep qualified function names, and schemas keep their namespace descriptions. Strict tools keep their grammar handling.
  • Malformed requests now get a 400: user null content, malformed tool-call lists, calls without a string id, name or arguments or type function, names outside 1-64 ASCII letters, digits, underscores and hyphens, tool replies without tool_call_id, and tool envelopes that aren't functions. ultra keeps its refusal.
• /v1/models: each model's entry lists the capabilities it was loaded with, read through the same accessor as request admission, so the listing and the refusals agree (#2157, part of #2139).
• Native metrics: TENSORFOLD_PHASE_METRICS=1 adds a bounded request queue-wait histogram and its vLLM alias. It's off by default (#2158, part of #2139).
• Engine refusals: an engine can refuse a request it can't serve as asked, in words the client can use.
  • The server answers 400 invalid_request_error before any header, on chat, completions, Responses and Messages, streamed or not.
  • An engine without the entry behaves exactly as before (#2198, #2082).
  • The refusal names the body field the client sent: response_format, guided_json and the other four, or text.format and output_config.format on the Responses and Messages routes (#2110, #2084).
• Tool calls before the thinking close: completed tool calls just before the thinking-close marker are returned as tool calls when no answer follows, whole and streamed. Calls followed by more reasoning or an answer stay in reasoning (#2313, fixes #2310).
• Flash Next on Metal at --parallel 1: response_format (json_schema, json_object) and tool_choice "required" or a named function are enforced, through the same grammar path --parallel 2 uses. Unconstrained requests take exactly the single-stream path as before (#1969, #1894).
• Penalties: native engines that can't apply presence or frequency penalties refuse nonzero values with a 400. Zero, null and omitted penalties keep working (#2166).
• Prompt cache: a restore the backend refuses tries the next shorter kept state before prefilling from the start (#2144). On CUDA, when a kept state is larger than the whole cache budget, the log line now says that TF_KV_SHARE=n would keep its prompt rows as the pool's pages, for families on KV pages; nothing else changes (#2333).
• DeepSeek-V4.1 tool calls: DSML calls stream from the invoke name, so a call whose first parameter is the long one streams while it is written (#2201, #1975).
  • GLM and XML calls have no such tag and still stream from the first parameter's end.
• GLM-5.3-Flash on CUDA:
  • TF_KV_SHARE=n with TF_KV_PAGES=on keeps GLM's prompts in the shared prompt store that Spark 2.5, Gemma 4 and Kolibri-1 use, with the same tokens, resume marks and memory. GLM's own store and TF_GLM_KV_SHARE go (#2211).
  • New and off by default: TF_GLM_PROMPT_TAIL=1 keeps a raw /v1/completions prompt's state at its last chunk start in the unpaged prompt cache, so a repeat or a longer prompt resumes there, with tokens unchanged. Any value other than unset, 0 or 1 is refused by name before the weights load (#2226).
• Metal request stats: a request's finished stats list its prompt pass widths (tensorfold.prefill_widths, one entry per completed target pass) on every Metal text family. This is telemetry only (#2293).
• Qwen3.8-27B:
  • A refusal that no --parallel or --context can fix names the wired-limit raise, and only suggests one that leaves the system room (#2123).
  • A prompt frame below 128 rows (TF_PROMPT_ROWS=16) is refused before loading, with the setting, the value and the fix named, instead of a bare "TargetBinding" (#2298).
• Startup settings: a malformed TF_ or TENSORFOLD_ setting stops the native server at startup with its name, the text it got and the forms it accepts, before any checkpoint or driver work. Empty or whitespace-only values still mean unset, and the documented lenient forms stay. --profile and the startup line report six more setting groups (#2405, part of #2370).
  • serve also refuses, by name, a TENSORFOLD_PREFILL_SHARE outside 0.01 to 1, a TENSORFOLD_PREFILL_POLICY other than fcfs, and TENSORFOLD_PREFILL_TOKENS or TENSORFOLD_PREFILL_BUDGET text that isn't a whole number. The lane host used to read each as its default without a word (#2369).
• Kept prompt states on disk:
  • a state is reused only by the same build and the same checkpoint files, so a rewritten checkpoint or a new build starts cold instead of reusing an old state (#2163, #2131);
  • what the store drops as it opens is now logged with its cause: a kept state whose file fails its check or misses a block, one line a cause, and the count of blocks no kept state names any more. What is restored is unchanged (#2397, part of #2349).
• GLM-5.3-Flash on CUDA at TP=2 and up: each rank names its startup phases (NCCL init, peer warm-up, weights, calibration, ready), prints NCCL's init device, version and log path, and warns once a minute while NCCL init or the warm-up is unfinished. Transfers are unchanged (#2354, part of #2347).
• API key file: a refused --api-key-file logs the path it was given, the mode the file system reported and which test failed, at startup and when a reload is refused (#2365).
• CLI: tensorfold models --json prints the cached checkpoints as one document (schema 1, and each checkpoint's repo, family and model_type). The plain listing is unchanged (#2364).
• Living Weights: --slide-drift-gate (opt-in, with --slide) measures each lesson's drift on unrelated replies and checks its recall before keeping it. Without the flag nothing changes (#2028, part of #1389).
• Cortheon:
  • a Windows path whose name holds a change word no longer turns a question into a change request;
  • fallback queries stop at a tabbed sentence end and drop a tabbed list mark;
  • search results with upper-case tags, HREF= or single-quoted links are kept;
  • a GitHub file under Blob is read as raw text;
  • an absolute link of another scheme stays as written (#2216).

Fixes

• CUDA on integrated GPUs such as the GB10: synchronous downloads stage through a per-thread pinned buffer. This guards against a driver fault where a copy into pageable memory could miss pages. Discrete cards keep the direct copy (#2103). Every other synchronous device-to-host copy now stages the same way (54 copies in families, tools and tests), with the bytes unchanged (#2289, for #2102).
• GLM-5.3-Flash on CUDA: an EXL3 checkpoint whose quantization_config sits inside text_config loads as EXL3. Admission already read it there, but the config parser read only the top level, so such a pack was admitted and then asked for NVFP4 dense weights at load (#2327).
• Qwen3.8-27B on Metal: a NaN or infinity in a value a row must not see no longer turns that row's attention output into NaN. This covers the prompt kernels, before M5 and on M5, and the M5 decode prefix. Finite inputs keep their bytes (#2106).
• DeepSeek-V4.1's NVMe session tier restores saved conversations again (#2182, above).
• HIP on RDNA 3.5 (gfx1150 to gfx1153): a prompt row gets the same bytes wherever its chunk starts. The 64-row prompt tile there takes each key past a row's causal edge as +0, since RDNA 3.5's dot2 rounds a -0 lane apart from +0. Its prompt bits change, and gfx1151's prefill is up to 0.7% slower at 2k and 8k. Other chips keep their prompt tile and bits. On every chip, a non-finite value at a key past a row's edge now adds nothing to that row (#2357, closes #2317).
• Flash Next on CUDA with TF_FLASHNEXT_KV_PAGES: a shared round's grouped draft head reads and writes each stream's own pages, so paged streams' replies equal their runs alone again, and paged decode runs as fast as unpaged. Default settings are unchanged (#2244).
• Qwen3.8-27B on Metal: a keep that fails part way (an allocation, a refused keep or path) returns its error instead of aborting the process, and after a failed allocation the round can keep again (#2376).
• Vision: PDF pages reach the vision tower in their true colours. The render flags set FPDF_REVERSE_BYTE_ORDER where printing was meant, which swapped red and blue on every page (#2387, for #2386).
• Clusters over RoCE: a Linux kernel built without transparent hugepages no longer refuses to start. prepareRegion keeps MADV_DONTFORK strict, asks for MADV_NOHUGEPAGE only where THP can apply it, and says at startup when it skips it (#2380, for #2378).
• Qwen4: the rotary table comes from core's exact helper, matching transformers' FP32 values. Every moved value is a 1-ulp move (#2223, part of #2000).
• Builds:
  • a plain zig build on Linux compiles again (#2362);
  • zig build -Dgpu=<gpu> builds the one-GPU server again (#2290);
  • zig build server-cpu compiles again (#2363);
  • zig build compile-dsv41-server and dsv41-serve build again, with the lanes module the DeepSeek server's /v1/systemone route reads (#2412).

Docs

• The optimization ledger: docs/optimization-ledger.md lists which speed and memory optimizations apply to which native family, and where each stands, with links to the PRs that measured them (#2232). The pull request pre-flight asks a PR that changes a family's optimization to update its cell (#2243).
• docs/api.md matches what the server reads and writes (#2299). Its Reasoning section says what a no-thinking GLM-5.3 prompt is on each engine (#2297, #1911):
  • native appends </think> to the template's render;
  • Python 0.6.6 strips the Reasoning Effort head.
• Contributing: authors post their own main-against-branch numbers first; our run follows once the review has no open asks (#2179).
• Flash Next:
  • the CUDA recipe gives TF_FLASHNEXT_CONFIDENCE's real default, -0.4, and what its sign means (#2296);
  • the slow drafted request after a draft: false one no longer reproduces, and the expert recipe lists peak memory at each --ssd-experts budget (#2237).
• Serving flags: ten native serve flags are now documented: --top-p, --top-k and --min-p as sampling defaults, --compact-at, --compact-keep and --compact-memory, Dory's --loops, the two silence-onset flags and Drama 3's --max-frames (#2345). docs/api.md lists DELETE /v1/slide/facts, and docs/control.md no longer says the native server refuses --spill-gib and --vision (#2341).
• Model recipes and flags, checked against the code:
  • the recipes match the native server: Gemma 4 serves M1 to M4 through the row path and enforces "required" tool calls, Nemotron refuses --tp above 1, qwen3_5 serves EXL3 packs on CUDA, the CUDA CLI applies --alias, Flash Next on Metal's draft levels are 3, 6 and 8, and more (#2361);
  • the serve flags match the CLI: --metrics-open opens /v1/metrics too, --learn-conversations on CUDA serves the Qwen3.5 hybrids and Flash Next as well as Nemotron, and the Metal Qwen3.5-2B keeps prompt states (#2367);
  • four subcommand flags: slide audit --threshold's range, benchmark --context and --kv-cache, info on Flash Next, and run --device and --carveout on CUDA builds (#2348).
• Hermes Agent: docs/hermes-agent.md shows how to use TensorFold from Hermes Agent through the community hermes-tensorfold plugin (a model provider and a dashboard tab), checked on a GB10 with Qwen3.8-Flash-Next (#2330, part of #2109).
• GLM on CUDA: the recipe notes that TF_GLM_CLEAR_THINKING is read by the native server too, not only the Python engine (#2350). It also notes that TF_GLM_CHUNK_SOLO=4096 made prompts 12-14% slower with NVIDIA's NVFP4 checkpoint on two GB10s at --parallel 1, so leave it unset there (#2288).

Internal

No served output changes from any of these.

• Tests:
  • the heaviest host test gates build in ReleaseSafe: the Qwen4 host gate's 113 checks take 18 seconds instead of 4 minutes on one 20-core host (#2249, part of #2242);
  • rollback conformance on the fake lane target, and 12 captured CUDA graph shapes replayed out of order (#2161, closes #2138);
  • the Metal kernel contract checks masked non-finite values across every family's prompt attention, with the remaining families registered as known (#2308);
  • the 27B's M5 decode control is fixed, so contract runs on M5 finish (#2303, fixes #2301);
  • the MOSS-Image teacher gate compares every step with the teacher (#2291);
  • other test fixes and re-baselined source pins (#2267, #2268, #2175);
  • the CUDA kernel contract releases each case's buffers when the case ends, so the full run fits in a GB10's memory (#1845);
  • test-metal-residency-owners-host compiles and passes again, with 68 tests (#2331, part of #2031);
  • tf-qwen27-reuse's yielded-pass check follows the host's lane loop on M5 chips (#2343);
  • tf-moss-image-blocks gates MOSS-Image's transformer block by block against the publisher's float32 run, on Metal and CUDA (#2294, part of #1529);
  • the resumed == fresh checks run a real fresh run and fail by name if it resumed: Qwen 3.5's two Zig checks (#2374, for #2353) and serve_check.py, through an optional cold server (#2381, for #2353);
  • exl3_load's allocation-failure test passes at any checkout path length; it failed at 70 of 102 lengths before (#2377);
  • the pull client's security tests build and run on Windows, on a portable fake hub, and CI runs them there (#2284);
  • test-quoted-markers-host checks with real tokenizers that quoted added-token spellings stay plain text through chat requests (#2400, part of #2399);
  • test-glm-snapshots-host builds again (#2403); four host steps no CI job runs build and pass again (#2409, part of #2031); the serving source pins take main's reviewed bytes (#2418, part of #2031);
  • the TP host reference adds bf16 and f16 sums in fp32 rank order and rounds once, as documented; sums at one or two ranks don't change (#2366, part of #1971).
• CI:
  • faster and cached CI (#2252, #2274, #2277);
  • the portable-names check runs on every pull request again (#2311);
  • main's own pushes reuse main's newest build cache, as a pull request does, with each saved cache pruned to about one build and a nightly run from nothing (#2402).
• Tools:
  • an XPU limits backend for the kernel contract (#2155);
  • a Living Weights keep-rate comparison tool (#2027);
  • image patch preparation rewritten from documented behaviour, with byte-equal output in 55,503 host cases (#2271);
  • tools/pr_check.py checks local changes against their merge base before a push, and the PR template asks for paired main-against-branch cells (#2355);
  • tf-xpu-contract-run runs the kernel contract's gates 1 to 4 on an Intel GPU through Level Zero, with planted controls and gate 5 checked live; its first run covers the Qwen3.8-27B glue on an Arc Pro B70 (#2269, part of #2080);
  • tools/zig/bench_sessions.py replays long agent sessions in turn patterns to measure the prompt cache, one JSON line per request with cached_tokens (#2359, for #2346).
• Code:
  • Nemotron's 16 Metal decode attention partials share one source, with the same kernels and bits (#2263);
  • 310 lean findings fixed, comments reworded to one line (#2292, #2295, #2309, #2315, #2318, #2320, #2323, #2326, #2329, #2332);
  • zig/build/cuda.zig goes from 600 to 467 lines, with the same steps in the same order (#2420);
  • the kernel contract's nvcc 13.0 resource records match what nvcc 13.0.88 builds for 57 CUDA kernels (226 per-SM slots), so the limits gate passes; no kernel, flag or launch changes (#2390);
  • the engine API carries an optional image capability beside speech; an engine without the entry is unchanged (#2248, part of #1573).
• Pipeline stages across machines: the stage protocol (boot description, round plan, rows layout and keep-alive), with golden bytes for every message, so a CUDA and a Metal stage can be built against the same bytes. Nothing serves through it yet (#2060, part of #2005). Clusters: MCDMA bring-up on real hosts, step 2a of #1594 (#2088).
• Developers: -Dnative-fp-decode=true is off by default and needs nvcc 13.2 or later. It builds CUDA's prompt16 FP4 tile and FP8 expert prompt tile with the hardware FP4 and FP8 decode on sm_120 and later. Outputs are bit-equal to the default build, and on a GB10 the FP4 prompt projections took 0.95 to 0.96 of the time (#2238, refs #1448).
• Changelog wording fixes with no code: #2278, #2286, #2312, #2371.

What can change for you

• Kept prompt states on disk: a state is reused only by the same build and checkpoint files, so after upgrading to 1.2.4 every kept disk state starts cold: the first long prompt prefills again, and is kept as before from then on. A rewritten checkpoint also starts cold. There's no setting to reuse old states (#2163).
• Startup settings: a malformed TF_ or TENSORFOLD_ setting now stops the native server at startup, naming the setting, the text it got and the forms it accepts; empty values still mean unset. Kept disk states of Qwen 3.5 and Qwen3.8-27B on Metal, and of Nemotron and Qwen 3.5 on CUDA, are prefilled once after upgrading (#2405). A TENSORFOLD_PREFILL_SHARE outside 0.01 to 1, a TENSORFOLD_PREFILL_POLICY other than fcfs, or TENSORFOLD_PREFILL_TOKENS or TENSORFOLD_PREFILL_BUDGET that isn't a whole number is refused at start; before, each was read as its default (a share of 0 meant whole prompt steps). Unset it, or set a valid value (#2369).
• Loading on CUDA: Gemma 4 and Qwen 3.5 read their weights directly by default, using about 0.6 GB more host memory while loading. TF_LOAD_DIRECT=0 keeps the old read-ahead (#2282).
• Flash Next on CUDA, prompt-cache marks: a mark inside a prompt chunk no longer cuts the pass there. Replies are unchanged. TF_FLASHNEXT_KEEP_INPASS=0 cuts at the mark as before (#1866).
• Flash Next on Metal with tensor units (M5), at an explicit --parallel 2 or more: replies take 0.8 of a filling prompt's time, and prompt steps grow to 512 rows, as on other Macs. The tokens are the same (#2394).
• GLM-5.3-Flash on CUDA at TP=2: a prompt step pairs two chunks when no other stream decodes. Replies are unchanged. TF_GLM_PROMPT_PAIR=0 turns it off (#1423).
• HIP on RDNA 3.5 (gfx1150 to gfx1153): prompt bits change, so replies can change where a token was a near-tie, and gfx1151's prefill is up to 0.7% slower at 2k and 8k. There's no setting to keep the old tile (#2357).
• Quantized Qwen3.5 checkpoints with joint_head_config.json on CUDA are now refused (QwenHeadNeedsBf16); use the BF16 checkpoint (#2383).
• Vision on PDFs: pages now reach the vision tower with red and blue the right way round, so replies about PDF pages can change (#2387).
• DeepSeek-V4 and V4.1 prompts: prompt ids change where the recorded encoder outputs differ from the old rendering: thinking markers, blank-line joins, tool schemas, the minimal and xhigh budgets, and developer instructions as system. So replies can change. Requests that were malformed and used to pass (see Serving) now get a 400 (#2193).
• GLM-5.3-Flash on CUDA, on a GB10 at two ranks or more: the L2 prefetch is on unless TF_GLM_L2PF=0. Replies are identical; it reads up to 12 MiB ahead on a side stream (#2162).
• DeepSeek-V4.x chat prompts: the native encoder renders chat even when the checkpoint ships a Jinja template. A pack that carries an older chat_template.jinja gets different prompt ids, effort defaults and tool rules, and so can get different replies.
  • To keep a template file, pass --chat-template.
  • TF_DSV41_SCHEMA_PROMPT=0 turns off the ## Response Format block that a request's response_format adds (#1698).
• Tool calls just before the thinking close: when no answer follows, they now come back as tool_calls instead of as markup in reasoning_content, whole and streamed (#2313).
• GLM-5.3-Flash on CUDA with TF_GLM_KV_SHARE: a server started with it set to anything but 0 or empty now stops before the weights load, naming TF_KV_SHARE. Use TF_KV_SHARE=n with TF_KV_PAGES=on instead (#2211).
• Presence and frequency penalties: on a native engine that can't apply them, a nonzero value is now a 400 instead of being ignored. Send 0 or omit them to keep the old replies (#2166).
• Qwen3.8-27B on Metal, a reply beside a long prompt: by default, a reply that would end soon now goes first. To keep the old order, use --decode-share, TENSORFOLD_PREFILL_SHARE, or any TENSORFOLD_PREFILL_ORDER other than srpt (#2224).
• Resumed turns are ranked by their uncached tail, so the order of waiting prompts can change; a rejected restore ranks the whole prompt again (#2276).
• Flash Next on CUDA at --parallel 1 on one GPU takes about 3.2 GiB more memory for its larger prompt passes. TF_FLASHNEXT_CHUNK_SOLO still sets the pass size (#2246).
• On integrated GPUs such as the GB10:
  • large synchronous downloads take longer: 212 µs instead of 85 µs at 4 MiB on an idle GB10 (#2103). Every other synchronous readback now stages the same way (#2289);
  • Qwen 3.5's prompt states sit in device memory, and the startup line says so (#2113).
• Nemotron on Metal: saved prompt states take their identity from the kernel source text, so after upgrading they aren't reused. The first long prompt prefills again, and is kept as before from then on (#2263).
• Qwen4: replies can change where a token was a near-tie, from the 1-ulp rotary-table moves. There's no setting to keep the old table (#2223).
• Qwen3.8-27B with TF_PROMPT_ROWS=16: it's refused before loading, by name, instead of failing later as "TargetBinding" (#2298).
• DeepSeek-V4.1 DSML calls: a call malformed after its name may already have streamed when the whole reply keeps it as text (#2201).
• Qwen3.8-27B on Metal: replies change only where a NaN or infinity already sits in a masked key (#2106).
• /v1/models: each entry gains the model's capability fields. Clients that read the listing strictly will see new keys (#2157).

Thanks

• @AbrahamCete (#2247, #2374)
• @AdrianBinDC (the #2310 report, fixed in #2313)
• @Arminova (#1866, #1942, #2244)
• @atakann (#2290, #2330)
• @Bizuayeu (#1423, #1580, landed in #2162; #2211, #2226, #2327, #2333, #2403)
• @BlivionIaG (#1514)
• @BobClawblaw (#2294, and finding and diagnosing #2301, fixed in #2303)
• @cshintov (#2106)
• @Defilan (#2412)
• @devopsfortunato (#2110, #2123, #2198, #2201, #2248, #2249, #2276, #2297, #2380)
• @ElderOrb (#2178, #2284)
• @gilby (the first Clef-Flash encoder port, #1361, in #2048)
• @gordonhatusupy (the #2386 report, fixed in #2387)
• @goshi (#2182)
• @jayleaton (#1698)
• @jishnuvenugopal (#2331, #2377)
• @jschmied (#2011, #2060, #2113, #2229, #2230, #2232, #2238, #2243, #2263, #2282, #2376, and the report behind #2362)
• @jungjipdo (#2229)
• @laidick (the #2256 report, in #2276)
• @laudney (#1792, #2048, #2262, #2357, #2383)
• @Maxrio9 (#1845)
• @ppardus (#2216)
• @shuuul (#2103, and the #2082 report)
• @ThinkOffApp (#2088, and the run that found #2397's dropped states)
• @tolewis (#2155, #2269)
• @Whq02 (#2363, #2364, #2365)