Skip to content

TensorFold 1.2.5

Choose a tag to compare

View all tags
@ashhart ashhart released this 08 Nov 17:02
· 974 commits to main since this release
v1.2.5
8be9c40

TensorFold 1.2.5

1.2.5 brings LFM2.5-24B-A2B on CUDA, Nemotron Parse and LightOnOCR-3 on HIP, and Clef's joint schema head on CUDA. It also adds TensorFold Web, a local web app. Qwen3.8-27B's MLX 8-bit checkpoints get faster prompts on Macs, and a new opt-in keeps its keys and values in pool pages. HIP's opt-in gfx12 matrix tiles get faster with the same bytes. Settings are stricter: DeepSeek-V4.1's settings are refused by name when malformed, and an explicit thinking off that a chat template can't honour is refused.

  • New:
    • LFM2.5-24B-A2B on one NVIDIA GPU, from the MLX 4-bit export;
    • NVIDIA's Nemotron 3.5 Parse on HIP;
    • LightOnOCR-3 (0.8B, 1B, 4B) on HIP;
    • Clef's joint schema head and /v1/systemone on CUDA;
    • TensorFold Web, opened with tensorfold web;
    • the HIP qwen3_5 engine reads a DFlash2 drafter's config and weights, and taps its target layers;
    • a CPU projection and RMS norm in the kernel contract.
  • Speed:
    • Qwen3.8-27B MLX 8-bit on an M5 Ultra: cold prompts are 18% faster and decode 7.6% faster.
    • Flash Next BF16 cold prompts on a GB10 take about 4% less time, and under 1% less again on top.
    • HIP on gfx12 with --policy matrix=on: reranking and embedding take 2.9-4.8% less time on an R9700, with the same bytes.
    • Qwen 3.5 and 3.6 on HIP: sampled streams with an MTP head make 4-12% more tokens a second.
    • Qwen3.8-27B prompt chunks on an M3 Pro: 2-3% faster served prefill, bit-equal.
    • --learn with 100,000 kept states starts about 5 s sooner.
  • Serving:
    • An explicit thinking off that the template can't honour is a 400.
    • A refused disk read tries one shorter kept state.
    • Qwen 3.5 and 3.6 streams on CUDA take less memory.
    • A request's stats list its prompt pass widths.
    • The PDF render worker no longer crashes on macOS.
    • New opt-ins: the Qwen CUDA history starts with drafts; Gemma 4's 2,048-row solo chunk; CUDA prompt-cache keep-others; Qwen3.8-27B Metal KV pages.

Where a speedup was checked for equal output, its line says so. Some changes alter settings, refusals, memory or startup by default; they're listed under What can change for you.

Headline numbers

Each number names its machine and its PR.

Release reads, each machine with 1.2.4 and 1.2.5 in alternating fresh servers (base, new, new, base, after an unscored warm-up):

  • Flash Next MLX 4-bit on a GB10, against 1.2.4: level, with the same tokens: every prompt and decode cell from 2K to 64K within 0.8%, and every sweep from one to eight streams within 1.1% apart from greedy chat at four streams (3.6% faster). The 64K long-context answer is right on both.

  • Qwen3.8-27B on an M5 Ultra, against 1.2.4: level, with the same tokens: every sweep within 0.4% and every prompt and decode cell from 2K to 64K within 1.2%, apart from one 1.2.5 run's first 2K prompt, which came up cold while the other 1.2.5 run matched both 1.2.4 runs.

  • Qwen3.8-27B on a GB10, against 1.2.4: level, with the same tokens: all 56 cells within -1% to +2%. The 64K long-context answer is right on both.

  • Nemotron 3.5 Lightning 30B-A3B on a GB10, against 1.2.4: level, with the same tokens: prompts within 1% and one stream level. Greedy chat at two to eight streams moved both ways (-2.8% to +1.5%), as one-prompt multi-stream cells do; decode after a 32K prompt resumed 10K tokens in was 2.5% lower in both 1.2.5 runs.

  • Qwen3.8-27B on an M3 Ultra, against 1.2.4: level, with the same tokens: every prompt and decode cell from 2K to 64K within 1.3%, and 2K prompts 2.1% faster in both 1.2.5 runs. Sweeps were within 0.4%, against one 1.2.4 run. The 64K long-context answer is right on both.

  • Qwen3.8-27B MLX 8-bit (groups of 64), on an M5 Ultra: cold prompts are 18% faster (649 to 768 tok/s at 2K) and decode 7.6% faster, with the same tokens and prompt state (#2518, fixes #2502).

  • Qwen3.8 Flash Next on CUDA, on a GB10 (NVFP4):

    • BF16 prompt projections take a 128 × 128 tile on sm_12x wherever it fills the GPU, with the same output bytes: cold prompts take 3.7 to 4.4% less time (4K to 34K tokens), replies unchanged (#1846, refs #1754).
    • Staging K32 fragments from the same K64 pack takes another 0.6 to 0.8% off cold prompts, with the same bytes (#1848).

  • HIP on gfx12 under --policy matrix=on, on a Radeon AI PRO R9700, with the same output bytes:

    • the WMMA dense tile's 64-deep K step: Qwen3-Reranker-8B ranks 16 documents in 0.694 s instead of 0.720 s, and Qwen3-Embedding-8B embeds 100 short inputs in 0.946 s instead of 0.974 s (#2280, part of #2245);
    • silu(gate) × up in the product's epilogue, on top: 0.657 to 0.661 s instead of 0.687 to 0.697 s, and 0.901 to 0.911 s instead of 0.940 to 0.948 s (#2344, part of #2245).

  • Qwen 3.5 and 3.6 on HIP, on gfx1151: sampled streams with an MTP head draft on the graphed head chain, with the same tokens: 4-12% more tokens a second at temperature 1.0 (#2455).

  • Qwen3.8-27B on Metal, on an M3 Pro: prompt chunks of 64 rows or more use half operands where a tile's inputs fit. Logits and tokens are bit-equal, and served prefill is 2-3% faster (#2435, part of #2089).

  • Qwen CUDA with TF_QWEN_REQUEST_STARTS=1, on a GB10 with Qwen3.8-27B 4-bit and its DFlash2 drafter: turns 2 to 6 of a six-turn chat reuse 1,450 to 1,635 prompt tokens and take 0.72 to 1.33 s instead of 1.78 to 2.10 s (#2176, carries #1335).

  • Gemma 4 on CUDA with TF_GEMMA4_CHUNK_SOLO=2048, on a GB10 with the 31B 4-bit release: a cold prompt's first token comes 4-5% sooner at 4K, 16K and 29K tokens, with the same replies (#2391).

  • Gemma 4 and Nemotron on CUDA, on a GB10: prompt chunks that nothing reads skip their last layers, so cold prompts start up to 3.9% sooner (Gemma 4 31B: 1.3% at 2K, 3.1% at 16K, 3.9% at 32K; Nemotron 3.5 Lightning level at 2K, about 2% sooner at 16K and 32K), with identical replies (#2432, refs #2165).

  • Qwen 3.5 and 3.6 on CUDA, on a GB10: each pooled stream takes 147 MiB less on Qwen3.8-27B and 61 MiB less on the 35B-A3B, with tokens and speed unchanged (#2447, part of #1943).

  • --learn with 100,000 kept states, on an M3 Ultra: the server starts about 5 s sooner (8.0x faster open), with the same states and files (#2316).

New models and backends

  • LFM2.5-24B-A2B on CUDA: served on one NVIDIA GPU from the MLX 4-bit export, on the shared CUDA kernels, with --parallel streams and copy drafts. It runs without a drafter or prompt cache (#1492, part of #1488).
  • Nemotron Parse on HIP: NVIDIA's nvidia/Nemotron-3.5-Parse is served natively from its FP32 safetensors on a Radeon AI PRO R9700 (gfx1201), with its RADIO image tower, convolution neck and mBART decoder. The token arrays match the published example's on three pages, plain and drafted, solo and concurrent (#1941, part of #1927).
  • LightOnOCR-3 on HIP: the native server reads lightonai/LightOnOCR-3-0.8B, -1B and -4B (BF16) and serves their OCR prompts, token-exact against Transformers apart from exact BF16 logit ties (#1965, for #1927).
  • Clef's joint schema head on CUDA: /v1/systemone answers on one NVIDIA GPU, as it already did on HIP (#2458).
    • Its readout streams fill in pieces of final-norm rows, 1,024 rows a piece unless the lane host caps it, as on HIP (#2384, part of #2324).
  • TensorFold Web: a local web app laid out like the desktop app (chat, models, learning, API, benchmarks and clusters) ships in web/.
    • tensorfold web opens it, and on first use offers to install it with uv, pinned to the same release.
    • It starts without network access and reaches a local engine directly, past any system proxy.
    • Adopted from @Stephane-Eden's TensorFold Studio (#1781).
  • HIP drafter groundwork for qwen3_5 (#1563):
    • the engine captures a drafter's requested target-layer taps in each stream's ring, with and without graphs (#2314);
    • it reads a DFlash2 drafter's config.json against its target, refusing one it can't run, and loads its weights as MLX affine-4. The server doesn't use the drafter yet (#2477).
  • CPU: Qwen3.5-2B's 4-bit, group-64 affine projection and an RMS norm run on the host's cores through the kernel contract's gates 1 to 4, with eight planted controls on any host with no device (zig build cpu-contract). Output bytes are pinned the same at 1 to 16 threads and across column splits on x86_64 and arm64 (#2478, part of #2448).

Speed and memory

  • Qwen3.8-27B on Metal, opt-in with TF_QWEN27_METAL_KV_PAGES=1: each stream's keys and values live in pages of the core KV pool, with greedy replies equal to the default on one and two streams. It has run only on Macs without tensor units, and an M5 refuses the flag by name (#2283, part of #1293).
  • Qwen3.8-27B on a Mac, disk staging: the staging permit widens to one whole state at its --context, within --prompt-cache-gib, so conversations past about 13K new tokens are saved instead of skipped with StagingFull (#2388).
  • GLM-5.3-Flash on CUDA with TF_KV_SHARE=n: the startup plan sets aside a prompt-cache slot for each kept state, so the cache keeps as many states as its slots hold, up to n, and the rest goes to the KV pool's pages. The startup line names the slots (#2358).
  • Lane host: a stream paid for at load is a standing lease in the KV ledger, so a family that reserves per request admits as many full-window requests as the window plan, not one fewer. No family on main reserves per request, so every current load is unchanged (#2434, part of #1943).
  • The speedups in Headline numbers: #2518, #1846, #1848, #2280, #2344, #2455, #2435, #2176, #2391, #2432, #2447, #2316.

Serving and API

  • Thinking off: an explicit thinking off that the chat template can't honour (a </think> marker, but no switch or closable generation prefix) is refused with HTTP 400 before submission.
    • Messages' count_tokens refuses it too.
    • A request that leaves thinking out, and a server or template default, never counts as an explicit off.
    • Model discovery's capabilities end with thinking_switch and thinking_applicable, and the benchmark clients read them (#2170, part of #2139).
  • Prompt cache on CUDA, opt-in: TENSORFOLD_PROMPT_CACHE_KEEP_OTHERS=1 stops a new prompt state from evicting another conversation's kept states.
    • A state that would fit only that way isn't kept, and is logged once.
    • A conversation's own older states still make room.
    • Tokens are unchanged (#2533, part of #2346).
  • Conversation states on disk: after a refused read, the server tries one shorter compatible state before prefilling from the start (#2442).
  • Stats: a request's finished stats list its prompt pass widths for Qwen3.8-27B, GLM-5.3-Flash and Nemotron (telemetry only) (#2304).
  • Qwen3.8-27B on Metal: the startup line reports how many admitted streams DFlash2 can draft for (#2526). Thanks to @mrv777 for the report.
  • Profiles: Dory's, Gemma 4's and Spark2.5's pool-page switches, and Nemotron's paging on Metal, are registered in the profile registry. The startup line and capabilities report them, and every default is unchanged (#2306, part of #1660).
  • Fabric: a pipeline stage's boot and round loop is one driver over an engine binding, which nothing uses yet. A stop that a middle stage receives now passes on instead of closing the ring (#2241, part of #2005).

Fixes

  • Vision on macOS: PDFs with text no longer crash the render worker. Each PDF renders in a fresh server process instead of a fork, under the same limits and deadline, and a crash is reported by its signal (#2393, fixes #2385).
  • Non-finite values on Macs: a non-finite value in a masked key no longer reaches prompt attention's output, for Nemotron (#2322) and LFM2 (#2328) (both part of #1009). Finite prompts run the same kernels as before.
  • Nemotron on Macs: the masked prompt shader calls the shared tf_paged_row helper again, so its prompt sources compile (#2548). A test now compiles Nemotron's prompt shaders on a Metal device (#2558).
  • Qwen3.8-27B on Metal: TF_PROMPT_ROWS below 128 gets the prompt floor's own message again (#2527).
  • GLM CUDA: the peer warm-up after NCCL init is bounded to 300 seconds; on timeout every rank stops, naming the missing peers and the transport observed (#2512, part of #2347).
  • Release checks: on a Mac release binary, the private-path check no longer reads the code signature's hash bytes as text (#2524).
  • Kernel contract: the Metal limits check finds LFM3's launches again (#2565).

Docs

  • The Qwen3.8-27B recipe says native serving reads MLX 8-bit with the same flags, measured against 4-bit on an M5 Ultra (#2506, refs #2502).
  • docs/evaluating.md says what to pin when testing TensorFold, how to check a release's signature, which tests need no GPU and which requests to replay. A root Dockerfile installs a verified Linux release archive (#2472).
  • /health's memory on Metal, and which Mac families report prompt_cache_plan (#2439).
  • WINDOWS.md: the host suite on a Windows machine with no usable GPU (#2479).
  • The README's and GLM CUDA recipe's Jammed XSFP4 row says it shipped in 1.2.3 (#2451, for #2041).
  • The optimization ledger's cells for #2011, #2162 and #2282 say they landed (#2431).

Internal

  • Packaging: each stable release publishes its signed Mac arm64 desktop runtime and catalog update from main's release tooling (#2465).
  • Tests and CI:
    • python3 -m pytest from the repository root passes on a machine with no GPU (#2471);
    • thirteen host checks that no CI job ran build and pass again (#2550);
    • a nightly job runs 246 host checks that no per-PR job ran (#2554);
    • test-glm-snapshots-host runs under test-cuda-host (#2426, refs #2031);
    • the Qwen4 and Dory RoPE table tests print only on failure (#2519, refs #1852);
    • the 27B Metal lane and served draft checks (#2493, fixes #2489).
  • Checks on contributions:
    • a host check that every includer of a CUDA function-body fragment declares its names (#2495);
    • compile-cuda-tests builds again (#2463);
    • lean flags each new getenv of a setting outside platform/environment.zig, by a TF_/TENSORFOLD_ name or a non-literal one; reads already on main, test files and test blocks aren't counted (#2521).
  • Records:
    • HIP record counts are pinned in their owning arrays (#2461);
    • the contract registry's record joins live in registry_joins.zig (#2422);
    • 13 nvcc 13.0 register-count slots are refreshed from the five-SM audit (#2515, refs #2395);
    • the CUDA limits audit reads sm_110 resource usage from compiled cubins (#2395, part of #2319).
  • KV pool: a family's pool group is sized by its bytes a row over its layers, and a paged CUDA load prints each group's rows a page and padding; every family stays one group, so nothing else changes (#2532, part of #2444).
  • File splits: metal.zig (#2425) and GLM's cuda_engine.zig (#2429) are split under 600 lines, unchanged.

What can change for you

  • An explicit thinking off: a request that turns thinking off where the chat template can't honour it now gets HTTP 400, where it used to be accepted and ignored. Leave thinking out of the request, or use a template with a switch, to keep it served. count_tokens refuses the same requests (#2170).

  • DeepSeek-V4.1 settings on CUDA:

    • a malformed TF_DSV41_* setting is refused by name before loading;
    • switches that used to read any text but 0 as on now take exactly 0 or 1, so a word such as false is refused rather than read the other way;
    • MHC_DEFER and COEF_LATE refuse no, which their old readers read as on;
    • a malformed TF_DSV41_TOOL_GRAMMAR is refused at server start unless TENSORFOLD_TOOL_GRAMMAR is set.

    Unset defaults are unchanged. Use 0 or 1 (#2379, #2516).

  • GLM-5.3-Flash on CUDA with TF_KV_SHARE=n: a value over 64 now stops the server, and a cache too small for one slot (for example --prompt-cache-gib 0) keeps no states, where before it kept them outside the plan (#2358).

  • GLM CUDA at two ranks or more: a peer warm-up that doesn't finish within 300 seconds now stops every rank instead of waiting (#2512).

  • HIP with --vision: a Qwen 3.5 checkpoint whose vision tower the HIP engine can't read (such as Qwen3.5-9B's 72-wide heads) is refused at start-up instead of answering each image with a 400 (#1965).

  • Flash Next on CUDA: the BF16 prompt tile is on by default on sm_12x where it fills the GPU, with the same bytes; TF_FLASHNEXT_PROMPT_BF16=0 turns it off. TF_FLASHNEXT_PROMPT_ROWS is gone (#1846).

  • Qwen 3.5 and 3.6 on CUDA: each pooled stream's rollback copy shares its verify log's bytes, so streams take less memory, with tokens unchanged (#2447).

  • Gemma 4 and Nemotron on CUDA: prompt chunks nothing reads skip their last layers by default, with identical replies; a drafter that reads those rows keeps them (#2432).

  • Qwen3.8-27B on Metal:

    • 8-bit checkpoints read their code bytes directly, with the same tokens and prompt state (#2518);
    • prompt chunks of 64 rows or more take half operands where they fit, bit-equal (#2435);
    • a Mac's disk staging permit is one whole state at --context, within --prompt-cache-gib (#2388).

  • Vision on macOS: each PDF renders in a fresh process instead of a fork (#2393).

  • Developers: the release packaging check is now tools/release/check_package.py (#2471).

  • /v1/models: each entry's capabilities gain thinking_switch and thinking_applicable. Clients that read the listing strictly will see new keys (#2170).

  • Request stats on CUDA: a finished request's stats gain its prompt pass widths for Qwen3.8-27B, GLM-5.3-Flash and Nemotron (#2304).

  • Startup lines: they now name registered pool-page switches (#2306), GLM's prompt-cache slots under TF_KV_SHARE (#2358), explicit DeepSeek settings (#2379), how many streams DFlash2 can draft for on the 27B (#2526), and a paged CUDA load's pool groups (#2532). Scripts that parse them may need updating.

  • Nemotron and LFM2 on Macs: replies change only where a NaN or infinity already sits in a masked key (#2322, #2328).

  • Conversation states on disk: after a refused read, the server reads one shorter compatible state before prefilling from the start, so a failed restore can take one more disk read (#2442).

Known issue

  • JPEG input with the Homebrew install: the server decodes JPEG only with libjpeg-turbo 3.1.4.1, so image tokens can't change underneath a model, and the formula didn't provide that release at first. It does now: the formula builds libjpeg-turbo 3.1.4.1 into the install, where the server finds it first. Run brew update && brew reinstall tensorfold if JPEG images are refused with a startup warning (#2581, thanks to @knicolov).

Thanks

Contributors

laudney, gordonhatusupy, and 10 other contributors
Assets 12
Loading