TensorFold 1.2.2

TensorFold 1.2.2

1.2.2 adds new models on CUDA and HIP. Qwen3.8-Flash-Next serves NVIDIA's own NVFP4 checkpoint at --tp 2. GLM-OCR runs
natively on HIP, and Qwen3.5-9B runs on Metal on the 27B's engine. Qwen3.8-27B EXL3 checkpoints prefill 38% to 42% faster on a
GB10, HIP decode is faster on Ryzen AI Max, and Flash Next reads its per-layer embeddings from an SSD much faster on Macs.
CUDA and Metal loaders release the page cache once the weights are loaded. CUDA loaders read checkpoints ahead, so they load
sooner. zig build -Dgpu=<gpu> builds a much smaller server for one GPU. Readers of safetensors and other checkpoint headers
across the engine now refuse a malformed file instead of crashing on it.

Where a speedup was checked for equal output, its line says so. Opt-in switches change nothing unless set. A few changes
move replies or refuse a start that used to go ahead; they're listed under
What can change for you.

Headline numbers

Where a number's replies were checked against its base, its line says how.

• Qwen3.8-27B EXL3 on a GB10: prompt prefill 38% to 42% faster from 2K to 64K tokens (494 to 705 tok/s at 2K), and decode
  5% to 12% faster, with the same tokens (#1467).
• HIP decode on a gfx1151: 7.5% to 18.9% faster on Qwen3.6-35B-A3B and Qwen3.8-27B at 32K and 64K, with identical
  tokens (#1932).
• Ryzen AI Max (RDNA 3.5): Qwen3.6-35B-A3B chat decodes about 18% faster with drafts on, and Qwen3.8-27B about 5%,
  with the same tokens (#1855).
• Flash Next on an M5 Max with --ple-on-ssd: decode 36% to 39% faster (75.8 to 105.1 tok/s on code), and a cold 2K
  prompt in a third of the time, with the same replies (#1462).
• Flash Next on a gfx1151: the first cold prompt runs in 28.55 s instead of 48.44 s, and the page cache falls from
  31.5 to 2.8 GiB (#1611).
• GLM on three GB10s at TP=3: cold prompts of 32K to 167K tokens finish 5% to 14% sooner, with identical replies (#1577).
• Flash Next on a GB10: a cold start drops from about 74 s to 48 s with the direct expert load (#1901). Cold prompts
  run about 4% to 5% faster, with identical tokens (#1764).

New models and backends

• nvidia/Qwen3.8-Flash-Next-NVFP4 on CUDA at --tp 2, on two GPUs, one rank a host (#1771, for #1754).
  • The BF16 attention, DeltaNet and shared-expert projections, the n-gram table (E4M3 rows with one BF16 scale) and the
    MTP head's E4M3 experts (128x128 block scales) load as stored.
  • Each rank owns half the routed experts whole, so replies are one GPU's bytes.
  • On two GB10s, drafted replies equal "draft": false (6 of 6), and concurrent replies equal their solo runs in 8 of
    8 cells.
• The CUDA server serves --tp, with --rank, --master and --master-port. Families that open without ranks
  refuse it by name, and --idle-unload is refused with it.
  • The ranks compute and compare a cached digest of every checkpoint byte, so TENSORFOLD_CHECKPOINT_SHA256 is no
    longer required (it still overrides the digest) (#1771).
• GLM-OCR on HIP: the publisher's BF16 zai-org/GLM-OCR, served natively. Its CogViT image encoder and GLM decoder
  run in Zig and HIP, with no Python or libtorch. Every reply was checked against the model card's Transformers examples
  (#1940, part of #1927).
  • Groundwork for the OCR models (#1929, #1930, #1931, #1933):
    • a family can declare that its replies carry no reasoning block;
    • the tokenizer supports NFKC and NFKD normalizers;
    • HIP has a shared rocBLAS BF16 projection, loaded as an optional library;
    • the prompt cache can count repeated keyed prompts as repeats.
  • No model already served decodes differently.
• Qwen3.5-9B on Metal runs on the 27B's engine, instead of being refused as an unqualified shape (#1826, part of
  #503). The dense Qwen Metal loader checks config-derived projection geometry through shared structural checks, and keeps
  the 9B and 27B qualification policy (part of #503).
• DeepSeek-V4.1 on Metal, part one:
  • its kernels, the decoder over every layer, Engram, prompt chunks and DSpark drafts, each checked against float64
    references (#1710-#1719);
  • its CSA2 attention, matvecs and glue kernels, under the kernel contract (#1709, #1710, #1711);
  • serving follows with #1720-#1722.
• Intel GPUs without DPAS (Gen12LP): a server built with -Dxpu-dpas=false serves Nemotron-H. It refuses the
  Qwen3.8-27B runner and Gemma 4 E4B by name, and Nemotron's prompt projection takes 8 x 4 tiles (#1692).
• Groundwork:
  • core/gguf.zig reads GGUF files with every length and offset checked, and repacks Q4_0, Q4_1, Q5_0, Q5_1, Q8_0,
    Q4_K, Q5_K, Q6_K, Q3_K and Q2_K blocks exactly into per-group affine weights. It is the first step toward native GGUF
    on Apple Silicon (#1926, part of #1920).
  • Ling's grouped expert routing runs on Metal, checked against a float64 reference (#1836, part of #1247).
  • MOSS-Image computes its sampling schedule natively from scheduler_config.json, bit for bit what its scheduler sets
    (#1670, part of #1529).
  • Living Weights on Gemma 4 and on Qwen4 dense, step 1 each: the attention backward passes on the host, checked against
    independent references. Nothing served changes (#2032, #2021).
• TensorFold Starlight, the enterprise platform, lives in its own repository,
  ashhart/tf-starlight, with its history (#1922, #1972).
  • It's one governed OpenAI- and Anthropic-compatible endpoint for a fleet of TensorFold engines.
  • It pins the TensorFold release it is tested with, and checks the engine contract against that release in CI.

Speed

• Qwen3.8-27B EXL3 on CUDA: prefill 38% to 42% faster from 2K to 64K tokens on a GB10, decode 5% to 12% faster, with the
  same tokens (#1467).
• HIP decode:
  • The planned decode score takes four keys a warp on long spans and two on short ones, with the same lane order and folds:
    7.5% to 18.9% faster on a gfx1151 at 32K and 64K, with identical tokens (#1932).
  • On Ryzen AI Max, one- and two-row decode launches use narrower stream tiles: about 18% faster on Qwen3.6-35B-A3B chat
    with drafts on, and about 5% on Qwen3.8-27B, with the same tokens (#1855).
  • Softmax stats and partial sums load sixteen values ahead and fold in the same order, with the same bits (#1961).
• HIP Qwen3.5-family servers price shared draft rounds at the row bucket they actually run. Qwen3.6-35B-A3B with MTP at four
  streams no longer drops into a slower drafting mode, and models load faster. Tokens are unchanged (#1830, fixes #1695).
• Flash Next on HIP:
  • n-gram drafts are gathered in parallel, and the checkpoint's weight pages are released after upload. On a gfx1151 the
    first cold prompt runs in 28.55 s instead of 48.44 s (#1611).
  • On RDNA 3.5 iGPUs (gfx1150 to gfx1153) prompts fill in 4,096-row chunks by default on one GPU, falling back to 1,024 when
    the larger chunk would admit fewer streams. Tokens are unchanged (#1651, fixes #1649).
  • Routing reads the checkpoint's stored BF16 router rows, with the same tokens, for about 2.5 MiB more a layer (#1614).
• Flash Next on CUDA:
  • Prompt passes do less work with the same bits, so cold prompts run about 4% to 5% faster on a GB10, with identical
    tokens (#1764).
  • The MTP head's experts run on a call's last row only, the one row that reaches the draft, and replies are unchanged
    (#1925).
  • A partly kept window's DeltaNet rows are applied at the start of the next window, lane buffers are reused across
    streams, and the shared expert runs on a side stream, all with the same tokens (#1763).
    TF_FLASHNEXT_KEEP_LATE=0, TF_FLASHNEXT_LANE_PARTS=0 and TF_FLASHNEXT_SHARED_SIDE=0 turn each off.
• Flash Next on Macs with --ple-on-ssd: per-layer embedding rows are read from disk in batches. Decode is 36% to 39%
  faster on an M5 Max (75.8 to 105.1 tok/s on code), and cold prompts take a third of the time at 2K tokens, with the same
  replies (#1462, fixes #1461).
• GLM on CUDA runs a prompt chunk's MTP pass on its last row past the caches. Cold prompts of 32K to 167K tokens finish 5%
  to 14% sooner on three GB10s at TP=3, with identical replies (#1577).
• Nemotron on CUDA drafts copies of earlier text in its lanes. A verified copy is tried first inside the existing verify
  windows, with the same acceptance rule, so replies are unchanged (#1974, part of #1910).
• Grammar masks for constrained streams are filled in parallel across streams, with the same masks and tokens. On an M5
  Ultra, Qwen3.5-2B serves 0.7% to 1.3% faster at 4 to 32 constrained streams (#1881, fixes #1455).
• Loading:
  • TF_LOAD_PREFETCH: Gemma 4 and Qwen 3.5 on CUDA read each layer ahead while the current one uploads, so the 27B loads
    about a third faster on a GB10. Replies are unchanged, and TF_LOAD_PREFETCH=0 is the old path (#1928).
  • Flash Next on CUDA reads its experts past the page cache with O_DIRECT, and keeps a load's shards open. A cold start on
    a GB10 drops from about 74 s to 48 s, with the same replies; TF_LOAD_DIRECT=0 keeps the mapped reads (#1901, fixes
    #1873).
  • On top of that, Flash Next's direct load reads the next chunk while the current one packs, so its experts load about
    12% faster, with replies unchanged (#1960).
  • Qwen3.8-27B with a DFlash drafter on Metal compiles the drafter head's pipeline at load, so the first request no longer
    waits on it (#1734).
• The kernel contract computes its affine and GLM float64 references on worker threads, so it finishes sooner, with the
  same reference bytes (#1776).
• Opt-in:
  • TF_QWEN_KEEP_LATE=1: Qwen3.5-family models on CUDA apply a verified window's kept DeltaNet rows inside the next
    round's slot scan, with the same bits.
    • On a GB10, Qwen3.8-27B MLX 4-bit decodes 1% to 8% faster (most at 16 streams, none slower), for about 66 MiB more
      a stream.
    • It stays off by default because, on by default, it failed 8 of 64 served chat requests on a GB10 with
      CUDA_ERROR_INVALID_VALUE. Replies are unchanged either way (#1824, #1958).
  • TF_FLASHNEXT_READOUT_ROWS=128 keeps a large shared decode round on the row kernels: the same bits, and up to 7% more
    throughput at 16 streams on a GB10 (#1741).
  • TF_FLASHNEXT_ATTN_LANES=1 runs a shared decode round's attention in one launch a kernel, instead of a launch per
    stream. The kernels are bit-equal to the per-stream launches (#1739).
  • TENSORFOLD_PREFILL_ORDER=srpt lets short replies decode ahead of a long prompt that is still filling. The prompt
    keeps at least a tenth of the GPU time, and replies are the same tokens (#1876).
  • Gemma 4, MiniMax M3, Kolibri-1, Spark and Dory on CUDA can opt into verified root and context copies with a four-token
    match. Their suffix-copy policy stays the default until each checkpoint passes its device campaign (#1910).
  • On M5 Macs, TF_ATTENTION_PV_FP32=1 keeps attention's P x V at full fp32 for Nemotron's prompt attention.
    • It's closer to float64: output cells off the nearest bf16 value fall from about 1.1M to 1.8K of 8.6M.
    • The cost is large: prompts are 30% to 87% slower at 2K to 32K (#1882).

Memory and long conversations

• Page cache on load: loaders drop their checkpoint from the page cache once the weights are loaded, so unified memory
  doesn't hold them twice. Files read at runtime (PLE rows, DeepSeek's Engram tables, Metal's SSD expert and PLE files)
  are kept.
  • On CUDA, every family's loader does this; a served Qwen3.8-27B on a GB10 frees about 15 GiB (#1884).
  • On Macs, the q27, Nemotron, LFM2 and Qwen3.5 loaders do it, with the same replies: on an M5 Ultra a loaded
    Qwen3.8-27B frees about 14 GiB, and Nemotron 3.5 Lightning about 17 GiB (#1885).
  • So do GLM, GLM-DSA and Flash Next on Metal (#1888).
• Gemma 4 on CUDA keeps its global layers' caches in pages of the core KV pool (#1619).
  • With TF_KV_SHARE=n, its prompts are kept and resumed as pool pages, with sliding-window tails kept per stream and the
    MTP assistant's pages bound to its cache.
  • Resumed, concurrent and drafted replies equal their plain runs. Unset, tokens and speed are unchanged (#1883).
• GLM on CUDA, opt-in: TF_GLM_KV_HOST keeps the KV pages' latent rows in pinned host memory, for a 1M-token window on
  two 96 GB cards. Unset, the engine is as before, with identical tokens at TP=2 on two GB10s (#1795, part of #1794).
• GLM on CUDA at TP=3, opt-in: TF_GLM_KDA_PAD=1 packs the fused KDA input to an even width, carried in the fingerprint
  every rank agrees on. Unset, the engine is unchanged (#1545).
• Nemotron on Metal, opt-in: TF_NEMOTRON_METAL_KV_PAGES=1 takes KV pages as a reply grows, so a short chat holds 1
  page instead of 17 (#1837). With it off, the paged checks cost one null check a pass (#1914).
• On a CUDA card too small for an EXL3 checkpoint's embedding beside its window, the Qwen3.5 engine maps the embedding
  rows from host memory instead of refusing to start. Where they fit on the card, they stay there (#1678).
• GLM-5.3-Flash on Macs with --learn-conversations: a prompt state restored from disk is charged at its full size. The
  prompt cache no longer grows past its budget, by about 2.5 GB a restored state at 130,000 tokens (#1818, part of #1572).
• Images in the prompt cache: a cached state keeps exact image identities (position, rows, grid and a content hash, never
  a path). So a state is never resumed for a different, moved or re-gridded image. Text-only prompts are unchanged (#1962).
• Lane hosts release a stream that finishes on its first token when its reply ends; before, it leaked until the server
  stopped. They also gain a byte ledger that families can opt into; no family uses it yet (#1945).
• Drama speech on CUDA, opt-in: TF_DRAMA_REFERENCE_CACHE=N reuses a catalog voice's reference prefill across requests,
  as Metal does (1 to 16 voices, each up to 61.4 MiB plus 46 KiB a saved row) (#1745).
  • It's checked by host tests against a fake lane, and not yet measured on a GPU.
  • The default 0 keeps none, and the default voice is unchanged.

Serving, build and API

• zig build -Dgpu=<gpu> builds one stripped server for one GPU, with every model family (#1936, fixes #1934).
  • It's about 69% smaller on CUDA (sm_80, sm_86, sm_89, sm_120, sm_121 or spark), and 58% smaller on HIP (a
    gfx name).
  • A CUDA build refuses a GPU of another SM at startup, naming both.
• serve --prompt-cache-disk DIR is a shorter spelling of --learn-conversations --learn-dir DIR, with the same limits
  (#1908).
• serve --profile stable names today's defaults explicitly (#1944, #1959).
  • Startup reports the profile, and any registered switch set away from it.
  • Seven CUDA KV switches, and GLM's TF_KV_PAGES, are read through one registry, with environment values winning byte
    for byte.
  • fast-exact and fast are refused until qualified.
• On HIP, an integrated GPU counts the card's own memory only when sysfs confirms the firmware carve-out. Otherwise it
  keeps the system's count, and the start-up line says which it used (#1963).
• CUDA kernels share their duplicated device primitives; the machine code is identical on sm_80, 86, 89, 120 and 121
  (#2018, closes #1409).
• HIP loads a shared expert wider than the routed experts beside the stack instead of refusing it, with the same tokens
  (#1612). Flash Next on HIP loads oMLX checkpoints whose MTP head sits at the top-level mtp.*, so their drafts work
  (#1613, fixes #1598).
• Cortheon:
  • reads a GitHub file as raw text from GitHub.com and www.github.com too;
  • keeps links whose scheme is written in capitals;
  • reads a link whose tag breaks after a carriage return (#1949);
  • finds closing tags in one pass, so many unclosed tags no longer take quadratic time (#1898).
• Compaction keeps a memory note sent as text parts instead of summarizing the conversation again (#1939).
• The proposal and pull-request templates ask for proof and an opt-in for any change to default-path output tokens,
  planted controls for new kernels, and the --version output (#1596).

Fixes

• Malformed checkpoint files are refused before anything large is read (part of #1872). Before, they could crash the
  loader, or read past a buffer.
  • Header lengths, entries and tensor ranges are checked through core's safetensors reader in the Metal vision towers'
    shard reader and the cluster planner (#1983), the Nemotron XPU loader and the XPU safetensors reader (#1919), the
    Living Weights view (#1923) and Flash Next's embedding shard on CUDA (#1917).
  • H3 on CUDA (#1906) finds each tensor by its real header entry, and loads spaced JSON headers.
  • Also covered: GLM's DFlash2 drafter on CUDA (#1904), DeepSeek V4.1 packs (#1903), EAQ containers (#1902), Gemma 4 on
    Intel GPUs (#1905) and pull's EXL8 width files (#1907).
• NaNs in host bf16 rounding:
  • sixteen copies (#1864, fixes #1863), and seven on load and inference paths (#1954), now round through core's shared
    conversion. A negative NaN no longer overflows in Debug builds, and every finite value gives the same bits.
  • Qwen 3.5 on CUDA keeps a NaN weight as NaN at load, instead of a finite value (#1986).
• GLM-5.3-Flash on CUDA refuses TF_GLM_CHUNK_SOLO=4096 beside --drafter at load. Before, it held 1.6 to 1.9 GiB of
  prompt buffers that no pass used (#1842).
• GLM on CUDA with TF_GLM_KV_SHARE no longer reports a balanced idle KV pool as uneven at close (#1993).
• CUDA's Qwen DFlash2 drafters must declare sliding attention on every layer, which the kernels run. A drafter that
  declares a full-attention layer is refused before loading (#1946).
• GLM refuses a checkpoint whose tensorfold_activation_dtype asks for anything but bfloat16, naming the key. Before,
  bf16 activations were served silently. docs/api.md says which engines serve /v1/decisions (#1987).
• Nemotron on Intel GPUs refuses a config with an unknown layers_block_type name at load, instead of serving that layer
  as attention (#1865).
• Metal: Qwen3.8-27B's keep gathers and scatters the binding's KV heads a row, so a keep stages what the layout holds
  (#2013).
• RoPE tables:
  • transformers' inverse frequencies are computed in one place, core.rope_freq.
  • Gemma 4 on CUDA (#1853), and lfm3, Spark 2.5 and the GLM vision tower (#1995), use it, with no value changes.
  • LFM3's backends share one copy of the frequencies and the layer schedule, bit for bit as torch's (#1839).
• Benchmark clients:
  • refuse malformed or non-finite --temperatures, and nonpositive --tokens, --reps or --levels, before any
    request (#1875, #1869);
  • the image session benchmark fails a turn whose stream ends early (#1916);
  • cache benchmark runs qualify only when a complete stream showed real text or reasoning (#1967);
  • tools/bench_image_session.py checks that image turns keep and resume their prompt (#1889).
• Tests and builds:
  • the CUDA contract's NVFP4 expert-waves control goes red on large GPUs such as the RTX 5090 (#1832);
  • Windows' cost-cache reads mid-replace count as a miss (#2001), and the server's linear-time tests count thread cycles
    (#1854);
  • zig build test-slide-xpu-host builds again (#1840);
  • test-info-config runs info's issue-934 test again (#2006);
  • a request-log test passes under any umask (#1900);
  • the XPU build makes one core module per device tree (#1918);
  • HIP gate-5 launch records are re-derived with ROCm 10.1's hipcc, record values only (#1979).
• --tp 2 on CUDA: a crash on the first prompt (a vtable built on the stack) is fixed (#1771).

Docs

• The GLM CUDA recipe says what TF_GLM_CHUNK_SOLO gains at TP=3 with TF_GLM_ROCE_MULTI: a little under half its gain
  without it (#1996).
• The Qwen3.8-27B recipe gains measurements on a 48 GB M5 Pro Mac mini with 1.2.1: 89.1 tok/s sampled code with DFlash2,
  against 86.5 on 1.1.0 (#1947).
• The Flash Next recipe covers native HIP on a Strix Halo iGPU, with speed and quality for three checkpoints (#1747).
• The cache benchmark receipts say which tool-call content they count (#1999).

What can change for you

These change what some requests get, or stop a start that used to go ahead:

• GLM with a non-bf16 tensorfold_activation_dtype is refused at load, instead of served in bf16 (#1987).
• A Qwen DFlash2 drafter that declares a full-attention layer is refused on CUDA (#1946).
• TF_GLM_CHUNK_SOLO=4096 with --drafter is refused on CUDA (#1842).
• --prompt-cache-disk with --learn-conversations or --learn-dir is refused (#1908).
• --tp on CUDA is refused by name for families that open without ranks, and --idle-unload is refused with it (#1771).
• A -Dgpu CUDA build refuses a GPU of another SM (#1936).
• Nemotron on Intel GPUs with an unknown layers_block_type is refused (#1865).
• On an XPU build without DPAS, the Qwen3.8-27B runner and Gemma 4 E4B are refused by name (#1692).
• A malformed checkpoint header is refused (part of #1872). H3 on CUDA could load the wrong tensor for a look-alike
  metadata entry, and now finds the real one (#1906).
• A checkpoint with a NaN weight: Qwen 3.5 on CUDA keeps it NaN instead of a finite value (#1986).
• Image prompts: a cached state is never resumed for a different, moved or re-gridded image, and image prompts' states
  are no longer saved to disk or learned (#1962).
• Cortheon fetches GitHub files from more hostnames, keeps capital-scheme links and reads more broken tags, so a turn
  can see different fetched content (#1949, #1898).
• Compaction keeps a memory note sent as text parts, instead of summarizing the conversation again (#1939).
• Benchmark clients refuse arguments they used to accept, and a cache run with only usage in its stream no longer
  qualifies (#1869, #1875, #1916, #1967).

Thanks

• @AbrahamCete (#1830, #1855, #1932, #1961)
• @Arminova (#1763, #1764)
• @BadAd84 (#1545)
• @BHCC2025 (#1467)
• @Bizuayeu (#1842, #1854, #1993, #1996)
• @BlivionIaG (#1776)
• @BobClawblaw (#1670)
• @cshintov (#2013)
• @diwakergupta (#1678)
• @fkiene (code from the cuda.fast submission in #1824)
• @garvrao80 (#1869, #1875, #1908, #1916, #1944, #1946, #1959, #1967, #1999)
• @gilby (#1577, #1611, #1612, #1613, #1614, #1651, #1709-#1719, #1747, #1876)
• @grantoverton (#1826)
• @jayleaton (#1596, #1745)
• @jishnuvenugopal (#1898, #1900, #2006)
• @jrb-ai (#1824, #1958)
• @jschmied (#1619, #1692, #1837, #1839, #1840, #1853, #1864, #1865, #1901, #1902, #1903, #1904, #1905, #1906, #1907,
  #1914, #1917, #1918, #1919, #1923, #1925, #1928, #1954, #1960)
• @kieranscotney (#1734)
• @laudney (#1771, #1929, #1930, #1931, #1933, #1936, #1940, #1945, #1962, #1979)
• @llstrk (#1963)
• @optimisme (#1739, #1741)
• @paragontasx (#1462)
• @pauldodd123 (#1947)
• @ppardus (#1939, #1949)
• @rootsbymenda (#1832)
• @russellmm (#1795)
• @ThinkOffApp (#1836)