TensorFold 1.2.1

1.2.1 adds new models on CUDA. MiniMax-M3.1-Flash runs over two GB10s, Gemma 4 31B drafts with Google's MTP
assistant, and Qwen3.6-35B-A3B gets a 1,048,576-token window and a DFlash2 drafter. MOSS-Image now runs end to end on
Metal and its denoiser runs on CUDA. Decode is faster on GB10 for MLX 4-bit models, and Gemma 4 E4B reads prompts 2 to
2.6 times faster on Intel's Arc Pro B70. GLM-5.3-Flash gets opt-in ways to keep long conversations, and the server
reports its load at GET /v1/load.

Where a speedup was checked for equal output, its line says so, and the opt-in switches change nothing unless set. A
few fixes change replies on purpose; they're listed under What can change for you.

Headline numbers

Where a number's replies were checked against its base, its line says how.

• Gemma 4 31B on a GB10, with its MTP assistant: 28.3 tok/s against 12.2 undrafted at one stream (#1385).
• Qwen3.6-35B-A3B (NVIDIA's NVFP4) on a GB10, with a DFlash2 drafter: 130 tok/s at one stream on a 4.2K-token
  prompt, against 99 with MTP (#1674).
• Qwen3.6-35B-A3B on one GB10 at 1M tokens: a 1,048,512-token prompt prefills in 26 minutes and finds a
  passphrase placed 5% in (#1675).
• Gemma 4 E4B on an Arc Pro B70: prompts read 2.0 to 2.6 times faster, with the same logits (#1746).
• Nemotron 3.5 Lightning on a GB10: +13.3% at 8 streams, from grouped 4-bit projections (#1783).
• GLM-5.3-Flash on GB10s at TP=3 with the 1M window: a 64k message sent again resumes in about 1.7 s instead of
  prefilling for 36 s (opt-in, #1732).

New models and backends

• MiniMax-M3.1-Flash on CUDA, at tensor parallel over two GB10s (sm_121, two ranks only; anything else is refused).
  • It reads Mia-AiLab's published EXL3 pack as stored. That's a lossy third-party quantization: against the release,
    teacher-forced over 2,406 tokens, its mean KL is 0.0788, with 89.9% top-1 agreement.
  • It serves the release's DSpark drafts, kept prompt states, tools, structured output and images. One stream with
    drafts ran 24 to 39 tok/s (#1388, after #1235).
• Gemma 4 31B on CUDA takes google/gemma-4-31B-it-assistant as --drafter. Drafted replies equal
  "draft": false, and four concurrent replies equal their runs alone. Other assistant types are still refused by type
  (#1385, for #1383).
• Qwen3.5 and 3.6 on CUDA past 262,144 positions: --rope-yarn 4, or a config with a YaRN block, raises the
  window to 1,048,576. Drafted, concurrent and resumed replies equal their plain runs. Without YaRN, replies are
  byte-identical to before. HIP, Metal, the DFlash2 drafters and other families refuse a scaled rope by name (#1675).
• Qwen3.6-35B-A3B on CUDA with a DFlash2 drafter: --drafter takes a DFlash2 model, such as
  incoai/Qwen3.6-35B-A3B-DFlash2. The checkpoint's own directory still drafts with its MTP layer. Every drafted reply
  equals its plain run (#1674).
• MOSS-Image on Metal, end to end: the prompt encoder (bf16 as published, under the kernel contract) and the image
  decoder (FLUX.2 small VAE, float32) now run inside the moss_image family, and tf-moss-image writes
  picture.png itself, with no torch step (#1687, #1504).
• MOSS-Image-Base's transformer on CUDA, through tf-moss-image, on an RTX 5090 and a GB10.
  • Every step's cosine against the publisher's torch bf16 pipeline is 0.99967 or better, and a 1024x1024 picture is
    44.0 dB (RTX 5090) and 45.9 dB (GB10) PSNR from torch's.
  • 50 guided steps at 1024x1024 take 24.6 s on an RTX 5090 and 69.6 s on a GB10, 1.26x and 2.03x faster than torch.
  • On CUDA, the prompt encoder and image decoder still run in torch, and nothing is served yet. See the
    recipe (#1528, for #1518).
• DeepSeek-V4.1's release checkpoint (deepseek-ai/DeepSeek-V4.1-Flash-1017) is read as published: FP8 blocks,
  MXFP4 routed experts and the FP8 Engram table, beside the EXL3 trellises. tf-dsv41-engram-pack writes the Engram
  tables as row shards with a checked manifest. The CUDA engine runs EXL3, so it refuses the release checkpoint by name
  before loading (#1705, for #1676). On Macs, its FP8 block-scaled rows and MXFP4 experts have their own Metal
  kernels under the kernel contract, with Kimi's left as they are, and its routed experts a grouped prompt GEMM; nothing serves through them yet (#1706, #1707, #1708).
• LFM3-1.7B and LFM3-VL-2B: their official MLX 4-, 6- and 8-bit checkpoints are admitted and checked ahead of
  their engines (#1513, part of #1491).
• XPU on Gen12LP (Iris Xe, DG1): Nemotron's kernels build, with a plain bf16 fallback for its two DPAS products. DPAS
  builds are unchanged (#1673).
• Qwen3.8-27B on two Macs, groundwork: the 27B family can load one rank's half of the checkpoint and run its
  layers with the pair's row exchanges, with one Mac's sums. Nothing serves through it yet, and on one Mac replies are
  the same tokens (#1433, after #1432).

Speed

• MLX 4-bit decode on GB10s: projections that read the same input run as one grouped launch, each output
  byte-equal to its separate launch, for the dense targets, Nemotron, the DFlash2 drafter, Gemma 4 and Spark.
  • Nemotron 3.5 Lightning: +13.3% at 8 streams, +5.9% at 2.
  • Qwen3.8-27B with DFlash2: +4.5% at 1 stream, +3.5% at 4, and +3.7% to +5.0% decoding after a prompt (#1783).
• Shared draft rounds price each stream's drafts first and skip proposal work that would get no verify rows, for
  Nemotron, Qwen's MTP, GLM, Gemma 4 and attached LFM2 drafters, on CUDA and Metal. Nemotron 3.5 Lightning with MTP:
  +9.0% greedy and +4.7% sampled at 16 streams on an M5 Ultra; +2.1% at 16 and +1.1% at 8 on a GB10 (#1779).
• Gemma 4 E4B on Intel GPUs: prompts read 2.0 to 2.6 times faster on an Arc Pro B70, and decode is 4% to 8% faster
  at 32K to 64K tokens, with the same logits (#1746, part of #1222).
• HIP prompt attention on gfx1151: cold first token is 1% to 5% faster from 2K to 64K tokens on Qwen3.8-27B, with
  the same output bits. The tile is now under the kernel contract (#1788).
• HIP prompt projections on gfx1151 round to the activation type in their tile instead of writing fp32 and
  launching a cast, with the same bytes. Cold prompts on Qwen3.8-27B are 2% to 3% faster (#1685).
• Qwen3.8-27B on M5 Ultra: a prompt chunk of more than 128 rows reads each 4-bit weight matrix once per chunk
  (#1738).
• Qwen3.8-27B on Metal with DFlash2: the drafter reads its sliding-window keys and values in place while the
  context hasn't wrapped, instead of copying them each round (#1727).
• Flash Next's MTP head norm: 1.4x to 2.5x faster on a GB10 and 4x to 5x on a Radeon AI PRO R9700, at 1 to 16
  rows, with the same bytes (#1702, fixes #1689).
• EXL3 MoE models on sm_120 and sm_121 (GLM-5.3-Flash and the Qwen3.5-MoE EXL3 packs): the grouped experts kernel
  loads a warp's next tile while it decodes the current one, with the same output bits. 11% faster at
  MiMo-V2.6-Flash's expert shapes on a GB10. GLM-5.3-Flash on two GB10s is within 1% of its previous speed (#1505).
• DeepSeek-V4.1 on CUDA: each window backs up only the ratio-2 recurrent carries, not every layer's sliding ring
  (89 device copies, 6.4 MB a window on Flash-1017) (#1752). DSpark draft passes no longer wait for every side stream
  (#1751).
• Flash Next on CUDA, opt-in: TF_FLASHNEXT_CHUNK_SOLO (up to 16,384 rows) lets a lone prompt take a larger chunk.
  At 4,096 rows on a GB10, its first token comes 5% to 8% sooner. It costs 2.25 GiB more scratch, and a request that
  arrives during a large pass can wait up to about 2 s (#1733, option C of #1554).
• GLM-5.3-Flash on CUDA, opt-in: TF_GLM_CHUNK_SOLO=4096, set on every rank, lets a prompt pass take 4,096 rows
  while no other stream is decoding, with the same bytes. On GB10s a cold prompt's first token comes 6% to 9% sooner
  from 8K to 128K tokens. It holds 1.6 to 1.9 GiB more for prompt buffers, and a request that arrives during a long
  pass waits for it to end (#1823, option C of #1554).
• Lanes, opt-in: an external drafter's own chain chances price its verify rows, and a stream given no rows is
  never offered a hold. Defaults are unchanged (#1381).
• Flash Next on Macs, opt-in: TENSORFOLD_PREFILL_POLICY=fcfs serves long prompts that arrive together
  oldest first, in chunks of TENSORFOLD_PREFILL_TOKENS rows (default 2,048) between decode rounds, with the same
  tokens. With eight 14K-token prompts at once on an M3 Ultra, the last user's first token comes after 93.6 s
  instead of 326.9 s. Each chunk pauses the streams decoding beside it for about 1.7 s (#1843).

Memory and long conversations

All of these are opt-in except Spark 2.5's paged layers and DeepSeek-V4.1's prefill planner. Unset, nothing else changes.

• GLM-5.3-Flash prompts in the KV pool: TF_GLM_KV_SHARE=n with TF_KV_PAGES=on keeps prompts as the pool's
  pages in the shared prompt store, on one GPU and at tensor parallel. Token-id prompts resume at every width (#1732,
  #1510).
• GLM-5.3-Flash prompt states in host memory: TF_GLM_HOST_CACHE=1 on every rank keeps them in pinned host memory,
  sized by --prompt-cache-gib. On discrete cards with little device room, a long conversation's state is kept instead
  of refused (#1736, #1735, #1510).
• GLM-5.3-Flash vision tower in host memory: with --vision, TF_GLM_VISION_HOST=1 leaves rank 0 holding 240 MiB
  for the tower instead of 1.05 GiB, and every rank gets that room back for KV pages and the prompt cache (#1760).
• Qwen3.8-27B and Qwen3.6-35B-A3B on CUDA: with TF_QWEN_KV_PAGES=n and TF_QWEN_KV_PAGES_ALL=1, the engine's
  own stream becomes a pool lane too, so no full-context window sits idle beside the pool: 1.90 GiB back for the 27B
  at --parallel 8 --context 16384 (#1750).
• Spark 2.5 on CUDA:
  • The full-attention layers' K/V live in pages of the core pool, leased per stream as it grows, so streams no longer
    reserve a whole context up front. Tokens are unchanged and speed is level on a GB10.
    TENSORFOLD_SPARK25_KV_PAGES=0 keeps the linear caches (#1653).
  • TF_KV_SHARE=n keeps prompts as the pool's pages, so a conversation resumes at the store's last 256-token mark
    without a full copy (#1756, part of #1657).
• DeepSeek-V4.1's prefill planner now sizes the indexer's scratch before it plans, so ragged prompts can't plan
  less than they use. It takes about 128 MiB more per rank on the Flash-1017 pack (#1681).

Serving and API

• GET /v1/load and scheduler load in /metrics: one JSON snapshot under the /metrics access rule.
  • It reports the server's identity, whether --idle-unload has parked the engine, requests in progress, queued
    prompt tokens, requeues and tokens as the work happens. On CUDA it also reports the KV mode, free lanes and
    GLM-5.3-Flash's page pool.
  • A value an engine can't count is left out, not shown as zero.
  • /v1/load answers in under a millisecond when idle, and replies are unchanged (#1655, #1558).
• Top-level thinking switch: the server reads {"type": "disabled"} or "enabled", as DeepSeek's API sends
  it. chat_template_kwargs still wins when both are set (#1742).
• Strict tools under tool_choice: "auto": a tool marked "strict": true is held to its schema on every family
  with a grammar runtime. TENSORFOLD_TOOL_GRAMMAR=1 holds every auto tools request the same way (#1688).
• DeepSeek-V4.x tools: tool_choice: "required" and named calls are held to the model's own DSML markup (#1684).
• DeepSeek-V4.x stats: replies carry the Spark recipe's per-response stats in their tensorfold block.
  TENSORFOLD_WIRE=spark turns them on for any family, and native turns them off (#1753).
• Living Weights: DELETE /v1/slide/facts takes a list of fact ids and removes them as one reversible change. A
  single delete is unchanged (#1816, part of #678).
• Living Weights on CUDA: Nemotron Omni learns through the shared Nemotron learner. A lesson that carries an
  image or audio is refused by name instead of training through the media towers (#1815, part of #1801).
• Windows: the grammar library loads as tfgrammar.dll, so structured output and required or named tool calls work
  there too (#1796, fixes #1789).
• Log times: TENSORFOLD_LOG_TIME=1 starts each log line with its UTC time to the millisecond. Off by default
  (#1690, fixes #1626).
• Grammar timing: lane round metrics now include the time spent filling grammar masks. Nothing is read when no
  stream is constrained (#1814, part of #1455).

Fixes

• Flash Next on a GB10 keeps its long-prompt speed. A loaded server left about 36 GiB of its checkpoint in the page
  cache, and on a GB10 that memory is the GPU's too: under 1 GiB stayed free, and from some point in a server's life
  long prompts ran at about 0.73x until a restart. The server now drops those pages once the weights are on the GPU,
  with the same tokens (fixes #1868).
• Session disk: a fresh start clears the .tfs and .tmp files earlier processes left, so restarts no longer
  fill the disk (#1680).
• DeepSeek-V4.1 at tensor parallel: every rank charges disk eviction the same size, so the ranks' caps no longer
  drift apart, and a restore checks the leader's identity first (#1679).
• CUDA graphs: an invalidated capture no longer leaves its stream marked as capturing, which had refused every
  later capture and host upload with UnownedCaptureSource (#1744).
• CUDA tensor parallel: a rank that can't bind its liveness port (NCCL_COMM_ID's port + 1, often held by an
  earlier server still exiting) now says so, instead of a generic "glue kernels" error. So do a rank that never
  arrives and a failed accept (#1813, fixes #1761).
• Mixed-bit Qwen3.8-27B on M5: checkpoints that mix bit widths in a fused projection, such as the oQ4 and oQ2
  quantizations, answer prompts longer than 128 tokens instead of failing with BadCoreProjectionRows. Replies keep
  main's tokens (#1822).
• Living Weights lessons no longer stop at their first step with LayerRoundsAgain (2 of 7 Nemotron lessons on a
  Mac, and both on a GB10, did before). A lesson that still can't round says so in words (#1810, fixes #1807).
• Old CUDA drivers: a driver older than CUDA 13.0 is refused at startup with a message naming R580 or newer. An
  R550 driver had crashed with SIGILL on the first module load (#1791, #1724).
• --cortheon: text beside an image counts toward the room for a fetched page. Progress lines no longer cut a
  character in half, and a fallback search that can't fit its first word is skipped instead of sent empty (#1677).
• Intel GPUs on WSL2:
  • tensorfold-xpu devices and the device tests no longer crash with an Iris Xe (#1672).
  • Free memory falls back to the device's total less what TensorFold allocated, so Nemotron with MTP can size its
    prompt cache (#1691).
• DeepSeek-V4.1 packs: with TF_DSV41_PF_4K=1, aot-needs lists the 2,048-row prefill variants too, so the pack's
  first operation no longer fails with MissingTritonVariant (#1683).
• Windows: the cost cache reads its tables through a shared file handle, so a table being replaced reads as a
  quiet miss, not an unexpected status (#1809, fixes #1805).
• Kernel contract on M5: the contract now states the tensor units' 10-mantissa-bit cut of fp32 operands for every
  Metal kernel with one, each with a control that fails without it. No output changes (#1748, part of #1546).
• Tests: the CUDA drafter cost-identity host test passes on filesystems with a coarse clock, such as ramfs and NTFS
  (#1770, fixes #1697).

What can change for you

These change what some requests get, or stop a start that used to go ahead:

• DeepSeek-V4.x without sampling fields, on a checkpoint with no generation_config.json, now samples at
  temperature 1.0 and top_p 1.0, as DeepSeek's model cards recommend, instead of decoding greedily. That file, the
  serve flags and the request still override them (#1725).
• Strict tools under tool_choice: "auto" are held to their schema (#1688), and DeepSeek's required and named
  tool calls to DSML (#1684).
• A top-level thinking switch now takes effect, where before it was ignored (#1742).
• Streamed reasoning: the joined delta.reasoning_content pieces now equal the finished reply's
  reasoning_content byte for byte, without the whitespace the finished reply strips. A streamed /v1/responses
  reasoning item matches the non-streamed one (#1743).
• --cortheon with images gets a smaller fetched page when the turn's text is long (#1677).
• --slide on a family that doesn't learn on that backend now stops before loading, on CUDA, Metal, HIP and XPU.
  Before, it served and learned nothing (#1800).
• A CUDA driver older than 13.0 is refused at startup (#1791).
• Living Weights lessons: new fits are written on the 4-bit grid before they're judged, so a lesson's steps and
  outcome can differ from before, and the same lesson can take different steps on separate runs (fixes #1807).

Known issues

• DeepSeek-V4.1: the prompt cache resumes only at 2,048-token boundaries (#1625).
• Living Weights on Nemotron 3.5 Lightning: a one-fact lesson is refused in a third to a half of runs, with or
  without --no-thinking. A refused lesson is taken out cleanly, and unrelated answers stay the same (#1587).
• Qwen3.8-27B learning on CUDA: the learner's gradient is 1.6 to 1.8 times too large at a prompt's last row. The
  learning gates stop the bad lessons, so serving without lessons is unchanged (#1806).
• GLM on Metal: an idle server's memory grows, about 4.6 GB an hour, until a restart frees it. With
  --learn-conversations, a prompt state restored from disk was charged less than it uses; the fix is in review
  (#1572, #1818).
• Qwen3.5-4B on a 4 GB GPU: reported to fit on 1.1.2 but to be refused at load from 1.1.5 on (RTX 3050 Laptop).
  The likely cause is prompt scratch sized for 4,096 rows whatever the context (#1827).

Thanks

• @AbrahamCete (#1685, #1788)
• @ajensenwaud (#1505)
• @akol1 (#1433)
• @atakann (#1690)
• @Bizuayeu (#1732, #1823)
• @BlivionIaG (#1381, #1513)
• @BobClawblaw (#1388, #1512, #1528, #1687, #1748)
• @florianbrede-ayet (the downstream fix #1679 adapts)
• @gilby (#1705, #1706, #1707, #1708)
• @grantoverton (#1822)
• @ivanfioravanti (#1843)
• @jayleaton (#1679, #1680, #1681, #1683, #1684, #1688, #1725, #1743, #1744, #1751, #1752, #1753)
• @Jenpo (corroborating #1680)
• @jschmied (#1513, #1653, #1672, #1673, #1691, #1733, #1750, #1756)
• @kieranscotney (#1727)
• @laudney (#1655, #1674, #1675, #1702)
• @mikolaj92 (the downstream fix #1680 ports)
• @Mirrdhyn (#1746)
• @Nipale-ai (#1738)
• @ppardus (#1677)
• @rootsbymenda (#1791, and the report of the diverging caps behind #1679)
• @russellmm (#1736, #1760)
• @ThomasHalwax (#1385)
• @wesley4546 (#1770, #1796)

Release reads against 1.2.0 (1 Nov: main at 9e8c368, then the #1868 fix A/B)

• Qwen3.8-27B MLX 4-bit on an M5 Ultra: GREEN. Prompts 9.4% to 10.8% faster at every length (for example 1,716 -> 1,901 tok/s at 2K), decode sweeps flat to +2%, decode after a prompt within the runs' own spread.
• Qwen3.8 Flash Next MLX 4-bit + MTP on a GB10: GREEN after a ten-run alternating check. Prompts and decode within 1.5% of 1.2.0. The long-prompt slowdown (#1868) hit both builds alike (1 of 4 scored runs on 1.2.0, 1 of 5 on main); it's fixed in this release.
• Qwen3.8-27B MLX 4-bit on a GB10: GREEN. Prompts level with 1.2.0's later runs (its first run is always its fastest), decode sweeps none slower.