TensorFold 1.0.12

TensorFold 1.0.12

1.0.12 brings NVIDIA's Nemotron 3.5 Super VL to CUDA, on one GB10 or two, with image input and NVIDIA's own FP4 and FP8 math. On Macs, Qwen3.8-27B and GLM-5.3-Flash read images, Nemotron serves oQ and other mixed-width checkpoints, and LiquidAI's LFM2.5-24B-A2B serves at 4, 6 and 8 bits. Recall, off unless you ask for it, lets drafts copy from text the server has read before, and Living Weights can now undo its lessons. Intel Arc and AMD GPUs get previews you build from source. In the release gate the 27B, Nemotron and Flash Next gave 1.0.11's replies token for token at 1.0.11's speed, and Super VL gave the same tokens on one GB10 and two.

Nemotron 3.5 Super VL on CUDA:

• nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4 (revision 82d93699) serves as stored on one GB10, or on two with rank 0 serving and rank 1 holding only its share of the experts (31.5 GiB, against rank 0's 40.1 GiB). Each expert's width is cut into two shares that every setup sums in the same order, so two GB10s give one GB10's tokens: all 17 replies to the eight release prompts matched, greedy, seeded sampled and with thinking on. MTP drafts use the target's own output head. Served at a 16,384-token window, drafted replies equal "draft": false, 2 and 4 concurrent streams equal their solo runs, and chats with regenerate, edit and delete equal a cache-off server's replies, on one GB10 and on two. The recipe has both setups.
• --precision checkpoint|full picks the math. checkpoint, the default on Blackwell, is NVIDIA's own: e4m3 rows into the FP8 layers and NVFP4 rows into the NVFP4 experts, on the FP8 and block-scaled FP4 tensor cores. full keeps bf16 rows (W4A16 and W8A16). Each mode is exact within itself, and they give different tokens from each other. On one GB10, checkpoint teacher-forced on full's greedy 256 tokens after three prompts:

Prompt	Mean KL (nats a token)	Largest KL	Same top token	NLL of the fed tokens, full to checkpoint
Python code	0.0036	0.25	255 of 256	0.0480 to 0.0482
Long prose	0.0137	0.22	246 of 256	0.1416 to 0.1514
Repetition	0.0057	0.50	255 of 256	0.0179 to 0.0252
All 768 tokens	0.0077	0.50	98.4%	0.0692 to 0.0749

• checkpoint read a cold 8,192-token prompt in 4.41 s against 7.96 s (1.81x, and 1.82x at 2,048 tokens), and decodes a little slower (52.3 against 50.1 ms a token serial), because it quantizes each row on the way in. tensorfold kl compares two precisions on the same tokens. Lightning's NVFP4 checkpoint keeps full for now.
• Speed, greedy, served, the mean of the eight release prompts:

Setup	Precision	MTP drafts	"draft": false
One GB10	checkpoint (default)	38.2 tok/s (29.8 to 49.0)	18.8 tok/s
One GB10	full	38.7 tok/s (30.2 to 50.8)	19.4 tok/s
Two GB10s	checkpoint (default)	46.8 tok/s (35.9 to 63.6)	20.2 tok/s
Two GB10s	full	48.1 tok/s (36.4 to 65.6)	21.4 tok/s

• Images run through the checkpoint's own RADIO tower and projector, in our kernels with the bf16 weights as stored. Each image is one tile of 1,024 to 13,312 patches, resized and normalized as NVIDIA's processor does: on a GB10 the pixels equal the processor's bit for bit on six pictures, five resizes and 3,000 patch grids, and the tower's error against a float32 run of the checkpoint's own modules is 0.99x to 1.03x that of their own bf16 run. A 13,000-patch image takes 0.87 s through the tower and projector. With an image, drafted replies equal "draft": false, four concurrent image requests equal their solo runs, a later turn resumes past its image while another image under the same placeholders resumes only the text before it, and two GB10s give one GB10's replies word for word. --vision-image-tokens defaults to 13,312 for Super VL, so up to four images keep the size NVIDIA's processor gives them: four full-size pictures (13,124 prompt tokens) got their first token in 11.45 s on one GB10. JPEG needs libjpeg-turbo 3.1.4.1 installed (PNG does not); video and audio parts are refused.
• Memory on one GB10: 70.8 GiB for the weights, 73.2 GiB with the image tower and its rows, 8 streams of 0.35 GiB at the default window, and about 24 GB of the GB10 left free. On two GB10s, before any weight loads, the ranks check that they run the same build, model, cut, precision and checkpoint bytes, so a copy one bit off stops both ranks with its name, and a rank that dies stops the other within about a second instead of leaving it waiting.
• On an RTX PRO 6000 Blackwell (sm_120), a build of the same work gave the GB10's KL and top-1 figures to the digit and the GB10's tokens in all 17 cells, and checkpoint read prompts 1.71x to 1.77x faster than full. This release's Linux x86_64 archive on that card gives the same 17 replies, passes the image cells, and decodes at 124.1 to 223.1 tok/s drafted and 62.2 plain.

Other new models and hardware:

• Qwen3.8-27B reads images on a Mac with --vision (--vision-max-images, --vision-image-tokens). Images in user messages and tool results are prepared as transformers' Qwen2-VL processor prepares them, and the checkpoint's image tower runs on the GPU in our kernels (bf16 weights as stored, fp32 sums and residual). Its rows take the placeholders' places with M-RoPE positions, through decoding, drafts and later turns. On an M3 Ultra six test pictures got the processor's grids, tokens and patches bit for bit, and rows closer to transformers' float32 tower than its bf16 tower is (mean row error 1.0 to 2.2% against 1.6 to 4.1%). With an image, drafted replies equal "draft": false, four requests at once equal each alone, and later turns resume past the image and equal a fresh server's replies; on an M5 Ultra the same checks passed, short of the fresh-server comparison. Text-only requests keep their tokens. The tower, its scratch and a request's image rows take 1.79 GiB at the default 4,096 image tokens, counted against the 70% load limit. It carries pieces of #726. Thanks to @BobClawblaw.
• LFM2.5-24B-A2B serves on Macs from LiquidAI's MLX checkpoints at 4, 6 and 8 bits, with drafts and prompt reuse. On an M3 Ultra one stream decoded at 194, 168 and 160 tok/s, cold prompts read at 4,421 tok/s at 2k tokens and 3,549 at 32k, and multi-turn chats gave a cache-off server's tokens at every step (#822, #823). It runs on new prompt tiles for Macs without tensor units (#821). Thanks to @BlivionIaG.
• GLM-5.3-Flash reads images and videos on a Mac with --vision. On an M3 Ultra with 8 of its layers, an image drafted equals "draft": false, four image requests at once equal each alone, and a later turn resumes past its image. --vision-urls fetches public HTTPS images with every address and redirect checked. Video needs the optional libtfvideo library over FFmpeg; in the author's runs on an M5 Ultra it decoded the same frames as PyAV on all 1,925 (#598, #614, #618). Thanks to @BobClawblaw.
• Nemotron serves oQ and other mixed-width MLX checkpoints on Macs: TensorFold's oQ4, oQ6 and oQ8, and any MLX affine checkpoint at 2 to 8 bits (not 7) in groups of 32, 64 or 128, each matrix at its own width. 1.0.11 refused all of them. A matrix at 4-bit in groups of 64 runs on the kernels it ran on before. On an M3 Ultra, oQ4, oQ6 and oQ8 each gave drafted replies equal to "draft": false on 17 prompts, 2 and 4 concurrent streams equal to their solo runs, and 12 resumed chat turns equal to a cache-off server's. The oQ checkpoints carry no MTP head, so copy mtp-4bit.safetensors from the 4-bit checkpoint into the folder to draft. Against mlx_lm's forward of the same checkpoint, teacher-forced on 3,072 positions:

Checkpoint	Mean KL (nats a token)	Same top token	Prompts of 8k to 32k tokens, against mlx_lm	Greedy prose / code, tok/s
oQ4	0.0054	96.6%	0.96x to 0.97x	162 / 249
oQ6	0.0050	96.2%	0.97x to 0.99x	117 / 183
oQ8	0.0053	97.0%	0.97x to 0.99x	111 / 174
4-bit (for scale)	0.0056	96.2%		188 / 268

CUDA and the Intel GPU engine read Nemotron at 4-bit in groups of 64 only and refuse other widths by naming the first such module, as --slide and learned neurons beside a shared expert do on Metal. tensorfold-native capabilities --json lists every MLX width for Nemotron, Flash Next and qwen3_5 checkpoints on Metal, and the app's download check reads that list, so it no longer turns away the 27B's oQ6e or Flash Next's oQ checkpoints, which the engine already served. It builds on the oQ kernels from @grantoverton's series (#716).

• d1-3B, Liquid AI's LFM2-VL decision model, answers POST /v1/systemone and /v1/decisions on CUDA, pictures included. On a GB10 it gave the reference's top answer on 98 of 99 questions, one question in 24.8 ms at the median (#842, #843, #844). Thanks to @plotarmordev.
• The Linux archives' CUDA kernels now carry Ampere sm_80 (the A100) beside sm_86, sm_120 and sm_121. ModelOpt FP8 and NVFP4 checkpoints still need compute capability 9.0 or newer, and a card below it is refused with the reason (#846). Thanks to @liumorrisclaw.
• Previews you build from source, not in the release archives. Intel Arc: Qwen3.8-27B and Nemotron 3.5 Lightning over Level Zero; on the authors' Arc Pro B70s the 27B with DFlash2 decoded code at 122.9 tok/s and chat at 99.3, and Nemotron ran plain greedy at about 173 tok/s (#802, #523, #547). AMD (HIP): Qwen3.5, Qwen3.6 and Flash Next, on one GPU or split over 2 to 8, with the author's checks passing on Radeon PRO W7800 (gfx1100) and V620 (gfx1030) cards and the kernel checks on an R9700 (gfx1201) too (#617, #621, #626, #631, #655, #697, #812 to #818). GLM-5.3-Flash on CUDA from NVIDIA's NVFP4 checkpoint at tensor parallel over hosts passed its lane checks on two GB10s, and its first four layers served; the whole checkpoint has not been served on our hardware yet (#804, #593). We have no Intel or AMD GPU, so those land on the authors' receipts and our host checks. Thanks to @Mirrdhyn, @BHCC2025, @BlivionIaG, @ThinkOffApp and @Bizuayeu, and to @jkuepker for the gfx1201 run.

Faster:

• Nemotron on CUDA: the router, top-k and norm kernels keep their bytes and take less time, so on a GB10 one stream decodes about 0.9% faster drafted from the router work and 0.2% to 0.5% from the norm work, with cold prompts and shared rounds level (#833, #796). Thanks to @jschmied.
• EXLR8's one-set projection takes 2% to 5% less GPU time at one row and up to 2% less at two on an M3 Ultra, with the same bytes (#789). Thanks to @mrv777.
• Flash Next on a Mac keeps its memory wired between requests for --keep-warm seconds (15 minutes by default): on an M3 Ultra a cached prompt's first token after 2 to 10 s idle came in 0.13 to 0.15 s instead of 1.95 to 2.61 s. The 27B's residency set now holds its kept prompt states too (#732). Thanks to @sethforprivacy, @BobClawblaw and @cwschroeder.
• Flash Next prefills each reply for the next turn in the background, as the 27B and Nemotron do: follow-ups on an M3 Ultra started in 0.17 to 0.19 s instead of 0.41 to 0.46 s, with the same tokens.
• LFM2.5-24B-A2B decode windows of 8 to 32 rows read each weight once through a matrix window kernel with the one-row twin's bits, windows of 16 rows or more read each picked expert once for all its rows, and its prompts run on an M5's tensor units at every prompt row; in single runs on an M3 Ultra, two shared 4-bit streams ran about 11% faster and four about 17% (#867, #868, #873). On an M5, core/affine_mm loads its code tiles in two 8-byte reads (#871). Thanks to @BlivionIaG.
• GLM-5.3-Flash's sparse prompt attention on a Mac reads a row's key list only as far as its last listed key, with the same bits: on an M3 Ultra cold prompts read 11.7% faster at 512 tokens and 7.3% at 2,048 (#898). Thanks to @harrisonfriia.
• Tool calls parse in one pass: a Gemma 4 call's strings, a reply of many DSML calls and a GLM call of many keys no longer search the call again for each piece, with the same arguments (#881, #882, #883). Thanks to @jishnuvenugopal.

Off by default:

• Recall (--recall) keeps a record of the token ids the server has read and written and lets drafts copy from it; every copied token is verified, so replies are the same tokens with it on or off. Four requests repeated after a restart ran 2.4x faster with Qwen3.8-27B on an M3 Ultra and 3.0x faster with Nemotron on a GB10, and fresh prose and chat stayed within 0.3% of Recall off. The record has a size cap, and DELETE /v1/recall empties it. Recall says what stays on disk.
• --learn-conversations keeps prompt states on disk, so a turn after a restart, a crash or an eviction can read them back: a 39,228-token Nemotron turn on a GB10 started in 0.25 s after a restart instead of 6.2 s cold, and a 21,306-token 27B turn on an M3 Ultra in 0.83 s instead of 103 s, each with the cold reply's tokens. Writes are capped a day.
• Living Weights keeps a way back. Before a folder's first lesson it keeps the files a lesson can change, as clones where the file system has them (5.3 s for 18.5 GB on an M3 Ultra, and no disk) or as copies (18.5 GB of disk on a GB10's ext4), and tensorfold slide rollback FOLDER puts them back after checking their hashes: the folder comes back byte for byte and gives the untaught replies.
• Living Weights deletes one fact with DELETE /v1/slide/fact/{id}, putting every bit back if its checks fail. On an M3 Ultra and a GB10 the deleted fact's answers stopped naming it and the 14 other replies stayed the same (#872). Thanks to @ThinkOffApp.

Fixes:

• Flash Next on a Mac samples when a request asks for a temperature, drafted, resumed and concurrent draws matching their references at the same seed; sampled prose runs at 88 tok/s against 96 greedy on an M3 Ultra. Speed-up mode stays greedy (#719). Thanks to @BobClawblaw.
• GLM-5.3-Flash samples every target row with the shared keyed sampler: the first reply token and plain rounds no longer fall back to greedy, and a drafted reply equals the same request with drafts off at any temperature. Two Macs still refuse sampled requests (#892). Thanks to @heitke.
• GLM-5.3-Flash's KDA and router kernels ask the Metal compiler for the 1,024 threads they launch with. An M1 Max had built some of them for fewer, which could leave outputs unwritten without an error; hc_expand_10, which offline M1 and M2 compiles built for 928, now asks too (#708). Thanks to @mrv777.
• GLM-5.3-Flash checkpoints with 5- and 6-bit experts gave empty replies from about 12,000 prompt tokens; the last expert's pairs now run (#864). A kept state that goes stale leaves the prompt cache at once, so a 3,794-token conversation resumed 3,781 tokens after an interruption instead of prefilling twice (#835). Thanks to @harrisonfriia and @heitke.
• LFM2's 4-bit decode twin turns each code into its integer as the prompt tiles do, so a tiny product no longer underflows to zero (#821). Thanks to @BlivionIaG.
• GLM-5.3-Flash loads on a Mac whose page cache holds other files: on an M3 Ultra with 158 GB of other checkpoints cached, a whole-model load is ready in 32.7 s where it stalled for minutes. A Nemotron load on Metal that fails after its kernels compile releases them (#875). Thanks to @jishnuvenugopal.
• Living Weights asks a fact's questions about other subjects by each wording's whole subject, keeping possessives and longer names intact (#714). Thanks to @cwschroeder.

Tools and API:

• GET /v1/models lists the reasoning efforts each model's template takes, so a client can build its effort picker from the server (#819). Thanks to @kelchm.
• tensorfold benchmark MODEL runs the community suite and saves a receipt. Nothing is uploaded unless you ask, after a preview and your consent (#717). Thanks to @ThinkOffApp.
• Replies carry the full SHA-256 of their token ids, and the benchmark tools count two replies as equal only by it (#862, #863). Thanks to @jayleaton.
• The benchmark clients send a bearer key from the environment variable --api-key-env names, never on a redirect, and keep it out of everything they print; a stream that ends without data: [DONE] counts as failed (#889, #890). Thanks to @alenperic.
• --record DIR writes each stream's committed tokens and the rows its drafter reads, for drafter training: Nemotron on CUDA records its final norm's rows, and the other families refuse the flag by name (#848). Thanks to @jschmied.
• The dashboard shows a filling prompt's tokens left and how long the prompt queue takes to drain, and on a Mac the Memory tile shows the footprint, its peak and the load limit (#738, asked for in #772). Thanks to @heitke.
• With no API key configured, /health names the release, the backend and the GPU (#854), and the API guide lists its fields (#866). Thanks to @ThinkOffApp and @jishnuvenugopal.

Safety:

• --slide starts only with an API key. A server holds its model folder while it runs, so a rollback refuses a folder a server holds and a --slide server refuses one a rollback holds. The hold is advisory: tools other than TensorFold do not see it.
• TENSORFOLD_REQUEST_LOG creates its file readable by its owner only (0600). A log that already exists keeps its mode; chmod 600 it once. Recall's files are owner-only too.
• Every residency set stays inside 70% of RAM with a 2 GiB margin, or is refused with a log line that says why. Image towers, their scratch and image rows count against the load limit and the prompt cache's budget.

For contributors:

• CUDA: ModelOpt NVFP4 checkpoints read as stored, fp32 outputs from the NVFP4 matmuls, expert packing without scratch copies, prompt entries that skip an expert, page-cache release after a load, family-declared prompt caches, and a rank-ordered reduce-scatter over NCCL (#758, #757, #763, #764, #766, #762, #642). Thanks to @Bizuayeu, @cwschroeder and @SvangenStudios.
• Metal: prompt tiles for Macs without tensor units, a dequantizer for any code width from 2 to 8 bits, and stacked experts for any checkpoint (#821, #810, #811). Thanks to @BlivionIaG.
• EAQ packs are checked whole before a loader reads one (#858). Thanks to @MiaAI-Lab.
• tf-merge-check --quantizer quantizes dense tensors to MLX affine 4-bit as MLX does and counts the codes, scales, biases and groups that equal a shipped checkpoint's (#709). Thanks to @philip-pentatonic.
• DeltaNet's wide prompt scan can keep states at marks inside a segment; no model calls it yet (#851). Thanks to @jschmied.
• Pictures decode in one shared place, PNG from the specification and JPEG through libjpeg-turbo 3.1.4.1 loaded at run time, to the pixels 0.6.6 got from Pillow. The server's JSON values are one module (#860). Thanks to @jayleaton.
• The kernel contract covers this release's new Metal and CUDA kernels against float64, and their launches against every chip they ship for, sm_80 included.

Behaviour changes:

• --slide refuses to start without --api-key, --api-key-file or TENSORFOLD_API_KEY.
• Flash Next on a Mac samples a request with a temperature, where it refused one before. Speed-up mode still refuses a temperature.
• Flash Next on a Mac keeps its weights wired between requests (171.7 GiB for the 6-bit checkpoint) for --keep-warm seconds after each request. FLASHNEXT_NO_RESIDENCY=1 turns that off.
• New request logs are created 0600.
• /health with no keys configured carries version, backend and device, and /v1/models entries carry a reasoning object.
• The Linux archives' kernels carry sm_80, and a ModelOpt checkpoint on a card below compute capability 9.0 is refused with the reason.
• Nemotron on a Mac admits mixed-width and 6- or 8-bit MLX checkpoints. On CUDA and the Intel GPU engine such a checkpoint is refused by name; the Intel GPU engine had no check and would have read it at the wrong width.
• A --slide folder holds a hidden .tensorfold-original/ snapshot, a full copy on file systems without clones. tensorfold info leaves hidden folders out of a model's size.

Known issues and owed items (docs/known-issues.md has the full list):

• Super VL: Lightning's NVFP4 checkpoint keeps full precision, and the llama.cpp-style 8-bit activation mode is not built yet. --learn-conversations keeps nothing on disk for Super VL yet, and --slide is refused for it. Its MTP head drafts an image's prompt from the placeholder ids, which costs acceptance but not exactness.
• Nemotron oQ: not run yet on an M5's tensor units; oQ6 and oQ8 decode within 5% of each other, paced by the routed experts' row kernels; the first prompt after a server start reads 0.3 to 0.5 s slower than on the 4-bit checkpoint.
• Images: Qwen3.8-27B reads images on Macs only (its CUDA build refuses --vision at startup), video parts are refused for the 27B and Super VL, and JPEG needs libjpeg-turbo 3.1.4.1, which TensorFold does not ship. tensorfold serve --help does not list --vision, --vision-urls, --vision-max-images, --vision-image-tokens or --vision-video-tokens, though the server takes them; docs/vision.md describes each.
• GLM-5.3 (EXLR8) decodes slowly next to the other models, and its replies past 2,048 tokens have not been compared with a whole-model reference. Qwen3.5-2B can keep adding digits to an integer field under a JSON schema. The 27B on an M1 to M4 Mac drafts greedy code shallower after a few prose replies, about 4% slower with the same tokens. On a 32 GB M5 or M6 Mac the 27B's Neural Engine prompt share can take the server past the load limit and into swap, because its fit check runs before the drafter and stream buffers load; TF_ANE_COLUMNS=0 keeps prompts on the GPU until the fix lands (#643).
• Flash Next: in speed-up mode a drafted request right after a "draft": false one decodes slowly; a turn more than --keep-warm seconds after the last request takes about 2 s to its first token; and on M1 to M4 its uniform 6-bit checkpoint gives different tokens from 1.0.9 since 1.0.10 (FZ_M3_LANES=1 keeps the earlier kernels).
• Living Weights: facts about one subject can overwrite each other and pile up badly past a few dozen, learned facts cost decode speed on a Mac, a rollback undoes every lesson at once, and a deleted fact's neurons are silenced rather than erased. Teach a copy of the checkpoint folder.
• --cortheon: the instruction filter is a heuristic that reads each line on its own.
• Since 1.0.11's notes the list adds Living Weights' rollback, delete and folder-hold limits, the snapshot's full copy on ext4 and the 27B's Neural Engine share on a 32 GB Mac, and Flash Next's slow first token now comes only after --keep-warm seconds idle, not a few seconds. One issue left the list: a lesson can now be undone.

Not run for this release:

• No M5-generation Mac and no two-Mac pair ran the release build.
• GLM-5.3-Flash, GLM-5.3, LFM2.5-24B-A2B, d1-3B, Nemotron's oQ checkpoints and the image input of the 27B and GLM-5.3-Flash were not served from the release build. GLM-5.3-Flash (8 of its layers), LFM2.5 at 4 bits and oQ4 were served from the release candidate's code with the tokens of their own gates, and the figures above come from each one's own gate.
• No A100 (sm_80), RTX 30-series (sm_86) or RTX 50-series and RTX PRO Blackwell (sm_120) GPU ran the release build. Its 44 CUDA kernel files carry code for each, checked with cuobjdump.
• The Intel Arc and AMD previews are not in the archives and were not part of the release gate. They rest on their authors' runs and our host checks.
• Recall, saved conversations (--learn-conversations), Living Weights and --cortheon were not part of the served gate. Their figures above come from their own gates before the build; for --cortheon, which sends real searches, the fuzz harness in safe and fast builds on macOS and Linux stands in.

Tested: the release build ran against 1.0.11 on an M3 Ultra (Qwen3.8-27B with its DFlash2 drafter, Nemotron 3.5 Lightning MLX 4-bit and Qwen3.8-Flash-Next 6-bit) and on a GB10 (Nemotron MLX 4-bit and NVFP4). Every case gave 1.0.11's tokens on both machines. Drafted replies equal "draft": false, resumed prompts equal fresh ones, concurrent requests and 1 to 8 streams equal their solo runs, and 26 structured-output cases are valid. One-stream decode stayed within 1.5% of 1.0.11 or ran up to 3.7% faster, 2 to 8 streams together within 3%, and prompt speed within 1% (Flash Next's 8k prompts 3.4% faster). 64k-token prompts answered on both machines, with Nemotron also answering at 128k and 256k on the M3 Ultra. Nemotron 3.5 Super VL ran from the release build on one GB10 and on two: the same tokens drafted and plain, at TP=1, at TP=2 and as the release candidate's build, and six pictures described the same drafted and plain. The server's golden tests and the fuzz harness, in safe and fast builds, pass on macOS and Linux, and a plain zig build builds every step on both.

Upgrading: --slide needs an API key now; nothing else to change.

Homebrew: brew upgrade tensorfold.