TensorFold 1.2.8

TensorFold 1.2.8

1.2.8 adds new models and hardens the server.

• Models: Recursive's Flint 40B-A2.5B on M5-class Macs; DeepSeek-V4.1 natively on Macs (experimental); the Qwen3 embedding and reranker families on CUDA; OCR through POST /v1/ocr on HIP.
• Server: you can cancel a request by its id. Request bodies, connections, templates, tokenizer patterns and grammars get fixed limits.
• Speed: most of the work is on prompts. Flash Next at --tp 2 on CUDA fills prompts 26.5% to 33.9% faster; Qwen3.8-27B MLX 8-bit prompts on an M5 Ultra are about 33% faster; GLM-5.3-Flash's MTP head drafts only as deep as it pays.

Where a speedup was checked for equal output, its entry says so. Some changes alter limits, refusals or startup by default; they're listed under What can change for you.

• New:
  • Recursive Flint 40B-A2.5B on M5-class Macs, natively from RecursiveSI/flint-int4;
  • DeepSeek-V4.1 on Metal for text prompts (experimental, TF_EXPERIMENTAL=deepseek);
  • Qwen/Qwen3-Embedding-* (/v1/embeddings) and Qwen/Qwen3-Reranker-* (/v1/rerank) on CUDA from the published BF16 weights;
  • POST /v1/ocr in Mistral's OCR contract on the native HIP engine;
  • POST /v1/requests/{id}/cancel;
  • --max-connections, --max-connections-per-address and --trusted-proxy;
  • opt-in TF_GLM_CONTEXT_CEILING, TF_DSPINE_WINDOW and TF_PROMPT_INTERMEDIATE.
• Speed and memory:
  • Flash Next on CUDA at --tp 2: cold prefill of 2k to 32k tokens is 26.5% to 26.8% faster on two GB10s and 30.9% to 33.9% faster on two RTX PRO 6000s.
  • Qwen3.8-27B MLX 8-bit on an M5 Ultra: cold prompts are about 33% faster.
  • Qwen3.8-27B with DFlash2 on an M5 Ultra: a copy-heavy greedy reply runs at about 792 tok/s instead of 241.
  • GLM-5.3-Flash with its MTP head: one greedy stream decodes 16% to 28% faster at TP=2 and 22-46% faster at TP=3.
  • GLM-5.3-Flash with TF_ROCE=1: decode is about 6% faster at one and two streams on three GB10s, and up to 12.9% faster on four.
  • Flash Next on HIP: cold decode at 4 streams is about 45% faster sampled and 18% greedy on a Radeon 8060S.
  • Video input decodes 2.1x to 3.4x faster on frame threads.
• Fixes:
  • Qwen 3.5 on CUDA no longer refuses at startup with --prompt-cache-disk or --learn-conversations.
  • GLM-5.3-Flash at four ranks on a switchless ring names the setting that fixes its start-up.
  • A later --compact-at replaces an earlier one.
  • A token id past the embedding table is refused on HIP and XPU.
  • libtfvideo.so links FFmpeg's x86_64 assembly.

New models and backends

• Recursive Flint on Metal: Flint (40B-A2.5B MoE) is served natively from RecursiveSI/flint-int4 on M5-class Macs, with its kernels and 2048-wide router under the kernel contract, a 262,144-token window, copy drafts and the prompt cache. FLINT_LOAD_LIMIT_GB caps its memory (#1177, fixes #1175). Thanks to @BobClawblaw.
• DeepSeek-V4.1 on Metal (experimental): with TF_EXPERIMENTAL=deepseek, tensorfold-dsv41 (zig build dsv41-metal-serve) serves the release checkpoint natively for text prompts, with DSpark drafts and the prompt cache.
  • On an M3 Ultra and an M5 Ultra with 512 GB, drafts make one stream 1.21x to 1.50x faster than --no-drafts and two streams up to 1.13x faster; three and four streams are level.
  • token_sha256 is equal in every cell (#1720, part of #1676). Thanks to @gilby, and to @3utsu for an independent M5 Ultra receipt.
• Qwen3 embeddings on CUDA: Qwen/Qwen3-Embedding-0.6B, -4B and -8B serve /v1/embeddings from the published BF16 weights.
  • On one GB10 the worst cosine distance to float32 is 1.55e-4, 1.34e-4 and 1.08e-4, against transformers' own BF16 at 1.88e-4, 1.79e-4 and 1.77e-4.
  • A request's vectors equal each input sent alone (#2285). Thanks to @Arminova.
• Qwen3 rerankers on CUDA: Qwen/Qwen3-Reranker-0.6B, -4B and -8B serve /v1/rerank, scoring a request's documents as one decision whose shared start runs once a pass.
  • On one GB10 the worst score from float32 is 5.5e-5, 6.2e-4 and 2.9e-4 on 16 documents of up to 2,579 tokens, against transformers' BF16 at 3.4e-5, 2.8e-4 and 2.2e-4.
  • A document scored alone equals its score in a request of 16 (#2995, part of #2261). Thanks to @Arminova.
• OCR on HIP: the native HIP engine reads page images through POST /v1/ocr in Mistral's OCR contract, so Mistral's Python and TypeScript SDKs work, with dots.mocr's reader giving a box for each layout block.
  • PDF documents, pages and content-changing options it can't apply are refused.
  • A page cut at the context limit fails with HTTP 422.
  • docs/api.md lists the fields, statuses and limits (#2570, #2440). Thanks to @laudney.
• dots.mocr's text decoder on CUDA: it serves text prompts on a GB10 from the publisher's BF16 weights, on a new shared engine for dense GQA decoders. --vision, prompt caching and fp8 caches are refused by name (#2673, part of #1927). Thanks to @laudney.
• MOSS-Image on Metal: tf-moss-image-generate makes a picture from options alone (--prompt, --negative, --width, --height, --steps, --seed, --guidance) at the card's defaults (#2719, for #1531). Thanks to @devopsfortunato.
• Groundwork that no served model selects yet:
  • a DFlash2 draft engine for chains in HIP's qwen3_5 engine: in the drafter test with Qwen3.8-27B on gfx1151, 0.86 of first drafts are right and a round makes about 5 tokens greedy (#2726, part 4 of #1563). Thanks to @AbrahamCete.
  • a bf16 grouped-experts format in the CUDA kernels (#1630). Thanks to @BobClawblaw.
  • a TCP transport for tensor parallel on hosts without RDMA, checked bit for bit over two hosts (#2718, part of #2065). Thanks to @Shoestring-dev.
  • pipelined rounds in the shared stage driver, with tokens checked equal to one engine (#2579, part of #2005). Thanks to @jschmied.

Server and API

• Cancel a request: POST /v1/requests/{id}/cancel cancels a running or queued request by its chatcmpl-, cmpl- or resp_ id. The reply ends cancelled with the tokens it had, and other streams are unchanged.
  • With API keys only a full or [admin] key may cancel.
  • A server without keys answers only direct loopback callers (#3193, fixes #3172).
• Connections: under connection pressure (96 of 128 slots open), one client address holds at most a quarter of the slots, and an idle keep-alive connection gives its slot up only when every slot is taken.
  • New flags: --max-connections, --max-connections-per-address and --trusted-proxy.
  • Below pressure nothing is refused (#3048).
• Limits on untrusted input:
  • a request body no route reads is capped at 64 KiB (#3029);
  • a chat template is held to a nesting depth, step count, stack depth and scratch memory, and renders byte-identically and as fast as before (#3072);
  • a tokenizer split pattern gets a work allowance of 2,048 instructions a byte (#3071), and a token id above 2^24 is refused at load (#3033);
  • a grammar's exploration stops at 2^38 steps, with no time limit, so the answer doesn't depend on the machine (#3037).
• Media, files and sessions:
  • a media URL whose host resolves to many addresses no longer hangs the fetch (#3035);
  • compaction notes and the fact graph are written owner-only through a renamed temporary file (#3031);
  • a crafted session chunk is refused, and the disk tier's files are owner-only (#3030);
  • pull refuses repository files under the engine's reserved names (#3032).
• Cluster:
  • on a real host, a node takes only the nodes its cluster file names (#3039);
  • a file pulled whole is checked against the manifest's hash (#3043), and a pull that would leave a file held only in part is refused unless allowed. No shipped command calls the cluster loader yet (#3074).
• Prompt cache:
  • Intermediate checkpoints (opt-in for Flash Next on CUDA): with TF_PROMPT_INTERMEDIATE=1, a bounded ladder of intermediate checkpoints lets a compacted prompt resume inside its retained prefix. On one GB10 with drafts off, a 104K-token request that kept its first 100K resumed 65,536 tokens instead of 2, cutting time to first token from 50.7 s to 19.9 s with the same tokens (#1953, fixes #1909). Thanks to @javierbits.
  • Shared pages: on the core pool, the radix store holds each pool page once, and a snapshot save can no longer make reclaim prune the leaf it is writing to (#2875, part of #2444). Thanks to @laudney.
• Settings:
  • TF_GLM_CONTEXT_CEILING (opt-in) opens GLM-5.3-Flash windows past 1,048,576 tokens (#2765). Thanks to @Bizuayeu.
  • TF_DSPINE_WINDOW=N (16 to 8,192, opt-in) narrows a Qwen3.8-27B DSpine drafter's window without editing the checkpoint (#2705). Thanks to @jschmied.
  • --profile fast refuses as not planned, and --profile fast-exact names the token gates it still needs (#2794). Thanks to @devopsfortunato.
• Cortheon:
  • page links survive whitespace around href or inside the URL (#3006). Thanks to @ppardus.
  • DuckDuckGo results are kept, with their ads still skipped, whatever the markup's spelling (#2750). Thanks to @ppardus.
• Startup lines:
  • Qwen3.8-27B on Metal says how many streams draft (#2806).
  • GLM-5.3-Flash with --vision and no video library says what --vision-video-tokens sets aside: 128 MiB a stream at the default 16,384, or 512 KiB at 64 (#2813). Thanks to @Bizuayeu.
  • DeepSeek-V4.1 prints its transport on both ranks (#2877).
  • The DFlash2 drafter for GLM-5.3-Flash gives its licence as Apache-2.0 (#3084).

Speed and memory

• Flash Next on CUDA at --tp 2: a prompt pass wider than 64 rows reduce-scatters its routed experts' slots, and replies keep their tokens.
  • On two GB10s, rank 0 sends 3.43 MiB a prompt token instead of 5.81, and cold prefill of 2k to 32k tokens is 26.5% to 26.8% faster.
  • On two RTX PRO 6000s it is 30.9% to 33.9% faster (#2977). Thanks to @swbiggart.
• Qwen3.8-27B on M5 Macs:
  • Prompt lane projections at every width take a chunk's 16-row blocks in one dispatch. MLX 8-bit cold prompts are about 33% faster on an M5 Ultra (769 to 1,029 tok/s at 2K, 769 to 1,027 at 8K), with identical tokens (#2851, for #2818).
  • With DFlash2, one greedy request at the default window uses the long-copy driver again: a whole-file rewrite runs at about 792 tok/s instead of 241 on an M5 Ultra, with the same tokens (#3069, for #3013).
• Qwen3.8-27B on Metal:
  • Prompt passes no longer write a buffer nothing read. On an M5 Ultra with the MLX 4-bit checkpoint, time to first token is 0.45% to 1.16% shorter (#2915, for #2905).
  • A client that sends the last reply back without its reasoning resumes at its history's end. On an M5 Max at 32k with a 5 GiB cache, turns 1-6 of a 7-turn chat took 16.5 s and 18.9 s of summed time to first token against 217 s, with identical replies (#2723). Thanks to @ThinkOffApp.
• GLM-5.3-Flash on CUDA:
  • With its MTP head and --mtp-confidence, a temperature-0 stream drafts the depth its acceptance supports. At --mtp-confidence 0.5, one greedy stream decodes 16% (code) to 28% (chat) faster at TP=2 on two GB10s and 22-46% faster at TP=3 on three, with replies unchanged (#2864). Thanks to @Bizuayeu.
  • With TF_ROCE=1 on every rank, three ranks or more take a one-shot RoCE all-gather, byte-equal to NCCL's. Decode is about 6% faster at one and two streams on three GB10s, and 12.9%, 10.4% and 6.6% faster at one, two and four streams on four. The target's tokens are unchanged (#2205). Thanks to @shanedertrain.
  • EXL3 prompt passes route their experts in one call, with bit-identical rows (#2852, for #2819).
• Qwen 3.5, 3.6 and 3.8 on CUDA, measured with Qwen3.8-27B (MLX 4-bit) on a GB10:
  • Small decode windows are no longer padded to 16 rows: about 1.5% faster at four streams (#2934, fixes #2909).
  • A one-row step no longer copies the recurrent state: 1-2% faster at one stream and about 4% at four (#2933, fixes #2908).
  • Both keep the same tokens.
• CUDA EXL3 grouped experts: Qwen3.6-35B-A3B EXL3 sizes its grouped storage for 36,864 pair rows instead of 163,904 (about 3 GB less), with bit-equal outputs (#2843, for #2823).
• HIP on gfx1151:
  • Flash Next's decode spreads its n-gram gather over the host workers. On a Radeon 8060S at 4 streams, cold decode is about 45% faster sampled and 18% greedy, and the cold 8K first token about 10% sooner, with tokens unchanged (#2695). Thanks to @gilby.
  • Flash Next's MTP head takes a prompt chunk on the prompt tiles, so cold prompts finish about 5% sooner at 8K and 32K. Replies are unchanged (#1878). Thanks to @gilby.
  • Flash Next's verify windows run their DeltaNet from the round's plan: decode is 2–5% faster at one to four streams, and drafted replies equal drafts off (#2728). Thanks to @gilby.
  • The routed prompt tile splits small experts: with Qwen3.6-35B-A3B (MLX 4-bit) at one stream, cold prompts of 64 to 256 tokens are 11% to 22% faster, with tokens equal to main's (#2878, fixes #2795). Thanks to @AbrahamCete.
• Intel XPU, Arc Pro B70, Qwen3.8-27B:
  • The prompt gdn_out kernel takes 22.6% less time (736 to 570 ms over a 32K prompt), with byte-identical outputs (#2921, refs #2859). Thanks to @dinadur.
  • A prompt chunk nobody reads stops its last layer early: the cold first token comes 1.3-2.5% sooner at 8K to 66K prompts, with identical replies (#2854, fixes #2849). Thanks to @SergiioB and @Mirrdhyn for the B70 runs.
• Video input: libtfvideo decodes MPEG-4 Part 2 and H.264 without B-frames on up to four frame threads.
  • The decoder alone is 2.1x to 3.3x faster on an M5 Max and 2.5x to 3.4x on an x86_64 Threadripper 3960X, with every intact frame equal to before.
  • H.264 with B-frames keeps main's slice-threaded path (#2863). Thanks to @swbiggart.
• DeepSeek-V4.1 assets: aot-needs plans the same Triton variants about 15 times faster (713 s against 45 s at a 65,536-token context) (#3007, found on #2270). Thanks to @Defilan.

Fixes

• Qwen 3.5 on CUDA: with --prompt-cache-disk or --learn-conversations, the startup probe no longer refuses with ProbeUnmarked (#2838, fixes #2804).
• GLM-5.3-Flash on CUDA at four ranks or more: when the split prefill's start-up exchange fails, as on a switchless ring of four, the refusal names TF_GLM_PREFILL_REDUCE=gather instead of a bare ncclGroupEnd error (#2868). Thanks to @Bizuayeu.
• GLM-5.3-Flash on CUDA, tensor-parallel abort: an abort reaches the communicator once, and a start-up failure after it no longer destroys the communicator again (#2918, part of #2560). Thanks to @laudney.
• Tensor-parallel shares in GLM-5.3-Flash: they go through one mesh layer, with tokens equal to main at TP=2. In any family on CUDA, an aborted tp_group call ends as Aborted instead of stopping the process (#2789, part of #2560). Thanks to @laudney.
• Token ids past the embedding table: they are refused before the lookup on HIP (#3046) and XPU (#3047). An 8-bit Qwen3.8-27B MLX checkpoint on XPU is refused at load by name (#3047).
• HIP ranks: a follower checks each step's counts against its message, so a malformed step is refused instead of read past (#3042).
• --compact-at: a later one replaces an earlier one, as for every other serve flag (#2883). Thanks to @laudney.
• Video on Linux: libtfvideo.so exports only its four tf_video_* functions, so on x86_64 it links FFmpeg's assembly. A CI check refuses a build that exports more (#2861). Thanks to @swbiggart.
• Flash Next settings: twenty-nine TF_FLASHNEXT_* settings refuse a malformed value by name at startup (#2585, part of #2370). Thanks to @devopsfortunato.
• Slide rollback: a fact graph is removed only when its own folders file names the folder being rolled back (#3073).

Docs and tools

• Docs:
  • The CUDA memory plan says what each setting decides and which file reads it (#3028). Thanks to @Bizuayeu.
  • The Intel Arc page notes that its B70 prefill row predates 1.1.0's more precise prompt DeltaNet (#2906). Thanks to @dinadur.
• Tools:
  • zig build tf-tp-check checks per-rank dumps of a mixed Metal and CUDA tensor-parallel run (#2321, part of #1236). Thanks to @ThinkOffApp.
  • core/lanes gains a shared depth sweep any core drafter can run (#2729, part of #2460). Thanks to @jschmied and @EugeneClaw.
  • tf-cuda-test flashnext-rank passes again on two GPUs (#2976, fixes #2975). Thanks to @swbiggart.
• CI and tests:
  • The serving pins run on every pull request (#3011).
  • The macOS nightly runs the Metal engine host tests (#2891).
  • A newer push to main cancels main's older run (#3230).
  • The CUDA source check passes again (#2902). Thanks to @Bizuayeu.
  • Main's nightly failures from #2585 and the re-pins are fixed (#3002).
  • Test fixes: #2946, #2889, #2865, #3010, #3255.
• Internal, with no change to what runs:
  • One source for shared Metal kernels: argmax (#2949), image-tower helpers (#2948), decode matvec (#2792), prompt GEMMs (#2787) and GLM's TP2 kernels (#2786).
  • One core reader for MLX affine modules (#2583) and one for block FP8 (#2892).
  • Unused kernels removed (#3065, #2816).
  • All of the above thanks to @jschmied.
  • Execution-witness groundwork: the kept prompt-state identity goes through any hasher (#2903), and admission moves to its own file (#2896). Thanks to @kotevcode.
  • Saved prompt states: Qwen3.8-27B and GLM saved prompt states on Metal are recomputed once after these kernel moves.

What can change for you

• Request bodies: a body that no route reads is capped at 64 KiB; a larger one gets HTTP 400 (#3029).
• Connections: under connection pressure (96 of 128 slots), one client address holds at most a quarter of the slots. --max-connections-per-address 0 turns the share off, and --trusted-proxy exempts a proxy (#3048).
• Chat templates, tokenizers and grammars:
  • A template past its nesting depth is refused at load (#3072).
  • A tokenizer split pattern that is too deep or too large is refused at load (#3071), as is a token id above 2^24 (#3033).
  • A grammar past 2^38 steps of exploration gets HTTP 400 (#3037).
• Invalid ids and settings:
  • Token ids past the embedding table are refused on HIP and XPU, and an 8-bit Qwen3.8-27B MLX checkpoint on XPU is refused at load (#3046, #3047).
  • A malformed TF_FLASHNEXT_* value is refused at startup by name (#2585).
  • --profile fast is refused (#2794).
• Qwen3.8-27B with DFlash2 on M5 Macs: one greedy request at the default window takes the long-copy driver; TF_QWEN27_LANES=1 keeps every request on lanes (#3069).
• GLM-5.3-Flash with TF_ROCE=1: three ranks or more use the RoCE all-gather. Drafter acceptance can move slightly, since the split drafter MLP sums in another order (#2205).
• pull: a repository file under the engine's reserved names is refused (#3032).
• Saved prompt states: Qwen3.8-27B and GLM saved prompt states on Metal are recomputed once after upgrading (#2816, #3065, #2786).

Thanks

• @3utsu (an independent receipt for #1720)
• @AbrahamCete (#2726, #2878)
• @Arminova (#2285, #2995)
• @Bizuayeu (#2765, #2813, #2864, #2868, #2902, #3028)
• @BobClawblaw (#1177, #1630)
• @Defilan (#3007)
• @devopsfortunato (#2585, #2719, #2794)
• @dinadur (#2921; and #2906)
• @EugeneClaw (#2729)
• @gilby (#1720, #1878, #2695, #2728)
• @javierbits (#1953)
• @jschmied (#2579, #2583, #2705, #2729, #2786, #2787, #2792, #2816, #2892, #2948, #2949, #3065)
• @kotevcode (#2896, #2903)
• @laudney (#2570, #2673, #2789, #2875, #2883, #2918)
• @Mirrdhyn (the B70 run for #2854)
• @ppardus (#2750, #3006)
• @SergiioB (the B70 run for #2854)
• @shanedertrain (#2205)
• @Shoestring-dev (#2718)
• @swbiggart (#2861, #2863, #2976, #2977)
• @ThinkOffApp (#2321, #2723)