Skip to content

TensorFold 1.2.6

Choose a tag to compare

View all tags
@ashhart ashhart released this 10 Nov 12:51
· 729 commits to main since this release
v1.2.6
6e9af93

TensorFold 1.2.6

1.2.6 makes the prompt cache hold up under many agents at once: long conversations survive 8 to 16 agents sending follow-ups together, and a follow-up's first token comes far sooner, with every reply unchanged. Qwen3.8-27B on CUDA can draft with Tencent's DSpine-b16 (opt-in), and Qwen3.8-Flash-Next and the Qwen DeltaNet prompts get faster on HIP. Spark 2.5 and Dory load faster on CUDA with direct reads, a new --unix-socket serves on a local socket, and POST /v1/images/generations serves an engine's image capability. Settings are more consistent: blank values read as unset across GLM, Gemma 4, Flash Next and the KV share switches, and GLM's 0/1 switches are refused by name when malformed.

  • New:
    • Qwen3.8-27B on CUDA drafts with DSpine-b16 when --drafter names it;
    • --unix-socket PATH serves on an owner-only Unix socket;
    • tool-call ids are reproducible for a replayed request;
    • tools/conformance.py runs a family's served exactness gates from one profile, with a Qwen3.8-27B Metal profile;
    • HIP builds the DFlash2 drafter's kernels for qwen3_5 (not launched yet);
    • POST /v1/images/generations for an engine with an image capability;
    • --instruct-sampling card serves the model card's instruct sampling with thinking off.
  • Speed:
    • Many agents at once: a follow-up's median first token at 16 agents drops from 546 s to 15.9 s on an M3 Ultra and from 105 s to 25.8 s on a GB10.
    • Qwen3.8-Flash-Next on HIP (gfx1151): cold prompts finish 29 to 41% sooner.
    • Qwen DeltaNet prompts on HIP (gfx1151): 4.6-6.0% less time on Qwen3.8-27B and 5.7-9.9% on Qwen3.6-35B-A3B.
    • Qwen3.6-35B-A3B MoE prompts on a GB10: 9.9-14.1% less time, from 2K to 64K.
    • Spark 2.5 and Dory on a GB10: cold loads take 2.53 s instead of 5.03 s and 3.79 s instead of 4.79 s.
    • GLM-5.3-Flash at TP=2 on two GB10s: prompts fill 13-23% faster, and one-stream code decodes 32% faster.
    • Qwen3.8-27B on Macs: sampled decode with DFlash2 is faster on an M5 Ultra and an M3 Ultra.
    • Flash Next on Metal: served decode is 2.4-6.0% faster on an M3 Ultra.
  • Serving:
    • JSON tool calls cut by max_tokens come back as calls.
    • Dory on CUDA keeps prompts as pool pages with TF_KV_SHARE=n.
    • Flash Next on Macs no longer fails a request with HTTP 500 after idle.
    • Drama 3 on CUDA opens again.

Where a speedup was checked for equal output, its line says so. Some changes alter settings, refusals, memory or startup by default; they're listed under What can change for you.

Headline numbers

Each number names its machine and its PR.

Release reads (1.2.5 against 1.2.6 on fresh servers, alternating; replies equal to 1.2.5 on every machine):

  • Qwen3.8-27B on an M3 Ultra: prompts 1.6 to 4.7% faster (4.7% at 64K); decode and 1-8 stream sweeps level.

  • Qwen3.8-27B on an M5 Ultra: prompts up to 2.6% faster (at 64K); decode and sweeps level.

  • Qwen3.8-27B on a GB10: level throughout.

  • Qwen3.8-Flash-Next on a GB10: level throughout.

  • Nemotron 3.5 Lightning on a GB10: level. One cell, eight streams sending the same greedy prompt, read 5.7% slower; with a prompt per stream the two releases are within 0.1%, so it was identical replies running in step, not a change.

  • Prompt cache with many agents, Qwen3.8-27B with DFlash2: each agent sends one ~17K-token prompt, then a follow-up on the same conversation. Every agent's replies equal main's, at every agent count (#2699).

    • On an M3 Ultra (default budget): a follow-up's median first token drops from 50.1 s to 1.42 s at 8 agents and from 546 s to 15.9 s at 16. Follow-up decode rises from 15.4 to 79.5 tok/s and from 5.9 to 88.5.
    • On a GB10: from 14.6 s to 7.7 s at 8 agents and from 105 s to 25.8 s at 16. Follow-up decode rises from 47.7 to 93.9 tok/s and from 33.6 to 100.8.
    • At 1 and 4 agents and on Qwen3.8-Flash-Next it's level with main, and first-turn speed is unchanged.

  • Qwen3.8-Flash-Next on HIP, on a Radeon 8060S (gfx1151): sparse attention runs a new kernel with the same sums. Cold prompts of 2,048 to 63,728 tokens finish 29 to 41% sooner, with replies unchanged. Other HIP chips take the kernel but weren't run (#1877).

  • Qwen 3.5, 3.6 and 3.8 on HIP, on gfx1151: the DeltaNet prompt recurrence runs at half a value head a block, with the same bits. A prompt takes 4.6-6.0% less time on Qwen3.8-27B and 5.7-9.9% less on Qwen3.6-35B-A3B (2K to 32K) (#2582, piece J of #1667).

  • Qwen3.5/3.6 MoE on CUDA, on a GB10: router calls of 257 rows or more run on the shared register tiles, with every logit's bits unchanged. Qwen3.6-35B-A3B NVFP4's cold prompts take 13.4% less time at 2K (0.382 to 0.331 s), 14.1% at 8K, 11.8% at 32K and 9.9% at 64K (15.899 to 14.323 s), with decode and replies unchanged. tf-cuda-test moe-router-bench measures the crossover on other GPUs (#2537, #2499).

  • Tokenizer memo, on an M5 Ultra: segments of 64 bytes or more are kept, not 1,024, with the same ids. On real agent conversations (Qwen3.8-27B tokenizer, up to 60K tokens), a warm encode of the eight longest takes 146-151 µs instead of 680-691 µs, and a one-off prompt is no slower (#2509).

  • Spark 2.5 and Dory on CUDA, on a GB10: loads use direct reads by default, with the same replies:

    • Spark X2.6 4B 8-bit: cold loads go from 5.03 to 2.53 s; resident memory peaks at 6.1 GiB instead of 9.4, and 0.4 GiB of page cache is filled instead of 4.2;
    • Dory NVFP4: from 4.79 to 3.79 s; resident stays at 7.2 GiB, and 1.3 GiB of page cache is filled instead of 3.1 (#2449).

  • GLM-5.3-Flash on CUDA, at TP=2 on two GB10s: decode windows reach 32 rows. Prompts of 2K and 8K tokens fill 23% and 13% faster, and one-stream code decodes 32% faster, with every reply equal to before (#2257).

  • Qwen3.8-27B on CUDA with DSpine-b16, on a GB10 (opt-in): every drafted reply equals the same request with drafts off. Against DFlash2, agent-shaped requests decode 2.4-19.8% faster, and chat and code 10-19% slower, which is why it is opt-in (#2633, refs #2592).

  • Qwen3.8-27B on Macs, sampled with DFlash2:

    • a verify window draws its rows on up to four host threads. One-stream sampled decode is 4 to 8.8% faster on an M5 Ultra, with tokens and errors unchanged (#2644);
    • the drafter waits only for its taps and head row. One stream on an M3 Ultra: chat 75.3 to 77.0 tok/s, code 106.8 to 109.8, with every token unchanged (#2646).

  • Qwen3.8-27B on Metal: a prompt pass that keeps no row's logits skips its last layer's attention, MLP and final norm, with the prompt state and tokens unchanged. Served prefill is 1.6% faster at 4K and 2.3% at 16K on an M3 Pro, and 1.2% at 8K and 1.5-1.6% at 16K on an M5 Ultra (#2436, refs #2229).

  • Flash Next on Metal, on an M3 Ultra: 6- and 8-bit row products at 2 to 4 rows take two outputs a simdgroup. A 2,560 × 6,144 product at 2 rows takes 23.5 to 28.7% less time across the three widths measured, and served decode is 2.4-6.0% faster, with every token hash unchanged (#2593, fixes #2553).

  • Dory on CUDA with TF_KV_SHARE=n, on a GB10 at a 4 GiB cache: a 25-turn agent loop to 34K tokens spends 10.7% less time to first tokens (about level up to 12K). 72 round-robin turns keep 29 states against 9, with the same replies (#2430, part of #1657).

  • --learn at 100,000 learned states, on an M3 Ultra: find, has and touch take 0.02-0.04 µs instead of 24-57 µs, and a reload's finds after another writer appended take 1.1 ms instead of 2.8 s, with the same records, order, clock and eviction victims (#2351).

New models and backends

  • DSpine-b16 on CUDA (opt-in): Qwen3.8-27B drafts with Tencent's DSpine-b16 when --drafter names its folder, never by default. Its code bridge and exact-sender table run on affine-4 projections. The draft tree walker moves into core/lanes (#2633).
  • HIP drafter groundwork for qwen3_5 (#1563):
    • the engine builds the DFlash2 drafter's kernels (tap gathers, key ring, dynamic conv, block attention), tested and in the kernel contract. Nothing launches them yet (#2545);
    • the load-time lane cost table times shared buckets with a distinct token a row, as shared rounds draft, and every row gets a distinct token for any vocabulary. Only the drafted width can change, never a token (#2652, #2661, part of #2605).
  • XPU DFlash2: Qwen3.8-27B's drafts go through the core Drafter and Drafted, and each stream's draft tree is keyed with its own request, where a sampled shared round read another's (#2672, #2460 item 5).
  • CUDA DFlash2 config: a target_layer_ids list that names one layer twice is refused, as on HIP (#2674).

Speed and memory

  • Prompt cache (#2699):
    • redundant states are evicted before each conversation's newest one;
    • admission never evicts a state that a queued request resumes;
    • concurrent keeps reserve their room;
    • the default budget scales with the room left after streams on Metal and unified-memory CUDA. Discrete CUDA hosts keep 8 GiB, and Flash Next's batch host 16 GiB.
  • GLM-5.3-Flash on CUDA: decode windows reach 32 rows, and MTP chains longer drafts (#2257).
  • GLM on CUDA, opt-in with TF_GLM_KV_LIVE_HALF=1: a kept prompt state holds only the live half of the double-buffered KDA recurrent state, with replies unchanged, drafts off and on (#2181, part of #2180).
  • HIP on gfx1151: a skewed-row matrix tile runs three blocks a WGP for prompts up to 8,192 codes deep, with the same results as the padded tile (#2087).
  • HIP uploads: a host-to-device upload copies in 64 MiB windows, so a multi-GiB tensor no longer has its whole source pinned by one copy (#2584).
  • Responses and Kimi chat: a long reply holds memory in proportion to its length, with every reply and streamed event the same (#2642).
  • KV pages: a window past the engine's own counts toward the startup plan only where the room past every stream's fixed part holds it whole. A band of rooms (for example at TP=3 with a 1M window and eight streams) no longer leaves the prompt cache at 0, where GLM's TF_KV_SHARE kept no state (#2568, fixes #2547).
  • The speedups in Headline numbers: #2699, #2537, #2509, #1877, #2582, #2449, #2257, #2633, #2644, #2646, #2436, #2593, #2430, #2351.

Serving and API

  • Image generation: POST /v1/images/generations serves an engine's image capability, one picture a request as a base64 PNG. An n other than 1, sizes past 1536 a side or 1,572,864 pixels, response_format url, unknown fields and a negative_prompt that is null or not valid UTF-8 are refused by name. A text model's routes are unchanged (#2253, part of #1573).
  • Instruct sampling: --instruct-sampling card (or field=value pairs) serves the model card's instruct sampling to sampled chat requests with thinking off that leave those fields unset. Request fields win, then the serve flags, then the pairs, and the server's own requests never take it (#2626).
  • Unix socket: --unix-socket PATH serves on a new owner-only Unix socket instead of TCP.
    • It refuses an existing path, a parent directory others can write to (unless sticky) and the TCP-only flags.
    • It checks each peer's user.
    • It is refused on Windows (#2389).
  • Tool-call ids: a reply's call ids are an HMAC-SHA-256 of the request under the server's key, so a request replayed to the same server gets the same ids. A stream names its calls as its whole reply does, and a Responses fc_ item id follows its call. The key is random per process unless TENSORFOLD_TOOL_ID_KEY_FILE names an owner-only file of at least 32 bytes. The call_ form is unchanged (#2653).
  • Cut JSON tool calls: a reply that max_tokens cuts inside a JSON tool call (the form Ripple-65B-Preview writes) returns that call with its arguments as far as written, whole and streamed alike, instead of leaving the markup in content. On /v1/messages, a whole reply with a cut XML call no longer returns a 500 (#2511).
  • Dory on CUDA: TF_KV_SHARE=n keeps prompts as the KV pool's pages in the shared prompt store, with at most n kept states, and turns Dory's pool pages on. With TENSORFOLD_DORY_KV_PAGES=0 it refuses to load, naming both (#2430).
  • With --cortheon: DuckDuckGo results and ads are matched regardless of letter case, Query and URL tool arguments are read, HTML's uppercase entities decode, and data: links are dropped (#2372).
  • Access log: control bytes in a request line appear as \xNN, so a request can't write terminal escapes into the log (#2542).
  • Living Weights on Nemotron (--slide): a lesson's pocket goes only into the layers that carry it and pass the full verifier. Corrections that overturn a shipped answer use an explicit additive fit, and serving without --slide is unchanged (#2025, part of #1968).
  • Diagnostics: opt-in startup receipts show Qwen3.8-27B Metal load phases and the first completed non-streaming reply (#2640).
  • Benchmark clients: bench_concurrent and bench_openai time content, reasoning and tool-call output separately and classify how each reply ended. tools/conformance.py --stream runs the exactness gates on streamed replies (#2625).
  • Conformance:
    • tools/conformance.py runs a family's served exactness gates from one JSON profile, and its own tests run in CI with no device (#2467);
    • a Qwen3.8-27B Metal profile, and a fresh_env key for the resumed gate on Metal (#2612);
    • the cancel and evict gates check that a cut reply and an evicted prompt leave the server as a fresh one, with a planted fault for each. The Metal profile skips evict with its measured reason; the CUDA profiles run both (#2611).
  • Evidence index: tools/evidence_index.py indexes what each family and backend has qualified, on which commit and at which grade (#2621). It reads saved native capabilities, including unknown and XPU chip labels, and invalid input stops with a bounded error (#2708).

Fixes

  • Drama 3 on CUDA opens again. Since 1.2.5 every open stopped with "unregistered strict environment parser"; both settings now have rows in the environment table (#2658).
  • Flash Next on Macs:
    • after idle, when macOS has unwired the model, a prompt chunk's later segment no longer sits in a GPU wait past macOS's limit and fails the request with HTTP 500 (#2594, fixes #2552);
    • a server that closes while replies wait now ends each of them, even short of memory (#2616).
  • GLM-5.3-Flash at TP=4: with TF_GLM_PREFILL_REDUCE=gather, four GB10s cabled as a switchless ring start and serve, where before every rank stopped within 122 s after ibv_modify_qp timeouts (#2569, refs #1971).
  • Linux reads:
    • Nemotron's io_uring checkpoint reads no longer hang at shutdown after a failed read (#2608);
    • an O_DIRECT read past a file's end reports end of file instead of a failed read (#2602);
    • a failed direct-read load names its reader's error, such as EndOfFile (#2610);
    • DeepSeek-V4.1's Engram shards on disks that need more than 512-byte direct reads are read through the page cache instead of failing (#2610).
  • Sessions: a restore whose page write fails mid-stream, or whose tier queue can't take it, no longer leaves the next restore or shutdown waiting forever (#2615).
  • File readers: a session, conversation state, EXL3/EXLR8 shape, Engram row, tokenizer id or Drama codec count whose size arithmetic would overflow is refused with the reader's own error, where a safe build used to panic. Every file accepted before reads the same (#2618). The same holds for --learn-conversations trailers (#2544).
  • Flash Next on CUDA: a verify window's and a shared pass's graph keys hash every buffer of a stream's tables. Nothing changes on today's placement (#2577, part of #2444).

Docs

  • A reply that quotes a template marker finishes with a plain request. A client that keeps ignore_eos should stop on the model's end-of-text and next-turn-open spellings and strip the trailing turn marker: behind an agent prompt, end-of-text alone let 7 of 9 replies run on, against 0 of 9 with both, on Qwen3.8-Flash-Next on an RTX PRO 6000 (#2437, refs #2399).
  • The Zig family recipe lists the core blocks on main with the call a family makes (#2468).
  • The LFM3 recipe records the qualification rules agreed on #1491, with an evidence ledger (#2416).
  • The image page says how the native server finds libjpeg-turbo 3.1.4.1 for JPEG input, and what each startup warning means (#2590, refs #2581).

Internal

  • Shared CUDA code (#548, #850, #2499): with every function's SASS or machine code unchanged:
    • RMS scales call tf_common::rms_scale (#2580, #2684);
    • stored partial-sum folds call tf_common::sum_rn (#2666);
    • Nemotron's router tiles move to common/router_tiles.cuh (#2536);
    • GLM's KDA chain and saved-row replay move to cuda.kda (#2142);
    • GLM's SwiGLU rows move to glm_swiglu.cuh, with a template for a clip after silu (#2627);
    • a shared MLA image (mla.cu) that no family calls yet, under the kernel contract (#2335);
    • the MoE routing glue gains group-limited selection in its sigmoid score form; Nemotron keeps its kernels and bytes (#2334);
    • CUDA EXL3 experts gain a clamped-silu finish kernel; existing EXL3 models are unchanged (#2336).
  • CUDA disk staging: a family's disk module may declare its own host staging permit for kept states. No family declares one yet, so every family keeps the 1 GiB permit (#2337).
  • KV pool: stores join an ordered chain of reclaim hooks; every load keeps its one store, so nothing changes yet (#2528).
  • Qwen 3.5/3.8 config readers: one table with a column per backend; each admits what it did (#2563).
  • Nemotron on Metal: the 8 group norms share one source, with the same kernels and bits (#2540).
  • Sampling: Qwen 3.5's host draw moves unchanged into core/lanes as sampling_row (#2606).
  • Tests and CI:
    • host pins for documented limits and defaults, a replay of cache eviction on agent traces, and a snapshot-index timing (#2305);
    • zig build sessions-writes measures DeepSeek-V4.1 session tier writes without a model (#2660);
    • zig build bench-stream-scan streams a 32K-token thinking reply through the chat, Messages and Responses routes (#2642);
    • runtime Metal libraries compile for each loader, and every launch mapping is checked on the host (#2601);
    • the Qwen4 QSA backward host test prints only on failure (#2588);
    • the release privacy check walks the tree in Zig, so it runs on native Windows (#2614);
    • the nightly Linux host job runs test-pdf-worker, test-cache-fit-host and a deepseek shard (#2599, #2598); the macOS nightly runs the thirteen held-back host checks (#2566); pytest has its own job (#2586).
  • Experimental: TF_TP_ROCE_RELAY=1 relays gathers through both neighbours on a four-rank RoCE ring with no opposite links, instead of falling back to NCCL. It's off by default, and two- and three-rank behaviour is unchanged (#2352, part of #1971).

What can change for you

  • Prompt cache budget (#2699): on Metal and unified-memory CUDA, the default budget now scales with the room left after streams, with the old default as a floor.
  • Persisted prompt states (#2287, closes #2131): the disk tier and --learn-conversations key each family's states by what its loaded run computed with (resolved arithmetic and kernel choices, every weight shard, external inputs such as the CUDA Qwen rotary table). Upgrading starts every kept state once cold, and so does changing a GLM or Qwen Metal prompt chunk height. GLM_EHPROJ_NAX takes only 0 or 1; other text, which used to read as on, is refused at start.
  • Spark 2.5 and Dory on CUDA: loads use direct reads by default. TF_LOAD_DIRECT=0 keeps the old mapped reads (#2449).
  • Nemotron on Metal: saved prompt states take their identity from the source text, so after upgrading they are not reused: the first long prompt prefills again (#2540).
  • DeepSeek-V4.1: prepared folders rebuild once after upgrading. At TP=2 on two GB10s that first boot took about 10.5 minutes to weights loaded, against about 5 later (#2325).
  • GLM-5.3-Flash on CUDA:
    • a malformed TF_GLM_* switch, TF_GLM_DFLASH2_LOG or TF_GLM_DRAFT_CONFIDENCE is refused by name when the engine opens. The seven 0/1 switches take exactly 0 or 1, where off, yes or 2 used to keep the default. Unset and blank keep every default (#2578);
    • decode windows reach 32 rows. The first load after upgrading times them: on two GB10s it took 100 s against 70 s, and 75 s on a later load (#2257).
  • Blank settings read as unset: an empty or whitespace-only TF_FLASHNEXT_KV_PAGES (#2619), TF_KV_SHARE, TF_QWEN_KV_SHARE, TF_QWEN_KV_PAGES (#2629), GLM's start settings (#2636) and TF_GEMMA4_CHUNK_SOLO (#2587) no longer stop the start. A blank GLM TF_KV_PAGES means auto, so at TP=1 a blank TF_KV_PAGES with TF_KV_SHARE or TF_GLM_KV_HOST=1 is refused by name, as an unset one is (#2629).
  • Config refusals:
    • Flash Next on Metal and HIP refuses a legacy rope_scaling (YaRN), and LFM2 and Kolibri-1 refuse both forms, where these readers ran plain RoPE (#2604);
    • a DFlash2 config naming a target layer twice is refused on CUDA (#2674).
  • Tool calls:
    • call ids are reproducible for a replayed request. A bad TENSORFOLD_TOOL_ID_KEY_FILE stops the start by name (#2653);
    • a JSON call cut by max_tokens returns as a call, not as text in content (#2511).
  • Benchmark clients: token ITL/TPOT is null where a chunk carries several tokens, and prefill_tps is deprecated in favour of prompt_ttft_tps (#2625).
  • Startup line: it names a --prompt-cache-gib only where it keeps all the states, and otherwise how many the room holds (#2568). Scripts that parse it may need updating.
  • Access log: control bytes in a request line are written as \xNN (#2542).

Thanks

Contributors

ElderOrb, laudney, and 19 other contributors
Assets 12
Loading