Skip to content

TensorFold 1.2.7

Latest
Latest

Choose a tag to compare

View all tags
@ashhart ashhart released this 11 Nov 17:18
v1.2.7
93a2e93

TensorFold 1.2.7

1.2.7 is mostly about resuming long conversations where they left off: Qwen3.8-27B on Metal resumes a follow-up at the end of its history even when the client drops earlier reasoning (first token in about 0.3 s instead of 85 s on an M3 Ultra), a failed disk restore falls back to the state already in RAM, and GLM-5.3-Flash on CUDA can resume a chat at its previous turn (opt-in). It also adds a WebSocket mode for the Responses API and LightOnOCR-3 image chat on CUDA, with speedups on Metal, CUDA and HIP and fixes for -Dgpu builds, damaged JPEGs and malformed GLM tool calls.

  • New:
    • Responses API WebSocket mode: GET /v1/responses with a WebSocket upgrade, so client.responses.connect() works unchanged;
    • LightOnOCR-3 0.8B and 4B serve image chat on CUDA from the publisher's BF16 weights;
    • TF_GLM_REQUEST_STARTS=1 (opt-in) resumes a GLM chat's next turn at its previous turn's history on CUDA;
    • TF_FLASHNEXT_SCALE_INDEX=1 (opt-in) reads Flash Next's 4-bit scales and biases through a 16-bit index on Metal (M1 through M4);
    • the dashboard and /stats show the prompt cache, its disk tier and, on Metal, Flash Next's SSD-expert cache;
    • a Qwen3.8-27B conformance profile for XPU on an Arc Pro B70;
    • docs/optimizing.md says how to approach speed work in TensorFold.
  • Speed and memory:
    • Qwen3.8-27B on Metal: a follow-up from a client that drops earlier reasoning gets its first token in about 0.3 s instead of 85 s on an M3 Ultra (34K history).
    • Flash Next on Metal: a copy stretch decodes 7.7% faster on an M3 Ultra; the opt-in scale index makes drafted decode 1.3% to 3.9% faster.
    • Qwen3.8-27B on an M5 Ultra: sampled decode with DFlash2 is 6% faster, and five cold long prompts in a row with a 14 GiB prompt cache reach their first tokens 3.5% to 5.9% sooner in total.
    • GLM-5.3-Flash with DFlash2 at TP=2 on two GB10s: four-stream decode is 3.7% faster greedy and 3.5% faster at temperature 0.7.
    • Qwen3.8-27B on CUDA beside DSpine: agent turns decode about 6-7% faster at one stream on a GB10.
    • HIP on gfx1151: 8K-token prompts reach their first token about 5% sooner on Flash Next and about 6% sooner on Nemotron 3.5 Lightning 30B-A3B.
    • Flash Next on HIP at tensor parallel 2 on two Radeon AI PRO R9700s: each card holds 0.74 GiB less.
  • Fixes:
    • zig build -Dgpu=<gpu> compiles again.
    • A damaged JPEG is refused instead of accepted with rows the decoder never wrote.
    • A malformed GLM tool call stays as text instead of becoming a call with corrupt arguments.
    • A failed round fails only its own streams, and a finished stream delivers its reply.
    • Two failed disk restores in a row resume from the state in RAM, not from token 0.
    • GLM-5.3-Flash thin loads with TF_GLM_LAYERS count only the weights they read at start-up.

Where a speedup was checked for equal output, its line says so. Some changes alter settings, refusals, memory or startup by default; they're listed under What can change for you.

Headline numbers

Each number names its machine and its PR.

  • Qwen3.8-27B on Metal, follow-ups without reasoning: a client that sends earlier replies back without their reasoning now resumes a follow-up turn at the end of its last message. On an M3 Ultra with --prompt-cache-gib 5 and a 34K history, such turns resume about 34,100 tokens instead of 7,424, with the first token in about 0.3 s instead of 85 s. Clients that keep the reasoning resume as before, and replies are unchanged (#2811, fixes #2791).
  • Flash Next on Metal (M1 through M4), grouped experts: a decode window of five rows or more runs its 4-bit experts grouped by expert, so rows that pick the same expert share its launch, and replies keep their tokens. A copy stretch decodes 7.7% faster on an M3 Ultra (256.6 to 276.3 tok/s); prose rounds, four rows or fewer, are unchanged (#2784, part of #2764).
  • Flash Next on Metal, scale index (opt-in): with TF_FLASHNEXT_SCALE_INDEX=1, on an M3 Ultra with the MLX 4-bit checkpoint, drafted decode on one stream is 1.3% to 3.9% faster (prose 137.3 to 139.1 tok/s, a copy-heavy edit 262.8 to 273.2) and "draft": false 0.8% to 2.2% faster, with every token unchanged. It costs 7.3 GiB more in use and about 4 s more load (#2760, part of #2751).
  • Qwen3.8-27B on Metal, sampled top-k: on an M5 Ultra with Qwen3.8-27B MLX 4-bit and its DFlash2 drafter, sampled verify rounds take 1.6 ms less and sampled decode is 6% faster, with the same candidates and replies; greedy is unchanged. The Qwen 3.5/3.6 CUDA engine shares this host draw for each first token and for grammar-masked windows; CUDA is not measured (#2507).
  • Qwen3.8-27B on Metal, keep-warm while a request runs: on an M5 Ultra with a 14 GiB prompt cache filling memory, five cold 24K- to 64K-token prompts in a row reach their first tokens 3.5% to 5.9% sooner in total; with the cache off, cold prompts are level. Replies are unchanged (#2725, fixes #2724).
  • GLM-5.3-Flash on CUDA with DFlash2, at TP=2 on two GB10s: with distinct prompts, four-stream decode is 3.7% faster greedy and 3.5% faster at temperature 0.7. One stream, the MTP head and every reply's tokens are unchanged (#2793, fixes #2720).
  • GLM-5.3-Flash on CUDA, request starts (opt-in): on two GB10s at TP=2 (EXL3 TR3, DFlash2), a chat of about 58.7K tokens resumed its later turns at 58,735 instead of 57,344, with replies identical to --prompt-cache-gib 0 and to the switch off (#2404, part of #2396).
  • Qwen3.8-27B on CUDA beside a DSpine drafter, on a GB10: the dense model copies from its context after 12 matching tokens instead of 20 (DFlash2 keeps 20), so agent turns decode about 6-7% faster at one stream, with chat and code level and greedy replies unchanged (#2703, after #2633).
  • HIP on gfx1151, routed prompt tiles: Flash Next's 8K-token prompts reach their first token about 5% sooner, and Nemotron 3.5 Lightning 30B-A3B's about 6% sooner. Served Qwen3.5/3.6 MoE prompts are unchanged, and replies are unchanged (#2576).
  • Flash Next on HIP, on two Radeon AI PRO R9700s at tensor parallel 2 with a 32k window: each card holds 0.74 GiB less, and --policy prefill_step=2048 loads where main refused it, with replies equal to main's (#2670).

New models and backends

  • LightOnOCR-3 on CUDA: the 0.8B and 4B models serve image chat on a GB10 from the publisher's BF16 weights. A plain BF16 Qwen3.5 checkpoint with an image tower the CUDA tower reads and no Clef joint head now opens as chat, its tied head packed from the embedding. The flags it is not qualified on (--drafter, --slide, --kv-cache fp8, --prompt-precision int8), towers the CUDA tower cannot read and a joint head beside a tied backbone are refused by name before the weights load. MLX, EXL3, ModelOpt and Clef checkpoints load and answer as before (#2597, part of #1927).
  • OCR kernel groundwork on CUDA (part of #1927): no model serves through these in this release:
    • tf_dots_qkv_rope adds dots.mocr's q, k and v biases and rotates its queries and keys in one pass at Transformers' BF16 rounding points, with a contract against FP64 up to 1,024 rows and positions up to 131,071 (#2709);
    • pattn_kernel gains instances at two query heads a block for head dims 128 and 64, the prompt attention that dots.mocr, GLM-OCR and Nemotron Parse need on CUDA (#2609).
  • XPU conformance: a Qwen3.8-27B profile (tools/conformance_profiles/qwen27-xpu.json: GPTQ 4-bit group 128 with its DFlash2 drafter, four streams on an Arc Pro B70). Its first turn carries a 600-item document because the XPU prompt cache keeps states only at multiples of 64 tokens (#2677, landed as #2690).

Speed and memory

  • GLM-5.3-Flash on CUDA, request starts (opt-in): TF_GLM_REQUEST_STARTS=1 cuts prompt passes at a request's message starts, so a chat's next turn resumes at its previous turn's history instead of the last 2,048-row point. GLM refuses it at startup with TF_KV_SHARE above 0 or with --learn-conversations (#2404).
  • GLM-5.3-Flash on CUDA with DFlash2: shared rounds size each stream's drafts by its last grant plus the slack, trimmed at verify, so a stream no longer stays at no drafts while its peers decode (#2793).
  • GLM-5.3-Flash on Metal with --vision: a prompt chunk sends the vision tower only the images that end after its start, so two interleaved image prompts no longer re-encode images their earlier chunks already injected (12 images through the tower instead of 18 in a host test). The rows a chunk reads are unchanged (#2744).
  • Lanes: where a family learns its shared rows' cost on the machine (GLM on Macs without tensor units), a fit that shows rows cost less than the table says now lowers their price, so a table timed on a slower build no longer keeps out drafts that would pay. While a stream count has no fit, its rounds reach the width they measure. Only how many drafts a round verifies changes; replies are unchanged (#2777, fixes #2776).
  • Qwen 3.5 on CUDA with --decode-share: the 512-row cap now reaches the prompt pass, cut inside the cold chunks on a 128-row grid. Decoders beside a long prompt wait one short step instead of a whole chunk, the long prompt's own first token comes later, and replies are unchanged (#2802, refs #840).
  • Flash Next on Metal (M1 through M4): TF_FLASHNEXT_GROUPED_ROWS sets the first grouped width, and 0 turns grouping off (#2784). The opt-in scale index sits beside the stored scales and biases, which prompt passes still read (#2760).
  • Qwen3.8-27B on Metal: while a request runs, the keep-warm ticker commits the residency sets' probe each interval, so the first command buffer after a wide prompt pass no longer waits in the kernel (#2725).
  • The speedups in Headline numbers: #2811, #2784, #2760, #2507, #2725, #2793, #2404, #2703, #2576, #2670.

Serving and API

  • Responses API over WebSocket: GET /v1/responses with a WebSocket upgrade takes response.create events and streams the same events as SSE, token-equal to it. previous_response_id continues store: false chains from a per-socket cache, with stream_id lanes, generate: false warmups and OpenAI's 60-minute limit, so client.responses.connect() works unchanged (#2749, fixes #2746).
  • Dashboard (--dashboard) and /stats: the prompt cache shows its states, hits, held bytes against budget, cached tokens, prefill reuse and its 64 most recently used states. Its disk tier, when present, shows reads and writes, latency, queue depth and read-ahead. On Metal, Flash Next's SSD-expert cache shows hits, evictions and occupancy. State ids are hashed only with --dashboard on (#2682, fixes #2655).
  • Living Weights: a lesson's subject is never a number group ("12 500", "7,400", including thin-space and no-break-space forms) or a sentence-opening determiner such as "Any" on its own (#2807).

Fixes

  • Builds: zig build -Dgpu=<gpu> compiles again. The -Dgpu products did not set is_release, which the server has read since 1.2.5, so every such build failed (#2879).
  • Images: a damaged JPEG whose decode stops before every pixel is written is refused as undecodable, where it could be accepted with rows the decoder never wrote. Clean JPEGs decode to the same bytes as before (#2759).
  • GLM tool calls: a call whose argument value runs into the next key/value pair, or never closes before the call ends, stays in content as text instead of becoming a call with corrupt arguments, whole and streamed. Other GLM calls parse as before (#2831, fixes #2801).
  • Failed rounds: when the backend fails a round, only the streams it held at the failure (paused ones included) and the prompts whose last piece the round planned fail with it. A prompt still filling keeps filling, and a stream that had already finished delivers its reply instead of an error (#2727).
  • Prompt cache: when two disk restores in a row fail (a damaged or missing state file), the request resumes from the shorter state it already had in RAM instead of prefilling from token 0 (#2694).
  • GLM-5.3-Flash on CUDA with TF_GLM_LAYERS=N: the start-up memory check counts the weights by tensor from the shard headers, only those the load reads, where it counted every checkpoint file. A 20-layer thin load of the 163.6 GiB checkpoint now counts 66.6 GiB without drafts (#2788, fixes #2745).
  • Flash Next on CUDA: a non-finite draft-head row fails as ProposalNonfinite instead of ProposalPaddingPick. Sampled draft candidates are checked for count, finite values and column range before sampling, and verify rows the same way before any output token is written (grammar-masked columns excepted). Finite rows pick exactly as before, and prompt passes and decode (chat and code, 1 to 8 streams) read level on a GB10 (#2753, refs #2734).
  • Qwen 3.5 and 3.8 configs: one tied-head rule reads tie_word_embeddings from text_config then the root, and a checkpoint that sets neither loads untied, as transformers does. Flash Next, LFM2, Kolibri-1 and Qwen 3.5 read rope_parameters, legacy rope_scaling and their top-level keys through one core reader, and every config they accepted parses as before (#2714, fixes #2561, part of #2460).
  • CUDA start-up: the note about pages parked on the kernel's per-CPU lists logs at debug level, so whether it shows no longer depends on the box's memory state. The budget still counts those pages (#2164, follow-up to #2127).

Docs

  • docs/optimizing.md says how to approach speed work in TensorFold: measure first, keep every token exact, and write what lands from our own code, with a worked example (#2740, #2739).
  • The optimization ledger marks the rows that have open PRs and adds DSpine as row 29 (#2681).
  • The GLM CUDA recipe covers a four-GB10 switchless ring (gather starts there, the NCCL settings it needs, and the opt-in relay with its measured effect), and docs/api.md gives presence_penalty's measured cost to drafting, about 4-10% of tokens a round (#2742, closes #1971 and #2645).

Internal

  • Shared CUDA code:
    • Flash Next's router logits template (route_logits) moves into common/router_mma.cuh and compiles to the same machine code on sm_80, sm_86, sm_89, sm_120 and sm_121 (#2556). GLM's router runs its eight K slices on the same tiles: every partial keeps its bytes, replies and speed at TP=2 on two GB10s are unchanged, and the unaligned copy tf_glm_router_part is gone (#2557, fixes #2535);
    • seven families (Spark 2.5, Kolibri-1, Dory, Gemma 4, Qwen4, Nemotron and MiniMax M3) take their sampled rule from cuda.sample.ruleFor, checked byte for byte against each old form over 25,480 settings, with the fatbins unchanged (#2680, #2460 item 3);
    • one core reader for ModelOpt projection parts serves Dory's NVFP4 and bf16 projections, Qwen 3.5/3.6's NVFP4 experts and check-precision, with the loaded weights unchanged (#2691, part of #2460);
    • one core walk copies kept prompt states, and GLM's copies on CUDA and Metal go through it, with the same copies in the same order (#2215, part of #1293);
    • opt-in build (-Dnative-fp-decode): qmmf's FP4 decode rows use the shared tf_common::e2m1_pairs, which equals the existing decode on all 2^33 inputs. The default build's 113 fatbins are byte-identical and GB10 decode is level (#2721);
    • a family can declare that it keeps no prompt states, so --prompt-cache-gib is refused by name before the weights load, and that its replies carry no reasoning block, as HIP already allows. Every family on main is unchanged (#2710);
    • kda.geometry refuses KDA heads or a conv window other than the kernels' with KdaGeometry. No family calls it in this release (#2338, refs #850).
  • Tests and CI:
    • the serving pins follow the sources main changed, so test-gradient-serving and test-qwen-serving-host pass on main again (#2738, #2798, #2872);
    • zig build test-qwen35-copies-host (#2836) and zig build test-fsr-cli-host (#2782) compile again, and zig build test-native no longer crashes in two Flash Next host tests on Metal (#2876);
    • host tests that rely on the operating system refusing access skip by name when run as root, as in containers (#2757, part of #2748);
    • the thinking benchmark check no longer flakes on busy Linux runners, and the image routes' fixture no longer hangs where socket buffers are small, as on macOS (#2743, closes #2735);
    • a HIP GPU test runs Flash Next's fused DeltaNet prompt prep and the chain through linear() and compares them bit for bit (#2706, after #2582);
    • tp.device runs tensor-parallel ranks as threads on one GPU, so DeepSeek-V4.1's exchanges are checked bit for bit at worlds 1 to 4, and zig build dsv41-test builds tf-dsv41-test. Serving still refuses three ranks (#2491, refs #2270);
    • host.runWith aborts and joins the ranks it already started when spawning one fails (#2700, follow-up to #2491);
    • the sessions-writes mixed load holds its pool to one slot's capacity, so it exercises spills to the tier (#2731).

What can change for you

  • Flash Next on Metal (M1 through M4): decode windows of five rows or more run their 4-bit experts grouped by expert by default, with replies keeping their tokens. TF_FLASHNEXT_GROUPED_ROWS=0 turns it off (#2784).
  • Qwen 3.5 on CUDA with --decode-share: a long prompt's own first token comes later, as decoders beside it wait one short step. TENSORFOLD_PREFILL_POLICY=fcfs now cuts Qwen's prompt steps beside decoders too, at its row count floored to 128 (#2802).
  • GLM tool calls: a malformed call stays in content as text, though a stream may already have sent the call's first pieces. A value containing the literal text <arg_key> now stays as text (#2831).
  • Images: a damaged JPEG is refused as undecodable (#2759).
  • Qwen 3.5 and 3.8 configs: a checkpoint that sets no tie_word_embeddings loads untied (HIP had loaded it tied; Metal's 2B still requires the key). A root and text_config that disagree, or a value that isn't a flag, are refused with BadTiedHead (#2714).
  • Flash Next on CUDA: a non-finite draft-head row fails as ProposalNonfinite instead of ProposalPaddingPick, and a refused row logs one line naming the stream and the draft length (#2753).
  • Prompt cache: after two failed disk restores, the reply's cached_tokens reports the position of the state it resumed from in RAM (#2694).
  • GLM-5.3-Flash on CUDA: with TF_GLM_LAYERS=N, the start-up memory check counts only the weights the load reads. Unset, it counts the files as before (#2788).
  • Plain BF16 Qwen3.5 checkpoints with an image tower on CUDA: they open as chat instead of the decision-only refusal (#2597).
  • Lanes: on families that learn row costs (GLM on Macs without tensor units), how many drafts a round verifies can change. Replies are unchanged (#2777).
  • CUDA start-up log: the parked-pages note logs at debug level (#2164).

Thanks

Contributors

wojo, laudney, and 18 other contributors
Assets 11
Loading