Skip to content

TensorFold 1.2.3

Latest
Latest

Choose a tag to compare

View all tags
@ashhart ashhart released this 05 Nov 17:36
· 464 commits to main since this release
v1.2.3
6ec5fa4

TensorFold 1.2.3

1.2.3 is mostly fixes, memory and serving work, with a few new paths.

  • HIP: dots.mocr, Qwen3-Embedding-8B and Qwen3-Reranker-8B are served natively, and Qwen 3.5 loads BF16 checkpoints through rocBLAS.
  • Metal: the Qwen3.8-27B engine serves Qwen3.5-MoE checkpoints, including Ornith-1.5-122B on one M5 Ultra.
  • PDF documents work as input with --vision --vision-pdf.
  • Intel GPUs: LFM3-1.7B runs on Iris Xe.
  • Kimi K3 and K3.1 are reachable through the native server on Metal, but only as groundwork. They have run on a synthetic graph, not on real weights; see below.
  • Speed: Flash Next stops a long prompt from stalling a running reply on Macs without tensor units, and decodes faster on a GB10, at --tp 2 on two GB10s and on a Strix Halo. A GLM cold prompt at TP=3 gets its first token sooner.
  • Memory and correctness: a prompt cache you size with --prompt-cache-gib now gets that size on CUDA KV pages. Cached-token counts can't exceed the prompt. A few loaders and decoders refuse bad input they used to accept.

Where a speedup was checked for equal output, its line says so. Opt-in switches change nothing unless set. A few changes move replies, change what a request returns, or refuse a start that used to go ahead. They're listed under What can change for you.

Headline numbers

Each number names its machine. Where replies were checked against the base, the line says how.

  • Flash Next NVFP4 on two GB10s at --tp 2: plain decode is 21% to 23% faster, drafted decode 9% to 11% faster at one stream and 76% to 77% at two, and prompts about 6% faster. Each rank holds 2.62 GiB less of the model and a stream's KV at 65,536 tokens takes 40% less, with replies token-equal to one GPU (#1784).
  • Flash Next on an M3 Ultra, --parallel above one: a 1,024-token reply beside a 64K prompt finishes in 8 s instead of 48 s. The prompt's first token comes at 54 s instead of 90 s, with the same tokens (#2167).
  • Flash Next on an M3 Ultra, a lone reply at --parallel above one: it runs as fast as at --parallel 1. Sampled replies are 12% to 19% faster than before and greedy ones about 4%, with the same tokens (#2199).
  • Flash Next on an M3 Ultra with --ssd-experts: about 11% less read from disk at 17 and 35 GiB pools, and a mixed workload took 8% less time at 35 GiB, with the same replies (#2209).
  • Flash Next FP4 on a GB10: one-stream decode up to 11.5% faster (chat +8.7%, code +6.3% at temperature 0), with every reply token-equal to the previous version (#1786).
  • Flash Next on a Strix Halo (gfx1151): the first request after start decodes at 35.8 tok/s instead of 24.4, and a cold 7K-token prompt starts in 28.9 s instead of 48.4 s, with every reply token-equal (#1615).
  • HIP on a gfx1151: Qwen3.8-27B at 16 streams goes from about 51 to 75 tok/s on code, with the same bytes. These are the contributor's own records, as we have no AMD GPU (#1694).
  • GLM-5.3-Flash on three GB10s at TP=3, with TF_GLM_ROCE_MULTI=1 and TF_GLM_CHUNK_SOLO=4096: a cold prompt's first token comes about 5% sooner, with the same tokens (#2116).
  • GLM-5.3-Flash on two GB10s at TP=2 with --drafter: TF_GLM_CHUNK_SOLO=4096 now serves, and a cold prompt's first token comes 6.1% sooner at 32K and 7.0% sooner at 64K, with the same tokens, for 80 MiB more a rank (#2219).
  • GLM-5.3-Flash on an M3 Ultra: about 14% more throughput at four streams when drafts land often, with the same tokens. This was measured on 40 of the model's 45 layers, not the whole model (#2111).
  • GLM-5.3-Flash csf0_v1 checkpoints on CUDA: about 13 GiB less across a TP=2 pair, with the same tokens (#2041).
  • Flash Next MLX 4-bit on a GB10, against 1.2.2: one stream is 3.5% to 6.6% faster and eight streams 1.1% to 3.8%, with prompts level and the same tokens. Decode after a long prompt is faster in 18 of 20 of our cells, by 2.3% to 14.8%; two cells where drafts land very often are slower, 32K by 5.2% and 64K by 3.1%, the cost of the same draft chain rule.
  • Qwen3.8-27B on an M5 Ultra, against 1.2.2: level, within 0.4% in every sweep and within 0.9% on decode after a 32K prompt, with the same tokens.
  • Nemotron on a GB10, against 1.2.2: level over 18 repeats a build at four streams, the one width where the same request can swing widely between runs, on both versions.
  • Qwen3.8-27B on a GB10, against 1.2.2: level, within 0.7% in every sweep and on every prompt and decode cell from 2K to 64K, with the same tokens.

New models and backends

  • dots.mocr on HIP: the publisher's BF16 dots-studio/dots.mocr OCR checkpoint is served natively, with its own template, tokenizer, image processor and EOS ids. It's token-exact against its model card's example (#1938, part of #1927).
  • LFM3-1.7B on Intel GPUs through XPU, Iris Xe first (#1696).
  • Qwen3.5-MoE on Metal: the Qwen3.8-27B engine serves qwen3_5_moe checkpoints: Ornith-1.5-122B on one M5 Ultra from the publisher's own 4-bit MLX weights, with routed and shared experts, its DFlash2 drafter, images, the prompt cache and several streams. The dense 27B is unchanged (#1125).
  • Qwen3-Embedding-8B on HIP: served as Qwen publishes it (BF16) through /v1/embeddings on a Radeon AI PRO R9700 (gfx1201). Inputs of up to 32,768 tokens batch to the same vectors as alone, and dimensions takes any width from 32 to 4,096 (#1978).
  • Qwen3-Reranker-8B on HIP: POST /v1/rerank answers with results (index, relevance_score, document), and takes top_n and return_documents. Each document's prompt is the model card's template, token for token (#1981).
  • GGUF, groundwork: Qwen3.8-27B on Metal gains a host adapter for GGUF metadata, tensor names and exact affine preparation. GGUF serving remains unavailable (#2064, #1920).
  • Qwen 3.5 on HIP loads BF16 checkpoints with their Hugging Face tensor names, and runs the BF16 projections through rocBLAS. MLX checkpoints are unchanged (#2046).
  • Nanbeige4.3-33B-INT4 on Metal: Nanbeige's own int4 release (compressed-tensors, groups of 32, 21.0 GiB) is served as published, read in place. Any other quantization of the family is refused by name. Against a float64 reference over 1,500 teacher-forced positions on an M5 Ultra, the engine's mean KL is 0.0044 with 97.1% top-1 agreement. Prompts need Metal 4 tensor units, so M1 to M4 Macs are refused at load (#1782, part of #1414).
  • Kimi K3 and K3.1 on Metal, groundwork only (#2221, #2192):
    • The native server renders chat, reasoning and tool calls for both.
    • They have run only on a synthetic four-layer graph. No real weights have run yet, and token ids against the release's tokenizer are still to be checked.
    • The full 1.45 TB checkpoint doesn't fit one Mac through this path.
    • K3.1 is capped at 2,047 tokens until its sparse index is qualified. A longer request is refused with that limit's message.
    • A K3.1 config's dsa_config is checked, and refused where unsupported, instead of being read as K3.
    • Images, external draft heads, TP, learning, disk recall, structured-output enforcement and budgets are refused by name.
  • The native tokenizer reads a Qwen2 checkpoint's slow-tokenizer files when it ships no tokenizer.json. Structured output still needs tokenizer.json (#2225).
  • Clusters: the fabric layer can link nodes over ConnectX (RoCE v2), the first step of #1594 (#1957).
  • Clusters, groundwork: the fabric layer gains a pipeline stage link, a TCP ring between ranks on Linux and macOS that carries each round's plan and hidden rows. Each link proves a shared key both ways before any frame. Nothing serves through it yet (#2012, step 1 of #2005).

Speed

  • Flash Next:
    • At --tp 2 on CUDA, each rank computes its own output columns of the dense projections instead of both repeating the whole projection, with the same replies as one GPU. The numbers are under Headline numbers (#1784).
    • On a GB10 beside another open stream, a greedy chain stops at 0.6 per draft, and a round with more than four streams drafting drafts greedy chains 6 deep. That fixes #1786's loss at eight greedy streams: MLX 4-bit chat goes from 288.8 to 303.8 tok/s and code from 336.9 to 358.5, FP4 stays level and every reply keeps its bits. TF_FLASHNEXT_GREEDY_WIDE sets the width and TF_FLASHNEXT_SHARED_DRAFTS=0 keeps the alone rule (#2160).
    • On a Mac without tensor units, with --parallel above one, a long prompt beside a running reply no longer slows both. Replies take 0.8 of the time by default, and the prompt's steps grow from 32 to 512 rows after eight. Numbers are above. --decode-share 0 keeps the old order (#2167, #1893).
    • FP4 on CUDA: drafted rounds send the first token earlier, keep the n-gram table resident, draft deeper on greedy requests and replay rounds through CUDA graphs (#1786).
    • On a Mac without tensor units, with --parallel above one, a lone reply runs as fast as at --parallel 1, and a second request joining it leaves it unchanged (#2199).
    • With --ssd-experts, the expert pool counts uses, so an expert used once leaves before one used again (#2209).
    • On HIP, the n-gram table's mapped tensors are read with random-access advice, so a gather's single-row reads fault only their own pages (#1615).
  • HIP:
    • A lane round of more than four rows reuses cached weights instead of reading them from memory again for each row block, and makes a chunk's code pairs once. On gfx1030, the 2x8 tiles that would spill keep their old loop (#1694).
    • Qwen 3.5's prompt convolution writes q, k and v to their own rows, which drops 144 of the 27B's 160 column copies, with the same results (#2117).
    • Qwen 3.5 prompts run the residual add and post-mixer norm in one launch, with the same bytes (#2099).
    • Qwen 3.5 decode reads the BF16 router rows the model already holds, with the same tokens (#2007).
  • GLM-5.3-Flash on CUDA:
    • At three ranks or more, where TF_GLM_ROCE_MULTI=1 opens the wire, TF_GLM_CHUNK_SOLO=4096 now puts a prompt chunk's exchanges in 8 pieces instead of 3.
    • TP=2 and every other combination keep 3. A set TF_GLM_OVERLAP_PIECES still wins, and rank 0 prints the count at startup (#2116).
    • TF_GLM_CHUNK_SOLO=4096 now serves beside --drafter (DFlash2), where it was refused, for 80 MiB more on each rank (#2219).
  • GLM-5.3-Flash on a Mac without tensor units learns what a shared round's row costs on that chip, instead of pricing it from an M5 Ultra's table, so it stops over-drafting with several requests at once.
  • CUDA on the RTX PRO 4500 (sm_120):
    • The dense 27B's MLP uses a 128-row tile, byte-equal to the 64-row one (#1730).
    • Attention takes the MT2 FP8 path from 8,192 cached tokens, token-equal to the default path (#1731).
  • Rerank on HIP: a request's documents go to the engine as one decision, and Qwen3-Reranker-8B packs them into as few prompt passes as fit, with the same responses byte for byte. 16 documents of about 260 tokens take 2.69 s instead of 3.22 s on a Radeon AI PRO R9700 (#2039).
  • HIP on the gfx1151 iGPU routes a many-expert prompt through a 64-row tile, with up to 12 expert pairs an item and prompt chunks capped at 1,024 rows. Other GPUs are unchanged (#1658).
  • Opt-in:
    • --policy matrix=on runs Qwen3-8B's BF16 projections on a new WMMA tile on RDNA4 (gfx12). On an R9700, Qwen3-Embedding-8B embeds 100 short inputs in 0.99 s instead of 2.47 s, and Qwen3-Reranker-8B ranks 16 documents in 1.39 s instead of 3.22 s, or 1.00 s with the batching above. The tile's fp32 sums are less precise than the dot2 chain's, so the default keeps the dot2 tile and its bits. gfx1200 is unmeasured (#2034).
    • TF_LOAD_PREFETCH adds checked read-ahead to CUDA loaders on Nemotron, Bolt, MiniMax M3, Kolibri-1, Spark 2.5 and DeepSeek-V4.1, off unless set. GLM keeps its default of 16 readers. This extends @jschmied's #1928 mechanism (#2033).
    • TF_GLM_COPIES=1 adds context copies for GLM on CUDA, verified and priced with the same row costs as drafts. Tokens are unchanged (#2146).
    • TF_KV_SHARE=n keeps Kolibri-1's prompts as its KV pool's pages, as Gemma 4's already can, with replies equal to a fresh prefill. It costs about 3.5% decode after a short cold prompt on Kolibri-1 (#1755, part of #1657).
  • llama with --drafter: a probe stays armed until a round verifies its drafts, so a request that once stood its drafter aside gets drafting back. On an M5 Ultra (Nanbeige4.3-33B with a 3B drafter), the case that had stood aside decoded at 17.15 tok/s instead of 11.86, and all 72 replies matched (#2169, closes #2168).
  • GLM on CUDA at TP>1 with DFlash2: the drafter reads complete prompt taps on every rank. It read stale rows under the split prompt reduce. Acceptance is back to the gathered level, with every reply unchanged (#1623).

Memory and long conversations

  • --prompt-cache-gib N on CUDA with KV pages now keeps N GiB before the pool sets aside windows past the engine's own. Before, a named cache could be cut to almost nothing at some room sizes. Unset, the order is unchanged. The startup line now counts the windows the pages hold (#2220).
  • GLM-5.3-Flash csf0_v1 checkpoints on CUDA: NVFP4 routed experts carry their scales at block 32 as a base plus a 4-bit offset. That saves about 13 GiB across a TP=2 pair, with the same tokens. Block-16 checkpoints are unchanged (#2041).
  • Kolibri-1 on CUDA keeps its full-attention layers' caches in pages of the core KV pool, as Gemma 4 does, with identical tokens when the prompt store is off (#1620, part of #1657).
  • Qwen 3.5 on CUDA: the mixers' and the dense MLP's scratch rows share one region. That frees 0.46 GiB on the 27B, with the same tokens and speed (#1476).
  • Saved prompt states on disk now include the profile settings that can change a family's arithmetic. Changing one of those settings starts a fresh disk namespace, so the first long prompt under it prefills again (#2131).
  • Disk prompt states at startup: when allocation fails, valid files are kept and the failure is reported, instead of the files being treated as damaged (#2134).
  • DeepSeek-V4.1 on CUDA reads --prompt-cache-gib as the kept states' resident limit on every rank (0: none resident in RAM; with TF_DSV41_SESSION_DISK an evicted chat is parked on disk). With TP > 1, rank 0 sends its limit to the others at startup (#2056).
  • GLM on CUDA with TF_GLM_KV_SHARE: a stream with an image now keeps and resumes its prompt, so one screenshot no longer makes every later turn fill the whole prompt again (#1833, for #1828).
  • Qwen4 and tensor-parallel loaders on CUDA: Qwen4 drops its checkpoint from the page cache after loading, and tensor-parallel loaders drop theirs only once every rank has finished, so ranks on one host no longer evict pages another is still reading (#1887).

Serving, build and API

  • PDF documents as input with --vision --vision-pdf (#1820).
  • Living Weights: tensorfold slide audit measures each learned fact's pocket neurons against its questions, subject contexts, other facts and 256 fixed general prompts. It's read-only: no weight, lesson or check changes (#1980).
  • Tool calls:
    • DeepSeek-V4 and V4.1 tool calls stream their arguments as they're written, from the first parameter's end, like GLM and XML calls. A call whose first parameter is the long one still arrives when that parameter closes (#1975).
    • GLM tool calls stream their arguments as the model writes them, and the joined pieces equal the whole reply's arguments (#2050).
    • In one-call tool mode, a stream sends any prose held after a hidden call envelope at the finish, so the streamed content equals the whole reply (#2049).
  • cached_tokens and prompt_tokens_cached_total can no longer exceed the prompt, whatever a backend reports (#1899).
  • Retryable 503s on the OpenAI-shaped routes name their wait in a fixed code: server_restarting, engine_loading, engine_reload_failed, engine_busy, engine_unavailable or memory_pressure. A parked model's reload now says engine_loading or engine_reload_failed instead of server_restarting, and a busy embeddings engine says engine_busy. Status codes, Retry-After and the drain's body are unchanged (#2202, for #2135).
  • /v1/embeddings: a batched request no longer hangs on a backend that prefills many prompts at once (#2047).
  • Decision routes include context_length_exceeded on context-limit errors for text-only prompts, keeping their status and text (#2183).
  • --reasoning-effort accepts minimal and max, from the shared request vocabulary. Template coercion and --no-thinking are unchanged (#2139).
  • capabilities and model discovery say what each server does (#2139):
    • HIP Qwen format discovery names each admitted MLX bit width and group, such as mlx-q4g64 (#2188).
    • HIP Qwen 3.5 and Qwen 3.5-MoE no longer list BF16, since those servers refuse a BF16 checkpoint for chat.
    • Fish Audio Drama 3 on Metal lists its NVFP4 expert weights beside FP8.
    • GLM-5.3-Flash on CUDA lists the EXL3 format its loader already serves.
    • Discovery omits reasoning for engines that disable it, including HIP OCR and scoring models.
    • It keeps reasoning.default_effort for templates that take reasoning_effort as free text.
    • Nemotron-H on Metal reports its 262,193-token limit in /v1/models when neither the checkpoint nor --context sets one.
    • The docs say what capabilities --json's env covers.
  • Model discovery reads one engine-info snapshot under the idle-unload lock, which avoids reload races (#2186).
  • Flash Next on Metal reports a filling prompt's progress, so the dashboard shows what's left, and TENSORFOLD_PREFILL_ORDER=srpt works from a server's first long prompt (#2194).
  • DeepSeek-V4.1 logs its resolved serving settings at start-up, so two runs can be compared setting by setting (#1395).
  • --profile stable now covers GLM's host KV tier (TF_GLM_KV_HOST, TF_GLM_KV_HOST_SHARED). Stable sets no value, so nothing changes by default (#1660).
  • Nemotron on CUDA: TF_KV_PAGES=0 and TF_KV_PAGES=all are now reported as overrides at startup and in capabilities, for Nemotron-H and Nemotron Omni, instead of "registered overrides: none". How the value is read is unchanged (#2061).
  • Clusters: cluster check, cluster status and serve's preflight exit 1 when a configured model can't be planned (#2069). The planner gives a dense model's rounds zero expert cost instead of dividing by zero, so its round times are finite (#2070).
  • Community benchmark receipts (v2) can record 1 to 8 loaded ranks and each node's hardware, in separate single-node and multi-node categories. v1 is unchanged, and v2 uploads stay off for now (#974).
  • tools/bench_fill.py measures a long prompt filling beside a running reply, in both arrival orders, with every reply checked against its run alone (#2154).

Fixes

  • GLM on CUDA at TP=2 with TF_KV_PAGES=on and TF_GLM_KV_HOST=1: a long first prompt no longer faults (#2173, fixes #2119).
  • Qwen 3.5 on CUDA: a kept path longer than the 48-row pending buffer is kept at once, which fixes the occasional INVALID_VALUE on deferred keeps (#1982).
  • GLM-5.3-Flash on CUDA reads block 32's scales only for a csf0_v1 checkpoint, and every rank refuses a scale offset past 0x7E by name (#2091). TF_GLM_PREFILL_REDUCE=ring warms up only each rank's ring neighbours at startup (#1971).
  • Qwen 3.5's attention-layer pattern (layer_types, full_attention_interval) is read the same way on Metal, CUDA, HIP and XPU. A layer_types list that contradicts the interval is refused, naming both keys (#2086, fixes #1863).
  • MiniMax-M3 on Metal refuses logprobs with an external drafter before prefill. Plain decoding keeps logprobs (#2139).
  • Six remaining fp32-to-bf16 conversions use core's shared rounding. Every non-NaN input gives the same bits, and a NaN now stays a NaN (#2052).
  • Dory on CUDA takes its RoPE tables from core's exact helper, matching transformers' fp32 values (#2035, part of #1852).
  • PNG pictures with a tRNS key past 255, or a 16-bit key, decode with Pillow's transparency. Pictures without such a key decode as before (#2100).
  • Qwen3.5-MoE's prompt sorts its experts' rows in one threadgroup, in the same order as the serial sort (#1799).
  • Cortheon keeps a source, link, page or query that differs only in spelling, and resolves . and .. without dropping empty path segments (#2036).
  • The kernel contract:
    • frees each kernel's device buffers once its gates are done, so the full suite fits on a 16 GiB card (#2042);
    • checks Qwen3.8-27B's attention_io at its other two head shapes, GQA16 and the 9B's 16 over 4 (#2063).
  • Tests and builds:
    • test-native no longer crashes on the Qwen27 lanes' facts (#2122);
    • six host-test steps that had gone red pass again (#2024);
    • the Recall tests pass on a machine with less than 4 GiB free (#2051);
    • passing test binaries no longer print, so zig build test shows failed command: only when a test fails (#2153);
    • XPU's dense 27B takes its kernel sizes from one shared header, with byte-identical SPIR-V (#1425);
    • HIP's OCR families share one host protocol, with the same replies (#2040).
    • HIP's tf_sum_partials records follow ROCm 10.1's hipcc, taking 8 limit problems off the contract (#2233).
  • Living Weights: a lesson's extra wordings are seeded from the lesson, so the same lesson samples the same wordings on each run. A refused lesson round names the sub-check that failed (#1587).

Docs

  • CUDA docs match the code. README, RUNBOOK and the recipes say which families refuse an A100 at startup, that no NVFP4 or FP8 model has run on sm_80 to sm_89, and that the largest recorded run is three GB10s (#2227).
  • Recipes:
    • The Qwen3.5-2B recipe says which listed format loads at its geometry: only mlx-q4g64 (#2139).
    • The Flash Next recipe says native Metal takes --ple-on-ssd (#2068).
    • The GLM CUDA recipe covers TF_GLM_OVERLAP_PIECES at TP=3 with TF_GLM_ROCE_MULTI (#2067).
  • README: the GLM-5.3-Flash Jammed XSFP4 row (#2114).
  • THIRD_PARTY_NOTICES.md names the model release and library code that some families' files follow, with the LFM Open License text (#2197).

What can change for you

These change what some requests get, or stop a start that used to go ahead.

  • --prompt-cache-gib N on CUDA with KV pages keeps N GiB first, so a command that already names N splits memory differently after the upgrade: a bigger cache, fewer pool pages. To get the old split, name a smaller N or leave it unset (#2220).
  • Saved disk prompt states for Qwen and Flash Next on CUDA from earlier versions aren't reused. The first long prompt after the upgrade prefills again, once (#2131).
  • Flash Next on a Mac without tensor units, --parallel above one: a running reply now takes 0.8 of the time beside a filling prompt. --decode-share 0 keeps the old order (#2167).
  • GLM-5.3-Flash at three ranks or more with TF_GLM_ROCE_MULTI=1 and TF_GLM_CHUNK_SOLO=4096 uses 8 exchange pieces by default. TF_GLM_OVERLAP_PIECES=3 keeps the old count (#2116).
  • Dory on CUDA: its RoPE values moved by up to 5 ulp to match transformers, so replies can change where a token was a near-tie. There's no switch back; the new tables are the reference's. Decode speed is unchanged within 0.3% on a GB10 (#2035).
  • A NaN in six fp32-to-bf16 conversions now stays a NaN, where it became a finite value before (#2052).
  • PNG pictures with a tRNS key past 255, or a 16-bit key, now decode with transparency, so a reply about such a picture can change (#2100).
  • DeepSeek-V4 and V4.1 with stream: false: a reply cut by length inside a DSML tool call now returns that call, not text (#1975).
  • Streaming clients get GLM tool-call arguments in pieces as they're written, and in one-call tool mode the held prose arrives at the finish (#2050, #2049).
  • cached_tokens is capped at the prompt's length (#1899).
  • Retryable 503s carry a code, and a parked model's reload is no longer reported as server_restarting (#2202, for #2135).
  • Decision routes add context_length_exceeded to a text-only context-limit error.
  • capabilities: HIP Qwen 3.5 and Qwen 3.5-MoE no longer list BF16, and discovery omits reasoning for engines that disable it (#2139).
  • /v1/embeddings on every family refuses a whitespace-only text, and names an input longer than the context with its token count (context_length_exceeded) before the batch limit (#1978).
  • /v1/rerank: only an engine that declares a reranker answers. Every other server returns HTTP 400, and a reranker server refuses the other model routes by name (#1981).
  • The lane host's score queue stops at a drain's deadline: the running score finishes and queued ones fail without running. This also reaches /v1/decisions on Flash Next on CUDA (#1981).
  • DeepSeek-V4.1 at TP > 1: a TF_DSV41_SESSION_RESIDENT_GIB set on rank 0 wins. Set on another rank, it must equal rank 0's limit or every rank refuses to start (#2056).
  • New refusals:
    • MiniMax-M3 on Metal: logprobs with an external drafter (#2139).
    • Qwen 3.5 on every backend: a layer_types list that contradicts full_attention_interval. XPU now checks a present config.json at load (#2086).
    • GLM-5.3-Flash on CUDA: a csf0_v1 scale offset past 0x7E (#2091).
    • Kimi K3.1: requests past 2,047 tokens, and an unsupported dsa_config (#2221, #2192).
  • Clusters: cluster check, cluster status and serve's preflight exit 1 when a configured model can't be planned (#2069).
  • Cortheon keeps sources and links that differ only in spelling, so a turn can see different fetched content (#2036).

Thanks

Contributors

diwakergupta, laudney, and 24 other contributors
Assets 11
Loading