TensorFold 1.1.2

TensorFold 1.1.2

1.1.2 keeps request text out of control tokens, decodes union tool parameters as JSON, takes Flash Next's CUDA prompts through faster tiles, and tightens native serving and memory admission.

Security and HTTP serving

• Chat prompts encode control-token spellings in request text as ordinary text, including messages, tool definitions, tool results, reasoning fields and decision labels (#1117). Templates still see the real text. A prompt that spells a control token can now count more prompt tokens because the spelling is ordinary text. A request whose text would still encode to an added-token id is refused instead, explicit --vision refuses to start when an image marker is not an added token, and LFM2's automatic image input turns off with a warning rather than failing the load.
• HTTP serving bounds total header and body read times and caps open connections at 128. A full server returns 503 with Retry-After; inference and streamed replies keep their existing timing policy.
• HTTP serving consumes and validates GET and DELETE bodies before routing, refuses incomplete or malformed request heads and folded header lines, and bounds all chunk framing together. An API-key file that a group or other users can read or write is now refused.
• TENSORFOLD_REQUEST_LOG writes only to a regular file that no one but its owner can read or write; it refuses a symbolic link or a shared file with one warning, and creates a new log 0600.
• HTTP serving closes unfinished replies, reports token-staging allocation failures and releases SSE frame storage after each write. Media and web fetches share one public address policy.
• Cluster checkpoint headers use the core safetensors parser and validate tensor ranges against the actual file length.
• Checkpoint shard indexes need checked JSON types and names inside the model folder, and checkpoint writes stage in an exclusive sibling file before replacing the target atomically. The secondary Metal, GLM, Flash Next and Kimi readers share the bounded safetensors header checks and refuse overlapping tensor payloads.
• Model pulls refuse unsafe portable paths and fallback blob names, follow nested snapshot links at their real depth, and never write past a file's declared size.
• Research web redirects that leave the first URL's scheme, host or port drop the request's extra headers, such as a configured search key, for the rest of the chain.
• Streaming research waits for the HTTP response head before sending chat, Messages or Responses keepalives. Late context-length refusals stay on the stream, with pre-token keepalives retained (#912).

Tool calls, reasoning and images

• Tool parameters are decoded as JSON when their resolved schema accepts the parsed non-string kind, including schemas built from oneOf, anyOf, allOf, a type list or a local $ref, streamed and not (#1121). Text that is not JSON, quoted strings and schemas that determine no kinds keep the model's text.
• --thinking-reserve N keeps max_tokens for the answer when thinking is on: lane engines allow up to N more tokens for reasoning, within the context window, and end the reply when the answer reaches max_tokens (#1064). It is off by default, and other engines say at startup that it is not applied.
• --loop-guard also closes reasoning that repeats a long passage verbatim (a whole period and at least 1,024 tokens), not only short cycles (#1075). It still acts only while the think block is open.
• With --vision, startup warns when the chat template renders an image part without an image placeholder, naming --chat-template, and an image request whose template writes none says so rather than reading as an engine failure (#1076).
• HTTP serving and the Kolibri runner share one end-of-sequence resolver that merges the model's and the generation config's EOS ids, so a declared second end token also stops HTTP replies. A present EOS entry with a negative, fractional, boolean or out-of-range id refuses loading.

CUDA serving and drafting

• Flash Next on CUDA takes prompt rows through factored tiles: dense matmuls, routed experts, hyper-connection read-out, router logits, the recurrence, indexer scores and 1,024-row attention launches, each bit-equal to the 1.1.0 kernel it replaces and registered with the kernel contract (#1046). The tiles run only where native glue owns every operation they replace, and TF_FLASHNEXT_CHUNK takes 1 to 16,384 rows, refusing anything else before loading; startup reports the chunk actually loaded. Measurements are below.
• Flash Next on CUDA serves structured output and required tool calls: the grammar masks the first draw and every draw in a verification window (#1062).
• Qwen3.5-family MoE models on CUDA route only a captured bucket's live rows through the experts, so padding rows no longer reach the router, top-k, plan or combine, and their experts' weights are not read (#1047).
• Nemotron on CUDA runs the RADIO vision tower once per image prompt instead of twice (#1057). Shared windows of several sequences on one GPU run the latent shared expert on its side stream again; tensor-parallel runs keep the main stream (#1058).
• GLM tensor parallelism on CUDA lets waiting ranks sleep between requests instead of polling at full load, waking for rank 0's next step; a rank that fails while the others sleep still stops them (#1053).
• GLM on CUDA keeps each committed prompt prefix in bounded, preallocated snapshot pages instead of its whole context arena, so saved prompt states stay inside the prompt-cache budget (#993).
• GLM on CUDA runs the uneven KDA input projections of TP=3 (22, 21 and 21 heads) on the existing row-major matmuls instead of refusing them; TP=3 device qualification is pending (#1067).
• Kolibri keeps its 16 verification rows past the advertised window in its caches, memory pricing and admission, so a request at the advertised edge no longer fails admission.
• Flash Next shares its existing proposal table and exact stored rows without changing its default IDs or packs.
• Shared draft rounds offer one deeper row every eighth round when the row cap allows, keeping normal pricing and trimming (#1059).
• Qwen3.8-27B CUDA sizes recurrent verification scratch by its live windows and keeps it between rounds, retaining fp32 state and exact rollback.
• Qwen3.8-27B CUDA can prefer verified context copies over neural proposals through the shared lane core. Copy rounds keep the drafter's committed target rows current, and neural drafting resumes after rejection.
• Qwen CUDA keeps DFlash and MTP proposal and live-context projections on native arithmetic regardless of shared frame size, so a stream's drafts do not depend on which streams share its rounds. On GB10 a shared frame's rows go through one launch of 32-, 64- or 128-row verify tiles, each row with the one-row decode's bits, and the target's wider verify frames take the same tiles. Prompt ingestion keeps its historical arithmetic. Measurements are below.
• Nemotron tensor parallelism requires NCCL 2.30.0 or later, sharing Flash Next's minimum. Older loaded libraries are refused before rank setup (#1060).
• Shared CUDA sliding-window attention uses bounded split launches, with sigmoid-routing and dense-row contract fixtures (#798). Tree and DFlash block attention gain exact-replay launch contracts and bounded resource records (#746).
• Optional GLM CUDA diagnostics under TF_GLM_PROF report per-rank prompt-chunk stages, transport phases and event waits without changing context ownership. On the 27B CUDA path, TF_TIMING reports offered and kept rows, capture bucket and forward time for each verification frame.

Shared serving

• Shared lane rounds let other streams use spare rows when a forced window does not fit, and shared prompt scheduling prefers admitted prompts with less work left, keeping foreground preference and bounded waits.
• Shared lanes reserve their commit storage before grammar transitions and token publication, so an allocation failure publishes no rejected token.
• Prompt-cache recall picks a disk restore by its estimated time saved over the best RAM state, within the memory budget. The tokenizer's memo cache keeps recently used segments when it evicts.

Metal and HIP serving

• Qwen3.8-27B on M1 to M4 processes prompt chunks through the M5's prompt-attention approach using simdgroup matrices (#918). Each row's bits remain independent of how the prompt is cut, error against float64 stays at bf16's half step, and decoded rows are unchanged.
• For stored bf16 eh_proj weights, GLM-5.3-Flash on Metal runs the MTP draft head's prompt-side eh_proj on a tensor-unit tile kernel by default; if its pipeline cannot build, it warns once and uses the scalar path, and GLM_EHPROJ_NAX=0 selects the scalar path. Quantized eh_proj weights keep their quantized path (#1066).
• On M1, quantized matmul tiles covering 16 rows use at most four simdgroups when the grid has at least 512 groups and the K-split count is at most 16; partial sums keep their fixed reduction order, so results are unchanged (#1055).
• Qwen3.8-27B on Metal takes prompt matmuls of 16 K chunks in 16 by 32 tiles, each simdgroup adding a pair of chunks, after a once-per-shape check on the chip that the pairs give the earlier tiles' bits.
• GLM-5.3 on Metal skips the sparse indexer's scoring pass while its 2,048-key top-k already holds every causal key, where those scores were never read; longer windows are unchanged (#1054).
• Flash Next on Metal releases its buffers, pipelines, mappings and files after a partial load or at shutdown.
• Flash Next Metal response prompts run through the shared lane host, with at most 32 prompt rows per choice beside a decoder. A long prompt started alone can hand unfinished work to the host when another request arrives, retaining its session and MTP lookahead (#1032).
• Plain Qwen3.8-27B Metal prompt caching can copy up to four interior DeltaNet marks without separate prompt calls, within the memory bound (#548, #851).
• Flash Next can reuse precise rotary values across target layers and MTP calls on tensor-unit Metal devices. The direct path remains in use when the optional table does not fit.
• The optional 27B Neural Engine share accounts for 32 MiB of private overhead and reserves its full size in the prompt-cache budget, including cold builds (#643). It remains off by default, selected through TF_ANE_COLUMNS; exact agreement with GPU output remains unqualified.
• Metal serving keeps weights, stream state and draft, image and speech working buffers resident between requests within the memory bound. GLM enables the shared set by default; GLM_RESIDENCY=0 turns it off.
• Flash Next Metal checks weights, startup scratch and requested storage against the RAM/GPU-bounded load ceiling before loading (#1039).
• Qwen on single-GPU HIP polls cancellation inside long prompt chunks, releasing cancelled work at layer boundaries (#1028). Short chunks and tensor-parallel runs still stop between chunks.
• HIP refuses --drafter before loading, since it does not run a DFlash2 drafter yet; the refusal leaves the engine's native drafting settings unchanged (#1044).

Benchmarks

• ROCm benchmark receipts name the GPU, read from rocminfo under HIP's device-visibility rules, and the CPU. An attached server can attest the checkpoint revision, GPU identity and receipt backend through a valid benchmark manifest; without one, the attached run stays unranked. Managed runs, inspect and publish say which ranking facts a receipt lacks, before any upload. ROCm board publication remains disabled (#1043).
• tensorfold benchmark keeps each attempt's draft counts (verify rounds, drafted and accepted tokens) in a local file beside the receipt, and reports no totals rather than overflowing when a sum would pass 64 bits; the published receipt is unchanged (#1045). A managed ROCm run refuses a named external --drafter and records none; an attached run does not configure the server's drafter (#1044).
• The Flash Next CUDA recipe adds single-GB10 measurements of the ModelOpt FP4-Local checkpoint, labelled with the earlier build they ran on (#1042).
• tf-qwen27-run --profile-prompt splits a timed Qwen3.8-27B prompt's GPU time by class (off by default; profiling slows the prompt). The cache and concurrent benchmark tools time a whole workload from its first request to its last completion and refuse incomplete throughput totals.

Checkpoints, builds and sessions

• Checkpoint hashing uses a bounded file pool and runtime SHA acceleration while preserving digest identities and atomic cache publication. On Linux, runtime SHA2 capabilities come through libc when libc owns process startup.
• Windows host builds and tests build again: dynamic libraries, clocks, environment reads and socket options go through the portable system layer, and POSIX-only tests skip on Windows, while Linux and macOS run what they did (#1034).
• Linux video looks for libtfvideo.so in the executable's ../lib directory, then beside the executable, then on the loader's path, as capability discovery does (#1107).
• The README opens with a pinned native-model quick start for Mac and Linux and a table of native model choices.
• Mac source-build instructions name full Xcode and its Metal Toolchain component. The build example uses the package manifest's version (#1049).
• The session prefix store reserves its free lists before publishing a slot, so removal and rollback no longer allocate (#637).

The app

• The app checks credentials, remote access, MCP launches and workspace files before use: restored masked credentials stay bound to their endpoint and account, revoking a paired device or rotating the owner credential ends its live streams, a stopped MCP server starts only after native confirmation of its whole launch configuration, and verified downloads refuse invalid sizes, looping pagination and links that leave the configured Hub endpoint.
• The app puts its planning-only note in the first system message, does not retry a recognized chat-template refusal, and retries a refused reasoning effort once at the nearest level the server names (#884).

Prepared work and qualification holds

• Exact stored proposal-head rows, shared scoring views and optional shared routing for native and external draft heads are prepared, with full or absorb-only fallback. Family defaults and precision are unchanged.
• Immutable shared proposal-vocabulary artifacts and an offline tools/draft_vocab.py responses builder are prepared; no artifact is activated and defaults are unchanged.
• Shared CUDA operators gain MLX's bf16 rounding steps, ready for families to adopt: residual tails with an optional branch norm and layer scalar (#1056), GeGLU and rounded SwiGLU rows (#1068), affine embedding rows at 2 to 6 and 8 bits with a bf16 scale (#1070), and q/k rows to head dimension 512 with a unit V norm (#1071). Each is registered with the kernel contract, existing kernels are unchanged, and the ck_lane contract fixture no longer reads past its zero buffer (#1073).
• An opt-in GLM CUDA FP8 latent-cache implementation is prepared behind --kv-cache fp8. BF16 remains the default; native quality and device qualification are pending.
• Shared serving and prompt scheduling have an embedding request contract with float/base64 output and declared MRL dimensions (#1035). Native EmbeddingGemma-2 loading and device qualification remain pending. Thanks to @jschmied for the RFC.
• Living Weights implementation and safety work already landed in 1.1.1. Model, retention and save/restart qualification remains pending; the #1029 lesson-counter repair is excluded and remains held.

Thanks to @Acermax, @aartiles, @Mirrdhyn, @BobClawblaw, @jschmied, @ThinkOffApp, @plotarmordev, @EugeneClaw, @Bizuayeu, @mrv777, @jayleaton, @akol1, @balancedscorpion, @cwschroeder, @hudsonwa, @tpischke, @astro-15-alive, @RS91Dev, @ss-cong, @Chris9121 and @Gogo6969.

Measurements

Qwen3.8-Flash-Next (MLX 4-bit with MTP) on one GB10 with CUDA, served with --context 131072 --parallel 2 and compared with 1.1.0 on the same machine and checkpoint. Replies were token-for-token equal to 1.1.0 at every prompt size: 24 of 24 rows at the default 1,536-token prompt chunks and 28 of 28 at the 16,384-token maximum. Concurrent streams at temperature 0 and 1 matched their solo replies, and a two-turn chat with about 32K tokens of history, resumed from its cached state four times in turn and twice at once, gave 1.1.0's reply every time.

Cold time to first token at the default prompt chunk with the prompt cache off, the median of three prompts (three 1.1.2 runs differed by at most 2.2%):

Prompt tokens	1.1.0	1.1.2	Speed-up
2,048	1.78 s	1.37 s	×1.30
8,192	6.84 s	4.93 s	×1.39
13,312	10.73 s	7.67 s	×1.40
32,768	27.48 s	19.28 s	×1.43
40,960	34.52 s	24.04 s	×1.44
65,536	56.68 s	38.83 s	×1.46

With TF_FLASHNEXT_CHUNK=16384 the first token can come sooner still, but its timing depends on the page cache: Flash Next reads its 29.8 GiB of n-gram embedding tables from the checkpoint files, and the larger prompt scratch leaves too little memory for all of them to stay cached. In the three runs that paged least, 1.1.2 took 4.32 to 4.41 s at 8,192 tokens, 16.43 to 16.53 s at 32,768, 32.86 to 32.90 s at 65,536 and 66.25 to 66.39 s at 128,000; in runs that paged heavily the same prompts took up to 4.95 s, 21.13 s, 49.08 s and 92.57 s. 1.1.0 reads the tables the same way and paged heavily during its own 16,384-chunk run, so the default-chunk table above is the like-for-like comparison.

The 1.1.0 figures come from one run on the same machine, and the comparison does not separate the part of the gain that 1.1.1 already shipped.

Qwen3.8-27B (MLX 4-bit with its DFlash2 drafter) on one GB10 with CUDA, served with --context 73728 --parallel 8, three repetitions a cell, compared with 1.1.1 on the same machine and checkpoint. Every concurrent reply equalled its solo run and its "draft": false run, and resumed chats equalled fresh ones with thinking off and on.

Streams	Chat, T=0 (1.1.1)	Chat, T=0 (1.1.2)	Code, T=0 (1.1.1)	Code, T=0 (1.1.2)
1	45.0	44.5	61.0	60.8
2	78.3	82.3	106.6	111.3
4	131.3	133.4	166.0	173.7
8	164.8	179.2	196.5	240.0

Aggregate decode tokens/s. At temperature 1 eight streams gave 142.5 chat and 229.8 code tokens/s (1.1.1: 140.1 and 196.3). Cold prompts at 32,768 and 65,536 tokens on the same 8-stream server took 136.6 s and 566.4 s to the first token (1.1.1: 149.9 s and 625.4 s).

The #1117 fix leaves prompt encoding cost unchanged: on five tokenizer and template pairs (Qwen3.8-27B, Qwen3.8-Flash-Next, Qwen3.5, GLM-5.3-Flash and Nemotron 3.5), a 400 KiB prompt encodes in 3 to 8 ms with or without a spelled control token, at or below the cost before the fix. Requests that spell no control token encode to the same tokens as before; a request that spells one can now get a different reply, because the spelling is ordinary text.

Known issues

• Fixed since 1.1.1's list: Flash Next on Metal no longer runs a long prompt to its end before other streams' rounds (#1032), a --cortheon keepalive no longer arrives before the HTTP response head, and Qwen3.6-35B-A3B with native MTP on CUDA (--drafter at the model's own folder) is no longer slower than the default at one stream: on one GB10 it decoded ×1.72 faster on code and ×1.46 on chat, with identical replies. At eight streams MTP gave up 22% to 30% of aggregate throughput, and the default still runs this model without MTP.
• Flash Next on one GB10 reads its 29.8 GiB of n-gram embedding tables from the checkpoint files through the page cache. With TF_FLASHNEXT_CHUNK=16384 they do not all stay cached: in our runs, time to first token at 65,536 tokens ranged from 32.9 s to 49.1 s, about 1.5 times. With the default 1,536-token chunks and the prompt cache off, three runs differed by at most 2.2%; saved prompt states (up to 16 GiB by default) use the same memory and can push table rows out at that chunk size too.
• Qwen3.8-27B cold prefill at 64K tokens on RTX 50-series GPUs (sm_120) measures about 2% slower than 1.1.0; 32K and shorter prompts, and GB10, are at parity or faster.
• Flash Next on Metal with an explicit --parallel 2 or more keeps its cache-count and lone-decode performance issue. The default --parallel auto is unaffected, and the shared-session repair is held for a later 1.1.x.
• No release binary serves DeepSeek-V4.1 checkpoints until its whole-model gate (#827) passes.
• Living Weights model, retention and save/restart qualification remains pending, and the #1029 lesson-counter repair remains held.

The full list is in docs/known-issues.md.

CI note (added 22 October). The Windows host-test job failed on this tag (run 39673838881). Windows is a build-and-test target, not a release platform: this release ships macOS and Linux archives, and the Linux job passed. The ten failing tests hit four Windows-only problems: replacing a shard file while it was still mapped, tests raising SIGTERM through the C runtime (which ends the process), CRLF checkouts of JSON test fixtures, and an expected / path separator. They were fixed after the tag in 91e77c1, with a fifth Windows-only test fix in f6786a8, and the Windows job has passed on main since.