Strata 0.1.44

Known issues, fixed in the coming 0.1.44.1:

• Macs other than M4 / M5 (reported on an M2 Max): answers can be garbage (#2323: one Metal kernel needs more threads than the chip allows and is skipped silently). Please wait for 0.1.44.1. The M4 and M5 Macs we tested are not affected.
• Building from source with CUDA 12.1 on Linux fails (#2322). Setup keeps your previous engine, so nothing breaks; please wait for 0.1.44.1.
• Quadro P6000 / older Pascal cards on CPUs without AVX2: the engine may stop at start (#2324). Stay on 0.1.43.1 until 0.1.44.1.

0.1.44: Unsloth's 4- to 8-bit files and NVFP4 packs in setup, Qwen3.6-35B-A3B, faster multi-GPU on AMD and Arc A-series cards, and a long list of fixes. Answers on the default models are the same as 0.1.43.1 (byte-identical in our identity tests on every machine we tested), with the exceptions listed under "Defaults that changed" (RDNA4, and the Unsloth files setup configures). Update with UPDATE.bat (Linux: ./update.sh); setup replaces the engine with 0.1.44.

New

• Unsloth's UD-Q4_K_XL, UD-Q5_K_XL, UD-Q6_K_XL and Q8_0 are regular choices in setup, next to UD-IQ4_XS (docs/QUANTS.md). UD-Q4_K_XL is no longer experimental or NVIDIA-only; the three larger files and Q8_0 are new, each pinned by size and SHA-256. Where the experts fit in RAM with 16 GiB to spare, setup keeps every expert in RAM (UD-Q4 on a 64 GB PC with the old budget rule against this rule, 5 interleaved pairs: decode x1.34-1.40, a 4K prompt x2.07). Images are on by default for these files.
  • Quality against Q8_0 (4,995 teacher-forced positions): UD-Q6_K_XL 94.6% same top token (mean KL 0.033), UD-Q5_K_XL 94.6% (0.038), UD-Q4_K_XL 93.1% (0.056), UD-IQ4_XS 88.9% (0.101), IQ3_S 87.7% (0.144), IQ3_XXS 84.5% (0.202), Q2_0 80.6% (0.315). tools/quant_quality.py reruns it on your own files.
  • Speed (one R9700, every expert in RAM): UD-Q4_K_XL 63 tok/s on a story and 85-88 on code; UD-Q5_K_XL 57-60 and 81-83; UD-Q6_K_XL 54 and 76; Q8_0 53-57 and 66-74. llama.cpp on the same box: 16-20 tok/s decode.
  • The K-quant prompt kernels are built by default; on NVIDIA they are a library of their own (strata_kq, shipped in the Windows zips), opened only when a model has these formats, so no other model pays for them. They are opt-in at run time (STRATA_MMQ_KQ=1): prompts x1.3-1.7 on AMD with every expert in RAM, with a prompt-path KL change we have not yet judged on enough text to make it the default.
  • Dense Q8_0 matrices read one row per block where that measured faster (RDNA4 and sm_86), the same bits as before: decode +2.8 to +3.8% on UD-Q4 (5/5 pairs).
• NVFP4 packs (docs/NVFP4.md; sergiywith's port of sergqwer's strata-nvfp4 work). The original model with NVFP4 experts, as ready-made packs (GPTQ by default, NVIDIA's own checkpoint second), pinned and SHA-256-checked, with the original's image encoder. RTX 50 cards have FP4 tensor cores; older RTX cards run the experts at 8-bit precision and UD-IQ4_XS is faster there, which is what setup's menu now says. Quality against Q8_0: 92.1% same top token (KL 0.070), between UD-IQ4_XS (88.9%) and UD-Q4_K_XL (93.1%). NVIDIA only; AMD and Intel setups do not list it. The route tail skip stays off for NVFP4 experts (it costs more quality there); STRATA_ROUTE_TAIL_SKIP=7 turns it on.
• Qwen3.6-35B-A3B (UD-Q4_K_XL, UD-Q6_K_XL, Q8_0), with the file's own MTP block for speculative decode. One strata binary now runs both families (the file's architecture decides). Setup offers it as the last family in the model menu and sizes it by RAM (32 / 44 / 52 GB). On an R9700 UD-Q4_K_XL: 3,466 tok/s on a 4K prompt and 83 tok/s decode (llama.cpp on the same card: 3,054 and 82), about 161 tok/s with --mtp 2; exact against the plain decode on our 10 parity prompts. On an RTX 3060 with 46 GB of RAM (hybrid mode, experts in RAM): 4K prompt 1.47x llama.cpp's best split. NVIDIA and AMD only; the Intel (SYCL) engine and the Mac engine refuse it with a clear message.
• Image generation is in preview (POST /v1/images/generations, off unless a config asks for it): setup does not offer it yet.
• A 780M iGPU joins the gfx11 WMMA kernels (decode x1.53 on a Radeon 780M, same answers), and the Vulkan image decoder works in tiles (a 1024x1024 image's VAE: 23 s instead of 45 s).

Faster

• AMD multi-GPU (RDNA4): the draft layer sits on the first card in the pipelined decode, the pipeline is the default from two stages, prompts read in 8192-token chunks across a split, and the PCIe share is measured in the pipelined loop as well. R9700 decode +2.5 to +4% from STRATA_HC_Q8 on gfx12 (see "Defaults that changed").
• Intel Arc A-series (Alchemist), by aslater3: the fused hyper-connection read without the private copy plus large-GRF mode (A750 decode 11.8 -> 14.5 tok/s, 5 of 5 pairs), more and smaller work-groups (13.3 -> 15.9 tok/s on top of it, 5 of 5), the prompt GDN recurrence with one barrier per token (a 4K prompt +4.6%; the kernel is 6x faster), and an expert cache sized from the driver's fdinfo when xe reports no free memory (Arc A770, #1545). All four are bit-identical to 0.1.43.1 on the A750 and on a B70, where they are off or unchanged. Unsloth's K-quant and Q8_0 expert formats decode each weight word once per call on SYCL, bitwise the same.
• Commit graph in one launch per layer kind (STRATA_COMMIT_BATCH, on by default, =0 keeps the per-layer loop) and the draft stream at high priority on gfx1200/gfx1201 (STRATA_MTP_HIP_PRIO=0 turns it off): the engine-wide latency per decode window, measured and bitwise the same.
• Two opt-ins for the expert arena, by RubenTorney (#1904, #1905); both are off by default and the default path is byte-identical to 0.1.43.1.
  • --adapt-async 1 now also works in the arena mode (one GPU): the adaptive tier's swaps run on a job thread instead of stopping the decode. On an RTX 5070 under Windows (Q2_0, arena mode, 5 interleaved pairs, three batches) decode is +4.2 to +5.2% (13 of 15 pairs faster), all of it on code and document prompts (+7 to +10%), the story prompt level; on an R9700 with 20K cache slots it is level except on a cold prompt (+12.9%). Answers with the tier on differ slightly from the blocking tier (deterministic run to run, not equal to it), which is why it stays opt-in. It also fixes a real race for the RAM-resident async tier: a K/V grow could lend a slot a swap was still writing.
  • STRATA_PIN_BY_NEED=1 (a Windows / WDDM feature, CUDA and HIP) registers the experts the GPU will not hold as pinned memory instead of a prefix of the arena, in the order of the expert profile. We added a second tier for the experts the adaptive tier has swapped out: on the 5070 the prompt read is +4.4% (5 of 5 pairs), +8.5% with a pin budget of half the arena (STRATA_ARENA_PIN_GIB=16); decode is within +-1%. STRATA_PCIE_MIN1=1 (send one miss over PCIe as soon as a layer has two, where the share otherwise rounds to 0) did not help on this card and is a separate switch.
• A server start that dies inside the GPU runtime's own set-up (a ROCm hsa_exception one start in twenty after a quick restart) is retried 2, 4 and 8 seconds later (STRATA_START_RETRIES).

Fixes

• A picture in a long conversation no longer loses the cached state (#2228). The server sends each picture's byte hash and the engine matches pictures by it (plus the grid), not by their embedding floats. If a conversation still re-reads its history, the engine now says why.
• Intel A770 under xe: decode never started on the first window (#2194). Reproduced on an A750 with STRATA_WINDOW_SYNC=0; the drain of the other queues is now the default on every Alchemist card (Battlemage is unchanged).
• Two-card Intel and NVIDIA splits: the auto split takes the placement with the most spare VRAM on the fullest card among near-ties (#1440); a refused SYCL start exits cleanly and setup writes NO_HOST=0 for a split whose experts exceed 80% of the cards' VRAM (#1980); the start-up VRAM check covers every card of a split (#2192); a card behind a slow link is penalised only against the fastest link and from a 4x gap on (#2007).
• The prompt path gives way to a short request at the next chunk (SuperDistance, #2170).
• A literal <tool_call>, <tool_response> or </tool_response> in a message is text (47Hunter47, #2026).
• Foresight exchanges keep the experts they move (#2134); the gfx12 16384 prompt chunk stays at 8192 when RAM is short (#2090); setup --update with an MTP folder shared by several configs (bulhenvaster, #2195); Windows AMD cards: name, VRAM and driver from the registry (#1587); a Windows host-function signal for the adaptive overlap, with the copy stream flushed after the swaps (#2188).
• Model files of another shape are refused at load with the metadata key that differs, geometry read from the GGUF (gt50, Shawn Jackson, #2179).
• Setup and tools: Debian 12/13 CUDA toolkit install (Gabriel Popa, #2172); a Nix flake (acobster, #2198); an elastic image encoder (i-Vladimir-i, #2196); --prefill-head-alias for 8 GB cards (Maxim-Mazurok, #2191); STRATA_POOL_UMWAIT (zsariboga, #2174) and env_to_user (zsariboga, #2175); the sm_61 (P40) wide dequant and an opt-in F32 dense path (Techannel, #2178); STRATA_VERIFY_ASYNC_LAUNCH (CC-David-CC, #2209); enable_large_pages.ps1 (dvasdekis, #2185). Raw NUL bytes are gone from two sources, which MSVC refused; the huge-page request is skipped when free 2 MiB blocks are scarce (#2126).

Defaults that changed (each has its variable to turn it back)

Answers from the GSQ-RCO packs (Qwen, Swift, Coder) are byte-identical to 0.1.43.1 in our identity gates on every box. The changes below touch RDNA4 and Unsloth's files only.

• Setup's size menu recommends IQ3_S (Qwen, Swift) and UD-Q4_K_XL / UD-IQ4_XS (Unsloth). They are marked "(recommended)" and are the default where the RAM fits them (IQ3_S replaces IQ3_XXS as the 64 GB default); otherwise the default is the largest size that fits. Every size stays a choice. On AMD and Intel setups NVFP4 is not listed and UD-IQ4_XS is the 4-bit suggestion.
• Unsloth files, setup writes three settings into the run config's env block:
  • STRATA_MMQ_KQ=1 for UD-Q5_K_XL and UD-Q6_K_XL (the K-quant prompt kernels: prompts x1.3-1.7 on AMD with every expert in RAM; their prompt-path KL against Q8_0 is equal or better than the FP16 path's, mean 0.038 vs 0.025 on Q5 and 0.028 vs 0.029 on Q6, median better on both). UD-Q4_K_XL keeps it opt-in: its mean is 0.095 against 0.081.
  • STRATA_HC_Q8=1 on NVIDIA for the UD-* files: the hyper-connection projections are read from the file's own Q8_0 bytes instead of a BF16-rounded copy. KL against Q8_0 0.0562 -> 0.0526 (p99 0.925 -> 0.716) on UD-Q4_K_XL, the hyper-connection read -36 to -39% per call (~2 ms per window). Answers differ from 0.1.43.1's on these files.
  • STRATA_ROUTE_TAIL_SKIP=0 where every expert is in RAM (the skip costs KL there: 0.056 -> 0.086 for +4-10% speed); the RAM-budget mode keeps the skip (+26-67% there).
  • Edit or remove the entries in the config's "env" block to undo any of them.
• RDNA4 (gfx1200/gfx1201): STRATA_HC_Q8 is on. The hyper-connection projections are read as Q8_0: R9700 decode +2.5 to +4%, mean KL 0.0003 to 0.0010 against the old read; answers can differ in near-ties. STRATA_HC_Q8=0 or STRATA_GFX12_DEFAULTS=0 restores 0.1.43.1's read.
• Everything else stays as it was: the NVFP4 tail skip (off), the NVFP4 FP4 prompt modes, STRATA_PROMPT_DIRECT and STRATA_COPY_HIP_PRIO are opt-in.

Mac (Apple Silicon, Metal): new, experimental

Strata now runs on M1 to M5 Macs (macOS 15 or newer, Command Line Tools, no Xcode). Run ./setup.sh: it builds the Metal engine from source on your Mac (about 7 minutes), downloads the model and starts the server (same API and chat page as on a PC). There is no ready-made Mac engine in the release zips. Nothing changes for the NVIDIA, AMD and Intel builds: we compiled the release with and without the Mac code and the engine objects are identical (CUDA, HIP and SYCL builds byte for byte; the Windows executables differ only in the linker's timestamp). Details: docs/MAC.md.

• Sizes: IQ3_S (default) and UD-Q4_K_XL (higher quality, 92.8% same top token as Q8_0) are the recommended ones; Q2_0 and IQ3_XXS for smaller Macs; Unsloth's UD-Q5_K_XL and UD-Q6_K_XL stream from the SSD on a 128 GB Mac. A 32 GB Mac streams the experts from its SSD (Q2_0 about 8 to 14 tok/s depending on free memory).
• Speed, M5 Max 128 GB, against llama.cpp Metal on the same GGUF files: decode with MTP 1.6 to 1.9x (Q2_0 107 to 114 tok/s, IQ3_S 95 to 103, UD-Q4_K_XL 74 to 94), prompts 1.5 to 1.6x (1,150 to 1,540 tok/s at 4K to 32K). UD-Q5_K_XL streamed: 46 to 51 tok/s and 1,070 to 1,110 tok/s prompts (llama.cpp 43 to 45 and 690 to 730); UD-Q6_K_XL streamed about 50 tok/s, which llama.cpp cannot run on this Mac. On a 32 GB M4 mini: 8 to 14 tok/s with the experts streamed from the SSD, against 4 to 6 for llama.cpp on the CPU.
• Same answers as the PC engines: greedy outputs agree with CUDA and HIP up to near-ties on every size; vision works (Q2_0 and UD-Q4_K_XL); the OpenAI/Anthropic server, tool calls and conversation cache are the PC's.
• Big-Mac memory safety: on a Mac with 64 GB or more, the engine keeps its whole footprint (expert arena, GPU buffers, everything the process holds) under 60% of the RAM and wires the filled arena, so a big file copy or download beside it cannot make macOS start compressing the model's memory. Without this a 128 GB M5 Max hit watchdog panics (a kernel timeout, the Mac restarts) while streaming UD-Q6_K_XL; with it a 30-minute soak with 20 GB and 40 GB file writes in the middle answered every request correctly with no swapping, at the same speed. The engine also refuses to start while a model repack runs, and setup refuses to repack with an engine running. Smaller Macs (the 32 GB M4) are unchanged.
• Limits: one request at a time, up to 32K context measured, Qwen3.6 and the Swift and Coder releases are not on the Mac yet, MTP is off on small streaming Macs. Tested on two Macs only; reports from other chips help (docs/COMMUNITY_BENCHMARKS.md).

Not in this release

• Unsloth's K-quant files have not been run on an Intel card end to end (the kernels pass their parity tests on an A750; a UD-Q4_K_XL run on a B70 is untested). Qwen3.6 does not run on Intel or Mac.
• #2189 (an elastic HIP allocator) and #2201 need a new look with the ROCm that setup installs.

---

Strata is free and open source. If it runs well on your PC, a coffee keeps the work on it going: