Strata NVFP4

Qwen3.8-Flash-Next (125B hybrid MoE) with NVFP4 experts on one RTX 20, 30, 40 or 50 card (12 GB of VRAM or more) and 64 GB of RAM or more, text and pictures. A fork of Niko1221/Strata, on upstream 0.2.0-beta (its engine, server and setup; not its app or installers). Setup installs, ready-made from Hugging Face, the original Qwen in NVFP4 (our GPTQ, or NVIDIA's checkpoint converted), censored as Qwen ships it; on ours an opt-in switch turns the censorship off. Upstream's GGUF models are still offered.

Accuracy against Qwen3.8-Flash-Next with every routed expert in Q8_0, speed on an RTX 5090 with 128 GB of RAM. Setup downloads the models it names; each Hugging Face card has its measurements and how to run the files by hand.

• Accuracy: how often the model's most likely next token is the Q8_0 model's. Measured on 5,153 tokens of code, docs, English and Ukrainian chats and agent transcripts, every model fed the same text. Every model ran with its experts on the CPU and the route tail skip off, so only the quantization differs. The Q8_0 model also has Q8_0 attention, Qwen's BF16 router and Qwen's own FP8 PLE, so it is the original's precision everywhere but the 8-bit weights.
• Speed: RTX 5090 (32 GB), Ryzen 9 9950X3D, 128 GB of DDR5-5600, Windows 11, Strata NVFP4 0.1.42-nvfp4.1's engine with the arguments 0.1.42-nvfp4.2's setup writes for each model. Median of 3 runs: 4 greedy chats (up to 1,500 tokens each) and one 32,007-token prompt.
• RAM: "RAM in use" is the engine and the server at the speed shown; "PC for this speed" is the installed RAM that holds it with Windows beside it. Smaller PCs still run the models, slower: the NVFP4 GPTQ and NVIDIA's on a 64 GB PC in the low-RAM mode decode ~115 tokens/s and read a 32K prompt in ~9 s (docs/NVFP4.md, "64 GB of RAM"); Unsloth's files go into a RAM-budget mode there and read experts from the SSD (UD-Q4_K_XL: 7-8.5 tokens/s at 64 GB, upstream's measurement); setup's minimum is 48 GB for them and the ISTA 2-bit files, 60-64 GB for the ISTA 3-bit ones.
• Unsloth's files keep every expert in RAM where they fit with 16 GiB more (setup from 0.1.42-nvfp4.2): from ~88 GB of RAM for UD-Q4_K_XL and ~68 GB for UD-Q3_K_XL. Below that, setup's RAM-budget mode reads the rest from the GGUF on the SSD. Setup before 0.1.42-nvfp4.2 used the budget mode on every PC; on 128 GB UD-Q4_K_XL then decoded 118 tokens/s and read the 32K prompt in 30 s (run setup again to switch an existing install).
• UD-Q3_K_XL's experts are i-quants despite the name: IQ3_XXS gate/up with IQ4_NL or Q8_0 down.
• The GGUF models run upstream's route tail skip by default: ~10% faster decode, and answers a little further from Q8_0 than the table shows.
• The two 128 GB models need a 128 GB PC (setup asks below that). Q8_0 runs there in the low-RAM mode by itself (100 GiB of experts locked in RAM); the mix keeps every expert in RAM.
• ¹ NVIDIA FP4-Local is NVIDIA's retrained 4-bit model (quantization-aware distillation): its experts and n-gram table were trained again for FP4, so it is a slightly different model rather than a rounding of the original. Its KL to the Q8_0 model is 0.051 (the NVFP4 GPTQ and NVIDIA's NVFP4: 0.038), while its perplexity on these texts is a little below Q8_0's. Measured on 2026-10-24 on the same texts against the same reference, with the engine of this fork's 0.2.0-beta-nvfp4.3. The NVFP4 GPTQ model gives the same accuracy on that engine (93.65%), and in the same runs it decoded 180-191 tokens/s, so the two run at the same speed within the noise. NVIDIA ships ModelOpt safetensors (95 GiB) that are converted on the PC (upstream Strata's tools/nvfp4_local_pack.py prepare, about 10 minutes, 99 GiB of files); this fork's setup does not install it. Its license, the NVIDIA Open Model Agreement, rules out the censorship switch.

• 0.2.0-beta-nvfp4.3: upstream 0.2.0's new interface, in your browser. The release folder's start-server.cmd now also starts upstream Strata's app page on http://127.0.0.1:8791 once the model is ready, and opens it: chat with pictures and the thinking switch, chats kept by the browser, skills, workspaces and the Monitor with the engine's own numbers. It runs on the folder's own node\node.exe and uses the server's model - it starts no engine of its own - and stops with the server. The old page stays at http://127.0.0.1:8080/. start-ui.cmd <address> opens it for a model something else runs (a tray); it never wakes a sleeping model except for a chat you send. A run config with "app_ui": true starts it with its server wherever that server is started from (a tray: the page lives while the tray's model is loaded), and "app_ui_open": false keeps the browser closed. Model downloads, tuning and updates stay where they are (prepare-model.cmd, update.cmd); upstream's desktop app is not part of it (details).

• 0.2.0-beta-nvfp4.2: this fork updates from this fork. At start the server asks GitHub, at most once a day, which strata-nvfp4 release is the newest. When it is newer than the installed one, the server's window and the web page's header say so, with the command that updates the folder (update.cmd in a release folder, UPDATE.bat in a clone). Nothing is downloaded or installed by itself; STRATA_UPDATE_CHECK=0 or "update_check": false turns the check off. strata update without --feed now runs this fork's updater instead of refusing; upstream's signed feed is used only when named. The engine is unchanged (details).

• On upstream 0.2.0-beta (0.2.0-beta-nvfp4.1). Upstream's engine, server and setup of 0.1.45 and 0.2.0-beta (the engine says 0.1.45.3) under this fork's NVFP4 models; upstream's new app, Terminal and installers are not part of this release, which stays the Windows zip. One GPU and one dialog give the same bits as 0.1.44.1-nvfp4.2 at 32K and in the kit's identity runs, and decode ~8% faster on the 5090 (203 against 180 tokens/s, 5 of 5 pairs) (the port and its checks). What you get from upstream:

  • the prompt tokenised from the last shared prefix and cheaper conversation-cache copies (#567/#1852, #1163, #1164), --conversation-cache-park-gain, the model card's sampling applied by the server per thinking state (thinking: the 1.0 / 0.95 / 20 setup used to write; not thinking: Qwen's instruct set), pinned memory kept free under Windows while the expert arena pins, so the buffers after it never fail;
  • in setup: the Gyro family (27.6 and 35 GB), downloads over 8 connections that resume, every file pinned by size and SHA-256, the RAM + VRAM memory tiers for the GGUF default, Qwen3.6-35B-A3B with pictures and several requests;
  • our earlier pull requests in upstream's form (IMMA prompt attention, the scattered MoE input, the fused up projection, the tier's eviction sync): this fork keeps its defaults on them. This fork's own: setup no longer writes a "sampling" block (the server's model card does the same); --family nvfp4 --model NVFP4-QAD (upstream's NVIDIA FP4-Local, prepared on the PC) is not offered here; strata update (upstream's signed engine feed) needs a feed named explicitly, so an install of this fork never switches to upstream's engine by itself.

• 0.1.44.1-nvfp4.2: Claude Code's system prompt is rewritten for the local model. The server recognises Claude Code's prompt by its billing header and drops what only Claude can use: its identity, Anthropic's security policy, the list of Claude model ids, the pronoun rule. The memory how-to is cut to its path. In their place comes a short note for a local agent: call only the listed tools, by their exact names, never as plain text. The harness, context management, plan mode, output styles and every other section pass as sent. The rewrite gives the same text on every turn, so the conversation cache keeps hitting. Claude Code 2.1.296's prompt goes from 1,361 to 676 tokens. "claude_code_prompt": "as_sent" in the config keeps it as it comes. The engine is unchanged.

• On upstream 0.1.44.1 (0.1.44.1-nvfp4.1). Upstream's hotfix: CPUs without AVX2 start again (#2324: the NVFP4 CPU rows' tables were built before main with AVX instructions, an illegal instruction on such a CPU, now none), a crashed engine start is named instead of retried as a GPU runtime start, the Responses API (Codex) honours the config's reasoning_effort, and setup run again over an installed model keeps its size instead of moving it to the new recommendation (here also for Qwen3.6, whose re-run kept neither its size nor its family). One GPU and one dialog give the same bits as 0.1.44 at the same speed (the port and its checks). New in this fork since 0.1.44:

  • Kept dialogs give RAM back when it runs short: below --conversation-cache-min-free-mib the parked dialogs leave RAM oldest first (to the disk tier when one is set), checked before every request and every 2 s while idle (on the 5090: three dialogs, 3.3 GB, released in 0.1 s).
  • The low-RAM mode pins its RAM copy in slices (Q8_0 on 128 GB: decode 53 tokens/s instead of 49, a 32K prompt in 8.5 s instead of 9.9) and keeps 1500 MiB of VRAM free under Windows for the desktop.
  • --rope-scaling dynamic for contexts past 256K: no scaling while a dialog fits the trained 262,144 positions (the same bits as none), yarn past them with the factor the length needs; past 262K as good as static yarn, which changes every short dialog too. Setup and Model settings write it on one NVIDIA GPU with one dialog at a time, static yarn elsewhere.
  • Every Qwen3.6-35B-A3B size can be installed on any PC: a size bigger than the RAM gets a note, not a question.

• On upstream 0.1.44 (prepared as 0.1.44-nvfp4.1, published within 0.1.44.1-nvfp4.1), which ships this fork's NVFP4 engine path. Upstream's "NVFP4 packs" are a port of this fork's work: its NVFP4 experts, MMQ tails, mixed formats and CPU rows are upstream's code now, and this fork runs upstream's copy. Upstream's --family nvfp4 --model NVFP4 | NVFP4-NVIDIA install this fork's qwen-nvfp4-gptq / qwen-nvidia-nvfp4 here, one install path (the port and its checks). One GPU and one dialog run as before: with the same expert cache, the default settings give the same bits as 0.1.43.1-nvfp4.1's engine. Upstream's changes that matter here:

  • Unsloth's UD-Q5_K_XL, UD-Q6_K_XL and Q8_0 are new sizes of --family unsloth, and UD-Q4_K_XL is no longer experimental; images are on by default for them. Upstream writes STRATA_HC_Q8=1 (and STRATA_MMQ_KQ=1 for Q5/Q6) into their configs. This fork's UD-Q3_K_XL stays a size.
  • Qwen3.6-35B-A3B (--family qwen36, Unsloth's GGUFs with the MTP layer, text only so far) is a new family with its own engine path. It runs here exactly as on upstream (the same tokens and logits, bit for bit): on an RTX 5090 UD-Q4_K_XL keeps every expert in VRAM and answers at ~410 tokens/s with its MTP drafts, a 33K prompt in 5.3 s. UD-Q6_K_XL does not fit a 32 GB card whole: 153 of its 256 experts per layer stay in VRAM, the rest run on the CPU, ~130-150 tokens/s. Setup recommends the largest size the RAM allows (Q8_0 from 64 GB); on a 32 GB card UD-Q4_K_XL is the fast one.
  • IQ3_S is the recommended GGUF size from 64 GB of RAM (it was IQ3_XXS).
  • A picture in a long conversation no longer loses the cached state (#2228); a long prompt gives way to a short request at the next chunk (#2170); a quoted <tool_call> / <tool_response> in a message is text (#2026).
  • Two GPUs: the auto split keeps room on its fullest card (#1440); a slow link costs more only from a 4x gap (#2007); the VRAM check covers every card (#2192).
  • Opt-in: --adapt-async 1 in the arena (#1904), STRATA_PIN_BY_NEED=1 on Windows (#1905), --prefill-head-alias.

• Also new since 0.1.42-nvfp4.2 (prepared as 0.1.44-nvfp4.1):

  • Several GPUs (layer split, #7): a prompt is read in chunks per stage so the stages work at the same time, a card on a slow link holds every expert of its layers, the CPU prompt share works on a split. On rented RTX 4090s a 32K prompt reads at 6,936 tokens/s on 2 cards (4,056 before; one card 5,472) and 9,703-9,997 on 4 (3,880 before); a pair with one card on a ~2 GB/s link is no longer slower than its first card alone.
  • Pinned memory on Windows: on a split each GPU page-locks its own layers up to its adapter's budget, not 8 GiB for all; where little can be page-locked the copies run faster (this RTX 5090 with nothing pinned: a 32K prompt at 87-88% of the pinned speed, 70-71% before).
  • Several dialogs at once ("parallel"): a window graph out of VRAM no longer ends the engine, and the slots' K/V grows with their context instead of taking it all up front (4 slots at 262K: 2.8 GiB instead of 14.6; one 24 GB card with 4 dialogs: 94 tokens/s in all where the engine ended before). parallel stays 1 by default.
  • Dialogs parked on the disk (opt-in --conversation-disk-dir, from xrip's upstream PR #1937): a dialog the RAM cache has no room for goes to a folder; a 100K dialog comes back in ~1 s instead of 11-12 s, also after the tray restarts the engine (numbers). For one user --conversation-cache-mib 8192 (4 dialogs in RAM) is enough; the disk tier is off by default.
  • Model settings has the dialogs and the context size (to 512K, with the RoPE extension past 262K); the controls are back inside the card.
  • The Windows zip ships strata_kq.dll (upstream's K-quant prompt kernels, for UD-Q5_K_XL / UD-Q6_K_XL and Qwen3.6).

• On upstream 0.1.43.1 (prepared as 0.1.43.1-nvfp4.1, published within 0.1.44.1-nvfp4.1). Upstream's changes that matter here (the port and its checks):

  • Non-finite logits are caught (#879, #1892): when a decode window's logits turn NaN, the server sends none of its tokens, drops the damaged caches and reads the prompt again from token 0, instead of answering "!". On by default (STRATA_NAN_GUARD=0 turns it off); this fork's fresh engine after a "!" reply stays as a backstop.
  • The PLE probe only warns (#1984, #2040): a drive it measures slow keeps the direct reads (STRATA_PLE_AUTO_MMAP=1 brings back 0.1.42's switch to mapped reads). Here it still runs once the expert arena is in, so the arena's own reads cannot make the SSD look slow.
  • Windows: the stall watchdog no longer ends the engine while it parks a conversation or waits for a draft with little free VRAM (#2049, #2065).
  • A request's smaller prompt chunk no longer ends the engine when its slot loan does not fit: the chunk is halved instead (#2090).
  • Server fixes: a quoted </tool_call> in a message stays text (#2026); a JSON tool call in a Markdown fence inside <tool_call> is read (#2037); a client that leaves while its pictures load stops the work (#1999).
  • Unsloth's files in the RAM-budget mode on Windows read the experts outside RAM in parallel: the GGUF views are closed once the file tier reads unbuffered (NTFS ran those reads one at a time).
  • BENCH.bat / ./bench.sh (in a clone): measures your install through the HTTP API in about 10 minutes and writes a report you can share.
  • The route tail skip stays off for this fork's packs. For the GGUF models upstream now leaves it off when 90% or more of the experts are in VRAM (#1884). Upstream's adaptive overlap (STRATA_ADAPT_OVERLAP=1) is opt-in.

• Also in 0.1.43.1-nvfp4.1:

  • The server's prompt encoder keeps one prompt per conversation (#1852, by gputier): a second conversation, such as a Claude Code subagent beside the main one, no longer pushes out the main conversation's kept prompt, so its next turn is not tokenized from the start again.
  • Not for WSL2: setup now frames a warning under WSL2, in --check and at install: the driver there page-locks only about 1 GB of RAM (Requirements, #7).
  • Two GPUs (layer split, #7): the expert arena loads on its own thread only with one GPU, so on a split it no longer takes page-locked memory before the second card's context exists; a refused hand-off buffer between the cards makes the prompt chunk smaller instead of stopping the start; and when the token embedding cannot be page-locked and the draft head runs on the other card, the engine says so at start instead of failing later with an illegal memory access; a later card's PCIe share starts from NVFP4's 0.25 like the first card's, not 0.55. One GPU runs exactly as before.
  • Tool calls: Python-style True / False become JSON booleans where the schema allows one (jarydkrish); a call missing its </tool_call> ends at its own </function> (upstream PR #1881, LeGeRyChEeSe); a quoted </tool_response> stays text (upstream PR #1840, Mert Özkan); opt-in "tool_name_repair": true maps a call to an undeclared tool onto the declared one it meant (ANBAL534).
  • Image URLs are fetched only from public addresses, re-checked on every redirect (shoftee); data: URLs and local files work as before.
  • A NaN stays a NaN in the PLE and prompt path's BF16 conversions, so the NaN guard sees it (sebastianrcnt); every other value converts to the same bits.
  • A stop token the model writes as text no longer cuts the reply: asked to write <|im_end|>, the model writes the token itself; where no turn can end (a tool call's argument, after an opening backtick or quote, after a space) it is now that token's text and the reply goes on ("stop_token_text": false turns it off).
  • Parallel dialogs (--batch with --conversation-cache-mib): conversation-cache and batch fixes from upstream pull requests #1503, #1163, #1164 (s.anastasiou, BlueKingMuch) and opt-in --prompt-cache-spread (#1959, xrip) (Several dialogs at once). One dialog runs as before.
  • A warning when --max-context is past the model's trained 262,144 tokens without rope scaling.
  • The Monitor shows every GPU: its role (layer-split stage, peer tier, helper cache, image encoder), VRAM, load, temperature, power and PCIe link (current / max, with the engine's measured bandwidth); the Disk card reads the model's drives. The Windows bundle's server env gets psutil (update.cmd installs it).

• Two more exact models for a 128 GB PC, and Unsloth's files faster (0.1.42-nvfp4.2).

  • --family qwen-nvfp4-mix: our GPTQ NVFP4 with Q8_0 experts in 38 of its 48 layers. It picks Q8_0's top token 95.0% of the time (GPTQ: 93.6%), at about half GPTQ's speed (91-95 tokens/s on an RTX 5090). Every expert stays in RAM: 106 GiB in use with images and the server.
  • --family qwen-q8: every routed expert Q8_0, the reference the others are measured against. 46-47 tokens/s, a 32K prompt in 10.3 s. Its 119.5 GiB of experts do not fit 128 GB beside Windows, so it runs in the low-RAM mode (100 GiB of them locked in RAM).
  • Both are the original, censored Qwen, with the censorship switch: 5 of 5 held-out harmful requests refused with it off, 0 of 5 with it on. Both share qwen-nvfp4-gptq's dense weights, n-gram table, embedding and draft head; an install after it downloads only the experts.
  • Below 128 GB, setup asks before installing them: they would run in the low-RAM mode, which was not measured with them.
  • The low-RAM mode also follows the pack's size: from 92 GiB installed, it now starts when the experts and 16 GiB more do not fit. In that case it leaves 10 GiB free, not 6. Below 92 GiB nothing changes.
  • The route tail skip is off by default for all of this fork's packs: NVFP4, Q8_0 and the mix. Before, only packs with NVFP4 experts left it off, so the all-Q8_0 pack would have had it on. The GGUF models keep upstream's default.
  • Unsloth's files keep every expert in RAM where they fit with 16 GiB more (a 96 GB PC for UD-Q4_K_XL). On a 128 GB PC, UD-Q4_K_XL now decodes 169 tokens/s and reads a 32K prompt in 4.9 s. Setup's old RAM-budget mode gave 118 tokens/s and 30 s, and it used more RAM (98.5 against 77.3 GiB). A 64 GB PC keeps the budget mode. To get the new config on an existing install, run setup again (START-HERE.bat --setup).
  • Unsloth's UD-Q3_K_XL is a size of --family unsloth, as a user asked. It picks Q8_0's top token 90.7% of the time, at 208 tokens/s and a 3.8 s 32K prompt, with every expert in RAM (57.5 GiB in use).

• On upstream 0.1.42 (0.1.42-nvfp4.1). Upstream's own changes now run here too:

  • The route tail skip (STRATA_ROUTE_TAIL_SKIP=7: a missed expert that every token of a verify window routes at rank 7-9 is neither fetched nor computed) is upstream's default for the GGUF models, and off by default for the NVFP4 models. On qwen-nvfp4-gptq it moves the answers by teacher-forced KL 0.035 against the skip off (top token the same 93.7% of the time), as much as the whole GPTQ quantization error against an all-Q8_0 reference (0.038), and thinking ends sooner (p(</think>) early x1.35). It makes decode 11.5% faster (193.7 against 174.0 tokens/s): "env": {"STRATA_ROUTE_TAIL_SKIP": "7"} in the model's config turns it on (numbers).
  • Decode's PCIe share is upstream's measured one (from the CPU pool's speed: 0.35 here); this fork's per-layer cost model is gone. The same speed as the old fixed share.
  • Four fixes this fork sent upstream are upstream's code now: the unbuffered start of the resident RAM mode, the page-file advice (a fixed 65536 MB), the K/V trim's residency upload, the prompt buffers' size.

• Censorship can be turned off (0.1.41-nvfp4.5). The original Qwen (qwen-nvfp4-gptq, and the original Qwen's GGUFs qwen and unsloth) now runs with or without censorship. The switch is off by default. To turn it on:

  • for one chat: the web app's "Disable censorship" switch;
  • for one request: the API field "uncensored": true;
  • as the default: install with START-HERE.bat --uncensored on, or set Disable censorship by default in the web page's Model settings (0.1.41-nvfp4.7; used from the model's next start).

  The engine removes the model's refusal direction at run time in layers 8-33 only; the weights are untouched.

  • With it on and thinking on, 0 of 104 English and 0 of 30 Ukrainian test requests were refused.
  • The model thinks as long as the original: 1,947 against 1,699 tokens on a long Claude Code request.
  • Never for NVIDIA's model: its license forbids it.
  • Details.

• orca-nvfp4 is removed from the fork (0.1.41-nvfp4.5). OrcaRouter's uncensored model, the former default, is no longer offered.

  • Its weight edit made it a much weaker agent. On the same Claude Code task it took 10-70 steps against the original's 175 and gave a clearly worse result. It thinks ~40% less.
  • Instead, use --family qwen-nvfp4-gptq --uncensored on: the original's weights, without the censorship.
  • An existing orca install keeps working.

• Recommended sampling (0.1.41-nvfp4.6). Setup writes the model's own temperature 1.0, top_p 0.95, top_k 20 into the config, as the default for requests that send none. Before, Claude Code decoded greedy.

• Model settings has a Disable censorship switch too (0.1.41-nvfp4.7): the default for requests that do not say, saved in the run config ("uncensored") and used from the next start; greyed out for a model without the refusal projection (NVIDIA's, Swift, the Coder).

The design is upstream's: the routed experts live in RAM, the most-used ones are cached in VRAM, the misses are computed on the CPU and fetched over PCIe in parallel with the GPU, and an MTP draft head speculates. Upstream's README is kept as README.upstream.md. docs/NVFP4.md has every measurement behind this fork, and the releases say what changed in each.

git clone https://github.com/sergqwer/strata-nvfp4
cd strata-nvfp4
START-HERE.bat

(Linux: ./setup.sh.) Setup checks the PC, asks which model, how much context and whether the model should read images, installs the engine and the model, and starts it on http://127.0.0.1:8080 (OpenAI and Anthropic APIs, a chat page).

• The NVFP4 models come ready-made: the expert pack, the dense GGUF, the FP8 n-gram table, the BF16 embedding and the MTP draft head are downloaded at a pinned revision and checked against their SHA-256; nothing is converted on your PC. qwen-nvfp4-gptq's routed experts are NVFP4 by our GPTQ from Qwen's BF16: 33.3% of plain rounding's error and 44.8% of NVIDIA's (NVIDIA's own is 74.4% of plain rounding's), lower in all 48 layers, at the same size and speed (179 against 175 tokens/s). qwen-nvidia-nvfp4 is NVIDIA's own checkpoint, quantized by NVIDIA and only converted here. The two share the n-gram table, the embedding and the draft head: the second one links them (The original Qwen in NVFP4). qwen-nvfp4-mix and qwen-q8 share all of qwen-nvfp4-gptq's files but the experts (Q8_0 for a 128 GB PC).
• qwen-nvfp4-gptq is the default where the PC meets the requirements; a PC below them gets a GGUF model as the default, and --family qwen-nvfp4-gptq --yes installs it anyway.
• Disable censorship (off by default): for qwen-nvfp4-gptq, qwen-nvfp4-mix, qwen-q8 and the original Qwen's GGUFs (qwen, ISTA's; unsloth) setup always loads this fork's refusal-direction projection (data/uncensor) and leaves it off: the web app's "Disable censorship" switch (Sampling) and the API field "uncensored": true turn it on for a request, and setup's question "Disable censorship?" (default no; --uncensored on|off) sets the default, "uncensored" in strata-<model>.json. Off, a request runs the original model. On, the engine removes the model's refusal direction from its residual stream after layers 8-33: with thinking on, 0 of 104 English and 0 of 30 Ukrainian held-out harmful requests refused (the original refuses almost all), KL 0.026 to the original, and its thinking as long as the original's (1,947 against 1,699 tokens on a long Claude Code request, log p(</think>) +0.019 nats). On the GGUFs, against each quant's own stock model (measured on Q2_0, IQ2_XS and UD-Q4_K_XL): thinking-on refusals 1, 0 and 0 of 104, KL 0.016, 0.017 and 0.028, log p(</think>) +0.027 to +0.040 on Q2_0 and +0.050 on UD-Q4_K_XL's thinking (upstream's vector +0.087 to +0.149 and +0.163); the same vector, unchanged, on all three. It replaces upstream's speed projection there. IQ2_XS loops in long greedy agentic thinking by itself (3 of 3 replays of the long request, with the switch on or off): choose another size for agents. Upstream's experimental_speed_projection is the same switch. Never for qwen-nvidia-nvfp4, where nothing is loaded: the NVIDIA Open Model License does not allow bypassing its safety guardrails. The vector is under the Qwen Community License 1.0; removing refusals removes a safety behaviour, and what the model writes with it on is your responsibility.
• orca-nvfp4 was withdrawn in 0.1.41-nvfp4.5: a much weaker agent. OrcaRouter's abliteration edits all 149 residual writers of the model. On the same Claude Code task it ran 10-70 steps against the original Qwen's 175, with a clearly worse result, and it ends its thinking sooner (log p(</think>) +0.119 nats; ~1,000-1,150 thinking tokens on the long request against the original's ~1,700). qwen-nvfp4-gptq with its censorship switch on is the uncensored model with the original's weights instead. An install of orca-nvfp4 keeps working and nothing of it is deleted; setup says so on each run.
• huihui-nvfp4 was withdrawn in 0.1.41-nvfp4.3: it often loops in long thinking (9 of 10 greedy replays of a real 72K-token Claude Code request, on this engine and on 0.1.40.3's; the other models none of 5). An install of it keeps working too.
• The engine is always this fork's: on Windows the ready-made one from this repository's releases (strata-windows-x64.zip, checked against GitHub's SHA-256), on Linux compiled from this source. Upstream's engine has no NVFP4 path, so setup never installs it and replaces one an older setup installed.
• AMD cards: the NVFP4 models run on NVIDIA cards only, and this fork publishes no AMD engine. For the GGUF models setup compiles one on Linux; on Windows build it first with tools\hip\build_windows.bat, then run START-HERE.bat --backend hip --prebuilt <its dist folder> (docs/AMD_HIP.md).
• Sampling: a request that sends none gets the model card's sampling from the server (upstream's, since 0.2.0-beta-nvfp4.1): with thinking temperature 1.0, top_p 0.95, top_k 20 (Qwen's generation_config.json), without thinking Qwen's instruct set. A request's own values win, and temperature 0 means greedy. A "sampling" block in strata-<model>.json (setup wrote one from 0.1.41-nvfp4.6 to 0.1.44.1-nvfp4.2, or your own) still wins over the card; setup writes none now and keeps the one an install has.
• Options: --family qwen-nvidia-nvfp4 picks a model without the menu, --uncensored on|off answers the censorship question (the switch's default), --check says what fits this PC, --dry-run shows what setup would download, install and write for an NVFP4 model (the engine's arguments too) and changes nothing.
• Updates: UPDATE.bat (./update.sh) runs git pull, then installs the engine the new version needs; the model files stay. The release also has a stand-alone Windows bundle (strata-nvfp4-v<version>-windows-x64.zip) that updates itself with update.cmd; its prepare-model.cmd downloads qwen-nvfp4-gptq ready-made, the files and revision setup pins, with the same censorship switch, off by default. strata update (python strata.py update) runs the same updater for the folder it is in: update.cmd's in a release folder, UPDATE.bat / ./update.sh in a clone; strata update --check only says whether a newer release (or, in a clone, newer commits) is there. Upstream's signed engine feed is used only when you name it (--feed).
• Update notice: at start the server asks GitHub, at most once a day, which strata-nvfp4 release is the newest (one small request, no token, nothing about the PC; the answer is kept in .update/update-check.json). When it is newer than the installed one, the server's window says so with the command that updates this folder, and the web page shows a small link in its header (hidden with its x until the next release; About lists it too). Nothing is downloaded or installed by itself. Off: STRATA_UPDATE_CHECK=0, or "update_check": false in the run config (About > Settings); --offline never asks.

• System: Windows 10/11 or Linux, run natively. Not WSL2. Under WSL2 the NVIDIA driver page-locks only about 1 GB of RAM (NVIDIA's known limitation: "pinned system memory ... availability for applications is limited"), and this fork needs page-locked RAM for the experts' RAM copy, for the GPU's reads of missed experts over PCIe, for KV streaming and for a layer split's hand-offs. Under WSL2 none of that can be pinned: the KV cache must stay in VRAM, every miss goes to the CPU, and a two-GPU layer split does not start (#7). On a Windows PC run START-HERE.bat in Windows itself.

• GPU: an NVIDIA RTX 20, 30, 40 or 50 card with 12 GB of VRAM or more, driver 580 or newer (CUDA 13). The release engine has code for sm_75, 86, 89 and 120, and the optional FP4 x FP4 prompt unit for 120a (RTX 50); the other cards read prompts on int8 tensor cores instead. The dense weights, the draft head and the K/V come first and the rest of the VRAM caches experts, so a smaller card decodes slower. Setup picks the context:

  card's VRAM	--max-context
  32 GB (RTX 5090)	262144
  24 GB (RTX 3090 / 4090)	131072
  16 GB (RTX 4080 / 5080 / 4060 Ti 16 GB)	65536
  12 GB (RTX 3060 12 GB / 4070)	32768

  12 GB at 262144 and 8 GB cards at any context stop with no VRAM is left for the expert cache. Slots and decode rates per card size: Other GPUs.

• CPU: x86-64 with AVX2, FMA and F16C (Intel Haswell, AMD Zen or newer); the release engine names any other CPU and stops. The CPU computes its share of the experts with kernels chosen at run time (CPU experts):

  • AVX-512 with F, BW, VL, VNNI and VBMI (e.g. Zen 4 and 5): NVFP4 rows in 512-bit lanes;
  • otherwise AVX2 + FMA + F16C (e.g. Zen 2 and 3, Intel Core 12th-14th gen): NVFP4 rows in 256-bit lanes, bit for bit ggml-cpu's own AVX2 dot. Where the CPU also has AVX-VNNI (Intel Core 12th-14th gen and Core Ultra), the GGUF models' i-quant and Q2_0 rows take AVX-VNNI copies of their kernels (the same sums);
  • AVX or SSE4.2 only: upstream's experimental older-CPU build, which setup compiles on that PC; slow (Older CPUs).

• RAM: 64 GB or more.

  • From 92 GiB installed every expert stays in RAM: 63 GiB of experts, ~67 GiB of physical RAM for the engine, ~71 with the image encoder and the server. A 96 GB PC qualifies: Windows lists it as 93-95.6 GiB.
  • Below 92 GiB (a 64 GB PC) the low-RAM mode starts by itself: the experts outside VRAM are pinned, hottest first, up to the free RAM less 6 GiB, and the rest are read from experts.bin when needed. --low-ram forces it, --no-low-ram turns it off, --ram-budget GIB caps it (64 GB of RAM, A 96 GB PC).
  • From 92 GiB it also starts where the pack's experts and 16 GiB more do not fit, and then leaves 10 GiB free: on a 128 GB PC (127.2 GiB listed) the all-Q8_0 pack (119.5 GiB) runs in it, the NVFP4 + Q8_0 mix (103.1 GiB) keeps every expert in RAM. Both are for a 128 GB PC (Q8_0 for a 128 GB PC).

• Page file: set a fixed 65536 MB (initial = maximum). Windows lets all programs together commit at most RAM + page file, and the engine commits ~100 GiB with every expert in RAM (~80 GiB in the low-RAM mode): the experts plus ~30 GiB that WDDM charges for the VRAM it uses. Nothing of the model is paged out, but short of commit the expert cache in VRAM is made smaller, and the model can run significantly slower (a chat's round 17% and 38% slower, measured) or not start. Below 60000 MB of page file in all for sure, the engine prints a WARNING at start and setup a framed warning (--check and the install). Only what is there for sure counts: a system-managed or growing file (initial below maximum) counts at its size now, since it may not grow in time while WDDM charges the VRAM (upstream #60), and the warning then says to make it fixed. To set it: Win+R sysdm.cpl > Advanced > Performance Settings > Advanced > Virtual memory > Change: Custom size, initial and maximum 65536 MB, then restart Windows.

• Disk: ~130 GB for an NVFP4 model (128.8 GB of files; the expert pack and the 51 GB n-gram table are most of it). Setup checks the free space first. Use the fastest NVMe drive you have: every start reads 63 GiB of experts.

RTX 5090 (32 GB, PCIe 5 x16), Ryzen 9 9950X3D, DDR5-5600, Samsung 9100 PRO, Windows 11. Measured with huihui-nvfp4 (withdrawn since; the original Qwen's packs have the same format and size) and the arguments setup writes (262K context, int8 K/V, images on). The card also drives the desktop, so runs vary by a few percent.

The 96 GB PC is the 128 GB one with 95.6 GiB reported as installed and 88 GiB of RAM left free; the last column is the low-RAM mode forced on it. A 64 GB PC runs the low-RAM mode with a smaller RAM budget, and a smaller card caches fewer experts: A 96 GB PC, 64 GB of RAM and Other GPUs have those runs.

Each change was measured (first-token KL against a reference, interleaved speed runs); the links go to the numbers.

• NVFP4 experts: 4.5 bits a weight (E2M1 codes, an FP8 scale per 16 values, an FP32 scale per expert matrix applied to each projection's output) instead of upstream's 2-4-bit GGUF quants (Where the global scales go).
• GPTQ re-quantization from BF16 (tools/requant.py), which made the packs above, calibrated on the engine's own MoE inputs; Q8_0 down in chosen layers is an option for a size budget (Re-quantized from BF16).
• Less rounding elsewhere: the n-gram table in its shipped FP8, the token embedding in BF16, RoPE angles from a float64 table, FP32-exact inputs to the prompt path's BF16 projections, int8 K/V behind a Hadamard rotation (Where precision is still lost).
• The prompt path on tensor cores: the experts' gate/up on RTX 50's FP4 tensor cores with two FP4 terms per activation, int8 elsewhere; the prompt attention on INT8 tensor cores; prompt chunks up to 32K (Prompt path precision, A round of kernels).
• NVFP4 CPU kernels for the CPU's share: AVX-512 1.8-3.7x ggml-cpu, AVX2 2.0-2.4x at 4-8 tokens (CPU experts).
• A faster start: the expert arena loads unbuffered on its own thread beside the dense weights, ~7 s for 63 GiB from a PCIe 5 drive (Loading).
• The K/V grows with the context (upstream's --kv-grow, on here): VRAM the window does not use yet caches experts, 7,060 -> 8,290 slots at 262K; --no-kv-grow allocates the whole window (Faster without changing an answer).
• Small prompt chunks shared with the CPU (this fork's change, upstream's default since 0.1.41); on top of upstream's, NVFP4 packs share chunks up to 4,096 tokens and the DeltaNet recurrence runs in chunks from 128 tokens (Small prompt chunks with the CPU).
• The adaptive VRAM tier re-ranks every 2 rounds, up to 192 swaps, counts x0.92 (upstream: 4 / 96 / x0.7), and admits the PCIe share's experts into VRAM (The adaptive VRAM tier, A second round of kernels).
• The low-RAM mode below 92 GiB installed, or where the pack's experts and 16 GiB more do not fit, read unbuffered (see Requirements).
• Q8_0 experts in this fork's packs (qwen-q8, all of them; qwen-nvfp4-mix, 38 layers beside GPTQ NVFP4) for a 128 GB PC (Q8_0 for a 128 GB PC).
• Every RTX 20-50 card with 12 GB or more: only the optional FP4 x FP4 unit needs Blackwell; each generation's code path was tested on the RTX 5090 (Other GPUs).
• Images on the CPU: the encoder runs in RAM, so the expert cache keeps all its VRAM; upstream's GPU encoder (FP16 flash attention) was up to 11% off an FP32 reference. A picture a tool returns (Claude Code's Read) reaches the encoder too (Images).
• A draft vocabulary with Cyrillic: an answer in Ukrainian decodes 2.11 tokens a round instead of 1.40; tools/draft_vocab.py --add cjk builds upstream's CJK subset (Upstream 0.1.28 merged).
• The server starts a fresh engine after a reply of "!" (below).

Setup writes strata-<family>.json (the server's config: the engine and its arguments) and run-<family>.bat (.sh) next to setup.py. START-HERE.bat starts the installed model again without questions; run-qwen-gptq-nvfp4.bat starts the server with that config without setup's checks (serve/server.py --engine strata --config strata-qwen-gptq-nvfp4.json --port 8080). A key or an "env" entry you add to the config stays when setup runs again; engine flags added to its "args" by hand do not.

Images: answer yes to setup's question (or --vision yes). The image encoder runs on the CPU while the engine waits; OpenAI image_url parts and Anthropic image blocks (a screenshot pasted into Claude Code) work.

Running the engine and the server without setup, and converting the model yourself: Build it yourself.

set ANTHROPIC_BASE_URL=http://127.0.0.1:8080
set ANTHROPIC_API_KEY=local
set ANTHROPIC_MODEL=strata-nvfp4
set ANTHROPIC_SMALL_FAST_MODEL=strata-nvfp4
claude

The server answers under its own model name whatever the request names. The model is hybrid: its Gated DeltaNet layers keep a recurrent state that cannot be cut back to a position, so the server keeps a checkpoint at every turn boundary, and an agent's next turn reads only its new tokens. An edit in the middle of a history falls back to the checkpoint just before it. /v1/messages/count_tokens is served. In strata-<family>.json:

• "anthropic_thinking": "on_request": a request that does not ask for thinking (Claude Code's small helper calls) gets none;
• "fit_max_tokens": true: a request whose max_tokens would run past the context is shortened to the room left instead of refused (docs/DETAILS.md).

This is not more memory, it is bigger pages. Windows maps memory in 4 KB pages; the CPU part of every token reads experts from the 63 GiB arena at DRAM speed, and each page needs a TLB entry. With 2 MB pages the CPU pool holds steady at ~7.5 ms a round on this machine, where 4 KB pages wandered between 7.4 and 11 ms. The decode rate moves little (4 KB pages cost a chat 0.4 ms a round here, a 32K prompt nothing); the point is the steadiness.

Windows gives large pages only to an account that holds Lock pages in memory (SeLockMemoryPrivilege), and puts a privilege into a sign-in's token only when that sign-in starts. So:

1. powershell -ExecutionPolicy Bypass -File tools\enable-large-pages.ps1 asks for admin (UAC) and grants it to the current user through secedit (works on Windows Home, which has no secpol.msc). The policy as it was is saved to %LOCALAPPDATA%\strata-large-pages\before.inf, and -Revoke takes it back.
2. Sign out and back in, or reboot. Locking the screen is not a new sign-in.
3. Check: the same script with -Check, or the engine's start line expert arena: ... large pages (2097152 B).

Without the privilege the engine says large pages refused ... VirtualAlloc error 1314 and runs on 4 KB pages. Error 1450 means the privilege is there but Windows found too few free 2 MB blocks (memory fragmented after a long uptime): it falls back the same way, and a reboot clears it. The arena is locked in RAM either way.

Engine flags go into the config's "args", environment variables into its "env" ({"STRATA_KV_ROT": "0"}). The switches kept only for A/B runs (each gives the same bits or a measured alternative) are in docs/NVFP4.md.

A long Claude Code conversation once turned into "!" mid-reply (token 0, what the sampler gives when no logit is finite), and the cause is not found yet (docs). Since 0.1.44.1-nvfp4.1 upstream's guard (#879) catches non-finite logits first: the window's tokens are not sent, the caches are dropped, the prompt is read again and the engine log has a non-finite logits (#879) line. The server still starts a fresh engine at the next request after a reply of "!". To find out what was damaged, give the engine STRATA_BANG_AUDIT=1 and STRATA_NAN_GUARD=0 ("env": {"STRATA_BANG_AUDIT": "1", "STRATA_NAN_GUARD": "0"} in strata-<family>.json; with the guard on, a reply never turns into "!", so the audit never runs). It costs nothing until a reply turns into token 0; then, once, it compares every expert in the VRAM cache with its RAM copy and with experts.bin and logs what differs (strata kvg audit lines). Please send us the engine log (strata-<family>.log) from the engine's start to the end of those lines (Upstream 0.1.41).

• Built and measured on Windows with one RTX 5090, a Ryzen 9 9950X3D and 128 GB of RAM. RTX 20/30/40 cards, smaller VRAM, 64 and 96 GB of RAM and smaller CPUs were tested on that PC through their own code paths and budgets (STRATA_EMULATE_CC, --vram-reserve-mib, a RAM ballast, STRATA_FORCE_AVX2=1, --pool-workers), not on the real hardware.
• The NVFP4 model runs on one GPU: the low-RAM mode and the growing K/V are one-GPU.
• Not for WSL2: the driver there page-locks only about 1 GB of RAM (Requirements).
• The low-RAM mode's unbuffered reads are Windows-only; elsewhere it reads through the page cache.
• Decode is bound by host RAM bandwidth: the CPU pool and the PCIe share read the same DDR5 (~84 of ~90 GB/s here). More VRAM for the expert cache or faster RAM is what moves it (A second round of kernels).
• The models are abliterated fine-tunes: they do not refuse. What they are used for is on whoever runs them.

Setup does all of this; START-HERE.bat --build compiles the engine from this source instead of taking the release's (it installs the build tools first, asking). By hand, clone llama.cpp at the pinned commit, then run release\build-release.cmd (the release engine: AVX2 ggml-cpu, sm_75, 86, 89 and 120, into build-release\):

git clone https://github.com/ggml-org/llama.cpp third_party\llama.cpp
git -C third_party\llama.cpp checkout 3cf03257f219afbe7334045ff7c6a06ac68c627d
release\build-release.cmd

It needs Visual Studio 2022 Build Tools, CUDA 13, CMake, Ninja and Python 3.11+. By hand in docs/NVFP4.md has the rest: a one-card CMake build, converting jpezzulli's ModelOpt checkpoint ("Prepare the model"), re-quantizing the experts from BF16 by GPTQ, and running the engine and the server without setup. Each Hugging Face model card also shows how to run its files by hand.

Releasing (maintainers): release/make_windows_bundle.py, then release/publish.py; their headers say what each checks.

MIT, as upstream (LICENSE, copyright Niko1221 and the Strata contributors). llama.cpp / ggml code compiled into the build is MIT as well.