DeepSeek-V4.1-Flash (552B) on one RTX 5090 with expert streaming: a port, its numbers, and what limits it

[JigSawPT]

I ported DeepSeek-V4.1-Flash to a llama.cpp fork and ran it on a single RTX 5090 (31.8 GiB of VRAM) with 125.7 GiB of RAM, with the routed experts streaming from NVMe through a VRAM cache and a pinned host tier. It runs exactly — logits within the reference implementation's own fp8 rounding floor — at 5.1 tokens/s on new content and 21 tokens/s on content already resident in the cache. Report, tools and raw results: https://github.com/JigSawPT/deepseek-v41-flash-on-5090

What it is. A new deepseek41 architecture on top of deepseek4.cpp (branch dsv41-porte of https://github.com/JigSawPT/llama.cpp, on b10269): the V4.1 compressor path and shared compressed caches, the hierarchical indexer's first level, the engram tables (189 GiB of n-gram memory) kept on disk and read by the host, the hyper-connection mix threaded one sub-layer ahead, and one defect that never raised an error — V4 normalises each head of q after wq_b, V4.1 does not (norm 181.019 = √(64×512) gave it away). The expert streaming is nibor1896's Crow patch series, credited. GGUFs are on the Hub: the target with the engram tables (502 GB, 11 shards, MXFP4 experts as a lossless repack of the released blocks, verified 480/480) and the model's own DSpark draft head (8 GB).

Correctness. Logit correlation against the reference at 1 401 tokens is 0.9967; the port against itself across two runs is 0.9959 — the divergence from the reference is indistinguishable from zero. What remains sits at the floor of the reference's fp8/fp4 arithmetic (one fp8 linear round-trips at 0.9999), and the MoE gate amplifies it because it is a switch: 208/240 top-6 choices agree, and the 6th-to-7th margin is about 1 %. Runs are not reproducible above ~1 024 tokens because expert-cache slot assignment depends on I/O timing; --moe-stream-io-threads 1 makes them bit-identical.

Where the token goes. Per remap call at a 72 GiB host tier: 20 % compute, 26 % PCIe, 54 % NVMe. The disk is at 43 % of what it delivers; the limit is that only ~3.8 disk requests exist in flight per layer, because a layer's requests can only leave after its router has run. Ceilings, measured: 6.2 tokens/s with no disk miss at all, 21.3 with everything resident — which a 105 GiB working set does not allow on 31.8 GiB of VRAM.

What did not pay, with numbers. A prefetch oracle with perfect knowledge of future routing: +30 %, but only at 5 tokens of look-ahead; at 1–4 remap calls it loses, which rules out layer-level predictors. Block verification with a perfect draft: +17–20 % and a plateau. An n-gram draft: −33 % on realistic content (and it exposed a rollback gap: need_n_rs_seq only covered model-based drafts — fixed for every type). The DSpark draft head: 51–97 % acceptance, neutral on mixed content, because on a disk-bound machine a verification step pays for the union of experts of its block. Encoder-only prefill: not exact below 2 432 tokens.

Two things I'd like from here. (1) Tests on other machines — Linux is untested on this branch. (2) The open conversion PR #28696 stores the engram tables differently from this branch; the layouts need reconciling before a runtime PR, and I'd rather do that with whoever is working on it than on my own.

Every number above names the tool that produced it in the repo; the negative results are in the README too, because each one closes a door.