DeepSeek-V4.1-Flash-1017 on two 16 GB RTX 5070 Ti with the routed experts on the CPU (SGLang + KTransformers, 1M context)

[05yuki]

https://github.com/05yuki/deepseek-v4.1-flash-1017-ktransformers runs DeepSeek-V4.1-Flash-1017 on one machine with ordinary GPUs:

• CPU: the 384 routed experts a layer run on the CPU in MXFP4, through KTransformers' kt-kernel.
• GPU: attention, the most-used experts and the prefill run on the GPUs, through SGLang.
• Disk: the engram tables (~200 GB) are read straight from the checkpoint.

The repository holds patches against SGLang main 9c0f6d6a and KTransformers 95009ea, plus a setup.sh that builds them and a serve.sh that sizes the launch to the machine.

Results

Hardware: 2 × EPYC 7452 (Zen 2, AVX2), 512 GB DDR4, 2 × RTX 5070 Ti 16 GB (SM120), TP2.

Every row was measured with the repository's tools/bench.py at tag 2026-10-23-0135. Each table comes from a single launch.

Decode, 1,024 tokens a reply:

Text	tok/s
Japanese prose	47.1
Chinese prose	40.6
Math	40.0
English prose	38.5
JSON	37.2
English technical prose	35.4
Python code	35.3

Prefill, one prompt of random token ids:

Prompt (tokens)	tok/s	Time
35,000	1,301	27 s
131,072	1,515	1.4 min
524,288	1,209	7.2 min
1,000,000	943	18 min

One card (TP1) also runs, at 32.0 tok/s decode on Japanese prose.

Accuracy

The served model is scored against the checkpoint's own inference/model.py. The reference runs one layer at a time on one 16 GB card. Both score the same 24 pieces of 480 tokens from two public-domain texts, one Japanese and one English, teacher-forced. Results at tag 2026-10-23-1702:

Run	Mean NLL	Minus the reference (SE over the pieces)
Reference	0.7185
Served, two GPUs	0.7181	-0.0004 (0.0040)
Served, one GPU	0.7194	+0.0008 (0.0050)

The scripts are in the repo, tools/ref-nll.py and tools/nll.py, so anyone can rerun this; the reference takes about 4 minutes and needs no server.

Upstream

Fixes we found along the way, sent as PRs:

• [Fix] moe: scale the KT routed output once when the top-k already folds the factor #45810: the KT path applied the routed scaling factor twice after [Refactor] moe: Derive layer contracts from the selected quant method #44993. NLL drops from 2.878 to 2.596.
• [Feat] deepseek-v4.1: the V4.1 KV cache layouts on SM120 / SM121 #45851: the V4.1 KV cache layouts on SM120 / SM121, through FlashInfer's SM120 sparse MLA.
• [Fix] deepseek-v4.1: score the torch indexer chunk in place #45745: the torch indexer scores its chunk in place (3 → 1 GiB).
• [Fix] kt: pass the GPU expert mask that kt-kernel's inference API takes #45183, [Fix] deepseek-v4: FP8 quantize-dequantize the attention output before wo_a for V4.1-Flash-1017 #45185, [Fix] kt: run with zero GPU experts (--kt-num-gpu-experts 0) #45188: the KT expert mask, the FP8 qdq of the attention output that V4.1-Flash-1017 needs, and zero GPU experts.
• [Feat] model-loader: make the unbalanced load barrier timeout configurable #45184: a configurable load barrier timeout.
• [Fix] deepseek-v4.1: size the KV pool by the ratio-1/2 layout's own entry bytes #45931: the pool configurator costs a ratio-1/2 compressed KV entry at 584 bytes, but V41_FP4 entries are 288. On one card this takes the pool from 98,560 to 180,480 tokens.

Earlier PRs from the same setup:

• [Feat] all-reduce: host-staged all-reduce for two GPUs without peer access #39605: a host-staged all-reduce for two GPUs without peer access (the repository's tier 2).
• [Fix] kt: read num_hidden_layers from text_config for multimodal checkpoints #41350: kt reads num_hidden_layers from text_config for multimodal checkpoints.
• On kvcache-ai/sglang:
  • fix(dsv4): honor --max-total-tokens for the compressed-attention pool kvcache-ai/sglang#83 (merged): the compressed-attention pool honours --max-total-tokens.
  • feat(dsv4): add SGLANG_V4_MARLIN_PARTIAL for partial-expert MoE kvcache-ai/sglang#84: partial-expert Marlin MoE.
  • fix(kt): apply the KT GPU-expert remap when there is no expert-location metadata kvcache-ai/sglang#97: the KT GPU-expert remap without expert-location metadata.
  • fix(kt): reserve VRAM for the generic full-GPU prefill slot, and honour it kvcache-ai/sglang#98: VRAM for the full-GPU prefill slot.
  • [perf] KT: the shard loaders do not read the CPU experts' checkpoint tensors kvcache-ai/sglang#103: shard loaders that skip the CPU experts.

GGUF work for the same CPU/GPU hybrid on other models: #43427, #43430, #43445, #43626, #43636, #43650, #43677.

The rest of the repository is specific to running experts on the CPU. It covers:

• streaming the experts for prefill;
• lending free KV pages to the most-used experts;
• an all-reduce through host memory for GPUs without peer access.

The tiers and the effect of each patch are in docs/patches.md.

If any of these would be useful upstream, we are glad to split them into PRs.