How fast can you run DeepSeek V4 Spark 0731 with CPU-GPU hybrid inference when VRAM is not enough?

[guqiong96]

I’ve been seeing a lot of discussions about running DeepSeek-V4-Flash-0731 on high-end setups like dual H100s or DGX Spark. But what if you don’t have that kind of hardware? Can you still run this beast on consumer-grade GPUs with limited VRAM?

I’ve been experimenting with a hybrid inference approach that offloads part of the MoE experts to system RAM + CPU when the model doesn’t fit entirely into GPU memory. Think of it as a “VRAM extender” for MoE models — the GPU still handles most of the compute, but the CPU steps in to help with expert layers dynamically.

I’m using a custom inference stack built on top of a modified sglang and vLLM backend, with a high-performance kernel library that handles the CPU-GPU scheduling and NUMA-aware memory management. The whole thing is designed to “just work” without requiring model sharding or complex distributed setups.

Benchmark Results (real runs on actual hardware)

Here are two configurations I’ve tested so far:

Open GPU Prefill, max_num_batched_tokens=8192 (Row 1), max_num_batched_tokens=32768 (Row 2)

Model	Version	CPU	Memory	GPU	Prefill	Decode	Speculative Decoding
deepseek-ai/DeepSeek-V4-Flash-0731	Lsglang-v1.4.7	EPYC 7642 *2	16 channels ddr4 3200	5060Ti * 2	850 t/s [input 32768]	28 t/s [input 32768]	30~46 t/s
deepseek-ai/DeepSeek-V4-Flash-0731	Lvllmds4-x-v2.3.9	EPYC 7642 *2	16 channels ddr4 3200	3090 * 2	1060 t/s [input 32768]	26 t/s [input 32768]	35~47 t/s
deepseek-ai/DeepSeek-V4-Flash-0731	Lvllmds4-v2.3.9	EPYC 9684x *2	24 channels ddr5 4800	pro 6000 * 1	3100 t/s [input 131072]	75 t/s [input 131072]	100~115 t/s
Software stack used: Lsglang Lvllm

[KushagraKanaujia]

Hey, building Throttle: semantic cache for self-hosted inference. Cuts repeat-query costs 40-60%, zero code changes. Works with vLLM, Ollama, SGLang, LMDeploy.

Open-source, free beta. Looking for users to test:

https://github.com/KushagraKanaujia/throttle