Update DeepSeek V4 Flash recipe to 0731 checkpoint with DSpark + NVFP4 KV + 1M context

[scarcer-1]

DeepSeek released DeepSeek-V4-Flash-0731 yesterday, a substantial update to the V4 Flash preview. The 0731 checkpoint delivers massive agentic benchmark gains — Terminal Bench 2.1 jumps from 61.8 → 82.7, DeepSWE from 7.3 → 54.4, Toolathlon-Verified from 49.7 → 70.3, Agents' Last Exam from 15.8 → 25.2 — and is broadly competitive with much larger models.

The current recipes/deepseek-v4-flash.yaml points at the preview checkpoint (deepseek-ai/DeepSeek-V4-Flash) and uses MTP speculative decoding with 2 draft tokens and FP8 KV cache. The 0731 release enables a significantly better configuration:

1. Update model reference → deepseek-ai/DeepSeek-V4-Flash-0731

The 0731 checkpoint has the same architecture as the DSpark preview but with improved weights. 167 GB, 48 safetensor shards, MIT license.

2. Switch speculative decoding from MTP → DSpark

The 0731 checkpoint ships with a built-in speculative decoding module (DSpark). The official vLLM serving command now uses:

--speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"greedy"}'

This is a departure from the current recipe's MTP-based approach and should yield substantially higher decode throughput.

3. Add NVFP4 KV cache support for full 1M context

The current recipe uses --kv-cache-dtype fp8 with max_model_len: 500000. The 0731 official recipe uses --attention-config '{"use_fp4_indexer_cache": true}' alongside FP8 KV, while tonyd2wild's DSpark recipe demonstrates nvfp4_ds_mla KV cache dtype achieving 1M calibrated context (pushed to 1.5M) on DGX Spark hardware with NVFP4 compression.

NVFP4 KV compression roughly halves the KV cache memory footprint per token, which is the critical enabler for hitting 1M context on the memory-constrained DGX Spark platform.

4. Additional considerations

• tonyd2wild's Patch 4 is required when swapping to 0731 weights with DSpark (vLLM's DSpark draft weight loader silently drops twelve shared-expert tensors on 0731, halving throughput without it).

Related artifacts:

• DeepSeek-V4-Flash-0731 on HuggingFace — official release with model weights, encoding tools, and vLLM serving instructions
• tonyd2wild/DeepSeek-v4-Flash-0731-DSpark-1M-NVFP4-KV-2x-DGX-Spark — verified recipe for DSpark + NVFP4 KV + 1M context on 2× DGX Spark, including the required Patch 4 fix

Proposed action: Update recipes/deepseek-v4-flash.yaml to target the 0731 checkpoint, add DSpark spec decode, add NVFP4 KV cache support (or a separate NVFP4 variant recipe), and raise max_model_len to 1,000,000 (or higher, per the tonyd2wild findings). The existing FP8 / MTP recipe could be retained as a fallback for users without NVFP4 support.

[alvin-000]

Thanks for sharing, I was looking at the Nvidia forum for how to fix the MTP not loading.

[phonosys]

a must have recipe!

[phonosys]

so far running great using the following params:

SPECULATIVE_CONFIG='{"method":"mtp","num_speculative_tokens":5,"draft_sample_method":"greedy"}';
./launch-cluster.sh exec vllm serve deepseek-ai/DeepSeek-V4-Flash-0731
--served-model-name deepseek-v4-flash-0731
--host 192.168.0.5 --port 8001
--trust-remote-code
--tensor-parallel-size 2 --pipeline-parallel-size 1
--kv-cache-dtype fp8_ds_mla
--block-size 256
--max-model-len 524288
--max-num-seqs 4
--max-num-batched-tokens 4096
--gpu-memory-utilization 0.80
--enable-prefix-caching --async-scheduling --enable-chunked-prefill
--tokenizer-mode deepseek_v4
--distributed-executor-backend mp
--tool-call-parser deepseek_v4 --enable-auto-tool-choice
--reasoning-parser deepseek_v4
--reasoning-config '{"reasoning_parser":"deepseek_v4","reasoning_start_str":"","reasoning_end_str":""}'
--default-chat-template-kwargs '{"thinking":false}'
--generation-config vllm
--enable-flashinfer-autotune
--nnodes 2

[doongkc]

@phonosys , which vllm build you got?
is it from this project?
I tried the lattest build, but it did not work.
When I set dspark speculative, a weird error reported

[phonosys]

dspark is crashing for me , tbh im not even running mtp (SPECULATIVE_CONFIG is not being used)
getting around 25t/s which im fine with .

[doongkc]

some update.
After use the repository tonyd2wild/DeepSeek-v4-Flash-0731-DSpark-1M-NVFP4-KV-2x-DGX-Spark
I build follow his guideline and successfully to host official 0731.

Some time speed will bump to 8x tok/s when high acceptance reate

Hope this project can have some issue fix for DS4, I still prefer the usage of eugr/spark-vllm-docker

[luck02]

Data point: 0731 running in production on 2x DGX Spark (GB10, TP=2 over CX-7) since today, built from this repo's main + the mods from #304. A few findings that may save others some time:

The "Patch 4" shared-expert loader fix is NOT needed on vLLM releases/v0.26.x. tonyd2wild's dropped-12-tensors bug (draft acceptance collapsing to ~26% at half throughput) doesn't exist there: the loader in vllm/models/deepseek_v4/nvidia/dspark.py handles the draft's shared experts via generic ("gate_up_proj", "w1"/"w3") stacked rules gated by is_layer_param — the markov collision was solved with the gate instead of narrowing the mapping. Verified in-image and confirmed empirically (no acceptance collapse, see numbers below). His patch targets runtimes with the narrowed _STACKED_PARAM_NAME_MAPPING table only.

Build: ./build-and-copy.sh --vllm-ref releases/v0.26.0 → vLLM 0.26.1.dev0 + FlashInfer 0.6.17. Both #304 FlashInfer TOPK=256 mods are still required against 0.6.17 (decode added dispatch entries + CUDA instantiations; prefill added the BF16 dispatch), and the d2t guard still applies. All three no-op cleanly if/when upstream catches up, so carrying them is safe.

Config (conservative bring-up, mp/no-ray default):

--kv-cache-dtype fp8 --max-model-len 262144 --max-num-seqs 4
--max-num-batched-tokens 4096 --gpu-memory-utilization 0.78
--speculative-config '{"method":"dspark","num_speculative_tokens":5,"draft_sample_method":"greedy"}'
--tokenizer-mode deepseek_v4 --tool-call-parser deepseek_v4 --reasoning-parser deepseek_v4
--enable-auto-tool-choice --generation-config vllm --load-format instanttensor

Results (warm): 32–40 tok/s single-stream, ~46% draft acceptance cumulative under mixed load, 8-request soak + 3-way concurrency clean, tool-calling and reasoning parsing working. Went with k=5 greedy per #304's sweep (didn't re-test the card's k=7). 0.78 util, not 0.80 — DSpark allocates draft buffers on first request, not at boot. No --async-scheduling (see #194).

Weight note: loading from an NFS-backed HF cache works fine — 155G streams at ~205 MB/s, ~13 min cold, one NAS copy serves both nodes.

Quality check for anyone wondering if the swap is worth it: blind 50-question pairwise grading vs the preview checkpoint came out 27–14–9 for 0731, winning every category. The preview's most common loss mode was scratch reasoning leaking into final answers; 0731 keeps the answer channel clean.

Recipe + mods are on my fork if useful: https://github.com/luck02/spark-vllm-docker/tree/dsv4-0731 (recipes/deepseek-v4-flash-0731-dspark.yaml).

[eugr]

This model is supported by b12x branch that I'm going to merge into main sometime next week hopefully and include it in my build pipeline. It should give you about 50 t/s on average.

You can try it now:

git checkout b12x
git pull
./build-and-copy.sh --exp-b12x -c
./run-recipe.sh deepseek-v4-flash-0731

[doongkc]

thanks for quick update @eugr
I have a point need to confirm, is your build return the correct thinking block?
When I try other vllm modification, it didn't generate the <think> token, that all reasoning content returned as normal output.
In the output, always exist the </think> token.

Quick check b12x branch, in recipe for 0731,

--reasoning-config '{{"reasoning_parser":"deepseek_v4","reasoning_start_str":"","reasoning_end_str":""}}' \
      --default-chat-template-kwargs.thinking=true \
      --default-chat-template-kwargs.reasoning_effort=high \

Is that mean the server skip to parse the thinking block while start/end string is ""?

Just building b12x now.

[s0301132]

thanks for quick update @eugr I have a point need to confirm, is your build return the correct thinking block? When I try other vllm modification, it didn't generate the <think> token, that all reasoning content returned as normal output. In the output, always exist the </think> token.

Quick check b12x branch, in recipe for 0731,

--reasoning-config '{{"reasoning_parser":"deepseek_v4","reasoning_start_str":"","reasoning_end_str":""}}'
--default-chat-template-kwargs.thinking=true
--default-chat-template-kwargs.reasoning_effort=high
Is that mean the server skip to parse the thinking block while start/end string is ""?

Just building b12x now.

@doongkc Hi, are you using claude code with API error "Content block is not a thinking block"? This is a known issue the model is not returning the standard thinking block and causing error, I'm using the local proxy to track all traffic and fill the thinking block if it's missing.

[doongkc]

hello @s0301132 ,

I'm not get that issue, claude code respone fine, but without thinking rendering

[eugr]

@doongkc - haven’t had any issues with reasoning parser not separating thinking properly. I haven’t tried Claude Code yet, but OpenCode, Hermes and other clients I tried all handled the thinking properly.

[eugr]

For those who don’t want to build - I pushed the latest version as eugr/spark-vllm-b12x

Haven’t had a chance to wire this pull to the build script yet.

[doongkc]

@eugr - just complete the build as your command.
I done some quick test. Something I found are:

• Thinking mode is not enabled (even kwargs replacement). Because not have any response that include the block.
• For same question (just simple like car wash question), the output for each thinking level is equivalent in length, not have any sign that the model think more about the question.
• Tried messages, reposones and chat/completion api type with different reasoning config.

Update: The think level worked when I request messages API, just my proxy server issue when convert api type
messages (agent tool) <--> proxy <--> responses (vllm).
Changed to messages (agent tool) <--> proxy <--> messages (vllm)
But I still can't test the think mode when request responses and chat/completion api

[s0301132]

@eugr - just complete the build as your command. I done some quick test. Something I found are:

• Thinking mode is not enabled (even kwargs replacement). Because not have any response that include the block.
• For same question (just simple like car wash question), the output for each thinking level is equivalent in length, not have any sign that the model think more about the question.
• Tried messages, reposones and chat/completion api type with different reasoning config.

Update: The think level worked when I request messages API, just my proxy server issue when convert api type messages (agent tool) <--> proxy <--> responses (vllm). Changed to messages (agent tool) <--> proxy <--> messages (vllm) But I still can't test the think mode when request responses and chat/completion api

Hi @doongkc, if you're using claude code with cc-switch, you can use my ds-proxy here: https://github.com/s0301132/ds-proxy, add your config.env file then host the proxy, config your cc-switch to use local route with OpenAI Chat Completions. So far I'm not encountering any thinking block issue after apply the proxy.