DFlash (Block Diffusion for Flash Speculative Decoding)

[wsbagnsv1]

Just saw this new repo on r/locallama and its seems promising, they (z-lab) claim to have a decent speedup even over eagle3, which is still a draft pr here. The idea is basically to not do a autoregressive draft model but a diffusion one to speed things up and ofc the stats vary but for Qwen3.5-35B-A3B for example they claim a 2-2.8x speedup over normal token generation with an acceptance length mostly beating even mtp other models are at times even better like qwen3.5 9b with up to 3.5x, as can be seen in their repo and huggingface. Im not sure if the devs here think its worth it rn with eagle3 having a draft pr already so ive created this as a discussion to just bring this up (;
The repo is available here
and the already available draft models can be found here

[jschoch]

there are a bunch of MTP PR's but they don't seem to be making much progress and seem to be EAGLE 3 focused. I'm also watching this diffusion draft approach closely.

[wsbagnsv1]

yeah thats why i didnt want to crowd the issues tab with another draft feature request although this one seems pretty neat (;

[ruixiang63]

For anyone interested in DFlash, please take a look at this PR: #22105 Now it comes to llama.cpp :)

[ruixiang63]

I’ve added dflash support for Qwen3.5/3.6 MoE. However, please note that the performance is currently not optimal. This is mainly due to the MoE architecture combined with the hybrid structure used in Qwen3.5/3.6 MoE, which is not yet well supported by llama.cpp. For more details on the performance limitations and underlying issues, please refer to my shared PR: #22105

[lym000000]

@ruixiang63 Thanks for the update!

Performance issue: noted on the MoE + hybrid architecture limitations with llama.cpp. Will keep an eye on #22105 for progress.

Suggestion: saw that the ggml team has been working on refactoring the codebase (e.g., CMake glob changes and bias tensor renaming). It might be easier for the ggml team if the PR is rebased on the latest upstream master to avoid conflicts with those structural changes.

[lym000000]

Don’t see it mentioned here yet, there's also DDTree for accelerated speculative decoding on top of DFlash.

• Page: https://liranringel.github.io/ddtree/
• Code: https://github.com/liranringel/ddtree
• Paper: https://arxiv.org/abs/2604.12989

[ruixiang63]

Yeah, thanks. My thought is to get the Eagle3 PR #18039 merged first, since the API refactoring also needs to happen there. My DFlash PR is based on Eagle3, so once that’s merged, I’ll rebase the DFlash PR on top of it.

[SunYong0821]

Yeah, thanks. My thought is to get the Eagle3 PR #18039 merged first, since the API refactoring also needs to happen there. My DFlash PR is based on Eagle3, so once that’s merged, I’ll rebase the DFlash PR on top of it.

Does it also work for dense models? like qwen3.6 27B

[gsrunion]

Data points from AMD Strix Halo (Ryzen AI Max+ 395, gfx1151, unified memory), Qwen3.6-35B-A3B + the published DFlash draft:

• UD-Q4_K_XL: 48 tok/s plain → 91-95 tok/s with draft-dflash on code/reasoning (64-74% acceptance); prose ~54 (31% acceptance).
• First spec-decode method that pays on this hardware — draft-simple, eagle3-style, and MTP all lost to serial draft overhead here.
• Anecdote (from a downstream FP4 fork build, not reproducible on stock): the draft also runs against a same-lineage sibling model (Qwen3.5-MoE-based fine-tune) at ~98% acceptance — drafts may be more family-portable than expected.

[d-shehu]

One problem I have with DFlash is the extra memory needed for the layer on top of the MTP which is baked into certain models like Qwen 3.6.

The other problem is when using tensor split, it significantly degrades performance. I don't fully understand the contention but unless I dump the DFlash layer into a single GPU rather than split it across both it makes slower than turning it off.

rocm & -sm tensor

[pich]

Adding a dense-model datapoint here, since most results in this thread are for MoE. I tested DFlash with Muse Glimmer 30B Q5_K_M on a single RTX PRO 4000 Blackwell SFF 24 GB capped at 70 W.

Regular decode: 17.98 tok/s
DFlash on coding: 80.87 tok/s (4.50×)
Best tuned run: 84.64 tok/s, 38.67% acceptance, average accepted prefix τ = 6.39
Mixed agent workload: 38.34 tok/s, 14.41% acceptance

I also filled the actual 262K context: at 262,116 input tokens it still decoded at 21.56 tok/s, using 22,920 / 24,467 MiB VRAM.

One implementation detail was measurable: moving draft argmax from the CPU path into the GPU backend graph improved the mixed benchmark from 36.30 → 38.34 tok/s (+5.6%). So at least on this dense 30B model, DFlash can pay off very significantly when the continuation is predictable. The gap between ~85 tok/s on code and ~38 tok/s on mixed workloads also shows how workload-dependent the headline speedup is. Full benchmark, patches and methodology: https://piszczek.pl/blog/dflash-changes-what-tokens-per-second-means