Skip to content

v0.6.5 - DFlash2 speculative decoding: 2x decode that holds at depth

Choose a tag to compare

View all tags
@Nathanw1014 Nathanw1014 released this 19 Aug 22:04
v0.6.5
1185a3d

v0.6.5 - DFlash2 speculative decoding: 2x decode that holds at depth

Adds DFlash2 speculative decoding, validated on gfx1151 / Vulkan-RADV. On Qwen3.8-27B this is
about 2x decode over base at every depth measured, and unlike DFlash v1 it does not fall apart
at long context.

Engine support is upstream ggml-org/llama.cpp#27342
by Jian Chen and is not our work. What is ours: cherry-picking it onto the Strix branch, validating
it on gfx1151 (688 test-backend-ops cases over the ops the new graph adds, drafter fully GPU
resident at graph splits = 2, no CPU fallback), a capability gate on the DFlash2 draft path, and
the draft-width tuning below.

Decode, tokens/s (Qwen3.8-27B UD-Q4_K_XL, f16 KV, Framework Desktop 64 GB, iommu=off)

context depth base DFlash v1 (n=5) DFlash2 (n=4)
0 11.81 21.09 (1.79x) 26.39 (2.23x)
8192 11.44 12.87 (1.12x) 21.58 (1.89x)
32768 10.54 10.75 (1.02x) 21.11 (2.00x)

DFlash v1 is worth essentially nothing by 32k (1.02x). v2 still returns 2.00x. The conv and
selector modules earn their keep exactly where the older drafter gives up.

Prefill is unchanged within noise (about 2 to 4 percent under base, the usual drafter overhead).

Using it

Drafter weights are inco's published GGUFs, Apache-2.0:
incoai/Qwen3.8-27B-DFlash2-GGUF

llama-server -m Qwen3.8-27B-UD-Q4_K_XL.gguf \
  -md Qwen3.8-27B-DFlash2-Q8_0.gguf \
  --spec-type draft-dflash --spec-draft-n-max 4 \
  -ngl all -ngld all -fa on

--spec-draft-n-max 4 is the better default for long context: at 32k, width 4 gives 21.11 t/s
against width 7's 16.32, so about +29 percent. They tie at d0 and 8k.

Read before quoting the numbers

  • The 2x depends on what you generate. These runs continued English prose (a Project
    Gutenberg book). Speculative decoding wins in proportion to how predictable the text is, so
    prose flatters it. On a code corpus the same config decayed about 47 percent from shallow
    context to 32k, against about 20 percent here. Neither is wrong, they are different
    workloads. Expect less on code.
  • The draft width optimum moves with workload, not only with depth. On code completion width 5
    to 7 wins at about 1.3k; through the chat template on the same code, where the model emits
    reasoning first, width 3 wins. Aggregate acceptance does not predict the optimum on its own.
    Tune per workload.
  • Upstream #27342 is still in review. If the drafter format changes before it merges, the
    published GGUFs may need re-downloading.

Full measurements and method: docs/dflash2-strix.md

Changes since v0.6.4

  • support DFlash2 (015f09c8a)
  • spec: capability-gate the DFlash2 draft path (0b0f35d0e)

Payload built from Nathanw1014/llama.cpp@0b0f35d0e (branch strix-halo-vulkan), bundled Mesa
RADV devenv driver as in v0.6.4.

Assets 4
Loading