POC of dspark on qwen3 8B - #2
Draft
Ankk98 wants to merge 30 commits into
Draft
Conversation
Port Qwen3 DSpark encoder/decoder graph and HF conversion with separate v_proj, Qwen residual/norm order, and optional rope_freqs for non-Gemma targets. Assisted-by: Auto Co-authored-by: Cursor <cursoragent@cursor.com>
Assisted-by: Auto Co-authored-by: Cursor <cursoragent@cursor.com>
Assisted-by: Auto Co-authored-by: Cursor <cursoragent@cursor.com>
Align vanilla harness prefill with DSpark spec on all backends. Use batched defer verify by default on non-CUDA at temp=0; CUDA keeps sequential fallback. Assisted-by: Auto Co-authored-by: Cursor <cursoragent@cursor.com>
20 code + 10 agentic prompts (thinking on/off), vanilla reference capture, confidence sweeps, and per-pipeline timing via compare harness JSON export. Assisted-by: Auto Co-authored-by: Cursor <cursoragent@cursor.com>
Support draft-mtp in compare harness via generic verify path. Benchmark driver compares vanilla vs MTP (when mtp-*.gguf exists) vs DSpark. Assisted-by: Auto Co-authored-by: Cursor <cursoragent@cursor.com>
Qwen3-8B has no inbuilt MTP; keep vanilla vs DSpark only. Assisted-by: Auto Co-authored-by: Cursor <cursoragent@cursor.com>
Exclude expected outputs, runs, and results CSV for Qwen3 evaluations from version control to streamline project management.
… details and token match analysis - Clarified command usage for short and long prompt runs. - Added comprehensive evaluation results including acceptance rates and speedup metrics. - Documented findings on token match correctness and bottleneck analysis. - Noted CSV header drift issue and recommended authoritative sources for results.
Run parallel verify logits on a scratch seq (seq_id+1) so draft positions never write main target KV. Append accepted tokens via process_committed with kv_append_only. Drop rolling snapshot and full prefix re-decode. Add trace helpers (DSPARK_TRACE_KV/VERIFY/ORACLE), harness oracle checks, and document root cause in dspark-benchmark-qwen3.md. Requires n_parallel>=2 for batched path; falls back to sequential when only one slot.
Parallel multi-row target forward does not reproduce causal vanilla decode at temp=0, causing token mismatches (e.g. code01 @ gen 162). Make sequential verify the default; keep parallel scratch path opt-in via DSPARK_VERIFY_PARALLEL=1.
Introduced a comprehensive document outlining the current status and future tasks for the DSpark Qwen3 project. This includes details on correctness verification, performance benchmarks, and a prioritized task list for both correctness and performance improvements. The document serves as a living plan following recent updates to the greedy verify mechanism.
Added a new function `common_speculative_dspark_context_reset` to clear target and draft KV and reset decode flags between benchmark runs. Updated relevant functions to utilize this reset mechanism, ensuring that stale K/V bytes do not cause false mismatches. Enhanced the verify path to assert canonical KV invariants during the verification process. This change improves the correctness of the DSpark benchmark by maintaining a clean state across runs.
…validation Enhanced the DSpark benchmark documentation to reflect recent changes, including a full suite baseline with 100% token match and necessary harness fixes for zero mismatches. Updated the benchmark script to ensure debug toggles do not leak into runs and set `DSPARK_BENCH_NPARALLEL=2` for consistent parallel execution. Adjusted the C++ test to ensure the correct `n_parallel` setting for scratch verification. This improves the clarity and correctness of benchmark results.
Extract explicit propose-verify-commit phases per the KV safety refactor plan: scratch verify logits, accept, and canonical commit live in dspark_target; draft propose/trim in dspark_draft; the benchmark harness calls dspark_pipeline_run. Ignore generated qwen3 eval CSV outputs.
…match detection Added functions to verify canary settings and compare logits during the verification phase. Implemented a mechanism to identify the first logit mismatch between batched and sequential decoding, improving the robustness of the verification process. Updated related functions to ensure correct handling of batch operations and synchronization. Adjusted comments for clarity regarding the verification phases and their implications on the canonical sequence.
Updated the verification functions to operate on scratch sequences instead of the main sequence, ensuring that the canonical KV remains unchanged during verification. Introduced state save and restore mechanisms to enhance the verification process. Adjusted related functions and comments for clarity, improving the overall robustness and correctness of the DSpark verification pipeline.
Added the function `dspark_draft_process_committed` to handle the processing of committed tokens within the draft phase. This function integrates with the existing pipeline to ensure that committed tokens are correctly injected into the draft KV, enhancing the overall robustness of the token verification and commitment process. Updated related functions to utilize this new processing step, improving the clarity and functionality of the DSpark pipeline.
…estigation doc Fix the root cause of batched-verify divergence on Vulkan: the last-layer inp_out_ids gather (ggml_get_rows feeding lm_head) triggers a different mul_mat shader on Vulkan between n_outputs=1 (single-token decode) and n_outputs=N (batched verify). The ~0.2 logit delta flips greedy argmax, breaking spec decoding at temp=0. Changes: - qwen3.cpp, gemma4.cpp: skip the inp_out_ids gather so lm_head always runs on the full [n_embd, n_tokens] tensor. This makes batched multi- token logits numerically identical to single-token decode on Vulkan. CPU was already fine; CUDA/FA expected to be fine too. - dspark_pipeline.cpp: fix dspark_now_ms() timing clock (was ~1000x too small due to static init + 1e-3 unit error). Remove fused_verify skip_host_logits from prefill (unused in default path). Remove trace env hack from pipeline run loop. - smoke_batched_logits_repro.cpp: fix scan mode pos bug (restore state after compare_row0_logits before sequential verify). - docs/dspark-investigation.md: full investigation log with experiments, root cause analysis, design options. Token match confirmed: code_500l n=200, code01_think_on n=250, code01_think_off n=200 (all on Vulkan Q4, scratch verify path). Assisted-by: Claude Sonnet
…yle) Replace the scratch-seq + per-token commit loop with a single batched llama_decode on the canonical sequence: decode [anchor, draft...] in one forward, greedy-accept from logits, trim rejected tail, process target features into draft. This is the vLLM / DFlash pattern: one forward per propose step delivers logits + KV + layer features. Correctness depends on the model-graph fix (qwen3.cpp/gemma4.cpp) making batched logits numerically identical to single-token decode. Token match: - code_500l (83% accept): match True, 1.57x speedup - code01_think_off (55% accept): mismatch at gen 41 - code01_think_on (35% accept): mismatch at gen 73 The divergence on low-accept prompts is the Vulkan mul_mat M=1 vs M=N numerics issue affecting ALL layers (QKV/FFN projections), not just the last-layer gather. The graph fix addressed lm_head, but internal-layer matmuls accumulate KV differences over ~40+ steps. The scratch-seq path (batched logits on scratch, single-token commit decodes on canonical) remains correct on Vulkan because it separates logits from KV. For 2x speedup on Vulkan, the backend mul_mat must be made numerically consistent for M=1 vs M=N across all layers.
The graph fix (last-layer gather removal) makes batched logits match sequential decode. But all internal-layer matmuls (QKV/FFN) also diverge between M=1 and M=N on Vulkan, causing KV drift over ~40 steps that flips greedy argmax on low-to-moderate acceptance prompts. Two architectures assessed: - Main-seq parallel verify (DFlash style): fast (1.57x on code_500l), matches vanilla at high accept rates, diverges at low accept. - Scratch-seq verify: correct on all prompts (token_match=True proven), but slow (0.4x) due to per-token commit decodes. The Vulkan matmul M=1 vs M=N precision issue is the root limitation preventing 2x speedup with vanilla-identical output. CUDA is expected unaffected (CPU confirms zero divergence through 90-step scan).
…lines The mul_mat_vec shader uses a specialization constant NUM_COLS (compile-time) for the number of output columns. Different ne11 values get different compiled SPIR-V with different loop unrolling and FP accumulation order, causing ~0.2 logit deltas that break batched row0 vs sequential row0 equality for speculative decoding. Add DSPARK_CONSISTENT_MMV=1 env flag: forces all ne11 values (1..8) to use the same max-columns pipeline (NUM_COLS=8), making per-column mul_mat results numerically identical regardless of batch width. This makes the smoke_batched_logits_repro scan pass (0 mismatches through 90 steps on Vulkan Q4), confirming the per-column mul_mat results are now consistent. However the MAINSEQ parallel-verify path still diverges because other ops (attention softmax, KV writes, etc.) also have n_tokens-dependent numerics not yet addressed. The flag is experimental: it changes the pipeline for ALL inference (including vanilla decode), trading a small throughput regression for numerical consistency.
The mul_mat_vec shader's NUM_COLS specialization constant creates different pipelines for different batch widths, causing FP divergence. DSPARK_CONSISTENT_MMV=1 forces same pipeline, fixing logits comparison but not the full forward pass KV consistency.
Extend the CONSISTENT_MMV fix to ggml_vk_guess_split_k (ensures split_k is identical for ne11=1 and ne11=N) and to ggml_vk_guess_matmul_pipeline / ggml_vk_guess_matmul_id_pipeline (uses consistent n for pipeline large/medium/small selection thresholds and tiling computation). These handle the non-vector matmul path (ggml_vk_mul_mat_q_f16), used by attention QK^T and AV computations where ne12*ne13 > 1. Smoke scan (row0 logits) passes with CONSISTENT_MMV but the MAINSEQ parallel verify path still diverges. Remaining ops likely affected: soft_max, and the fact that different n_tokens dispatch different workgroup counts in the N dimension of various matmuls and reductions.
…roach) Change the CONSISTENT_MMV pipeline selection from NUM_COLS=8 (force all batches to use max-columns shader) to NUM_COLS=1 (force all batches to use the single-column shader). The NUM_COLS=1 shader has the same shared memory layout as single-token decode, so per-column FP is identical. The column dispatch loop (CPU-side) submits one dispatch per token, each with ne11=1. This change alone makes the pipeline selection use index 0 (NUM_COLS=1) when CONSISTENT_MMV is active. The dispatch functions (ggml_vk_mul_mat_vec_q_f16 and ggml_vk_mul_mat_q_f16) still need column-loop additions to actually process multiple columns, which will be added in a follow-up.
Add per-column dispatch loops inside ggml_vk_mul_mat_q_f16 (matmul path) and ggml_vk_mul_mat_vec_q_f16 (mmvq path) when the env var DSPARK_CONSISTENT_MMV=1 is set. - matmul path: column loop decomposes ne11 (>1) into per-column dispatches, computing batch_stride from nb[1] and the quantized or f16 prealloc buffer layout. Only affects safe cases (f32/f16 src1 type, single ne2*ne3). - mmvq path: column loop decomposes the batch_n group into per-column dispatches with adjusted subbuffer offsets. Dispatches z-dimension set to 1 (single column per dispatch). Both loops adjust only buffer offsets (vk_subbuffer), never tensor metadata fields (ne[], view_offs, nb[]). This avoids the storage buffer alignment issues that made per-element view_offs tuning Smoke scan: 0 logit mismatches through 90 batched-vs-single steps. MAINSEQ pipeline: token-match for first 40 steps, diverges at gen 41 due to non-matmul ops (softmax, elementwise) still having batch-width-dependent dispatch on Vulkan. Assisted-by: Claude Sonnet
Ankk98
marked this pull request as draft
July 13, 2026 11:59
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Work in progress