Conversation
* Add new command-line options for DSpark, including --input-ids, --dspark-confidence-threshold, and --dspark-disable-markov. * Introduce common_load_input_ids_json function to load token IDs from a JSON array. * Extend common_params_speculative structure to include DSpark-specific parameters. * Implement DSpark speculative decoding logic in common_speculative.cpp. * Add Gemma4DSparkModel class for model conversion and integration. * Update documentation to outline DSpark implementation plan and validation steps. This commit lays the groundwork for integrating DSpark speculative decoding into the llama.cpp framework, enhancing model performance and flexibility.
…eculative.cpp * Introduced conditional logging to display draft and target IDs during the speculative decoding process when the DSPARK_DEBUG environment variable is set and the number of propose steps is 12 or fewer. * This enhancement aids in debugging and understanding the acceptance process in speculative decoding.
* Introduced a new structure for managing GPU resources for the DSpark Markov head, allowing heavy computations to be offloaded to the GPU when available. * Added initialization and cleanup functions for GPU resources in llama-model.cpp. * Updated common_speculative_impl_draft_dspark to utilize GPU for bias calculations, improving performance during speculative decoding. * Modified tests to ensure compatibility with the new GPU offloading feature, enhancing the overall robustness of the speculative decoding process.
* Enhanced the handling of logits by introducing a softcap mechanism and integrating Markov bias calculations into the greedy sampling path. * Improved performance by conditionally applying softcap and Markov bias based on temperature and draft probabilities. * Streamlined the process for sampling tokens, ensuring efficient computation during speculative decoding.
* convert: add dsv4 conversion * add basic setup * add llm_graph_input_dsv4 * add save-load state * add sinkhorn eps - correction by @fairydreaming * add rope fix * cleanup dead code * fix bugs * support pro model: added by @fairydreaming * remove redundant V cache * Chat template * remove debugging leftovers * Add mechanism for inlining templates based on architecture * s/deepseek-v4-flash/deepseek4/g * s/deepseek-v4-flash/deepseek4/g continued * enable graph reuse * enable FA * fix test llama archs * rename * compatibility with antirez ds4 GGUFs * simplified set_gguf_parameters() by calling super class method, replaced moe.score_func with expert_gating_func. * reserve worst-case kv-cache * revert max split inputs * address review comments * add padding to enable FA * pad only the final value of plan.n_kv to 256 * remove built-in cpp chat template * cont: remove cpp built-in template * rm outdated test * replace ggml_view_3d() with ggml_reshape_3d() Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * only support n_seq=1 for now * remove unused var * cont: remove unused var * use scale bias * use correct ptr for can_reuse * remove gen-chat-inline-templates.py * simplify graph reuse * cont: cleanup * remove unused inputs * enable partial checkpointing * add correct shape for kq_mask + set llama_model_n_swa to 0 for dsv4 * precompute source_idx + add comment about dummy write * support multi-seq * remove restored_trim_pos * use split_equal when possible * fix indent * address review comments * use LLM_KV * fix ci --------- Co-authored-by: Piotr Wilkin <piotr.wilkin@syndatis.com> Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com> Co-authored-by: Xuan Son Nguyen <son@huggingface.co> Co-authored-by: fairydreaming <166155368+fairydreaming@users.noreply.github.com> Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* Introduced a new GPU-based block sampling mechanism that fuses the Markov chain processing into a single graph dispatch, significantly improving performance during speculative decoding. * Added profiling capabilities to track performance metrics for draft-forward and sampling steps. * Updated the common_speculative_impl_draft_dspark structure to accommodate the new block sampling logic and integrated it into the decoding process. * Enhanced the llama-ext.h and llama-model.cpp files with new functions for initializing and managing the GPU block sampler. * Modified tests to include timing statistics for draft and verification steps, aiding in performance evaluation.
* Updated the handling of the `common_speculative_process` function across multiple test files to ensure it processes the correct batch of tokens. * Introduced new timing metrics for target decoding, acceptance, and processing steps to enhance performance profiling. * Adjusted output logging to provide clearer insights into the timing of various speculative decoding phases, aiding in performance evaluation.
Single-pass batched target verify with post-decode sync, truncated process() on the committed prefix, and consolidated verify_batched API. Assisted-by: Auto Co-authored-by: Cursor <cursoragent@cursor.com>
- Single GPU sync for multi-layer feature reads (no_sync layer inp API) - Optional split verify via ctx_tgt_feat + shared KV (DSPARK_SPLIT_VERIFY=1) - Benchmark harness: spec-first, warmup, draft-max 5 for smaller verify batch Assisted-by: Auto Co-authored-by: Cursor <cursoragent@cursor.com>
Cap draft decode at n_draft+1 (keeping fused block_gpu per length), shrink proposals adaptively on low hit-rate prompts, default n_max=4, dedupe verify sync, and run vanilla before spec in compare_vanilla_speculative. Assisted-by: Auto Co-authored-by: Cursor <cursoragent@cursor.com>
Expand adaptive n_max when hit rate > 72%, add decode_submit_ms to verify timing, restore spec-first bench order, default n_max=3 (best coding sweep), and add docs/dspark-benchmark-experiments.md for experiment tracking. Assisted-by: Auto Co-authored-by: Cursor <cursoragent@cursor.com>
Print vanilla vs spec prefill and generation tok/s in compare harness, skip redundant process() GPU fence on short verify batches, tune adaptive n_max thresholds (0.50/0.65), and expand benchmark doc with measured pp/tgp numbers. Assisted-by: Auto Co-authored-by: Cursor <cursoragent@cursor.com>
Assisted-by: Auto Co-authored-by: Cursor <cursoragent@cursor.com>
Run vanilla before spec with KV reset and 3s cooldown, document actual pp/tgp for both paths, retract spec-first inflated speedups, set n_max=4 as fair coding sweet spot (1.61x, token match YES). Assisted-by: Auto Co-authored-by: Cursor <cursoragent@cursor.com>
Assisted-by: Auto Co-authored-by: Cursor <cursoragent@cursor.com>
Assisted-by: Auto Co-authored-by: Cursor <cursoragent@cursor.com>
…tic (ggml-org#25005) * vulkan: extract flops calculation into function * use flops instead of matmul src0 tensor size for submission threshold * use unsigned ints
Assisted-by: Auto Co-authored-by: Cursor <cursoragent@cursor.com>
…g#25131) Signed-off-by: Adrien Gallouët <angt@huggingface.co>
Fix defer-layer D2H to copy only accepted rows from the full verify batch. Continue generation when confidence truncation yields an empty proposal. Document confidence-threshold sweep (-c 512) and add opt-in fused verify argmax. Assisted-by: Auto Co-authored-by: Cursor <cursoragent@cursor.com>
Introduce a new document for DSpark benchmark experiments on NVIDIA CUDA, detailing the environment setup, performance metrics, and comparisons with Vulkan. Update the existing benchmark experiments document to reference the new CUDA results, highlighting a ~2.0x coding speedup on RTX 3090.
Default sequential target verify on CUDA builds (batched multi-token forward diverged from vanilla). Add shared prefill API, committed re-decode path for non-defer backends, and CUDA-aligned vanilla prefill in the compare harness. Assisted-by: Claude Sonnet Co-authored-by: Cursor <cursoragent@cursor.com>
…park` Introduce a new comprehensive reference document comparing NVIDIA's upstream vLLM DSpark integration with the llama.cpp port on branch `ft-dspark`. The document outlines architectural choices, performance metrics, and key differences between the two implementations, providing insights into their respective approaches to non-causal attention, CUDA graph usage, and model coverage. Related documentation links are also included for further context.
…org#24588) * HIP: keep MMQ for gfx900 MoE and Q8_0, use hipBLAS for dense K-quants Assisted-by: GitHub Copilot CLI * HIP: tighten conditional block to be explicitly for gfx900 * HIP: Further simplified gfx900 conditional block * removed unnecessary comment
* vulkan: roll bk loop in matmul for asahi linux * vulkan: fix inline comment * vulkan: revert BK-loop unroll change * vulkan: edit spirv directly for asahi roll bk loop * vulkan: remove trailing whitespace at the end of comments
Make greedy verify sequential by default for token-match correctness; keep batched defer/fast paths behind DSPARK_VERIFY_FAST=1 with KV tail trim and committed-batch process fixes. Add CUDA VPS setup script, P0 batched-logits repro harness, and benchmark/docs updates. Assisted-by: Auto Co-authored-by: Cursor <cursoragent@cursor.com>
…itional options. Introduce flags for skipping sync, model downloads, builds, and verification, along with a reboot option for GPU issues. Update usage documentation to reflect new capabilities.
Keep DSpark layer taps enabled during batched verify decode, process accepted rows without a second target re-decode, and fall back to sequential verify on CUDA when DSPARK_VERIFY_FAST is set. Assisted-by: Auto Co-authored-by: Cursor <cursoragent@cursor.com>
Ankk98
marked this pull request as draft
July 22, 2026 18:52
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
10 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is a work in progress.
This is a proof of concept branch, meant for validating and benchmarking dspark algo. The plan to to create a new branch and PR once the validation is complete. This PR is just for viewing diff.
Here, I am trying to implement DSpark Speculative Decoding Algo for Gemma 4 12B QAT variant.
Key Benchmarking Results
Workload Fair best
Coding 1.57x (-c 512)
Agentic 1.40x
Notes
Testing Hardware
References
Contact
Feel free to reach out to me for anything at https://x.com/ankk98