Feature Request: Integrate Speculative Decoding for Token Generation Speedup (Default & Configurable)

[sawansri]

Overview

We should integrate and expose llama.cpp's Speculative Decoding features natively within Lemonade.

At a high level, Speculative Decoding accelerates generation by guessing (drafting) multiple upcoming tokens and then having the main model verify/accept them all at once in a single batch.

On UMA architectures like Strix Point/Halo, models run from shared LPDDR5X memory, making memory bandwidth the primary bottleneck. Speculative decoding is particularly well-suited here since the RDNA 3.5 (GPU) compute is largely idle during standard decoding, and the batched verification step can exploit that spare compute to extract multiple accepted tokens from what is effectively a single memory pass over the weights.

There are two main approaches (more details in the llama.cpp docs):

1. Draft Models: Uses a secondary, much smaller model to guess the next tokens.
2. N-gram / Self-Speculative: Looks for patterns in the current prompt/context window to guess upcoming tokens (e.g., ngram-simple, ngram-mod, ngram-map-k4v). These methods have a very slight overhead and are typically lighter to run compared to draft models, check llama.cpp docs for more details.

Early Benchmarks

• Model: GLM 4.7 Flash (No Thinking)
• Task: Simple sorting algorithm implementation + code revision
• Standard Generation: ~50 tok/s
• With Speculative Decoding: ~120 tok/s
• Settings Used --spec-type ngram-mod --spec-ngram-size-n 24 --draft-min 48 --draft-max 64

Implementation Plan

Here is how I think we should integrate this optimization into Lemonade

• 0. Add Speculative Decoding to Docs
  If we don't have something similar already, we should create an "Optimizations" section in the docs covering various ways users can configure their setups to get better performance compared to the out of the box defaults. Need to be careful here not to be repetitive and keep it high level (point users back to llama.cpp docs for more details).

• 1. Enable a Zero-Config Default
  We should enable a lightweight, speculative method (like --spec-type ngram-mod or ngram-simple with conservative draft bounds) by default for standard chats. This ensures every Lemonade user gets "free" out-of-the-box speedups without needing to download draft models or tweak settings.

• 2. Frontend / UI Controls
  Expose these settings in the Lemonade WebUI model configuration.

  • Add a dropdown for the user to select the Speculative Mode (Auto (N-gram), Custom Draft Model, Disabled).
  • Allow attaching a specific GGUF as a draft model.

• 3. Specialized Recipes for Coding Agents
  Agentic workflows (like tool-calling, MCP, and code generation) produce highly repetitive text (JSON schemas, blocks, boilerplate code). Since we already maintain model configurations in lemonade-sdk/recipes (like the claude-code recipes), we simply need to edit these existing recipes. We can inject highly aggressive n-gram mapping flags (e.g., ngram-map-k4v or ngram-mod with a high --draft-max) directly into the agent/coding recipe defaults.

Qwen Family Support

Speculative decoding support for Qwen 3 Coder Next and the Qwen 3.5 family will be enabled as part of an upcoming llama.cpp PR ggml-org/llama.cpp#19493).

Would love to get everyone's thoughts on this and how we should properly integrate into Lemonade!

[m13v]

Great writeup. The n-gram speculative decoding results on UMA architectures are impressive. One thing worth mentioning for the coding agent recipes: tool calling responses tend to have extremely repetitive JSON structure (same schema keys, same formatting), which makes n-gram speculation hit rates very high. We see similar patterns in our AI agent where structured tool outputs dominate the token stream.

For the UI controls, I would suggest also exposing a "speculation overhead" metric alongside tokens/second so users can see the tradeoff. On some prompts (highly creative, low-repetition text), aggressive speculation can actually slow things down due to wasted draft tokens. Making that visible helps users tune it for their specific workload rather than guessing.

[m13v]

For context on the tool calling JSON repetition patterns I mentioned, here is how our agent structures tool calls that would benefit heavily from n-gram speculation: https://github.com/m13v/fazm/blob/main/Desktop/Sources/Providers/ChatToolExecutor.swift - the structured output format is very predictable and speculation-friendly.

[Bobettemcnugget]

Adding a concrete, benchmarked data point for the custom-draft-model path on Strix Halo, since this is exactly the UMA/RDNA 3.5 case described in the issue.

Setup: Ryzen AI Max+ 395 (gfx1151, 128 GB), Lemonade 11.5.1, llamacpp:rocm b1305 (llama.cpp c745be2, which includes the DSpark merge #25784). Target: DeepSeek-V4-Flash-0731 UD-IQ3_XXS (~97 GiB). Draft: am17an/DeepseekV4-Flash-20260731-DSpark (~11 GB).

Result (running llama-server directly, bypassing Lemonade):

Baseline decode: ~15 t/s
With DSpark draft (--spec-type draft-dspark --spec-draft-n-max 3): ~21.4 t/s (predicted_per_second: 21.38)
Draft acceptance: ~53% (draft_n: 462, draft_n_accepted: 245)
Both models resident fit in the 120 GiB GTT ceiling at 65536 ctx (~113/125 GB observed), no OOM.

So a ~40% decode speedup, on the exact hardware profile this issue targets: RDNA 3.5 compute is idle during standard decode and DSpark exploits it.

The blocker is purely the missing recipe field. --model-draft / -md / --spec-draft-model are on the reserved-args list, so they can't be passed via llamacpp_args, and 11.5.1 exposes no draft_model field to set one (grep of the install shows speculative only in vllm_model_config.json, nothing for the llamacpp path). --spec-type/--spec-draft-n-max are accepted in llamacpp_args, but without an attachable draft model they're inert — attempting to load errors with:

Failed to load model: Invalid custom llama-server arguments:
Argument '--model-draft' is managed by Lemonade and cannot be overridden.

A recipe-level field to attach a GGUF draft model (mirroring the structured speculative_config the vLLM path already has) would make this a config change. The measured payoff on Strix Halo is real and sizable; the only thing standing between users and it is the field. Happy to test a build that adds it.