Characterization (reference): GPU-target + NPU-draft speculative decoding on Lunar Lake — works, but consistently slower than GPU-only (0.55–0.74× throughput). Methodology + numbers

[blairducrayoppat]

TL;DR. On an Intel Core Ultra 7 258V (Lunar Lake) I measured speculative decoding with the target on the GPU and the draft model on the NPU. With an NPU-legal (symmetric INT4) draft it runs end-to-end and produces correct output — but it is a consistent net slowdown (0.55×–0.74× of GPU-only throughput, i.e. 1.35×–1.8× slower), whereas the same draft on the CPU gives a 1.35× speedup. The cause is simple and reproducible: the NPU runs the small draft model ~3.6× slower per token than the CPU does, and speculative decoding only pays off when the draft is cheap. This is reference data that confirms OpenVINO's documented guidance (use a CPU draft for a GPU/CPU target, or put both models on the NPU) — not a bug report. The related thread (openvino#34450) is resolved/closed; this just quantifies the symmetric-draft path it points to.

Raw data: the per-config measurements (JSONL + CSV + dataset card) are published as a dataset → https://huggingface.co/datasets/blairducrayoppat/npu-specdecode-lunarlake

Setup

• Hardware: Intel Core Ultra 7 258V (Lunar Lake), Arc 140V iGPU (16 GB), NPU (AI Boost). Windows 11 build 26200 (25H2). NPU driver 32.0.100.4778, GPU driver 32.0.101.8826.
• Software: OpenVINO 2026.1.0, OpenVINO GenAI 2026.1.0.0. Python LLMPipeline.
• Target (GPU): Qwen3-14B, INT4 (asymmetric per-group — fine on GPU; only the NPU rejects asymmetric, which is why only the draft needs --sym).
• Draft: Qwen3-0.6B, INT4 symmetric — optimum-cli export openvino --model Qwen/Qwen3-0.6B --task text-generation-with-past --weight-format int4 --sym --group-size 128 --ratio 1.0. The --sym flag matters: per-group asymmetric INT4 is unsupported on the NPU (it produces u4 zero-points the NPU compiler rejects; see #34450). This symmetric export is my working recipe that satisfies the NPU constraint (verified: 0 u4 zero-point constants, 196 i4 weight constants in the IR).
• Target and draft share the same Qwen3 tokenizer (vocab/merges byte-identical; identical token IDs on a 6-string check) — a requirement for speculative decoding.
• Method: greedy, ignore_eos=True, fixed 128-token budget, 1 warm-up + 3 measured reps, each config in a fresh process. ignore_eos forces a fixed decode length so per-config throughput is compared over identical work rather than over runs that stop at different points. Metrics from PerfMetrics / SDPerModelsPerfMetrics. Construct/compile time is measured once and excluded from throughput (the slowdown is steady-state per-token cost, not compile). Single prompt.
• On num_assistant_tokens (NAT): on the stateful path this is an initial/seed value the pipeline adapts each round (grows on a full accept, shrinks otherwise, capped relative to the seed) — so NAT below is the seed, not a fixed per-round proposal count. Both the CPU and NPU drafts use this same stateful path, so the comparison is apples-to-apples.

Constructing it (the one gotcha)

A heterogeneous NPU draft only constructs via the stateful speculative pipeline. Passing a scheduler_config selects the continuous-batching pipeline, whose paged-attention scheduler the NPU doesn't support — on 2026.1 that raises RuntimeError: ... Option not found: scheduler_config (on current master the property is silently dropped on the NPU branch instead). Omit scheduler_config and it works:

import openvino_genai as ov_genai
draft = ov_genai.draft_model(r"<qwen3-0.6b-int4-sym>", "NPU")          # draft on NPU
pipe  = ov_genai.LLMPipeline(r"<qwen3-14b-int4>", "GPU", draft_model=draft)  # no scheduler_config
cfg = ov_genai.GenerationConfig()
cfg.max_new_tokens = 128
cfg.ignore_eos = True            # fixed budget, to match the tables below
cfg.num_assistant_tokens = 3     # seed; the stateful pipeline adapts it per round
print(pipe.generate("What is 2 + 2? Give the answer and one sentence of explanation.", cfg))

(The assistant_confidence_threshold form of dynamic drafting is continuous-batching-only, so it's unavailable on this stateful NPU-draft path.)

Result 1 — the draft model alone, per device

Same Qwen3-0.6B-symmetric model, run standalone (no speculative decoding), 128 tokens, 3 reps:

draft device	throughput	latency/token	construct
CPU	106.8 ± 0.7 tok/s	9.4 ms	1.3 s
GPU	150.0 ± 0.9 tok/s	6.7 ms	3.5 s
NPU	29.2 ± 0.1 tok/s	34.3 ms	13.7 s

The NPU runs this small model ~3.6× slower per token than the CPU (and ~5× slower than the GPU). That's expected: the NPU is built for sustained large-matmul throughput, and pays a high fixed per-inference dispatch cost that dominates on a tiny model invoked repeatedly. NPU first-compile also adds ~12 s.

Result 2 — speculative decoding (GPU target + draft)

128 tokens, greedy, 3 reps. NAT = num_assistant_tokens seed (see Method). Speedup vs the 12.2 tok/s GPU-only baseline. Acceptance rate = accepted ÷ draft-proposed (this matches GenAI's own definition; single-prompt, illustrative).

draft device	NAT seed	throughput	speedup	acceptance	compile
none (GPU only)	—	12.2 ± 0.1 tok/s	1.00×	—	7.8 s
CPU	3	16.5 ± 0.5 tok/s	1.35×	0.40	9.3 s
CPU	5	14.7 ± 0.9 tok/s	1.20×	0.32	9.4 s
NPU	1	9.0 ± 0.0 tok/s	0.74×	0.60	22.2 s
NPU	3	7.1 ± 0.0 tok/s	0.58×	0.35	21.8 s
NPU	5	6.7 ± 0.1 tok/s	0.55×	0.32	22.0 s

The CPU draft accelerates; the NPU draft decelerates at every setting. The NPU draft is least-bad at NAT seed=1 — the opposite of normal spec-decode tuning — because fewer draft tokens means less of the expensive NPU per-token cost.

Why

Speculative decoding nets a win only when the time to draft a few tokens is much smaller than the target verification it amortizes.

• CPU draft: ~9.4 ms/token; drafting ~3 tokens ≈ 28 ms — cheap next to the ~53 ms batched target forward per round (and the 82 ms/token a no-draft run pays). The accepted tokens are nearly free → 1.35×.
• NPU draft: the same model is ~34 ms/token standalone, and ~48–61 ms/token inside the heterogeneous loop (the measured draft_tpot from SDPerModelsPerfMetrics; the delta over the 34 ms standalone is cross-device GPU↔NPU synchronization). Drafting ~3 tokens ≈ 150 ms — already more than the ~63 ms target forward it's trying to amortize. Every round adds more latency than it removes.

Crucially this is not an acceptance/quality problem: NPU-draft acceptance is comparable to the CPU draft (it's the same draft model; row-to-row differences are single-prompt noise) — ~0.60 at NAT seed=1. The draft proposes good tokens; it just costs too much to produce them on the NPU. Output is correct: under greedy, the NPU-draft and CPU-draft outputs are token-identical to each other (704/704 chars at 256 tokens). Speculative decoding is theoretically lossless w.r.t. the target's greedy decode; in practice the batched verification pass uses a different floating-point reduction order than the sequential no-draft path, so a single near-tie argmax can flip — which is why both spec-decode paths diverge from a no-draft run at the same token. It is not an NPU defect.

How this lines up with the documented design

This corroborates OpenVINO's existing guidance rather than contradicting it:

• The speculative_decoding_lm samples (C++ and Python) default both models to CPU and state: "CPU, GPU and NPU can be used. For NPU, the preferred configuration is when both the main and draft models use NPU." — i.e. the NPU is positioned as a whole-LLM device, not as a draft behind a non-NPU target.
• OpenVINO's NPU speculative-decoding (FastDraft) story is likewise the both-models-on-NPU route.
• On #34450 an Intel maintainer pointed to the CPU as the draft device, and the disposition was that NPU LLM support is limited to the documented NPU recipe (per-group asymmetric INT4 is unsupported).

So if you want to free the CPU by drafting elsewhere: today, on this hardware, a GPU target wants a CPU (or GPU) draft, not an NPU draft.

Caveats

Single machine, single prompt, n=3 (tight variance, but not a broad sweep); greedy; fixed 128-token budget; Windows; OpenVINO 2026.1 (behavior may differ on 2026.2/master). The acceptance numbers are prompt-dependent and illustrative — the robust finding is the throughput / per-token-latency result, which is device physics, not prompt-specific. These numbers use default NPU prompt/response budgets; on this stack a non-default NPUW MAX_PROMPT_LEN triggered a 0xC0000005 access violation, so reproduce with defaults first.

Reproducer

The core is the ~15-line snippet above, plus the same call with "CPU" / no draft. Full harness (bench_community.py, draft_standalone.py, bench_aggregate.py) and raw per-rep JSON (128-token budget) available — happy to share; the aggregated per-config numbers (JSONL/CSV) are in the dataset.

Open questions (forward-looking)

1. Is GPU-target + NPU-draft intended as a supported configuration, or primarily a consequence of "at least one model on NPU" routing to the stateful path?
2. Is continuous-batching speculative decoding (and dynamic assistant_confidence_threshold) on the NPU roadmap?
3. Would a short note in the speculative-decoding docs — "the draft belongs on a low-latency device (CPU/GPU); an NPU draft behind a non-NPU target is a net slowdown on current hardware" — be worth adding?

AI assistance: I used an AI coding assistant to help build the benchmark harness, cross-check every number against the raw result JSON, and draft this write-up; all measurements were produced by running the harness on my hardware and validated against the on-disk result files. Per the OpenVINO AI Usage Policy.