Evaluate a load-aware speculative-depth governor

[randomvariable]

Objective

Evaluate a model-informed speculative-depth governor that avoids throughput collapse under concurrency.

Current behaviour

Scheduler selects K from exactly one runtime signal — the count of scheduled requests:

• vllm/v1/core/sched/scheduler.py:1192 — num_spec_tokens_for_schedule keyed on len(num_scheduled_tokens)
• vllm/v1/spec_decode/dynamic/utils.py:12,77 — config validated and expanded into a dense batch_size -> K lookup

It does not consult acceptance rate, per-position acceptance, target/draft latency, draft cost, queue wait, KV pressure, request properties, or MoE routing. The selected K is batch-wide, never per-request.

K=0 is intentional and must stay so: both vllm/v1/spec_decode/step3p5.py:321,341 and vllm/v1/spec_decode/llm_base_proposer.py:580,607 run the cache-sync forward before returning empty drafts. It means "no draft tokens", not "no proposer work".

Blocker: telemetry does not exist yet

Speculative metrics are aggregate interval counters, not live policy inputs:

• vllm/v1/spec_decode/metrics.py:26,82 — drafts, draft tokens, accepted tokens, per-position counts over a logging interval
• vllm/v1/core/sched/scheduler.py:1766,1785 — per-request acceptance is computed transiently, then folded into SpecDecodingStats; no retained request-keyed history

Absent entirely: target-forward duration, draft-forward duration, GPU-time attribution. No closed-loop controller is possible until these exist. This is the first work item.

MoE routing telemetry exists but is output-oriented, not scheduler-facing — vllm/model_executor/layers/fused_moe/routed_experts_capturer.py:58,120 captures per-token/per-layer expert IDs only under enable_return_routed_experts, with no per-expert load histogram or routing-cost estimate.

Existing half-built machinery

vllm/v1/worker/gpu/spec_decode/dspark/speculator.py:83,93,218,221 contains a per-request confidence-scheduled verification-length concept that is not usable policy machinery: synthetic SPS pending calibration, and the computed ell is never wired to verification length. Reconcile this before building anything new — it is the same shape as the per-request budgeting candidates below.

Scope: one control loop, not six

Six scanned research candidates were mapped here. Verified against the code, four are the same control loop with different input signals:

Candidate	Verdict
Interpretable latency model (arXiv:2605.15051)	This issue — telemetry precondition not met
EcoSpec (arXiv:2607.12696)	Collapses in — MoE routing cost is one cost predictor
D-Cut (arXiv:2607.14647)	Collapses in — per-request budget acts after drafting
Sparse Glimpse (arXiv:2607.27735)	Collapses in — marginal acceptance scoring is still depth control
AngelSpec (arXiv:2607.25852)	Out of scope unless multi-proposer switching is in scope
Lossy-verification guardrail (arXiv:2607.26627)	Out of scope — correctness, not performance

All four reduce to choose speculative depth by expected benefit versus cost, differing only in which signal feeds the estimator. Tracking them separately would create four issues for one mechanism.

AngelSpec selects the drafter family, not depth, requiring a proposer-family selection hook at vllm/v1/core/sched/output.py:267 / vllm/v1/worker/gpu_model_runner.py:5030. No common controller can currently select between proposer implementations. Separate issue if pursued.

Lossy verification is a correctness property — preserving the target distribution — and does not belong in a performance issue, where it risks being deprioritised behind throughput work. Current code has rejection sampling and block verification (vllm/v1/worker/gpu/spec_decode/rejection_sampler.py:79,149) but no runtime distribution-drift monitor. Separate issue.

Drafter classes

Evaluate each separately; do not transfer fitted latency or acceptance parameters across classes. Native MTP for Qwen3.6, GLM-5.2, Nemotron 3 Ultra, Hy3, and Step-3.7-Flash; external MTP for Gemma 4; EAGLE for Mistral Medium 3.5; DFlash for Laguna S 2.1. Step-3.7-Flash is its own drafter class and attention-layout row.

Do not name gpt-oss, Kimi K3, MiniMax M3, Granite 4.1, Inkling, or MiMo-V2.5 without a configured speculative mechanism.

References

• Interpretable latency model for speculative decoding, arXiv:2605.15051: https://arxiv.org/abs/2605.15051
• Upstream regression: [Performance]: Dynamic speculative decoding (num_speculative_tokens_per_batch_size) causes catastrophic aggregate-throughput collapse under concurrency at the batch-size threshold (MTP, V1/PIECEWISE) vllm-project/vllm#49548
• Related fork issue: Evaluate regime-aware KV-cache scheduling for batched inference #6

Proposed work

1. Add the missing telemetry: per-request acceptance history, target/draft forward durations, GPU-time attribution.
2. Reproduce the batch-threshold throughput collapse with current V1 speculative decoding.
3. Fit or implement a policy using observable batch size, draft latency, verification latency, acceptance, and cache pressure.
4. Compare against fixed draft depth and current threshold behavior.
5. Keep the first implementation scheduler/policy-only.

Acceptance criteria

• Reproducer captures throughput and p95/p99 latency across concurrency thresholds.
• Before evaluation, record concurrency points, run duration, and fixed maximum throughput/p99 regressions allowed versus no speculation.
• Policy converges without oscillation as batch size changes.
• Single-request acceptance/correctness remains unchanged.

Target hardware

• CUDA sm_121a (GB10 / DGX Spark) — throughput collapse under concurrency at sm_121 memory bandwidth
• ROCm gfx1151 (Strix Halo) — same concurrency collapse pattern on unified memory with shared bandwidth