Prototype buffered linear-attention state updates for Qwen hybrid models

[randomvariable]

Objective

Prototype an IO-aware buffered/chunkwise speculative-window serving path for Qwen3.5/3.6 Gated DeltaNet (GDN) hybrid decoders. This issue remains valid, but persistent KVBuffer-style support is not code-ready as a small patch.

Verified Status

Qwen hybrid decoder routes linear-attention layers to QwenGatedDeltaNetAttention. Prefill already chunks with chunk_gated_delta_rule and commits final state. Ordinary decode performs one fused recurrent update per request/token. Speculative decode already has accepted-prefix bookkeeping plus state rewind/acceptance support.

Existing MambaSpec, mamba_utils, MambaStateCopyFuncCalculator, GPU hybrid state adapter, and GDN metadata provide reusable state-copy foundations. They do not provide KVBuffer/deferred-commit/direct-from-buffer behavior: no decode-buffer metadata or persistent buffer ownership contract exists.

Upstream PRs vllm-project#48792 and vllm-project#49887 remain open ReplaySSM work. They are upstream-watch only, not dependencies.

Narrowest Safe Hypothesis

Implement only a CUDA-only, explicit capability/config-gated Qwen GDN speculative-window prototype:

• Bound a buffer beside convolution and temporal state; both states are included.
• Buffer ownership lasts only for current verification window; no persistent state beyond it.
• Commit only accepted prefix at existing num_accepted_tokens boundary.
• Keep ordinary one-token decode, prefix-cache reads/writes, preemption, KV connectors, and CUDA graphs on existing paths until dedicated lifecycle and graph-ownership work exists.
• Do not claim generic linear-attention support.

Options

Option	Verdict
CUDA-only GDN verification-window prototype	Narrowest safe implementation target.
Persistent KVBuffer/deferred commit/direct-from-buffer serving	Requires lifecycle, ownership, metadata, cache, preemption, connector, and graph work; not a small patch.
Generic linear-attention framework	Out of scope; behavior and state contracts are model-specific.
ReplaySSM upstream work (vllm-project#48792, vllm-project#49887)	Watch only; not a dependency.

Key Hazards

• Convolution and temporal state must advance together.
• Accepted-token count can be 0..N+1.
• Prefix-cache hashability and copy-on-write correctness.
• Abort/preemption slot lifetime.
• Mixed state_indices ordering.
• Fixed-shape CUDA graph capture.
• BF16/FP16 rounding and TP-sharded state.
• CUDA-only capability gating must fail closed.

Acceptance And Promotion Requirements

Before promotion beyond prototype, validate:

• Kernel outputs plus convolution/temporal-state equivalence across dtypes, window lengths, and flush boundaries.
• Mixed prefill/decode/speculative metadata and every accepted-token count.
• Prefix-cache partial hits and copy-on-write.
• Abort and preemption.
• Eager and CUDA-graph behavior, or explicit safe graph rejection.
• TP=1 and TP>1.
• Qwen3.5-0.8B correctness, then Qwen3.6 target correctness.
• Baseline versus prototype latency, accepted-token throughput, memory, concurrency, and prefix-cache cold/warm measurements.

Tracked Research Candidates

Bole: Efficient Tree Speculation for Hybrid-Attention Language Models (arXiv:2608.01651)

Logged 2026-08-04 daily scan. Bole performs tree speculative decoding through recurrent linear-attention layers by transforming the linear recurrence into a tree-structured closed form, verifying all proposal nodes in parallel and losslessly encoding speculative state updates as token-level factors (reconstructing only the selected state after sampling). Reports 3.4-7.7x faster linear-attention tree verification, 82-99x lower transient state memory, up to 4.72x decode throughput.

Mechanism verification (2026-08-04): tree verification through linear layers is GENUINELY ABSENT here. BailingLinearAttentionMetadataBuilder._build_for_cudagraph_capture asserts max_query_len <= 1 + num_spec_tokens (vllm/v1/attention/backends/linear_attn.py:172), and the per-request state index tensor is single-chain (max_bs, 1 + num_spec_tokens) (linear_attn.py:144). EAGLE3 tree speculation exists for full-attention layers (gpu_model_runner.py:643) but is not extended into the linear path; no proposal_tree/tree_mask symbols exist under vllm/v1. Bole's factorised state encoding is the most direct external reference for this issue's buffered-state-window objective.

Status: watch / candidate for prototype. Code not yet released; revisit when available. Kernel must be retargeted to SM12 warp-level mma (100 KB/SM, TN-only) and separately designed for gfx1151 (WMMA 16-element floor, 64 kB/WG LDS). Scope: tree verification through recurrent LINEAR layers only — not GQA/SWA/MLA target verification, which already supports tree masks.