[Spec] Add DSpark: confidence-scheduled speculative decoding

[sglang-bot]

DSpark speculative decoding — semi-autoregressive block drafting + confidence-scheduled, variable-length verify.

Follow-up work (separate PRs)

Kept out of this PR to limit its footprint; planned as follow-ups after it lands:

• The num_tokens_per_bs family (runners / graph runners / spec helpers) is misnamed — the value is per request, not per batch. Renaming it to num_tokens_per_req is mechanical (~240 internal occurrences).
• The fa3 backend rewrites speculative_num_draft_tokens in place for draft workers; it should keep the server-args value verbatim and read a derived num_tokens_per_req field instead.
• The per-algorithm hooks in speculative_hook.py duplicate the shared num_steps/topk forcing and spec batching defaults across algorithms.
• The DSpark Triton kernels live outside the sglang.kernels ops layout; migrating them waits on untangling their ScheduleBatch coupling.
• The dsv4 backend carries algorithm-keyed instance state: the draft-block attention support and the needs_cpu_seq_lens opt-out are keyed on flags fixed at construction. These are the only instance-keyed modes in the attention layer, where every other speculative path dispatches on per-batch data (spec input types, ragged layouts) or capability methods.
• The decode cuda-graph runner drives ragged verify as a parallel token-bucket-keyed mode beside the bs-keyed graphs, with its own capture/admission paths. Unifying the two keyings — uniform verify windows as the special case of token keying — would remove the duplicated capture machinery, but re-keys every algorithm's replay path and needs its own performance validation.
• The final shape of the shared draft-worker construction layer (draft_worker_common, shared with DFLASH) is undecided: a stateless builder module as today, or a shared block-draft worker base class owning the common identity/lifecycle code; the module name should follow whichever outcome.
• Hand-built DSpark draft batches leave can_run_dp_cuda_graph unset, and the dp-attention sync gate ANDs it into graph admission, so the dense-draft block never replays cuda graphs under dp attention. Whether that conservatism is required (the draft forward is attention-TP-local) or a missed graph path needs GPU validation.
• The confidence-relay staging (device-to-host confidence transfer for the scheduler) lives in the shared overlap-utils module while its only producer and consumer are DSpark components, and the planner imports its constants back from there; it also allocates its CUDA buffers lazily on first use. Relocating it into the DSpark component package (with construction-time allocation) would shrink the shared-file footprint.
• The dsv4 attention-metadata builders (causal expansion, page-table/positions) were rewritten as triton kernels serving every dsv4 path, speculative or not. Triton-vs-torch parity is CI-covered; a dedicated performance validation of the non-speculative dsv4 paths on the new builders is still to be done.
• The SGLANG_DSPARK_* environment variables sit inside the generic test-and-debug block (plus one under spec config) instead of a dedicated section, and three boolean perf toggles (FAST_KERNEL, FAST_SAMPLING, FP32_LM_HEAD) carry no verb category token, unlike the sibling OPT_* entries. Renaming becomes an alias-deprecation exercise once released, so the regrouping and renames should happen soon after landing.

---

CI States

Latest PR Test (Base): ✅ Run #29208178181
Latest PR Test (Extra): ❌ Run #29208178114

[Phoenix3334]

Cross-reference: compact CUDA-Graph stress investigation (#31023 / #31455)

Cross-referencing a downstream DSpark stability investigation because it exercises paths beyond the blog's Figure 5 happy path.

Mode distinction

The official Figure 5 command uses:

SGLANG_RAGGED_VERIFY_MODE=compact
no SPS table
bs=1, input/output=256/256
cuda-graph-max-bs=4

So this is the compact verify-all uniform-ragged path, with default overlap compared against --disable-overlap-schedule.

The investigation in #31023 has two relevant reproducer families on an earlier development snapshot:

A. compact/no-SPS, bs=64, repeated and fresh-service CUDA-Graph replay
B. compact+SPS, c256, 60k/1k, 2560 requests, variable verify lengths and sustained multi-tier replay

The residual failure is an asynchronous CUDA illegal-memory access in target verification after the earlier cross-TP planning divergence was fixed by #31195. A strong Graph-OFF control survived, while completed default-off diagnostics did not find an actionable error in the covered host-visible metadata, explicit covered index bounds, translated out_loc, SITE4 resource structure, or result-copy handoff. The exact in-Graph faulting kernel/resource remains unknown.

Version caveat

The positive evidence was collected in a v0.5.14-series base environment while importing source at A1 22cfd54c, later with local default-off diagnostics. This is not yet a claim that clean v0.5.16 is affected.

A clean v0.5.16 regression matrix is now being prepared:

1. the official-scale compact/no-SPS happy path;
2. compact/no-SPS bs=64 stress replay;
3. compact+SPS c256 60k/1k;
4. Graph-OFF / overlap-OFF / static controls.

If clean v0.5.16 does not reproduce repeatedly, the next action will be a bisect from 22cfd54c to v0.5.16 rather than carrying the historical diagnosis forward.

Upstream question

Is there already an internal or public stress/regression suite covering both of the following on TP8/B300?

compact verify-all/no-SPS + sustained CUDA-Graph replay
compact+SPS variable verify + high-concurrency long-context replay

Any known post-22cfd54c change expected to harden these paths would also help narrow the release regression.

[Phoenix3334]

Follow-up from #31023: clean v0.5.16 shape-specific decode Graph capture failure at bs=60

A clean downstream regression on DeepSeek-V4-Pro-DSpark / TP8 has isolated a deterministic startup-time failure outside the blog's Figure 5 shape range.

What passes

The exact Figure 5 startup scale reaches readiness:

SGLANG_RAGGED_VERIFY_MODE=compact
no SPS table
CUDA Graph ON
overlap ON
--cuda-graph-max-bs 4

So this is not a report that the official batch-1 Figure 5 happy path is broken.

What fails

Using the same compact/no-SPS mechanism family but expanding the decode Graph capture set:

cuda-graph-max-bs	Result
4 / 8 / 16 / 32	READY
64	bs=64 captures, then illegal memory at bs=60
128	larger shapes through bs=64 capture, then illegal memory at bs=60

Tested environment:

image: iregistry.baidu-int.com/nccl/sglang:v0.5.16-cu130
sglang: 0.5.16
CUDA: 13.0
PyTorch: 2.11.0+cu130
model: DeepSeek-V4-Pro-DSpark
TP: 8
mem-fraction-static: 0.82
chunked-prefill-size: 4096

Each arm used a fresh server; GPU memory returned to 0 MiB afterward. The exception is surfaced from decode CUDA-Graph capture, but the exact faulting kernel is not yet known.

The detailed evidence and interpretation are tracked in #31023. The current classification is an expanded-shape startup regression, separate from the historical runtime compact+SPS Bug 2 unless a shared cause is later demonstrated.

Two upstream questions:

1. Is bs=60 included in an existing DeepSeek-V4-Pro-DSpark TP8 CUDA-Graph capture regression on the pinned dev-dspark/692c5f7d environment or current main?
2. Are there known backend-specific shape constraints around this capture tier that should be encoded in the capture-shape generator or test suite?

A reproduction on the official pinned image/source would help distinguish a core SGLang/DSpark issue from the specific v0.5.16-cu130 runtime stack.

[Phoenix3334]

Final downstream regression update: large-shape capture failure is non-deterministic, not one bad shape

A clean v0.5.16 downstream regression on DeepSeek-V4-Pro-DSpark / TP8 has now completed the key controls.

Passing paths

Official Figure 5-scale path:
  compact / no SPS / max-bs=4 / bs1 / 256-256 -> PASS

Low-tier path:
  decode capture bs<=32 -> PASS
  compact+SPS sustained replay with max-running-requests=32:
    32/32 + 320/320 requests
    100% cuda graph: True
    zero illegal-memory / scheduler exceptions

Failing larger-shape capture path

Across otherwise equivalent clean launches, the first illegal-memory exposure moved:

bs=60  / TP7 / Triton surfacing point
bs=96  / TP4 / TRT-LLM fused-MoE routing surfacing point
bs=112 / TP6 / DeepGEMM surfacing point

An explicit --cuda-graph-bs-decode list that skipped bs=60 still failed at bs=96; bs=96 had captured successfully in another run. Approximately 42 GB remained free per GPU at failure.

This rejects the hypothesis of one uniquely corrupt shape. It instead indicates a non-deterministic illegal-memory fault in the larger DSpark decode CUDA-Graph capture path. The reported kernels are asynchronous surfacing locations, not confirmed faulting kernels.

The historical compact+SPS c256 runtime Bug 2 remains inconclusive on v0.5.16 because the required larger Graph range fails during startup, while limiting capture to 32 forces the c256 workload predominantly onto eager decode.

Current classification:

large-shape startup capture failure: confirmed
low-tier capture/replay: stable in tested matrix
historical runtime Bug 2 fixed: not established
shared root cause: plausible, not proven

A useful upstream regression would cover repeated TP8 capture of the larger decode shape set in addition to the published small-batch Figure 5 path.