docs: DSpark speculative-decoding feasibility & adaptive-placement spec

[pekkah]

Summary

• Surveys DeepSeek's DSpark/DeepSpec speculative-decoding framework (DFlash parallel backbone + rank-256 Markov head + confidence-scheduled verification) and checks it against what SharpInference already has (MtpDecoder, SpeculativeDecoder, TierPlanner, HardwareProfile).
• Specs a DSparkPlacementPlanner that auto-decides whether the draft head runs on GPU, CPU, or is disabled, based on measured VRAM/RAM headroom — with user-overridable CLI flags/env vars (--dspark-place, --dspark-verify-len, --dspark-min-confidence, SHARPI_DSPARK_*) following the existing pinGpuLayers/SpecType precedent.
• Lays out a phased rollout (safetensors loader → CPU-only draft pass → decoder → placement planner → CUDA/heterogeneous benchmarking → confidence scheduler → server integration) and calls out open risks (undocumented Markov-head math, no DSpark head for MoE architectures, unvalidated PCIe-cost heuristic for CPU-draft/GPU-target placement).
• Docs-only change; no code in this repo is touched.

Went through an internal review pass (fact-checking claims against the actual source, plus a design-quality pass) before opening this PR; fixes already folded in:

• Corrected the --draft-model + Vulkan-target claim (rejected outright today, not routed to a CPU draft).
• Filled in the missing dspark_qwen3_14b_block7 param count.
• Pointed the safetensors-loader work at the existing SafetensorsLoader in SharpInference.Diffusion instead of proposing a redundant one, and called out the NativeAOT/trim-safe JSON parsing constraint from CLAUDE.md.
• Fixed the RAM-headroom formula (LayerPlacement has no per-CPU-layer byte field today) and had the planner reuse TierPlanner's reserved-VRAM-floor logic instead of re-deriving it.
• Spelled out that the runtime free-VRAM recheck is a concrete Phase 3 step, and noted the coupling between placement margins and the variable verify length from §7.

Test plan

• N/A — documentation only, no build/test surface affected.

---

Generated by Claude Code

[gemini-code-assist]

Code Review

This pull request introduces a feasibility and adaptive-placement specification for DSpark Speculative Decoding, detailing placement options, auto-placement algorithms, user overrides, and a phased rollout plan. Feedback on the specification highlights a logical conflict in subtracting the expert cache budget from free VRAM, the risk of using total system RAM instead of actual available RAM for headroom calculations, and a technical inaccuracy regarding PCIe round-trip overhead for independent draft models.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

There is a logical conflict in subtracting targetPlacement.ExpertCacheBudgetBytes from hardware.VramBytes when computing vramFree for DSpark:

1. Since ExpertCacheBudgetBytes is defined as the remaining VRAM after trunk weights, KV, and reserved space are subtracted, subtracting it here would reduce vramFree to nearly zero (or just the scratch buffer size), preventing DSpark from ever being placed on the GPU for MoE models.
2. If DSpark were to be supported on MoE models in the future, the DSpark draft head should carve its allocation out of the remaining VRAM before the SLRU expert cache budget is determined, rather than treating the entire expert cache budget as an untouchable prior allocation.
3. Since DSpark is currently inapplicable to MoE architectures (as noted in §9), this term is always 0 for all currently supported targets anyway, making its inclusion in the formula unnecessary and potentially confusing.

[gemini-code-assist]

Using hardware.RamBytes (which represents the total physical system memory or container limit) as the baseline for ramFree is highly optimistic and risky.

In real-world environments, a significant portion of system RAM is already consumed by the OS, other processes, and the current process's own non-trunk allocations. Assuming all non-trunk RAM is free can easily lead to over-allocation, triggering heavy disk paging/swapping or Out-Of-Memory (OOM) crashes when loading a multi-gigabyte draft model (1.4B–3.4B parameters).

To make this safer and more robust, consider:

1. Querying the actual available/free physical memory instead of total memory. In .NET, GC.GetGCMemoryInfo() provides MemoryLoadBytes (used physical memory), so TotalAvailableMemoryBytes - MemoryLoadBytes can be used to estimate the actual free RAM.
2. Applying a much larger safety buffer or a conservative scaling factor if total RAM must be used as the baseline.

[gemini-code-assist]

The assertion that "every draft step round-trips the current hidden state/token across PCIe and back" is inaccurate for independent draft models like DSpark:

1. Unlike draft frameworks that depend on the target model's hidden states (such as EAGLE), DSpark is an independent draft model (using a DFlash backbone + Markov head) with its own weights.
2. Therefore, the CPU draft model only needs the last accepted token ID(s) to seed its generation, and it produces $k$ candidate token IDs. The only data transferred across PCIe are these small token ID arrays (integers), which has negligible latency (microseconds).
3. The actual bottleneck for a CPU-draft / GPU-target setup is the CPU execution latency of the 1.4B–3.4B parameter draft model itself, not the PCIe transfer overhead.

Correcting this in the spec will help align the performance heuristics with the actual hardware bottlenecks.