perf(laguna): DSpark Markov head + spec-verify stack - 206 to 249 tok/s (RTX 3090)#482
Conversation
- default the plain-mul_mat MMVQ ncols ceiling to 3: measured crossover on RTX 3090 (Q4_K_M/Q6_K dense GEMVs) - MMVQ wins at ncols<=3, MMQ at 4-8. Laguna w6 chain 199->237 tok/s, Qwen3.6 chain 127->137 tok/s. Env-overridable via LUCE_MMVQ_MAX_NCOLS for other architectures. - extend the MUL_MAT_ID MoE fast path from 8 to 16 tokens (host+device caps, launch_bounds, dispatch); keeps CUDA-graph capture for 9-16 token spec-verify batches. ne2<=8 behavior byte-identical. - LUCE_MMQ_DP_MAX_NE1 tuning env (default off; documented negative result on sm_86).
Incorporates #460 and #461 rebased onto main: converter embeds the optional dflash.dspark.* tensors, the draft loader binds them (old GGUFs unaffected), and the Laguna greedy-chain decode applies the low-rank previous-token Markov correction. LoRA loader ported to the GgufMmap API introduced on main.
…ates One graph on the draft stream (base logits via a single lm_head matmul + unrolled Markov argmax->get_rows chain, async readback, one sync) replaces the per-candidate graph rebuild and host logits round-trip. The same builder serves DDTree candidate generation (markov-corrected top-K, same output contract as project_hidden_to_topk). Kill switches: DFLASH_LAGUNA_FUSED_DSPARK=0, DFLASH_LAGUNA_DSPARK_TREE=0. Measured (laguna-xs2 Q4_K_M + v24 drafter, RTX 3090, HumanEval): chain accept +2.2pts at w3; DDTree commit 3.83->4.43 at budget 12; verify-width sweep moves the chain optimum to w6.
Within a (width, mask-window) key the verify graph is structurally identical every step - kv position enters only through input data - so cache the built graph and skip the per-step host rebuild + allocator pass. Outputs are byte-identical (same graph, same data); DFLASH_LAGUNA_PERSIST_VERIFY=0 restores the rebuild-every-step path.
q4-mix quantizes the drafter backbone to q4_0 and keeps the dflash.* heads at q8_0 (the Markov/projection bias precision is what near-tie corrections depend on). Measured acceptance-neutral: f16 236.6 -> q8_0 241.3 -> q4-mix 249.6 tok/s (Laguna-XS.2 + v24 drafter, RTX 3090, HumanEval, verify width 6).
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
7 issues found across 17 files
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
- draft LoRA loader: overflow-safe gguf_tensor_in_file() bounds check with the gguf_bounds_error() diagnostic; validate tensor type and byte size before resizing the host buffer so corrupt metadata cannot force a large allocation - laguna verify: key the persistent step-graph slot on backend/weights/cache identity, and restrict reuse to the kv_idx input-data path (NO_KVPAD / PAD_CPY fallbacks bake kv_start into the graph, so they rebuild per step) - server: reject --draft-lora for non-laguna targets and for layer-split placement instead of silently ignoring it - laguna backend: drop the dead active_draft_lora_ member - quantize_dflash_draft.py: np.asarray() the metadata array parts before .item() (robust across gguf-py part representations; output verified byte-identical to the shipped q4-mix draft) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Pushed 8c02575 addressing all 7 cubic findings:
Validation on RTX 3090 (lucebox, CUDA 13.2, sm_86):
Environment note, unrelated to this PR: fresh builds of both this branch and its merge-base 737cd47 produce deterministic gibberish laguna output on lucebox itself (accept 16.7%, avg_commit 1.00, same hash across CUDA 12.0 and 13.2 toolchains and persist on/off), while a Jun 23 binary from the laguna-ddtree lineage serves the same GGUFs correctly at 176 tok/s. The problem pre-dates this PR (window: hub 0788827..737cd47 and/or ggml 574be613..ac06e543, possibly interacting with the reinstalled box stack: kernel 6.17, driver 595). Deserves a separate issue; this PR's numbers come from the cloud 3090 per GATE_RESULTS. |
… KV types Since #455, resolve_laguna_kv_types() routed laguna's KV dtype through resolve_kv_types() whenever any DFLASH27B_KV_* env var was present. dflash_server auto-sets DFLASH27B_KV_TQ3=1 for max_ctx > 6144 (a VRAM heuristic for the qwen 27B family), so every laguna server run at the default max_ctx=8192 silently got a TQ3_0 ternary KV cache instead of Q8_0 and produced degenerate output on every decode path (accept 16.7%, avg_commit 1.00). Laguna now honors only the explicit per-axis DFLASH27B_KV_K / _KV_V overrides (--cache-type-k/v), logs when they take effect, and ignores the legacy F16/Q4/TQ3 shorthands. Verified on RTX 3090 (lucebox): default-env laguna-xs2 + v24 q4mix w6 chain goes from deterministic garbage to coherent 272 tok/s at 71.1% accept / avg_commit 4.27; output hash identical with DFLASH_LAGUNA_PERSIST_VERIFY=0; explicit q4_0 override honored and logged; ctest 12/12. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Follow-up: the "broken laguna output on lucebox" from my previous comment is root-caused and fixed in ac531b0. It was not box-specific at all. Root cause (a Fix: laguna now honors only the explicit per-axis With this, the PR stack validates end to end on real hardware (lucebox RTX 3090, CUDA 13.2, default env):
Bisect trail for the record: main@737cd47f (PR base) already reproduced the garbage; commit bisect converged on #455; file-level grafts isolated |
There was a problem hiding this comment.
1 issue found across 1 file (changes from recent commits).
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="server/src/laguna/laguna_backend.cpp">
<violation number="1" location="server/src/laguna/laguna_backend.cpp:73">
P1: Missing (K, V) pair validation after env-var overrides. If a user sets `DFLASH27B_KV_K=tq3_0` and `DFLASH27B_KV_V=q4_1`, the unsupported pair passes silently and fails later with an opaque CUDA kernel error instead of a clear startup diagnostic. Add `dflash::is_supported_kv_pair(k_type, v_type)` check and abort with the same supported-pairs listing that `resolve_kv_types()` provides.</violation>
</file>
Tip: Review your code locally with the cubic CLI to iterate faster.
Re-trigger cubic
Extract the supported-pair check from resolve_kv_types() into dflash::validate_kv_pair_or_abort() and call it from the laguna per-axis override path, so an unsupported combination (e.g. DFLASH27B_KV_K=tq3_0 DFLASH27B_KV_V=q4_1) aborts at startup with the supported-pairs listing instead of failing later in a CUDA kernel. Verified on lucebox 3090: invalid pair aborts with the listing; valid q4_0/q4_0 override still honored, logged, and coherent; default q8_0 path untouched; ctest 12/12. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
ae89446 addresses the follow-up cubic finding on the KV override path: the supported-pair check is extracted from Verified on the 3090: invalid pair aborts with the listing, valid |
Stage the per-step graph inputs (embed, positions, kv_idx, masks, feat rows) in a slot-owned pinned host buffer and upload them with ggml_backend_tensor_set_async on the backend stream. A plain tensor_set from pageable memory costs a staged DMA plus a stream synchronize per call (~6 pairs per decode/verify step); the single sync inside ggml_backend_graph_compute() now covers ordering. Output hashes are byte-identical; worth ~1-2 tok/s at w6 and grows with context length (the mask upload scales with kv_pad). Add DFLASH_LAGUNA_STEP_PROF=1: per-step wall breakdown of the spec chain (draft forward / DSpark heads / verify / other), printed with the spec summary. Measured on RTX 3090 at w6: verify 11.8 ms (11.6 ms pure GPU, 98% busy), draft 2.2 ms, heads 0.75 ms, other 0.4 ms of a 15.2 ms step - the basis for the current kernel-optimization roadmap. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
b9e1146 adds two things from a kernel-optimization pass toward 300 tok/s on the 3090:
Measured state at w6 (282.6 tok/s, laguna-xs2 + v24 q4mix): step = 15.2 ms of which verify graph = 11.8 ms (11.6 ms pure GPU at 98% occupancy of its window, confirmed via Inside the verify GPU time, top items per step: MoE expert matmuls ~3.8 ms ( Negative result worth recording: routing K-quant mmid to the MMQ path at batch 7 (crossover 8 -> 2) was a clear regression, verify 11.8 -> 17.5 ms - MMQ tile machinery across 256 experts costs more than MMVQ's redundant reads at 512-dim experts. Reverted; the winning move for the MoE 3.8 ms is an expert-grouped MMVQ variant (read each selected expert once, apply to all its tokens), which is a real kernel project, est. 1.5-2 ms/step. Related: Luce-Org/lucebox-ggml#31 fixes the mmid CUDA-graph gate to mirror the dispatch (MMQ path is sync-free), which keeps tree-verify widths replayable; no pin bump needed for this PR. Realistic roadmap from 283: expert-grouped mmid MMVQ (+~1.7 ms) -> ~305-315; rope qk fusion + KV-quant batching (+~0.8 ms) -> ~320-330. Beyond that acceptance becomes the binding constraint. |
There was a problem hiding this comment.
2 issues found across 2 files (changes from recent commits).
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="server/src/laguna/laguna_backend.cpp">
<violation number="1" location="server/src/laguna/laguna_backend.cpp:937">
P2: The per-step profiler under-reports `verify` time when DDTree mode is active. `prof_verify_ms` is only accumulated around `target->verify_batch(...)` (line 937), but the DDTree branch (`args_.ddtree_mode`) calls `verify_tree(...)` and then `continue`s before reaching this accumulation. Meanwhile, `n_draft_steps` is incremented in both branches (line 896 for DDTree, line 1062 for non-DDTree), so the final `[step-prof]` report divides by a denominator that includes unmeasured DDTree steps, artificially deflating the per-step verify average and inflating `other`. Consider profiling the DDTree verify path (e.g. wrapping `verify_tree` with `prof_lap`) so all decode modes report consistent per-bucket breakdowns.</violation>
</file>
<file name="server/src/laguna/laguna_target_graph.cpp">
<violation number="1" location="server/src/laguna/laguna_target_graph.cpp:1212">
P2: The `laguna_host_stage` helper makes the decode/verify hot paths fail hard when a backend does not expose `ggml_backend_dev_host_buffer_type()` (or when the allocation fails). Previously these paths used `ggml_backend_tensor_set()` from caller memory and worked on any backend. Consider falling back to synchronous tensor uploads when `laguna_host_stage` returns `nullptr`, or gating the cached-graph optimization on backend host-buffer support.</violation>
</file>
Tip: Review your code locally with the cubic CLI to iterate faster.
Re-trigger cubic
| return false; | ||
| } | ||
|
|
||
| if (step_prof) prof_verify_ms += prof_lap(); |
There was a problem hiding this comment.
P2: The per-step profiler under-reports verify time when DDTree mode is active. prof_verify_ms is only accumulated around target->verify_batch(...) (line 937), but the DDTree branch (args_.ddtree_mode) calls verify_tree(...) and then continues before reaching this accumulation. Meanwhile, n_draft_steps is incremented in both branches (line 896 for DDTree, line 1062 for non-DDTree), so the final [step-prof] report divides by a denominator that includes unmeasured DDTree steps, artificially deflating the per-step verify average and inflating other. Consider profiling the DDTree verify path (e.g. wrapping verify_tree with prof_lap) so all decode modes report consistent per-bucket breakdowns.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At server/src/laguna/laguna_backend.cpp, line 937:
<comment>The per-step profiler under-reports `verify` time when DDTree mode is active. `prof_verify_ms` is only accumulated around `target->verify_batch(...)` (line 937), but the DDTree branch (`args_.ddtree_mode`) calls `verify_tree(...)` and then `continue`s before reaching this accumulation. Meanwhile, `n_draft_steps` is incremented in both branches (line 896 for DDTree, line 1062 for non-DDTree), so the final `[step-prof]` report divides by a denominator that includes unmeasured DDTree steps, artificially deflating the per-step verify average and inflating `other`. Consider profiling the DDTree verify path (e.g. wrapping `verify_tree` with `prof_lap`) so all decode modes report consistent per-bucket breakdowns.</comment>
<file context>
@@ -915,12 +927,14 @@ bool LagunaBackend::do_spec_decode(int committed, int n_gen,
return false;
}
+ if (step_prof) prof_verify_ms += prof_lap();
int accept_n = 1;
int bonus_tok = -1;
</file context>
| // kernels, dominated step wall time. Staging in pinned memory and uploading | ||
| // with tensor_set_async on the backend stream leaves the single sync inside | ||
| // ggml_backend_graph_compute() as the only per-step synchronization. | ||
| static uint8_t * laguna_host_stage(ggml_backend_t backend, |
There was a problem hiding this comment.
P2: The laguna_host_stage helper makes the decode/verify hot paths fail hard when a backend does not expose ggml_backend_dev_host_buffer_type() (or when the allocation fails). Previously these paths used ggml_backend_tensor_set() from caller memory and worked on any backend. Consider falling back to synchronous tensor uploads when laguna_host_stage returns nullptr, or gating the cached-graph optimization on backend host-buffer support.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At server/src/laguna/laguna_target_graph.cpp, line 1212:
<comment>The `laguna_host_stage` helper makes the decode/verify hot paths fail hard when a backend does not expose `ggml_backend_dev_host_buffer_type()` (or when the allocation fails). Previously these paths used `ggml_backend_tensor_set()` from caller memory and worked on any backend. Consider falling back to synchronous tensor uploads when `laguna_host_stage` returns `nullptr`, or gating the cached-graph optimization on backend host-buffer support.</comment>
<file context>
@@ -1203,6 +1203,31 @@ LagunaGraphOutputs build_laguna_graph(
+// kernels, dominated step wall time. Staging in pinned memory and uploading
+// with tensor_set_async on the backend stream leaves the single sync inside
+// ggml_backend_graph_compute() as the only per-step synchronization.
+static uint8_t * laguna_host_stage(ggml_backend_t backend,
+ ggml_backend_buffer_t & buf,
+ size_t need) {
</file context>
|
Completing the kernel-optimization ledger from the last comment, after exhausting the bit-exact space (all tested on the 3090 at w6, byte-identity gated on output hashes): Already banked by this PR stack (verified active): q/k rope fused at graph level (#479 fused-qk), q8_1 activation memoization, CUDA-graph replay on verify (93%) and draft chain, MMQ dense crossover at width>3. New negative results, for the record:
Where this leaves the step (15.2 ms, 283 tok/s): verify 11.8 ms of which MoE ~3.8 (roofline, frozen), dense MMQ stack ~3.7 (the only real kernel headroom, but any change alters rounding), small ops ~1.7 (dual K/V set_rows fusion is the last bit-exact item, worth only ~0.3 ms). Conclusion: under strict output-byte identity the ceiling is ~290 tok/s with this drafter. The two honest paths to 300+: (a) a small-batch dense matmul kernel for sm_86 (accepting rounding changes, gated on acceptance parity like this PR's other kernel swaps), or (b) acceptance: avg_commit 4.5 at the current step time clears 300 with zero numeric risk - a v27 drafter tuned for w6 acceptance is the cheapest 300. |
Remove the switchable draft LoRA adapter support that was folded in from #460: the --draft-lora flag, the per-request extra_body.draft_lora selector, the GGUF LoRA adapter loader/merger, and the DraftLoraSpec plumbing through the factory and backends. #460 stays open for a future pass; this PR ships only the DSpark Markov head and the spec-verify stack. The draft-variant container in LagunaBackend stays (single "base" variant) so a future variant mechanism can slot back in without re-plumbing the loader. Verified on the 3090: output hashes byte-identical to the previous commit, 282 tok/s at w6, DSpark head active, --draft-lora now rejected as an unknown option, ctest 12/12. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
55e53f0 removes the draft LoRA variant support that had been folded in from #460 (deferred by request): the Validation on the 3090: output hashes byte-identical to b9e1146 ( Note: this also obsoletes two of the earlier cubic fixes by deletion (the LoRA loader bounds/resize hardening and the |
|
55d7b69 lands the QKV projection fusion discussed in the perf thread: attn_v joins the #479 load-time weight fusion in the 20 layers where it shares the q/k quant type (Q6_K attn_v layers keep qk+v). One projection matmul per fused layer at decode widths; prefill keeps split weights, same kill switches as qk. Safety gate (paired same-session A/B vs Benchmarking caveat worth recording: this 3090's suite baselines from earlier sustained runs were depressed by VRAM heat (the known 0x20 SW-slowdown behavior), which made naive cross-session deltas look like +20-30%. All claims above are from paired runs on an equally-warm card; treat absolute suite numbers as session-relative. |
|
7005de6 bumps the ggml pin to lucebox-ggml Final benchmark battery on the complete stack (QKV fusion + dual set_rows + pinned staging), RTX 3090, w6, v24 q4mix:
Byte-identity chain: dual set_rows verified hash-identical to the QKV-fusion build on code/math/long prompts; QKV fusion itself was gated on paired A/B acceptance parity plus unchanged suite accuracy (its numeric deltas even left the code-prompt text byte-identical). |
1 participant
Summary
Laguna spec-decode performance stack: 206 → 249.6 tok/s on HumanEval (laguna-xs2 Q4_K_M, RTX 3090, greedy), with every step gated on output-hash identity or measured acceptance parity.
Commits
MUL_MAT_IDfast path.dflash.dspark.*tensors are optional. (The draft LoRA variants from Add switchable draft LoRA variants for Laguna #460 were initially bundled here and later removed in 55e53f0; Add switchable draft LoRA variants for Laguna #460 stays open for a future pass.)DFLASH_LAGUNA_FUSED_DSPARK=0,DFLASH_LAGUNA_DSPARK_TREE=0.DFLASH_LAGUNA_PERSIST_VERIFY=0kill switch.quantize_dflash_draft.pywithq8_0andq4-mixschemes; q4-mix keeps thedflash.*heads at q8_0, which is what preserves acceptance (64.0% vs 63.5% f16 at w6).Production recipe
No env vars required; the width-6 optimum follows from the cheaper verify slope (2.0 → 1.13 ms/width) plus the Markov head's deep-candidate corrections (depth-4 marginal acceptance 38% → 55%).
Correctness methodology
Notes