row/KERNEL-MARLIN-DENSE-DIRECT: dense-marlin port arbitrated NO-GO (4-run ubench) - #50
Closed
localai-bot wants to merge 1 commit into
Closed
row/KERNEL-MARLIN-DENSE-DIRECT: dense-marlin port arbitrated NO-GO (4-run ubench)#50localai-bot wants to merge 1 commit into
localai-bot wants to merge 1 commit into
Conversation
… measured NO-GO (#50) Arbitrated the named MXFP4 residual-#1 lever (route E=1 dense projections through vLLM's DENSE marlin_gemm direct-A / m_block_size_8 instead of our MoeGroupedGemmNvfp4Marlin E=1) with the same-tool per-shape ubench the lever protocol mandates BEFORE a large port — did NOT vendor the ~2000-line dense kernel. Re-ran dgx:~/mxfp4-nsys/mxfp4_marlin_ubench.py (reference apply_fp4_marlin_linear = the exact kernel a port would vendor, vs ops.moe_wna16_marlin_gemm E=1 = the 1:1 twin of our production kernel) 3x fresh on the idle GB10; with #46 = 4 same-tool runs. Per-STEP moe/dense mean: M=1 0.997, M=2 1.007, M=4 1.010 (all noise, sign of delta flips run-to-run) and M=8 1.039 (the only reproducible signal: m_block_size_8 8-row tile vs our moe_block_size=16, ~+0.33ms/step = ~0.8% of a c8 step). The +7-9% cross-tool figure (#47/#48) is the nsys-vs-torch-profiler tool bias, NOT a recoverable kernel/dispatch cost. VERDICT: NO-GO. The port cannot reach >=1.0x on any axis (recovers ~0.8pp of the 5.8% c8 gap at M=8 only, nothing at c1-c4); vendoring 2000 lines of delicate dense marlin + a NON-byte-exact reroute of the CUDA-graph-captured decode for that would gold-plate a measured near-dead hypothesis (#46 caution). Recoverable slivers named (all cheaper than the port): force moe_block_size=8 at M=8 in DenseAlignFor (engages our already-vendored m_block_size_8 kernels, ~5 lines, no new vendoring); drop the per-call defensive workspace memset; the ~0.7ms/step host/sched slice. Box left clean (both locks free, worker down, disk 21G). FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
Collaborator
Author
|
Landed on main as the squash (mudler authorship), with the NOW/state fold included. 🤖 Generated with Claude Code |
mudler
added a commit
that referenced
this pull request
Aug 6, 2026
…xposed (#50) row/KERNEL-MARLIN-DENSE-DIRECT squash, records-only. 4-run same-tool per-shape ubench (dense marlin_gemm vs our E=1 grouped twin): M=1/2/4 within +-1% with run-to-run sign flips (true zero; the +7-9% per-call was nsys-vs-torch cross-tool bias), M=8 reproducibly +0.33ms/step from m_block_size_8 vs our block-16 padding. NO-GO on the ~2000-line port; the sliver is a ~5-line DenseAlignFor block=8 tweak on already-vendored kernels. Remaining MXFP4 terms: that tweak (~0.8pp at c8), the per-call ws Memset drop (micro), and the ~0.7ms/step host/sched slice. NOW + state folded same-change. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
mudler
added a commit
that referenced
this pull request
Aug 6, 2026
The #50 landing appended to the row without compacting the superseded binding text, pushing STATUS 91 chars over its ratchet (my landing chain gated doc-checkpoint but not public-doc-tables - both now required). FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
mudler
added a commit
that referenced
this pull request
Aug 6, 2026
…ve EVERY tput axis vs #49, c1 crosses to parity+; goal still <1.0x on c2-c8 Clean-checkout binding grid at the committed sha d3b412f (not an overlay) via mxfp4-online-serving-grid.sh, oracle vLLM 0.25.0 (FlashInfer mxfp4 disabled), single-load/arm, drop_caches+mincore. #44 model gate re-passed inside the grid; 24/24 legs failed:0; reps tight (ours c8 CoV ~0.4%). Grid EXIT=1 is the gate-FAIL signal (gate_pass:false) + single-model cross-summary wait, not a crash. Binding (ours/vLLM normalized ratio) vs #49 tput [0.990/0.922/0.930/0.942]: total_token_throughput c1 1.005 c2 0.925 c4 0.939 c8 0.953 median_tpot_ms c1 1.002 c2 0.922 c4 0.915 c8 0.939 mean_ttft_ms c1 1.034 c2 0.962 c4 1.004 c8 0.999 median_itl_ms c1 1.004 c2 0.919 c4 0.920 c8 0.929 peak host-mem ours 35.2 GiB vs vLLM 76.7 GiB = 2.18x LESS VERDICT: c1 PASSES every axis; c2-c8 BELOW on tput/tpot/itl (best c8 0.953), TTFT parity c4/c8. gate_pass FALSE => MXFP4 parity goal NOT DONE (below-floor on c2-c8). The byte-exact slivers moved EVERY tput axis up vs #49 (c1 +1.5pp crosses to parity+, c4 +0.9pp, c8 +1.1pp, c2 +0.3pp), no regression. Residual map (c8 ~4.75% gap): (1) block=16 padding CLOSED (sliver a); (2) grouped-Marlin decode +7-9% per-call (E=1 indirect sorted_token_ids gather + fp32 C_tmp vs vLLM dense marlin_gemm direct-A, #46/#50); (3) host slice. Host-slice attribution (step 2, VT_LOOP_TRACE under decode): every window interval_ms ≈ step_ms (delta ≤0.02ms; admits=0 pure-decode windows interval-step ≈ 0), so the born-on-runner engine-core loop has negligible host overhead — the #47 ~0.7ms/step residual is NOT an engine-loop lever (it is the shared async frontend or the cross-tool attribution boundary). Records: STATUS, BENCHMARKS, benchmark-record, state, NOW. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
mudler
added a commit
that referenced
this pull request
Aug 6, 2026
…l 58% (#52) row/QUANT-CT-MXFP4-C8-DIFF squash, records-only. First same-tool nsys both engines at c8 (vLLM offline in-process, identical prompts, M=8). Fair decomposition of the ~3ms TPOT gap: MARLIN +1,377us (58%) = STRUCTURAL - 5 grouped-E1 GEMMs/layer (gate+up unfused, 144 CTAs) vs vLLM's 4 dense fused (48 CTAs), +25% GEMM count/step; settles #50 (the isolated-shape ubench was blind to count+CTA structure). Flash +658us (28%, +11%/call at IDENTICAL grid, unresolved params diff). Glue +290us (12%, portable-fusion class). The #51 binding ran EAGER: classic-dense decode graph is opt-in OFF; graph-ON = byte-coherent, gap 1184->305us (=vLLM), TPOT -1.3%, SACRED gate owed before flip. STATUS ratchet lowered 287838->287832. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
mudler
added a commit
that referenced
this pull request
Aug 8, 2026
…ory + landed-vs-claimed audit + TP-W plan; DSpark rider (task #287) The dedicated TP spike at parity pin 555967922 (SHA-verified before reading), building on the #50-era TP spec, the scale-out W1/W2 lanes and the #164 mode enumeration instead of re-deriving them. Records-only: no source, no CMake, no test changes. S1 — a 14-row at-pin inventory of everything TP touches (GroupCoordinator + TP group init, device communicators incl. custom-AR csrc + pynccl's 18-symbol C surface, the four parallel-linear classes with bias/loader/shard_id semantics, vocab embed + LM head + logits gather, sharded weight loading per class, head split + the KV-cache-per-rank implication for our paged runner, MoE TP-vs-EP, the multiproc executor and the V2 runner, sampler at TP), each with our seam cited and a REUSE/PARTIAL/NEW verdict; ~40% of the end-to-end surface is already landed or directly reusable. The landed-vs-claimed audit is explicit: vt::Communicator + OpId routing + the per-device registry + the NCCL TU + the four all-reduce seams are REAL, but the tp handle dead-ends at the layer boundary (qwen3.cpp defaults), no production loader passes tp into LoadMergedBf16RawNK, and the only tp>1 driver is the toy MLP test. S2 — recommendations: thread-per-rank orchestration (recorded deviation; multiproc returns at multi-node), an additive TP>1 branch through ModelRegistry::Forward with TP=1 byte-identical, tensor_parallel_size on vllm_model_params at the next ABI bump (base v14), and the per-weight-class sharded-load map (GGUF x TP explicitly deferred). S3 — the TP2-on-CPU token-exact gate over the W1 in-process transport (synthetic-dense CI tier + real-0.6B env-gated tier), real multi-GPU enumerated PENDING-HW. Honesty finding recorded: upstream's own TP=2 test (test_basic_correctness.py:204) compares greedy tokens vs HF, NOT vs TP1; and at the pin use_all_gather() defaults True, so every TP rank all-gathers full logits and samples identically. S4 — ranked bricks TP-W0..W7 with effort/dependency/gate; TP-W1..W4 and TP-W7 are CPU-completable now. RIDER (USER 2026-08-08): DSpark speculator re-grounded at the pin (specs/dspark-speculator-note.md): DSparkSpeculator(DFlashSpeculator) one-parallel-pass block drafting, method "dspark" V2-runner-only, draft models for BOTH our registered target families (Qwen3 + Gemma4), slotting beside our landed MTP + DFlash lanes. SPEC-DSPARK + feature-matrix rows refreshed at the pin; full scope stays a future spike. The brief asked for new rows; both rows already existed, so they were refreshed in place (disclosed deviation). Records: roadmap ROAD-V1-D2 + the TP W-plan sub-table, PAR-TP / BACKEND-DISTRIBUTED-TP / SPEC-DSPARK / feature-matrix rows, coordination closed-claim note (rows keep READY/INVENTORIED by design, so no active-claims row), parity-ledger + state appends, NOW refresh, STATUS/BENCHMARKS checkpoint edits (superseded scale-out narrative compacted to stay inside the STATUS ratchet). FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-fable-5 [ClaudeCode]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Records-only arbiter: the +7-9% was cross-tool bias; M<=4 true-zero (sign flips), M=8 +0.33ms/step = the block-16 padding sliver, capturable by a ~5-line DenseAlignFor tweak on already-vendored kernels. NO-GO on the ~2000-line port. Remaining: block=8 tweak + ws-memset micro + the ~0.7ms host slice.
🤖 Generated with Claude Code
https://claude.ai/code/session_014fZAcg1WcU8V629k6HWKys