DSV4-Flash active-LoRA zero-K3: full qualification closed (K3 = 0 at 64-decision promotion) by kiddyboots216 · Pull Request #45 · togethercomputer/xorl

kiddyboots216 · 2026-08-11T16:58:44Z

Summary

Trainer half of the DSV4-Flash active-LoRA exact lane on the admitted WORLD8 program: official geometry, rank-1/alpha-1 active adapter, native FP8 dense and MXFP4 routed serving values, single-request eager decode, temperature 1/top-p 1, and MTP disabled.

The implementation includes fail-closed admission, native payload ownership, the 948-factor dsv4_expert_banks adapter contract, exact attention/indexer/router/head paths, backward and update ownership, independent raw-FP32-logprob replay tooling, and strict checkpoint/runner contract validation.

Unified canonical combine

DSV4 uses #44 canonical_moe_fold_v1, the same balanced adjacent-pair BF16 fold as Qwen/GLM. dsv4_native_combine.py retains DSV4 variable-row all-to-all transport, reshapes rank-ordered arrivals, and feeds them to the shared fold. Serving #17 refuses the off-path reduce_scatterv optimization and reaches the corresponding canonical post-experts combine with an explicit engagement log.

Campaign 2 also closes two latent value-dependent gaps:

Trainer pad-to-48 is retired so trainer and serving launch identical M; the shared serving primitive chunks the pinned geometry to 10 tokens so an expert cannot span completion-order-dependent row blocks.
Exact-head forward values use the serving batch-invariant BF16 log_softmax; the surrogate VJP remains unchanged.

Qualification

The byte-changing Campaign 2 migration completed the full ladder: base ruler at 4 and 64 decisions, A joins with a correctly-diverging perturbed control, training/update gate, trained-adapter B join, and a 64-decision promotion replay with every decision-time raw-logprob byte equal and k3_max = 0.0. Campaign-1-comparable decode throughput was 5.5 tok/s.

The official restack preserves that qualified runtime byte program. A new replay is not required merely because commit SHAs changed, and this head remains qualified without a SHA-only requalification run.

Campaign-only launch recipes, cluster paths, raw traces, replay artifacts, and the lane log remain outside the public PR diff. The production qualification summary is in docs/k3/DSV4_FLASH_LORA_ZERO_K3_PLAN.md.

Stack

Base: Unify canonical MoE folding across Qwen and GLM #44 at 5bb7a3f37e6f08d9c3299b69264a2ad24f7846b8
Serving dependency: DSV4-Flash exact active-LoRA serving surface (zero-K3 lane, serving half of xorl#45) xorl-sglang#17 at 4f6dcd2d49205b578bc75471129f4514f5eb7499
Head: 3a8627c1ba8a05bf4fa19a463814a9ae5831912c

Validation

Final focused DSV4 model/runner/checkpoint/combine suite: 147 passed.
Changed Python files: Ruff passed.
Range-diff preserves all 30 prior payload commits, with expected canonical-fold conflict context and gitlink changes.
Additive review fixes harden exact adapter/runner contracts and remove six stale loader locals.

Known boundary

Scheduler-local sampler LoRA liveness means DP-idle ranks remain base-only under the admitted routed_dp_rank=0 contract. The trainer mirrors this exactly. A gather-aware upstream fix would restore the full factor surface and requires A/B requalification.

kiddyboots216 · 2026-08-11T17:01:47Z

Serving-side pin for this lane: togethercomputer/xorl-sglang#17 (the submodule gitlink on this branch references its head).

broly-code-security-scanner · 2026-08-11T17:02:51Z

Broly Security Scan

Note

Summary

0 actionable finding(s) in this PR
1 total in scan · 0 dismissed false positives

1 finding(s) below the medium reporting threshold are not listed above — see the repository Security tab for the full set.

No finding is at or above high, so this check is not blocking. The findings above are still tracked and reported.

Note

Re-scan this PR anytime with /broly scan — useful after /broly undismiss, or to refresh findings without a new push.

Broly — SAST (zai-org/GLM-5.2) · Secrets · SCA · IaC · GH Actions · Base Images · Supply Chain Threats · Exploit Chains · Adversarial Verification

We're continuously improving Broly's accuracy and finding quality — your feedback is valuable. False positives, missed findings, bugs, and feature requests all welcome.

kiddyboots216 · 2026-08-11T21:12:27Z

Successor for the MoE-combine reconciliation: #48 unifies DSV4 onto the canonical fold (with xorl-sglang#18) and carries the full requalification — K3 exactly 0.0 at the 64-decision trained-adapter promotion replay. The architecture-scoped NCCL-order combine documented here is retired on that branch.

The Qwen3.5/GLM exact lanes reduce MoE partials with the canonical adjacent-pair BF16 fold (#44/#16); DSV4 reproduces the pinned NCCL-tree contributor order [1..7, 0]. These are different byte programs, so the DSV4 combine (and the variable-row gather primitives it needs) now lives in dsv4_native_combine.py with the order baked in and the unification hazard documented; ep_native_combine.py and its tests return to main's content, removing the #44 -> #45 textual conflict entirely. Pure code motion: the 4-decision base ruler replay was re-verified byte-equal (K3 = 0 x4) after the relocation.

+
+    repo = Path(__file__).resolve().parents[1]
+    model_path = Path(args.model_path).resolve()
+    tokenizer = AutoTokenizer.from_pretrained(str(model_path), trust_remote_code=True, local_files_only=True)


The Qwen3.5/GLM exact lanes reduce MoE partials with the canonical adjacent-pair BF16 fold (#44/#16); DSV4 reproduces the pinned NCCL-tree contributor order [1..7, 0]. These are different byte programs, so the DSV4 combine (and the variable-row gather primitives it needs) now lives in dsv4_native_combine.py with the order baked in and the unification hazard documented; ep_native_combine.py and its tests return to main's content, removing the #44 -> #45 textual conflict entirely. Pure code motion: the 4-decision base ruler replay was re-verified byte-equal (K3 = 0 x4) after the relocation.

…ification harness Trainer-side exact contract: fail-closed official-geometry admission and WORLD8 RCA topology resolver, 948-factor FP32 inventory bind, native block-FP8 and MXFP4 payload retention with BF16 placeholder strip, serving-value attention/indexer/lm-head forwards with trainer-owned VJPs, ordered EP native combine, dsv4_expert_banks export, and the decode-cache diagnostic scorer. Qualification harness: frozen-trace capture/replay scripts, complete-adapter generator (zero/distinguishable/perturbed), Marlin LoRA and hash-topk gates. Environment contract: DSV4 exact kernels run in the pinned SGLang torch-2.11 environment; sglang-kernel is dropped from the torch-2.12 profile (its compiled extension is torch-2.11 ABI and only fails at load time) with a fail-fast import+real-op smoke test, a deferred quack CuTe import so xorl imports under SGLang's cutlass-dsl pin, and a raise for DSV4 exact server training without the active-LoRA lane (a base-only trainer would silently pair the exact trunk with a non-exact LM head).

…ess) Frozen sampler base denominators (4- and 64-decision, byte-stable across repetitions), Marlin MXFP4 base/LoRA gate PASS, hash-topk layers 0-2 PASS, base-ruler replay first-divergence reports (decision 0, trunk-scale deltas in both whole-sequence and decode-cache scoring), trainer component dumps for layers 0-2, and the lane log with environment recipes and the node-100 window-claim protocol.

…ine, node contention protocol

…opulation - Exact router GEMM: the trainer computed the gate as a cuBLAS BF16 GEMM widened to FP32; the sampler's deterministic contract patches torch.mm to the batch-invariant persistent Triton GEMM. Call that kernel directly (never a global mm patch under grad). Verified byte-equal router logits against the sampler dump. - Decode-cache scorer: dummy DP ranks (no valid labels) now declare num_samples=0 so they contribute zero rows to the EP-gathered expert batch, matching serving idle ranks; fused_marlin_moe is not row-count invariant (results/dsv4_flash_zero_k3/marlin_rowcount_ab.py). - Sampler post-experts combine pinned to the ordered reverse-rank chain (submodule commit); base denominators re-frozen byte-stable x3. - Current frontier recorded: layer-1 attention diverges structurally with a clean q-chain; operand isolation in progress.

named_buffers() deduplicates shared tensor objects, so the loader's buffer snapshot restored each lru_cache-shared freqs_cis table under only its first FQN (layers 0 and 2); every other DSV4 layer ran RoPE on zeroed storage — the rope slice of q/k was exactly 0 on layers 1 and 3-42, the base-ruler first divergence after the router repair. Rebuild one table per distinct (base, yarn, head_dim, device) tuple post-materialization and re-share it across layers; cloning per layer instead would add ~11 GB/GPU (1M-position complex64 tables x 43 layers) and OOMed the WORLD8 load.

…code by prefix - The serving contract pins NCCL_ALGO=allreduce:tree; its bitwise EP8 result is a left-associative BF16 chain in contributor order [1..7, 0]. The trainer's combine now takes an explicit chain_order (Qwen3.5's captured [7..0] default untouched) and DSV4 passes the tree-captured order. The earlier sampler-side ordered-all-reduce patch never engaged on this path (layer-level reduce) and is reverted to avoid drift in other configs. - The decode-cache scorer replays each DSV4 decision as its full prefix and keeps only the decision position: the exact attention recomputes decode cache bytes per position, and Marlin batches pad to the 48-row qualified geometry, so no incremental KV carry is needed. Base-ruler decode comparison after these plus the RoPE rebuild: decision 0 within one BF16 ulp, k3_max 0.057 (was 244310); layers 0-1 byte-equal end to end; frontier now the first C4 compressed-attention layer.

The HF checkpoint ships compressor.ape (and indexer.compressor.ape) in the natural layout: the serving model applies its own hotfix permutation at load and the exact C4 path re-applies it before the compress kernel. Un-doing a hotfix that was never applied fed the kernel a permuted table (order-3 permutation, so the double application landed on t^-1(A) instead of t(A)). With this fix the trainer's exact_compressed_attention program was byte-verified stage-by-stage against a live serving probe. Loader test updated to the natural-layout contract; A-join capture scripts staged.

…kernel The deterministic contract patches standalone RMSNorms to the batch-invariant Triton kernel; sgl_kernel/native rmsnorm differs by one BF16 ulp at rounding boundaries (layer-4 q_norm). Also stash rebuild args on compressor/indexer freqs tables (126 shared tables re-registered after load) and adopt the isolation agent's compressor kv_score fix: the compress-score GEMM now calls matmul_persistent (the sampler's interposed mm) instead of the cuBLAS fp32-out branch. Layers 0-3 byte-equal end to end after these.

The serving stack constructs its model under the CUDA default device, so torch.polar rounds with device intrinsics; the trainer built the same tables on CPU (glibc libm), differing by one fp32 ulp on ~15% of table components and tripping BF16 rounding boundaries sporadically (layer-4 q rope, one element on the 10-token ruler). With the table, ape, kv_score, and q_norm fixes the 4-decision base ruler is byte-equal at decisions 0 and 1; decisions 2-3 sit at ulp scale pending M=1 decode segments.

…he state Each decode decision now runs the literal serving decode kernels (flash_mla_with_kvcache over paged FP8 caches; SWA stream with descending window indices and 128-slot pages; compressor decode state machine with kv-score ring and block-boundary pooling; complete-block compressed stream) over per-layer carried state seeded by the byte-proven prefill path. The full-prefix replay was not row-count stable (bucketed kernels), which held decisions 2-3 at ulp scale. BASE RULER CLOSED: the 4-decision replay is byte-equal at every retained decision (decode_comparison.byte_equal=true, K3 = [0.0, 0.0, 0.0, 0.0]) against the frozen TP8 sampler trace, trainer denominator byte-stable. Offline: 204/204 stage checks byte-equal (layers 0-4 x decisions 0-3 x all boundary stages); 110 + 12 tests pass in both profiles.

…o under isolation Sampler side: zero-adapter capture byte-equals base (E2E no-op), base capture byte-equals the frozen ruler across a server restart, nonzero is distinguishable, the single-factor perturbed control is detected in the bytes. Trainer side: the zero-adapter session replays the zero capture byte-equal (K3=0). The nonzero session diverges at layer-0 MoE via nonzero-rank shared/routed LoRA partials (rank 0 byte-equal; rank 1 at one-ulp scale) — per-rank factor slicing under isolation.

Serving gates LoRA per scheduler batch, so DP-attention idle ranks compute base-only expert partials even for EP-gathered real tokens; under the pinned routed_dp_rank=0 contract only rank 0's shared slice and expert shard carry the adapter. The trainer now threads lora_live=(ep_rank==0) through the shared-expert TP partial and routed Marlin functions (zero factor substitution forward; exact-zero factor grads backward). ADAPTER A JOIN CLOSED: A1 all-zero factors byte-equal both engines; A2 deterministic nonzero factors byte-equal on all four decisions (K3 = [0.0, 0.0, 0.0, 0.0]); the perturbed negative control (one routed lora_B +2^-9) is detected at decision 0 in both directions. LANE DECISION to surface: under this serving behavior 7/8 of the routed bank factors and shared slices [256:2048] never influence serving and now receive exact-zero gradients — likely an upstream sampler gap (LoRA batch_info is scheduler-local, not gather-aware) worth an upstream fix before scaling the lane beyond single-request DP8.

…ptimizer step - Payload access inside surrogate VJPs now unshards/reshards explicitly (module __call__ fires FSDP pre-forward hooks, fatal during autograd). - Compressed-attention surrogate allows genuinely-unused inputs (C128 with no complete block leaves the compressor input out of the graph; its exact gradient is zero). - The exact-lane q_norm is a proper autograd Function: batch-invariant serving kernel forward, native-recompute surrogate VJP backward. - Gradient checkpointing disabled for the qualification lane (the exact mHC Function's saved tensors do not survive checkpoint recompute; activations at ruler scale are trivial) — recorded as a known interplay issue. Gate results: forward_backward loss 2.4731 over the 4 ruler decisions with the ownership validator green; AdamW step applied; post-step replay diverges from the pre-step trace (factors moved) while the trainer stays byte-repeatable.

- dsv4_expert_banks exports are BF16 serving views regardless of preserve_lora_dtype (the sampler's fail-closed loader rejects FP32); the serving artifact is the {adapter_config.json, adapter_model.safetensors} pair, not the full checkpoint dir (optimizer shards are trainer state). - B1: post-optimizer-step factors exported, loaded by the sampler, fresh 4-decision capture byte-stable x3; trainer replay byte-equal, K3 = 0 x4. - B2 PROMOTION: fresh 64-decision prefix, denominator byte-stable x3; trainer replay byte-equal at every retained decision, K3 = 0.0 at all 64, trainer denominator byte-stable.

Base ruler, A join with negative control, training gate, B join, and the 64-decision promotion replay all hold with K3 exactly zero; measured endpoint throughput recorded; fail-closed behavior exercised. Includes the throughput harness and B-join capture scripts.

The Qwen3.5/GLM exact lanes reduce MoE partials with the canonical adjacent-pair BF16 fold (#44/#16); DSV4 reproduces the pinned NCCL-tree contributor order [1..7, 0]. These are different byte programs, so the DSV4 combine (and the variable-row gather primitives it needs) now lives in dsv4_native_combine.py with the order baked in and the unification hazard documented; ep_native_combine.py and its tests return to main's content, removing the #44 -> #45 textual conflict entirely. Pure code motion: the 4-decision base ruler replay was re-verified byte-equal (K3 = 0 x4) after the relocation.

Replaces the retired NCCL-tree contributor-order chain on both sides: serving routes the gated post-experts combine through tensor_model_parallel_canonical_moe_all_reduce and the trainer folds its variable-row exchanged partials with canonical_moe_fold_v1. Bytes change; the unified program requires full requalification.

--repeats N catches rare completion-order flips (the 74-token layer-40 routing flips ~31% per call); the nonzero LoRA discriminator fills every expert's factors so it stays meaningful when a captured real routing never selects rank-local expert 0. Gitlink: canonical-fold serving surface + exact Marlin chunking.

Under the chunked exact-Marlin program the pad to 48 rows made the trainer's decode segment run its live row in an M=10 chunk (1 live + 9 masked) while serving launches M=1; the two geometries diverge by one ulp value-dependently (first at decision 39 of the campaign-2 64-decision base ruler). The trainer now launches exactly the serving runner's M per gathered segment; every M <= 10 launch is single-block by construction under the shared chunker.

Serving's deterministic mode interposes log_softmax with the batch-invariant Triton kernel; its BF16 rounding differs from ATen's on boundary values (proven offline at decision 39 of the campaign-2 64-decision base ruler: one entry of the row flips by one bf16 ulp, and the f64 truth sits 5.5e-8 past the rounding boundary — ATen rounds correctly, the wire contract follows serving). The exact head's forward value now uses the serving kernel; the surrogate VJP keeps FP32 reference math. Campaign 1 never sampled a boundary value. Also: model-level tail diagnostic captures (hc_head_output/final_norm under pseudo-layer -1) used for this localization, an @file form for replay loss-fn params, and the replay wrapper default-expansion fix (${4:-{...}} appended a stray brace to every explicit params arg).

kiddyboots216 mentioned this pull request Aug 11, 2026

DSV4-Flash exact active-LoRA serving surface (zero-K3 lane, serving half of xorl#45) togethercomputer/xorl-sglang#17

Open

github-advanced-security AI found potential problems Aug 11, 2026

View reviewed changes

kiddyboots216 changed the base branch from main to feature/canonical-moe-fold-unification-20260810 August 11, 2026 18:02

This was referenced Aug 11, 2026

Unify canonical MoE folding across Qwen and GLM #44

Open

docs: refresh public documentation for current code #42

Open

github-advanced-security AI found potential problems Aug 11, 2026

View reviewed changes

Comment thread scripts/qualify_dsv4_marlin_lora.py Fixed

kiddyboots216 mentioned this pull request Aug 11, 2026

Unify DSV4 onto the canonical MoE fold and requalify to K3 = 0 #48

Closed

github-advanced-security AI found potential problems Aug 11, 2026

View reviewed changes

Comment thread scripts/capture_dsv4_exact_trace.py Fixed

github-advanced-security AI found potential problems Aug 11, 2026

View reviewed changes

Comment thread scripts/reproduce_dsv4_hash_topk.py Fixed

Comment thread scripts/replay_dsv4_exact_trace.py Fixed

Comment thread scripts/replay_dsv4_exact_trace.py Fixed

kiddyboots216 force-pushed the feature/canonical-moe-fold-unification-20260810 branch from e08c0a4 to 5bb7a3f Compare August 12, 2026 02:29

kiddyboots216 force-pushed the feature/dsv4-flash-lora-zero-k3-20260810 branch from d09fb81 to 3a8627c Compare August 12, 2026 02:29

github-advanced-security AI found potential problems Aug 12, 2026

View reviewed changes

Comment thread scripts/capture_dsv4_exact_trace.py Outdated

repo = Path(__file__).resolve().parents[1]

model_path = Path(args.model_path).resolve()

tokenizer = AutoTokenizer.from_pretrained(str(model_path), trust_remote_code=True, local_files_only=True)

kiddyboots216 force-pushed the feature/canonical-moe-fold-unification-20260810 branch from 5bb7a3f to 0f41afe Compare August 12, 2026 03:15

kiddyboots216 force-pushed the feature/dsv4-flash-lora-zero-k3-20260810 branch from 3a8627c to bfddfc1 Compare August 12, 2026 03:17

github-advanced-security AI found potential problems Aug 12, 2026

View reviewed changes

Comment thread scripts/capture_dsv4_exact_trace.py Fixed

Comment thread scripts/capture_dsv4_exact_trace.py Fixed

kiddyboots216 force-pushed the feature/canonical-moe-fold-unification-20260810 branch from 0f41afe to 65ae56a Compare August 12, 2026 03:19

kiddyboots216 force-pushed the feature/dsv4-flash-lora-zero-k3-20260810 branch from bfddfc1 to b9fa32a Compare August 12, 2026 03:19

github-advanced-security AI found potential problems Aug 12, 2026

View reviewed changes

Comment thread scripts/capture_dsv4_exact_trace.py Fixed

kiddyboots216 force-pushed the feature/canonical-moe-fold-unification-20260810 branch from 65ae56a to ecfe560 Compare August 12, 2026 04:02

kiddyboots216 added 5 commits August 12, 2026 04:02

Record base-ruler RCA progress: stage-1 byte-clean, JIT-cache discipl…

7fd8496

…ine, node contention protocol

kiddyboots216 added 25 commits August 12, 2026 04:02

Record the base-ruler divergence burn-down table

818f75a

Update burn-down: table provenance fixed, decisions 0-1 byte-equal

e03a30a

Mark the lane log closed

b086868

Ignore the exported serving-adapter artifact directory

7812a9f

Slim DSV4 campaign artifacts and harden trace loads

45b3325

Normalize DSV4 exact trainer checks

0f041ab

Record the campaign-2 canonical-fold qualification (K3 = 0.0 x 64)

6e269f7

Drop campaign-only hash-topk replay helper

5983512

Harden exact DSV4 adapter and runner contracts

b0ea0c6

Remove stale DSV4 loader locals

35a5739

kiddyboots216 force-pushed the feature/dsv4-flash-lora-zero-k3-20260810 branch from b9fa32a to 35a5739 Compare August 12, 2026 04:02

github-advanced-security AI found potential problems Aug 12, 2026

View reviewed changes

Comment thread scripts/qualify_dsv4_marlin_lora.py Fixed

Harden DSV4 qualification inputs

f55f5ab

kiddyboots216 requested a review from qywu August 12, 2026 04:11

Navigation Menu

Uh oh!

DSV4-Flash active-LoRA zero-K3: full qualification closed (K3 = 0 at 64-decision promotion) - #45

DSV4-Flash active-LoRA zero-K3: full qualification closed (K3 = 0 at 64-decision promotion)#45
kiddyboots216 wants to merge 31 commits into
feature/canonical-moe-fold-unification-20260810from
feature/dsv4-flash-lora-zero-k3-20260810

kiddyboots216 commented Aug 11, 2026 •

edited

Loading

Uh oh!

kiddyboots216 commented Aug 11, 2026

Uh oh!

broly-code-security-scanner Bot commented Aug 11, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kiddyboots216 commented Aug 11, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

kiddyboots216 commented Aug 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Unified canonical combine

Qualification

Stack

Validation

Known boundary

Uh oh!

kiddyboots216 commented Aug 11, 2026

Uh oh!

broly-code-security-scanner Bot commented Aug 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Broly Security Scan

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kiddyboots216 commented Aug 11, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

kiddyboots216 commented Aug 11, 2026 •

edited

Loading

broly-code-security-scanner Bot commented Aug 11, 2026 •

edited

Loading