feat(aprender-serve): heavy CPU-vs-GPU per-stage diff harness — M-MOE-SUB-3

[noahgift]

Summary

M-MOE-SUB-3 heavy diagnostic harness that runs CPU-traced + GPU-traced forward bodies (M-MOE-SUB-2 step (a) PR #1516 + step (b) PR #1523) with a SaveTensorPlan capturing MoeRouter and MoeFfnOut for every layer, then computes per-layer per-stage cosine similarity to identify the first layer where the GPU diverges from the CPU. Per contracts/trace-moe-gpu-sub-stages-v1.yaml v1.2.0 step M-MOE-SUB-3.

Why

falsify_qw3_moe_gpu_parity_001_cosine_vs_cpu observes 100% NaN at lm_head from forward_qwen3_moe_cuda while CPU forward_qwen3_moe produces finite output on the same input. The first 9 stages of the GPU forward path are CPU-identical; the only GPU-specific stage is the MoE FFN. This harness pinpoints WHICH layer of the MoE FFN first diverges → that layer is the M-GPU-MOE-1.4 bug-origin candidate.

What ships

• New heavy test falsify_moe_sub_002_cpu_gpu_traced_per_stage_diff in crates/aprender-serve/tests/qwen3_moe_gpu_per_stage_diff.rs.
• 6 light unit tests for the verdict classifier (Match, Diverge, NanGpu, NanCpu, NanBoth, Missing) — all 6 pass.
• Skip-if-not-present pattern for the canonical 17.3 GB Qwen3-Coder GGUF (matches qwen3_moe_gpu_parity convention).
• #[ignore] + #[cfg(feature = "cuda")] gating per heavy-test convention.

Invocation

cargo test -p aprender-serve --features cuda \
  --test qwen3_moe_gpu_per_stage_diff \
  -- --include-ignored --nocapture

Sample output (when the heavy test runs)

layer | moe_router (cos / verdict)        | moe_ffn_out (cos / verdict)
------|-----------------------------------|-----------------------------------
L00   | 1.000000 MATCH                    | 1.000000 MATCH
L01   | 0.998765 MATCH                    | 0.997123 MATCH
...
L17   | 0.998123 MATCH                    |             NanGpu      ← first divergence!

M-MOE-SUB-3 bisection summary:
  first DIVERGE on moe_router  : None
  first DIVERGE on moe_ffn_out : None
  first NaN_GPU on moe_router  : None
  first NaN_GPU on moe_ffn_out : Some(17)

If first_NaN_GPU(moe_router) == 0: bug is in the F32 router @ hidden CPU dot product (unlikely).
If first_NaN_GPU(moe_ffn_out) == 0 and moe_router @ L0 is finite: bug is in expert_swiglu_cuda.
If first_NaN_GPU(moe_ffn_out) > 0 and earlier layers MATCH: bug is layer-N specific (rare).

What the harness does NOT do (yet)

• Does NOT assert pass criteria. The full FALSIFY-QW3-MOE-GPU-PARITY-001 cosine threshold lives in qwen3_moe_gpu_parity. This is a diagnostic harness for the M-GPU-MOE-1.4 NaN/Inf bisection.
• Does NOT clean up /tmp/moe-sub-{cpu,gpu}-<pid>/ dirs (operator inspects them for raw bytes if cosine is ambiguous).

Falsifier

FALSIFY-MOE-SUB-002 (byte-identity preservation for existing stages). This PR ships the harness; an operator-dispatched run on lambda-vector RTX 4090 + cached 17.3 GB Qwen3-Coder GGUF (~30-60 min wall) will produce the divergence table. Threshold-based discharge becomes meaningful AFTER the M-GPU-MOE-1.4 bug is fixed.

M-MOE-SUB-3 status after this PR

Step	Status
Test harness authored	SHIPPED (this PR)
Run on lambda-vector + interpret table	operator-dispatched
Promote to FALSIFY-MOE-SUB-002 DISCHARGED	gated on M-GPU-MOE-1.4 fix

Test plan

• cargo test -p aprender-serve --features cuda --test qwen3_moe_gpu_per_stage_diff → 6/6 light tests pass; heavy test ignored as expected
• cargo fmt --check (file-level)
• cargo build -p aprender-serve --features cuda --tests --test qwen3_moe_gpu_per_stage_diff → compiles
• Auto-merge once required checks pass
• Operator dispatches --include-ignored on lambda-vector → divergence table

🤖 Generated with Claude Code