[RFC / Benchmarks] Fitting 284B DeepSeek-V4-Flash in 57.61 GiB RAM (256K ctx, 0 GPUs): SwiGLU-H2 Hessian Quantization, AVX-512BW _1x8 Register Tiling (+77.1% pp), & _MM_HINT_NTA L3 Isolation

[synchopate]

Hi everyone,

Over the past few weeks we have been working on fitting and accelerating 284.33B DeepSeek-V4-Flash (61 layers, 256 routed experts, top-8 active, MLA + Engram + MQA8 Sparse Indexer) and 30B-A3B Hybrid MoEs (Nemotron-3.5-30B-A3B, Qwen3-30B-A3B) on standard dual-channel 64 GB DDR5 CPU workstations (0 GPUs) at full 256K context.

All patches, reproducible standalone C transcoders, and AVX2/AVX-512BW/NEON kernels are open-source (MIT) in synchopate/llama-modchip. Below is a purely technical breakdown of the 5 most impactful low-level optimizations, with exact microbenchmark telemetry and file references in case any of these are useful to upstream ggml-cpu.

---

1. Closed-Form Second-Order SwiGLU Hessian Diagonal (SwiGLU-H2) + Block-64 Q2_0 (3.75 bpw) Transcoding (-20.31 GiB)

Standard IQ1_S / IQ2_XXS conversion of 284.33B DeepSeek-V4-Flash occupies 76.87 GiB on disk, which immediately triggers Linux OOM killer on a 64 GB RAM box (or collapses into 133 tok/s SSD page-cache thrashing). However, naively dropping routed experts to 1-bit without cross-projection sensitivity weighting destroys coding/math perplexity because SwiGLU couples the Gate ($W_g$) and Up ($W_u$) projections multiplicatively:

$$y = \text{SiLU}(z_g) \odot z_u, \quad z_g = W_g x, \quad z_u = W_u x$$

In scripts/transcode_dsv4.c (transcode_moe_exps, lines 610–742), we derive the exact expected diagonal of the Gauss-Newton Hessian over an isotropic spherical activation prior $x \sim \mathcal{N}(0, \sigma_x^2 I)$:

$$\mathbb{E}[\text{SiLU}(z_g)^2] = \frac{7}{20}\sigma_{z_g}^2 = 0.35,||W_g^{(r,:)}||_2^2 \sigma_x^2$$

$$\mathbb{E}[\text{SiLU}'(z_g)^2] \approx \frac{31}{100} = 0.31$$

This gives a closed-form per-row importance metric across all 256 routed experts in layers 3–60 without requiring an external calibration dataset:

• Up-Projection Row $r$ Importance: $I_{\text{up}}(r) \propto \sqrt{0.35} \cdot ||W_{\text{gate}}^{(r,:)}||_2$
• Gate-Projection Row $r$ Importance: $I_{\text{gate}}(r) \propto \sqrt{0.31} \cdot ||W_{\text{up}}^{(r,:)}||_2$

Combined with transcoding the dense Q5_K MLA key-LORA (attn_k_b, 3.70 GiB -> 1.62 GiB) and Sparse Indexer (indexer.wk, indexer.weights_proj) into a custom 64-weight block GGML_TYPE_Q2_0 (3.75 bpw, 30 bytes/64 weights) with a 24-iteration alternating Lloyd-Max solver:

• File size: 76.87 GiB $\rightarrow$ 56.56 GiB (60,729,886,080 bytes, -20.31 GiB / -26.4%)
• Live process VmRSS (/proc/pid/status at 256K ctx): 57.61 GiB (60,408,048 kB), leaving 6.39 GiB of free headroom inside 64 GiB RAM with 0 bytes of swap.

---

2. Whole-Model 4-Level Cache-Tiled Prefill (ZMM Registers $\rightarrow$ L1D $\rightarrow$ L2 $\rightarrow$ L3) (+77.1% Prefill Throughput)

On CPU, multi-token prompt evaluation (pp) in MoE models suffers from severe DDR5 read amplification: token-major execution streams the same quantized expert/attention weights from RAM repeatedly across tokens, while standard step_k = 2 (.out_1x2) matrix-matrix kernels only reuse a decoded weight super-block across 2 activation columns.

In patches/deepseek-v4-flash-fractal-256k.patch (ggml/src/ggml-cpu/arch/x86/quants.c and ggml-cpu.c), we implemented a 4-level cache hierarchy optimization across all 6 quantized tensor types (Q2_0, IQ1_S, IQ2_XXS, IQ1_M, Q5_K, Q6_K):

1. Level 0 (ZMM Register-Stationary _1x8, _1x4, _1x3, _1x2 Multi-Column Kernels):
   • Upgraded all 6 formats from 256-bit AVX2 to 512-bit AVX-512BW (__m512i), folding two 256-element QK_K super-blocks per ZMM iteration and keeping 8 ZMM accumulators (acc0..acc7) pinned in registers across 8 activation columns (step_k = 8). This cuts weight decompression + maddubs overhead by 4.0x vs _1x2 (8.0x vs 1x1).
   • Zero-Remainder {8, 4, 3, 2} Greedy Column Decomposition (ggml_compute_forward_mul_mat_one_chunk): In 256-expert MoEs (top-8), individual experts frequently receive odd token counts ($M_e \in {3, 5, 6, 7, 9, 11}$). With power-of-two stepping ({8, 4, 2, 1}), odd tails drop to scalar 1x1. Adding _1x3 enables exact decomposition ($5 = 3+2$, $7 = 4+3$, $9 = 3+3+3$, $11 = 8+3$), eliminating 1x1 fallback for every batch size $M_e \ge 2$.
   • 7-Instruction BMI2 _pdep_u64 + SWAR XOR-Parity Sign Expansion (IQ2_XXS): Replaces the 8-load keven_signs_q2xs[aux32] lookup table bottleneck with _pdep_u64(s56 ^ (s56 << 1) ^ ... ^ (s56 << 6), 0x0101010101010101ULL) | (s56 & 0xFEFEFEFEFEFEFEFEULL) followed by _mm512_movm_epi8, expanding 64 signs into a ZMM register in 7 ALU cycles with zero L1D cache ports used.
2. Level 1 & Level 2 (32 KB L1D + 1 MB L2 Token-Stationary Q8_K Tiling + AVX-512 quantize_row_q8_K):
   • Vectorized quantize_row_q8_K (quants.c:505) with 16 ZMM registers + _mm512_cvtepi32_epi8, keeping the Q8_K activation tile (8 cols × 9 KB = 72 KB) hot inside each core's 1 MB L2 cache.
   • Added token-to-core affinity (by_tok = (n_tokens >= nth)) in ggml_compute_forward_dsv4_hc_pre_f32 and ggml_compute_forward_dsv4_hc_post_f32 (ops.cpp), preserving per-core L1D/L2 locality across the 4-stream Hyper-Connection (HC) residual width expansion/collapse.
3. Level 3 (64 MB L3 MQA8 FlashAttention Target-Query Blocking):
   • DeepSeek-V4's Sparse Indexer and MLA KV-cache share keys/values across $G = 8$ query heads per KV head (ne02 / ne12 = 8). Our MQA8 fast-path (ops.cpp:9452) loads each cached KV token once from L3/RAM and evaluates all 8 query heads simultaneously in registers (8x KV bandwidth reduction), while tiling by target query token ic (n_queries >= nth) so each thread keeps its 8 query heads (8 × 576 × 2B = 9.2 KB) inside its 32 KB L1D.

Prompt Length (pp)	Baseline v2.3.0 (tok/s)	Whole-Model 4-Level Cache-Tiled v2.4.0 (tok/s)	Speedup (100% Bit-Exact)
pp = 298	25.20 tok/s	43.16 tok/s	+71.3%
pp = 392	25.10 tok/s	43.03 tok/s	+71.4%
pp = 403 (Slot-0 Warm)	25.05 tok/s	44.36 tok/s	+77.1%
pp = 580	24.73 tok/s	43.98 tok/s	+77.8%

---

3. Single-Cycle AVX2 Variable Bit-Shift dot_q2_0_q8_0 (+55%) & 3D L1D Tiling (TILE_I0 = 64)

For CPUs without AVX-512 (or in our standalone Q2_0 AVX2 path in patches/q2_0-avx2-3d-tiling.patch):

• Single-Cycle _mm256_srlv_epi32 Unpacking: Instead of 4 sequential scalar shifts (>> 0, >> 2, >> 4, >> 6) + _mm256_set_m128i lane insertions to unpack 2-bit weights, we broadcast the 128-bit qs payload (_mm256_broadcastsi128_si256) and apply a single-cycle per-dword variable shift _mm256_srlv_epi32(q256, _mm256_setr_epi32(0, 0, 2, 2, 4, 4, 6, 6)) (+55% isolated kernel speedup: 4.42 -> 2.85 cycles/block).
• 3D L1D Cache Tiling (TILE_I0 = 64): Standard 2D tiling streams the entire x activation vector (16 KB for K = 16384) for every output row, overflowing Zen 5's 48 KB L1D (11.2 GB/s L1D-to-L2 spill traffic per token). Adding an inner output-row tile (TILE_I0 = 64) bounds the working set to 18.5 KB < 48 KB L1D, reducing x reloads by 64x.

---

4. _MM_HINT_NTA L3 Isolation (HARPCache) + 8-Step Sparse Indexer Lookahead Prefetch (ic_pf = i + 8)

• Non-Temporal Streaming for Cold Routed Experts (HARPCache): During decode ($M=1$), streaming 1.88 GiB of cold routed MoE expert weights per token through standard temporal cache lines evicts the hot Q8_K activation vectors, shared experts, and MLA KV-cache from the 64 MB L3 cache. Inserting _mm_prefetch(..., _MM_HINT_NTA) in the expert dot-product inner loops isolates cold expert weights to a single L3 way while pinning hot activations in L1D/L2/L3.
• 8-Step Lookahead Hardware Prefetching for Sparse Attention (ic_pf = i + 8): In ggml_compute_forward_flash_attn_ext_f16_one_chunk (ops.cpp), when the index_mask tensor ([n_kv, n_queries]) selects top-$k$ sparse KV tokens, irregular row jumps stall the hardware prefetcher. Scanning ahead to ic_pf = i + 8 and issuing _mm_prefetch(k_data + kv_idx_pf * nb11, _MM_HINT_T0) hides the ~90 ns DDR5 latency behind ongoing AVX-512 FMA execution.

---

5. $L_2$-Quadrature Adaptive MoE Pruning (p_early = 0.025) + Gate-First Lazy SwiGLU

In Nemotron-3.5-30B-A3B (128 experts, top-6 active) and DeepSeek-V4-Flash (256 experts, top-8 active), trailing experts in the softmax tail frequently contribute $< 1%$ of the layer norm.

• Variance-Exact $L_2$-Quadrature Rescaling: When pruning trailing experts whose weight falls below $p_{\text{early}} = 0.025$, standard $L_1$ renormalization ($\sum w_i = 1$) inflates output activation variance by 1.89x, causing downstream RMSNorm distortion. Since quasi-orthogonal expert outputs add in quadrature ($\text{Var}(\sum w_i E_i(x)) \approx \sigma_E^2 \sum w_i^2$), rescaling kept weights by the $L_2$ ratio $\alpha_{L_2} = \sqrt{\sum_{\text{all}} w_i^2 / \sum_{\text{kept}} w_i^2}$ preserves exact output variance (KL = 0.000517 vs 0.004810 for $L_1$).
• Gate-First Lazy SwiGLU (3.16% DRAM Elimination): Evaluates ffn_gate_exps first, computes SiLU(gate) in L1D, and skips fetching ffn_up_exps and ffn_down_exps whenever $\max_j |\text{SiLU}(g_{e,j})| < 10^{-3}$.

All code and clean git patches are available in patches/. Happy to split any of these (especially the AVX-512BW _1x8/_1x4/_1x3/_1x2 quants.c kernels or the MQA8 FlashAttention fast-path) into standalone PRs if there is interest!