Skip to content

v2.2.0 — Block-64 2-Level Hierarchical Non-Linear MoE Quantization (14.35 GiB Fractal-64), 100/100 Quality Gate, & 80.14 tok/s on 8-CCD Threadripper

Choose a tag to compare

View all tags
@synchopate synchopate released this 02 Nov 23:13
· 29 commits to main since this release
v2.2.0
2145e5e

DRAFT Release Notes — llama-modchip v2.2.0: Block-64 2-Level Fractal-MoE Compression & 100-Point Quality Gate

Status: Verified on live Zen 5 hardware (CT 100, AMD Ryzen 9 9950X, 64 MB L3, 256K context). Saved locally and on NAS (/mnt/nas_share/nemotron-30b-cpu-256k/). Not pushed to GitHub (origin/main) until final approval.

1. Headline Results (Live Hardware Verification on CT 100)

Starting directly from the unquantized 58.84 GiB BF16 master weights (NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16.gguf), our Block-64 2-Level Fractal Compressor (Fractal-64-BF16) shrinks the model to 14.35 GiB (14,698 MiB, 3.90 bpw overall / 3.75 bpw on MoE experts) — saving -3.24 GiB on disk and -6.01 GiB of idle Resident RAM (VmRSS) while scoring 100.0 / 100.0 on the Quality & Tool-Calling Gate and running +28.0% faster (25.02 tok/s vs 19.54 tok/s).

We also ship Fractal-MXFP4-16.0GiB (15.95 GiB disk / 17.46 GiB idle VmRSS) for users re-quantizing directly from an existing Q4_0 GGUF without downloading the 58.84 GiB BF16 master, achieving 26.08 tok/s (+33.5% faster) and 100.0 / 100.0 quality parity.

Metric / Suite Baseline-Q4_0 (Official GGUF) Fractal-MXFP4 (from Q4_0) Fractal-64-BF16 (from BF16) Fractal-64-BF16 Delta
GGUF File Size on Disk 17.59 GiB (18,015 MiB) 15.95 GiB (16,328 MiB) 14.35 GiB (14,698 MiB) -3.24 GiB (-18.4%)
MoE Expert Format & Bitrate Q4_0 (4.50 bpw) MXFP4 (4.25 bpw) Fractal-64 (3.75 bpw) -0.75 bpw on experts
SQNR on True BF16 Expert Weights 21.02 dB 18.51 dB 17.78 dB +1.53 dB vs Q3_K (16.25 dB)
1. Agentic Tool-Calling (30 pts) 30.0 / 30.0 30.0 / 30.0 30.0 / 30.0 100.0% parity
2. Executable Code Gen (30 pts) 30.0 / 30.0 30.0 / 30.0 30.0 / 30.0 100.0% parity
3. Long-Context Needle (8.5K) & Math (20 pts) 20.0 / 20.0 20.0 / 20.0 20.0 / 20.0 100.0% parity
Quality & Tool-Use Subtotal (80 pts) 80.0 / 80.0 80.0 / 80.0 80.0 / 80.0 100.0% parity
Decode Speed (tok/s) 19.54 tok/s 26.08 tok/s 25.02 tok/s +28.0% (+33.5% MXFP4)
Idle Resident RAM (VmRSS, 256K ctx) 21.41 GiB 17.46 GiB 15.40 GiB -6.01 GiB (-28.1%)
Post-Benchmark Peak VmRSS (256K ctx) 21.71 GiB 18.86 GiB 17.27 GiB -4.44 GiB (-20.5%)

2. Technical Architecture of the Block-64 2-Level Fractal Compressor

2.1 Solving the 1856 × 2688 Super-Block Alignment Trap

  • In NVIDIA-Nemotron-3.5-Lightning-30B-A3B, the 46 sparse MoE layers (128 experts per layer, 90.1% of total parameters) have inner dimensions n_ff_exp = 1856 (29 × 64) and n_embd = 2688 (42 × 64).
  • Neither dimension is divisible by QK_K = 256, which causes all standard 256-block K-quants and I-quants (Q3_K, IQ3_S, IQ3_XXS) in llama.cpp to fail alignment and fall back to Q4_0.
  • By designing our 2-level fractal block around QK2_0 = 64 (30 bytes per 64 weights = 3.75 bpw), every 1856 and 2688 row divides evenly (29 and 42 blocks per row) with zero fallback tensors.

2.2 Three-Scale Self-Similar Representation (30 bytes / 64 weights)

  1. Super-Block Scale (ggml_half d, 2 bytes): Captures the macro-scale dynamic range across all 64 weights (d = max_b(s_opt[b]) / 19.0f).
  2. Shifted 4-Bit Sub-Block Scales (uint8_t scales[4], 4 bytes):
    • Divides the 64 weights into 8 sub-blocks of 8 weights (b = 0..7).
    • Maps each 4-bit nibble sc[b] ∈ {0..15} to an integer multiplier mult[b] = 4 + sc[b] ∈ {4..19}, eliminating dead near-zero sub-scale codes {0..3} and increasing effective scale resolution by +31%.
  3. Non-Linear Dyadic Codebook (uint8_t qs[16] + uint8_t qh[8], 24 bytes):
    • Encodes 64 3-bit indices (0..7) mapped to the non-linear symmetric dyadic lattice {-14, -8, -4, -1, +1, +4, +8, +14}.
    • Achieves 17.78 dB SQNR on real BF16 Nemotron-3.5 MoE weights — +1.53 dB higher than Q3_K (16.25 dB) and +2.91 dB higher than IQ3_S (14.87 dB).

2.3 Zero-Overhead Fused AVX2 + _MM_HINT_NTA Dot-Product Kernel

  • In ggml_vec_dot_q2_0_q8_0_generic (ggml/src/ggml-cpu/quants.c), the 64 high bits in qh[8] are interleaved in 2 instructions via _mm_unpacklo_epi64(qh_lo, _mm_srli_epi16(qh_lo, 1)), combined with qs[16], and translated to signed int8_t fractal values via _mm256_shuffle_epi8(lut256, idx).
  • After _mm256_maddubs_epi16, each 8-weight sub-block occupies 4 consecutive int16_t lanes. We multiply by mult[b] ∈ {4..19} directly inside _mm256_madd_epi16(p01, vmult01) (vpmaddwd), fusing sub-block rescaling and int16 -> int32 horizontal accumulation into a single 1-cycle instruction.
  • Cold MoE expert blocks are prefetched with _MM_HINT_NTA (non-temporal cache-bypass hint), preventing the 46 MoE layers from evicting the 6 GQA KV caches, 23 Mamba-2 SSM states, and shared experts from the 64 MB Zen 5 L3 cache.

3. Scaling Projection: DeepSeek V4 Flash & DeepSeek V4 Pro on CPU

Because our Block-64 2-Level Fractal Compressor (3.75 bpw on routed experts / 3.90 bpw overall) operates on 64-element blocks without requiring 256-element K-quant alignment and starts directly from BF16/FP8 master weights:

Model Architecture Total / Active Params Master BF16 / FP8 Size Standard Q4_0 / Q4_K Size Our Fractal-64 (3.75 bpw MoE) Target CPU RAM Tier
Nemotron-3.5-Lightning-30B-A3B (Verified Live) 30B / 3B 58.84 GiB (BF16) 17.59 GiB 14.35 GiB (15.40 GiB idle RSS) Fits easily in 24–32 GB RAM
DeepSeek V4 Flash (Fine-Grained Hybrid MoE) ~200B–236B / ~16B–21B ~200–236 GiB (FP8) ~118–135 GiB ~91–107 GiB Fits inside 128 GB DDR5 consumer workstation (9950X)
DeepSeek V4 Pro / V3-671B (Full Flagship MoE) 671B / 37B ~671 GiB (FP8) ~380–404 GiB ~305 GiB (or ~248 GiB with 3.0 bpw 2-Level Fractal) Fits inside 256–384 GB EPYC / Threadripper

4. Files Included in v2.2.0 (Backed Up to NAS)

  • patches/zen5-nemotron-256k-optimizations.patch — Updated with block_q2_0 (30 bytes, Block-64 2-Level Fractal), quantize_row_q2_0_ref, dequantize_row_q2_0, AVX2 + _MM_HINT_NTA ggml_vec_dot_q2_0_q8_0_generic, and _MM_HINT_NTA ggml_vec_dot_mxfp4_q8_0. Passes git apply --check cleanly against d2f83055.
  • scripts/build_fractal_moe_gguf.sh — Supports --profile fractal64 (14.35 GiB from BF16), --profile mxfp4 (15.95 GiB from Q4_0), and --profile mxfp4_q5 (16.10 GiB).
  • benchmarks/eval_quality_and_tools.py — 100-point automated Quality, Tool-Calling, Executable Coding, Long-Context Needle (8.5K tokens) & Speed/RAM release gate.
  • workspace/eval_baseline_q4_0.json, workspace/eval_fractal_mxfp4.json, workspace/eval_fractal_64_bf16.json, and workspace/QUALITY_GATE_REPORT.md.
Assets 2
Loading