Skip to content

v2.5.0 — LUT-VQ3-64 (3.00 bpw AVX-512 VBMI), Windows 11 1M-Context Port, & Metal Q2_0 Fast Decode

Choose a tag to compare

View all tags
@synchopate synchopate released this 11 Nov 07:45
· 13 commits to main since this release
v2.5.0
e0a005f

llama-modchip v2.5.0 — LUT-VQ3-64 (3.00 bpw AVX-512 VBMI), Native Windows 11 1M-Context Port, & Apple Silicon Metal Q2_0 Fast Decode

v2.5.0 introduces LUT-VQ3-64 (GGML_TYPE_VQ3_0 = 43, 24 bytes / 64 weights = 3.00 bpw), a 3D Lloyd-Max Vector Quantizer decoded directly inside 6 AVX-512 ZMM registers via VPERMI2B, paired with Fractal-64 (GGML_TYPE_Q2_0 = 42, 30 bytes / 64 weights = 3.75 bpw) in domain-aware hybrid GGUF checkpoints. This release also adds the native Windows 11 patched engine port (1M-token context verified on 64 GB RAM), fixes the Windows x64 MinGW-w64 AVX-512 ABI/alignment crash (commit 54c89bf), and merges the register-tiled Apple Silicon Metal Q2_0 fast-decode kernel (+86% on M1 Max).

1. LUT-VQ3-64 (GGML_TYPE_VQ3_0 = 43, 3.00 bpw) & Fractal-64 (GGML_TYPE_Q2_0 = 42, 3.75 bpw)

  • block_vq3_0 Layout (24 B / 64 W = 3.00 bpw):
    • 2 B FP16 super-block scale d
    • 3 B (s24[3]) storing 8 × 3-bit unsigned sub-block scales (mult = 2 + u_b ∈ {2..9}, one per 8-weight sub-block)
    • 19 B (qs[19], 152 bits) storing 20 7-bit 3D Lloyd-Max centroid indices (140 bits, weights 0..59) + 2 6-bit 2D tail indices (12 bits, weights 60..63)
  • Register-Resident AVX-512 VBMI Decode: Holds the entire 128-entry 3D int8 codebook (384 bytes) in 6 ZMM registers across multi-row/multi-column GEMV/GEMM kernels (_1x8, _1x4, _1x3, _1x2, 1x1), resolving 64 byte lookups per _mm512_permutex2var_epi8 (VPERMI2B) instruction with zero L1/L2 cache traffic, plus portable AVX2 and scalar fallbacks.
  • Direct Signal Fidelity on Uncompressed BF16 MoE Weights (blk.1.ffn_up_exps.weight, $N=4.19\text{M}$):
Quantization Format Block Geometry Effective Rate SQNR vs BF16 Cosine Similarity Relative $L_2$ Error
BF16 Master 128 B / 64 W 16.000 bpw $\infty$ 100.00% 0.00%
Q4_0 Baseline 18 B / 32 W 4.500 bpw 21.09 dB 99.61% 8.82%
Fractal-64 (GGML_TYPE_Q2_0 = 42) 30 B / 64 W 3.750 bpw 17.75 dB 99.15% 12.96%
Standard ggml Q3_K 110 B / 256 W 3.438 bpw 16.30 dB 98.84% 15.32%
LUT-VQ3-64 (GGML_TYPE_VQ3_0 = 43) 24 B / 64 W 3.000 bpw 13.57 dB 97.78% 20.97%
Standard ggml Q2_K 84 B / 256 W 2.625 bpw 10.15 dB 95.37% 31.08%

2. Shipped Hybrid Checkpoints & Full-VQ3 Ablations

A. NVIDIA-Nemotron-3.5-30B-A3B (52 Blocks: 23 Mamba-2 SSM + 23 MoE + 6 GQA)

  • Shipped Hugging Face Checkpoint (Synchopate/NVIDIA-Nemotron-3.5-30B-A3B-VQ3-64-CPU-GGUF, 13.30 GiB / 14.28 GB): Keeps boundary MoE layers (blk.1, 3, 49, 51) and all 23 ffn_down_exps tensors in Fractal-64 (3.75 bpw), and encodes the 19 interior blk.6..47.ffn_up_exps tensors directly from BF16 into VQ3_0 (3.00 bpw).
  • Full 23-Layer up_exps Ablation (--all-up-exps, 13.08 GiB disk / 16.04 GiB VmRSS on 16C Zen 5):
Metric (benchmarks/eval_quality_and_tools.py) BF16 (58.84 GiB) Q4_0 (17.59 GiB / 19.86 GiB BF16-ref) Fractal-64 (14.36 GiB) LUT-VQ3-64 (13.08 GiB --all-up-exps / 13.30 GiB Hybrid)
Peak Resident RAM (VmRSS, 256K Context) 59.40 GiB 21.71 GiB 17.27 GiB 16.04 GiB (-5.67 GiB vs Q4_0)
Raw Single-Token Decode (N=1) 7.82 tok/s 19.54 tok/s 22.25 tok/s 22.44 tok/s (+14.8% vs Q4_0)
End-to-End Decode (Stockfish-PVS) 7.82 tok/s 19.54 tok/s 25.02 tok/s 24.18 tok/s (+23.7%, 70.19% accept)
Quality Gate (Tool + Code + Needle + Telemetry) 100.0 / 100.0 100.0 / 100.0 100.0 / 100.0 100.0 / 100.0

B. DeepSeek-V4-Flash-0731 (284B Total / 13.17B Active, 43 Layers: 3 Hash + 40 Learned Top-6)

  • Shipped Hugging Face Checkpoint (Synchopate/DeepSeek-V4-Flash-0731-VQ3-64GB-CPU-GGUF, 56.40 GiB / 60.56 GB, 3 shards): Keeps 100% of MLA attention projections (attn_q_b, attn_output_a, attn_output_b, attn_kv, compressor_*) in Fractal-64 (3.75 bpw), boundary shared experts (blk.0..4, 39..42) and all ffn_down_shexp in Q5_K/Q6_K, and encodes the 68 interior blk.5..38 ffn_gate_shexp & ffn_up_shexp projections into VQ3_0 (3.00 bpw).
  • 255-Tensor Aggressive Ablation (--aggressive, 56.07 GiB disk / 58.31 GiB VmRSS on 16C Zen 5): Accelerates raw N=1 tg64 decode from 7.74 tok/s to 9.05 tok/s (+16.9%) and pp128 prefill from 37.78 tok/s to 41.11 tok/s (+8.8%), with 10.4 ms /dev/shm Claude Code 40-tool Slot-0 restore (0.16s TTFT).
  • MLA Runtime Requirement: Always pass matching KV cache types (-ctk q8_0 -ctv q8_0 or -ctk q4_0 -ctv q4_0) and keep native Top-6 routing unpruned (do not set LLAMA_MOE_TOP_K or DSV4_MOE_TOPK at load time).

3. Native Windows 11 Port (1M Context) & x64 AVX-512 ABI Fix

  • Windows 11 Patched Engine Port (@MR1KRAKEN, PR #27 / Issue #28): Added windows/build_patched_engine.ps1 -Target dsv4|nemotron, windows/modchip_dsv4.ps1, windows/AVVIA_DSV4.bat, and docs/WINDOWS_PORT.md, verified up to 1,048,576 (1M) tokens on an 8C/16T AMD Ryzen 7 9800X3D (64 GB DDR5, 21.3–22.6 tok/s pp512, 6.35–8.04 tok/s tg128).
  • Windows x64 MinGW-w64 AVX-512 Fix (commit 54c89bf, @khairullinr Issue #31): Fixed GCC UCRT64 XMM6..XMM15 callee-saved register clobbers across 16 MB stack probes (-Wl,--stack,16777216 + non-inlined SIMD kernel helpers) and replaced aligned vmovdqa64 .rdata loads with unaligned vmovdqu64 (_mm512_loadu_si512) for PE/COFF 16-byte section alignment compatibility.

4. Apple Silicon Metal Q2_0 Fast Decode & Community Benchmarks

  • Apple Silicon Metal Q2_0 Fast-Decode Kernel (@austindixson, PR #24): Register-tiled kernel_mul_mv_q2_0_f32 shader boosting Apple M1 Max decode by +86% (57.65 tok/s) and reaching 78.60 tok/s on Apple M3 Max.
  • Apple M5 Max 64 GB DeepSeek-V4 284B Port (@notactuallytreyanastasio): Added apple-silicon/build-dsv4-engine.sh, apple-silicon/run-dsv4.sh, and apple-silicon/results/dsv4-m5max.md (169.4 tok/s pp512, 27.0 tok/s tg128, 0 swapouts), plus M5 Max CPU DRAM bandwidth telemetry (286.5 GB/s, @kulesh Issue #21).
  • Community CPU Leaderboard: Added AMD Ryzen 7 5700X (101.93 tok/s prefill / 19.66 tok/s refactor, @germanpsycho Issue #23) and AMD Ryzen 5 2600X (21.06 tok/s prefill / 10.42 tok/s refactor, @lawsochi Issue #26).

5. Build, Quantization, & Release Assets

  • Self-Contained LUT-VQ3-64 Encoder & Scripts: vq3/vq3_enc_lib.c compiles without external -lggml-base dependencies and is built automatically by scripts/build_nemotron_vq3.py, scripts/build_dsv4_vq3.py, and ./modchip quantize --vq3-nemotron|--vq3-dsv4.
  • Linux Build: 
    # Nemotron-3.5-30B-A3B (Fractal-64 + LUT-VQ3-64 + RouteCrack/Stockfish-PVS)
TARGET_MODEL=nemotron bash scripts/build_zen5_llama.sh

# DeepSeek-V4-Flash 284B (MLA + Fractal-64 + LUT-VQ3-64)
TARGET_MODEL=dsv4 bash scripts/build_zen5_llama.sh
  • Windows 11 Build (MSYS2 UCRT64): 
    powershell -ExecutionPolicy Bypass -File windows\build_patched_engine.ps1 -Target dsv4
  • Attached Release Assets:
    • llama-server-modchip-linux-x86_64 & llama-cli-modchip-linux-x86_64: Prebuilt Linux x86_64 binaries (commit 54c89bf, supports both Nemotron-3.5 and DeepSeek-V4 across Fractal-64 Q2_0=42 and LUT-VQ3-64 VQ3_0=43).
    • deepseek-v4-flash-fractal-256k.patch (alias llama-modchip-dsv4-10tps.patch), zen5-nemotron-256k-optimizations.patch, lut-vq3-64-avx512-vbmi.patch, and vq3_codebook.h.
Assets 16
Loading