Skip to content

llama-modchip v3.0 — Unified 16GB / 32GB / 64GB MaxQ Engine

Latest

Choose a tag to compare

@synchopate synchopate released this 16 Nov 21:06
· 1 commit to main since this release

v3.0 — Unified 3-Tier RAM Lineup (16 GB Nemotron-3.5 · 32 GB & 64 GB Qwen3.8-Flash-Next + DeepSeek-V4 284B), Zero-Collision ABI (types 44/45), 512-Bit Q1_0/Q2_0/GDN Engine, Zero-Copy 3D MoE --repack, & Static-libgomp GLIBC_2.30 Binaries

Release Date: November 2026
Highlights:

  1. Donor-Free Qwen3.8-Flash-Next (MaxQ-64GB & MaxQ-32GB Aligned) + Opt-In Heretic-2 Abliterated Twins & Complete 3-Tier RAM Matrix:
    • MaxQ-64GB (57,969,748,352 B = 53.9885 GiB, 1,086 standard ggml tensors, max_type = 42, 0 custom types):
      • Shard 1 (46,449,692,736 B = 43.2596 GiB, SHA256 = f2b8bc79acb4b4d79033acd91c9c360ba96da92cd5841512befc450e15f4786f) + Shared Shard 2 (11,520,055,616 B = 10.7289 GiB, SHA256 = 86b79714867743ddf5911ee7dff2717ae275f870551d86469cbdbf026c6b552e, per_layer_token_embd.weight [256, 320001536] in Q1_0 = 41) + optional mmproj-Qwen3.8-Flash-Next-BF16.gguf (907,542,880 B, SHA256 = ef2af1d6aa8a22740c7252bc7ce40c5c2f1249aa8d9c8499a4d4572c3cfdf9df).
      • Compresses the 320M-row Per-Layer Embedding (PLE) table from 26.82 GiB (IQ4_NL) to 10.73 GiB (PCA-Wiener D=256 + RaBitQ Q1_0), reinvesting +16.09 GiB into all 512/512 unpruned routed experts (K = 640), FP64 Woodbury DualGram signed-scale Q2_0, and the built-in blk.48 NextN Multi-Token Prediction (MTP) module (0.81 GiB).
      • Blind wiki-alt.txt (40 × 512 chunks, 0.000000% 8/13-gram overlap with calibration) vs. Donor-Free BF16 Reference (Mean PPL(base) = 2.776624 ± 0.054868): Achieves Mean PPL = 3.432500 ± 0.070829 (3.43825 ± 0.07119 under v3.0 default --repack), Mean KLD = 0.497117 ± 0.009892, Median KLD = 0.105952, and Same top p = 78.314 ± 0.408%, outperforming ISTA-DASLab GSQ-RCO-Coder-IQ1_M (54.397 GiB / 58,408,584,928 B, 256/512 pruned experts, no MTP: Mean PPL = 5.091640 ± 0.121418, Mean KLD = 0.930241, Median KLD = 0.296442, Same top p = 69.382%) by -1.6591 PPL (3.53× smaller PPL degradation over BF16), -46.6% Mean KLD, and -64.3% Median KLD while 0.41 GiB smaller.
    • MaxQ-32GB (38,341,683,424 B = 35.7085 GiB, 1,128 standard ggml tensors, max_type = 42, 0 custom types):
      • Shard 1 (26,821,627,808 B = 24.9796 GiB, SHA256 = 06cb65f9cf6e510cd16500ca1719701a0ffa13dcaa4c4d0cc135e4f2071a8b4d, 42 ConeCollapsed K = 512 layers with 42 × F32 ffn_down_exps.bias + 7 K = 640 boundary layers + blk.48 MTP) + Shared 10.7289 GiB Shard 2 (86b79714...) demand-paged via NVMe mmap.
      • 32 GB Consumer Exclusivity (18-Checkpoint Hugging Face Audit): Across all 18 published Unsloth and ISTA-DASLab Qwen3.8-Flash-Next GGUFs (54.40–329.72 GiB), 0 / 18 fit inside a 32 GB host (<= 28.5 GiB). MaxQ-32GB passes strict Linux Cgroup v2 enforcement (memory.max = 28.5 GiB, MemorySwapMax = 0, oom_kill = 0) at VmHWM = 26.91–27.44 GiB <= 28.5 GiB with blind wiki-alt.txt Mean PPL = 5.812001 ± 0.140197 (5.80406 repack) and Mean KLD = 1.006916 ± 0.014944.
    • Opt-In Gated-Residual Over-Projected Abliterated Twins (qwen38-64gb-ablit & qwen38-32gb-ablit, v3.0-r2):
      • Applies TPE-optimized signed multi-layer over-projection (beta = 1.83 - 2.39 across blk.31..47 from trohrbaugh/Qwen3.8-Flash-Next-heretic-2, achieving 0 / 100 refusals at 0.0818 KLD by compensating for Qwen3.8's widened multi-stream Gated Residual branches, plus signed beta = 2.2508 blk.48 NextN MTP alignment so speculative drafting stays synchronized on uncensored completions) directly onto virgin MaxQ-64GB and MaxQ-32GB (blk.0..30 & token_embd 100% untouched):
        • MaxQ-64GB-Abliterated: Shard 1 = 46,449,692,736 B (1,085 tensors: 27 edited [16 blk.32..47 ssm_out/attn_output + 5 blk.31..35 ffn_down_shexp + 5 blk.31..35 512-expert ffn_down_exps + 1 blk.48 MTP attn_output], 1,058 bit-identical to MaxQ-64GB; SHA256 = 1dac27e81a3a73dcec21edea78f224eb48393a726e3f9ee7676c603d3174e82f) + Shared Shard 2 (11,520,055,616 B, SHA256 = 86b79714867743ddf5911ee7dff2717ae275f870551d86469cbdbf026c6b552e) + optional mmproj (907,542,880 B, SHA256 = ef2af1d6aa8a22740c7252bc7ce40c5c2f1249aa8d9c8499a4d4572c3cfdf9df).
        • MaxQ-32GB-Abliterated: Shard 1 = 26,821,627,808 B (1,127 tensors: 32 edited [27 weight tensors + 5 blk.31..35 F32 ffn_down_exps.bias aligned with signed layer beta_m = 0.449..1.710], 1,095 bit-identical to MaxQ-32GB; SHA256 = 09b54c335505f9f609415dbc774868ad5ae188f17e873dead3ec743ec84e1009) + Shared Shard 2 (86b79714...) + optional mmproj (ef2af1d6...).
      • Gated-Residual Over-Projection, blk.48 MTP Synchronization & Edition Positioning: Because Qwen3.8-Flash-Next routes activations through widened multi-stream Gated Residual Hyper-Connections (hc_attn_*, hc_ffn_*), single-branch beta <= 1.0 projection leaves parallel residual branches carrying refusal state at late layers (blk.32..47). v3.0-r2 encodes Heretic-2's TPE-optimized over-projection (0 / 100 refusals vs. 99 / 100 upstream at 0.0818 KLD) plus signed beta = 2.2508 blk.48 NextN MTP synchronization (81–85% draft-mtp acceptance on live multi-turn tool-calling at 13.2–14.8 t/s, VmHWM = 46.34 GiB on 64GB and 28.40 GiB <= 28.5 GiB on 32GB), positioning Aligned MaxQ-64GB / MaxQ-32GB as the default pure software-engineering flagships and *-ablit strictly for opt-in security research, reverse engineering, red-team auditing, and uncensored legal/policy analysis (configure --temp 0.6 --top-p 0.95 --top-k 20 --presence-penalty 0.2 --repeat-penalty 1.05 on 32GB presets to prevent low-bit greedy tail loops).
    • 16 GB NVIDIA-Nemotron-3.5-30B-A3B Donor-Free MaxQ-16GB + Zero-Recompute 42/43 -> 44/45 Fleet Unification:
      • NVIDIA-Nemotron-3.5-30B-A3B-MaxQ-16GB (14,382,336,640 B = 13.3946 GiB < 13.50 GiB, SHA256 = 26b884d70a52046a7198c4e01cac397af8bf44584b04cdcf59f896145587db67): Built 100% donor-free from nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 (401 standard ggml tensors: {F32: 237, Q4_0: 70, Q8_0: 48, MXFP4(39): 28, Q2_0(42): 18}, max_type = 42, 0 custom types; 0.000000% 8/13-gram calibration overlap vs wiki-alt.txt) for native unmodified llama.cpp, Apple Silicon Metal, CUDA, Vulkan, and LM Studio. Certified under hardware-enforced Linux Cgroup v2 (MemoryMax = 15.0G, MemorySwapMax = 0, -c 32768 -ub 512 -fa on -ctk q8_0 -ctv q4_0, oom = 0) at /proc/<pid>/status VmHWM = 14.3040 GiB <= 15.0 GiB (RssAnon = 0.9097 GiB + RssFile = 13.3942 GiB; process VmHWM quoted over cgroup memory.peak = 0.949 GiB ARC/page-cache delta), delivering 188.29 t/s mean (194.05 peak) pp512 and 15.10 t/s mean (15.75 peak) tg128 pure N=1 (3/3 bit-exact) plus 27.80 t/s (94.44% accept) on MiniLRU code refactoring (merge_intervals + OpenAI multi-turn tool-calling PASS).
      • Zero-Recompute 42/43 -> 44/45 Migration & Transparent 40 × 512 wiki-alt.txt Ladder (vs. 16.07 GiB MXFP4 ref 6.5962 ± 0.1729): Patches strictly 47 header bytes (46 ttype + 1 general.file_type, 0 payload bytes altered) across VQ3-64 (14,282,559,040 B, SHA256 = 04a460cdd831002d80b2f47ec5d70e7a92227abcf0a57f31ad73fe0bd50e711f, PPL = 6.7614, Mean KLD = 0.0871 — highest-accuracy 16 GB CPU-Turbo flagship), Fractal-64 (15,420,034,624 B, SHA256 = 3a3a09eac29b5906397ca6a93d2ff0f566fca0bdd365294d067aeb5d0ebf4a1b, PPL = 6.6533, Mean KLD = 0.0688), and Abliterated-VQ3-64 (14,282,559,488 B, SHA256 = 75da8372f3896bfacd103f3cb85bf22b729d39cb1efce82fb3c838ab2af53e57), alongside universal std-ggml MaxQ-16GB (PPL = 7.3129 ± 0.1923, Mean KLD = 0.1722, Median KLD = 0.0698, Same top p = 83.431%) and 64 GB DeepSeek-V4-Flash-0731-MaxQ-v4 (57.99 GiB, 6112e887... / 5d85c740... / a8654d4d...). (Note: Nemotron's default Jinja template enables <think>...</think> reasoning; set max_tokens >= 4096 for thinking or pass enable_thinking: false / --reasoning-budget 0 for concise direct/JSON replies.)
  2. Unified llama-modchip v3.0 Engine Patch (patches/llama-modchip-v3.0.patch) & Zero-Collision Custom Types (44 & 45):
    • Zero Upstream Type Collision: Preserves upstream GGML_TYPE_Q2_0 = 42 (18 B / 64 W = 2.25 bpw, QK2_0 = 64) and GGML_TYPE_F8_E4M3 = 43 untouched; relocates Fractal-64 (30 B / 64 W) to GGML_TYPE_FRACTAL_64 = 44 and LUT-VQ3-64 (24 B / 64 W) to GGML_TYPE_VQ3_64 = 45. A single binary built from ggml-org/llama.cpp @ de32b268c4ad55d45f3b7b16c029f4e077f4ba74 + llama-modchip-v3.0.patch (SHA256 = b14475aa5350d632551b5f4261017e5ccb648750f8c6c9975255a171c367fbf0) runs the entire 16 GB / 32 GB / 64 GB fleet across Linux, macOS Metal, Windows, CUDA, and Vulkan.
    • Zero-Copy 3D MoE --repack Cure & q8_cache / params->wdata Aliasing Fixes: Fixes upstream ggml_compute_forward_mul_mat_tiled / mul_mat_id_tiled q8_cache workspace aliasing (ggml_cpu_get_q8_wdata), clears last_mmid_ids_data on every non-MUL_MAT_ID node, fixes repack.cpp stride/unaligned-load bugs, skips heap repacking on 3D MoE tensors (ne[2] >= 64) so zero-copy mmap is preserved (MaxQ-64GB RssAnon reduced 5.4× from 9.81 GiB to 1.81 GiB, draft-mtp acceptance restored from 0% to 93.00%, batch=1 --repack vs --no-repack KLD = 0.00000 ± 0.00000), and aligns MADV_HUGEPAGE inward to 2 MiB boundaries (src/llama-mmap.cpp) so lazy PLE ranges are never pre-faulted.
    • 512-Bit Q1_0/Q2_0/MXFP4 Kernels & Multi-Threaded GATED_DELTA_NET: Adds AVX-512 VNNI/BW single-col & multi-col (1x8/4/3/2) kernels for Q1_0 (41), signed-scale Q2_0 (42), overflow-free Q8_0, and MXFP4, eliminates inner barriers in multi-threaded 8-zmm FMA GATED_DELTA_NET, restores 4*nth dynamic causal-triangle load balancing on FLASH_ATTN_EXT prefill (neq1 > 1), and widens Stockfish-PVS v2 / RouteCrack token IDs to 32-bit llama_token (248,320 vocab safe).
  3. Empirical Speedups, Transparent Attribution, & Runtime Defaults (modchip, install.sh, scripts/llama_modchip.py):
    • Combined v3.0 Engine + Non-Convoy -t $((N_CORES - 1)) --poll 0 Throughput (16 Zen 5 vCPUs, default --repack ON):
      • MaxQ-64GB (draft-mtp): pp512 39.24 -> 71.97 t/s (+83.4%), tg128 7.329 -> 12.662 t/s (+72.8% vs Stage 4C --no-repack, 4.95× vs broken pre-v3.0 repack, peak 12.82 t/s, 1.988× over v3.0 none 6.368 t/s, 93.00% MTP accept, VmHWM = 45.08 GiB).
      • MaxQ-32GB (draft-mtp): pp512 61.39 -> 74.55 t/s (+21.5%), tg128 9.455 -> 11.454 t/s (+21.1% on p512, 14.29 t/s avg / 14.77 t/s peak on inplace_edit, 1.718× over v3.0 none 6.668 t/s, VmHWM = 26.91 GiB <= 28.5 GiB).
      • (Transparent Attribution Note: Stage 4C ran oversubscribed -t 16 --poll 50 --no-repack on 16 vCPUs, whereas Stage 4D-1 ran -t 15 -tb 15 --poll 0 --poll-batch 0 with v3.0 default --repack ON; isolated non-convoy threading accounted for +25.9% pp / +31.3% tg on 32GB none, and ./modchip + install.sh now default llama-server to -t $((N_CORES - 1)) -tb $((N_CORES - 1)) --poll 0 --poll-batch 0.)
    • 248K Speculative Shootout & RC_TABLES_DIR=/nonexistent Isolation: Built-in draft-mtp won cold single-turn rep 0 tg128 on 6 / 6 (100%) of (edition, prompt) pairs; zero-weight Stockfish-PVS v2 (--spec-type ngram-simple --spec-ngram-simple-size-n 3 --spec-ngram-simple-size-m 3 --spec-ngram-simple-min-hits 1) achieved 10.58–12.82 t/s (1.67×–1.87×) on cold rep 0 source-in-context refactoring with +5.6% to +11.0% faster prefill and -0.53 to -0.85 GiB lower VmHWM. ./modchip and start_server.sh automatically export RC_TABLES_DIR=/nonexistent whenever serving non-Nemotron models so ./workspace/routecrack Nemotron tables are never loaded by mistake.
    • /dev/shm Hybrid State Save/Restore (MaxQ-32GB, 512 tok): Restores the 233.14 MiB (244,461,872 B) 36 GDN + 12 QSA state in 38.49 ms avg (38.16 ms min, 6.11 GiB/s, 161.96× faster than 6.18s prefill; ~19 ms / 120.6 MiB with --ctx-checkpoints 0) with 100% bit-exact continuation (SHA256 = 8e5a6444...).
    • Sampling & <think> Guidance: MaxQ-32GB (1.63 bpw: 2/8 blind pass at greedy temp=0, 3/8 greedy repetition loops without penalties vs. MaxQ-64GB 5/8 pass [5/5 within 4,096 cap, 0/8 loops]) defaults to --temp 0.6 --top-p 0.95 --top-k 20 --presence-penalty 0.2 --repeat-penalty 1.05 in ./modchip; when enabling <think> mode, set max_tokens >= 16384 (or enable_thinking=False for <= 4096 caps). Unpatched upstream llama.cpp users should pass --no-repack (-nr); not needed on v3.0.
  4. 12 Static-libgomp Portable GLIBC_2.30 Linux x86_64 Binaries (SHA256SUMS):
    • Links libgomp.a, libstdc++.a, and libgcc.a statically (NEEDED = [libdl.so.2, libpthread.so.0, libm.so.6, libc.so.6, ld-linux-x86-64.so.2], max_glibc <= 2.30, 0 external OpenMP runtime needed) and removes internal weight-building scripts from the public runtime tree:
      • 54af3232375ede820f91143c106a594c83b815a3dca66fe98ab08bcb8b63d4f0 llama-server-modchip-linux-x86_64-avx512
      • 0f0c07f97a98d8aeb9a012264fdeff5a787d2104a2afb122052bfe0a21c2d1a9 llama-server-modchip-linux-x86_64-avx2 (and llama-server-modchip-linux-x86_64)
      • 531d870988dcc580322341d2bbd7f0cf6b6cc3b3376a2e86d092a021f7cddba2 llama-cli-modchip-linux-x86_64-avx512
      • 49a918206afe371d16ab14fd66d9cc87e35c7f8c0471c2cc13c759f2e0350d03 llama-cli-modchip-linux-x86_64-avx2 (and llama-cli-modchip-linux-x86_64)
      • 7ad8253fb9c3a779be58122adc51ea9786bb391acb49d3cbfa6280cedb01e9d8 llama-bench-modchip-linux-x86_64-avx512
      • fa8a055799bb2f65df8c6077f1377b81bfeda4efe3851639d05c1b44c3fa45e8 llama-bench-modchip-linux-x86_64-avx2
      • 003f452008972fb8696f7ea2800cfede360ba2a2c32f0a605fba2463b66bcadb llama-perplexity-modchip-linux-x86_64-avx512
      • f77c3581d37fa6920daec6938abc5767675b430130eed03ec05e68c6e6d4b624 llama-perplexity-modchip-linux-x86_64-avx2
      • befa4842fa09a061d4fe3376d226ed4ad2dafa8c149db10c06b37f85e59e3f94 llama-quantize-modchip-linux-x86_64-avx512
      • 521d9bf53f888c71e158732cd2c0b14e80a5e3d2005aed5f7a3e204ebfba9bbb llama-quantize-modchip-linux-x86_64-avx2

Official MaxQ v3.0 Hugging Face Lineup & Collection