Skip to content

Releases: synchopate/llama-modchip

Release list

v2.4.0 — DeepSeek-V4-Flash 284B Kernel & Mathematical Co-Design (+68% Prefill, 9.1–10.1 tok/s Decode, L2-Quadrature MoE), Windows 10/11 Hybrid GPU+CPU Pack, & 80.8 tok/s AMD Vulkan MXFP4

v2.4.0 — DeepSeek-V4-Flash 284B Kernel & Mathematical Co-Design (+68% Prefill, 9.1–10.1 tok/s Decode, L2-Quadrature MoE), Windows 10/11 Hybrid GPU+CPU Pack, & 80.8 tok/s AMD Vulkan MXFP4 Latest
Latest

Choose a tag to compare

View all tags
@synchopate synchopate released this 04 Nov 00:24
v2.4.0
a80966d

⚡ v2.4.0 — DeepSeek-V4-Flash 284B Kernel & Mathematical Co-Design (+68.3% Prefill, 9.06–10.12 tok/s Pure CPU Decode, $L_2$-Quadrature MoE Tail Pruning, wo_b * wo_a SVD Spectrum), Native Windows 10/11 Hybrid GPU+CPU Pack (windows/), & 80.8 tok/s AMD Vulkan MXFP4

Release Date: November 2026
Highlights:

  1. ⚡ DeepSeek-V4-Flash 284B (13.17B Active) Kernel & Graph Co-Design (+68.3% Prefill & +53.3%–71.2% Pure Decode — patches/deepseek-v4-flash-fractal-256k.patch):
    • AVX-512 vprorvq (_mm512_rorv_epi64) Q2_0 Dot-Product, In-Register AVX2 Fast Walsh-Hadamard Transform (GGML_OP_HADAMARD), & Vectorized HC_PRE/HC_POST: Replaces scalar 64-bit qh extraction in ggml_vec_dot_q2_0_q8_0_generic (ggml/src/ggml-cpu/quants.c) with single-instruction AVX-512 variable 64-bit rotation (_mm512_rorv_epi64), keeps all 9 butterfly stages (s = 1..256) of the 512-point Fast Walsh-Hadamard Transform (396 calls/token) inside AVX2 YMM registers (ops.cpp), and vectorizes Sinkhorn-20 Hyper-Connections (hc = 4) across the 4096-dim embedding.
    • Zero-Rebuild Dynamic CSA/LID Skip (ggml_dsv4_skip_csa_comp), 8-Slot Q8_K/Q8_0 Activation Cache, MUL_MAT_ID Row-Group Reuse, & Barrier Elision: Eliminates full 4,551-node compute-graph rebuilds on non-compressor decode steps (7 out of every 8 tokens) by tagging dummy CSA/LID compressor nodes with flags |= 0x100 (GGML_TENSOR_FLAG_DSV4_SKIP), caches quantized Q8_K/Q8_0 input activations across shared-input projections (s_q8_cache_buf), reuses MUL_MAT_ID expert row mappings (g_tls_mmid_ids_valid) between up_exps and gate_exps, and elides internal threadpool barriers on small matrices (nr0 < 4096).
    • Verified Speedup on 16C/32T Zen 5 (AMD Ryzen 9 9955HX, 0 GPUs): Accelerates 512-token prompt prefill (pp512) from 24.70 tok/s (20.73 s) to 41.58 tok/s (12.31 s, +68.3%) and pure lossless single-token decode (N=1, zero speculation) from 5.91 tok/s (169.3 ms/tok) to 9.06 tok/s (110.33 ms/tok, +53.3%) (and 104.67 ms = 9.55 tok/s on CSA/LID compressor steps).
  2. 🧮 Mathematical Structure of DeepSeek-V4-Flash: $L_2$-Quadrature Adaptive MoE Tail Pruning (9.68–10.12 tok/s) & Exact SVD of wo_b * wo_a:
    • Why MoE Tail Pruning Requires $L_2$-Quadrature Rescaling (Not Linear $L_1$ Rescaling): Layers 0..2 (dsv4_hash_layer_count = 3) use deterministic token-ID hash tables (ffn_gate_tid2eid) with flat weights (~0.25 each) and are preserved at 6/6 experts. In the 40 learned-router layers (il = 3..42), the 6 expert output vectors $\mathbf{e}_1(\mathbf{x}), \dots, \mathbf{e}_6(\mathbf{x}) \in \mathbb{R}^{4096}$ are nearly orthogonal ($\langle \mathbf{e}i, \mathbf{e}j \rangle \approx 0$), so their norms add in quadrature ($|\sum w_i \mathbf{e}i|2^2 \approx \sum w_i^2 |\mathbf{e}i|2^2$). Rescaling kept experts via $L_2$-quadrature ($\alpha{L_2} = \sqrt{\sum{i=1}^6 w_i^2 / \sum{i=1}^k w_i^2}$) instead of linear $L_1$ sum ($\alpha{L_1} = \sum{i=1}^6 w_i / \sum{i=1}^k w_i$) preserves the exact balance against the unscaled dense shared expert (ffn_out = moe_out + ffn_shexp) and cuts KL-divergence by -58%.
    • Cross-Platform Zero-Overhead Runtime Control (/tmp/dsv4_moe_cfg.bin & Windows %TEMP%\dsv4_moe_cfg.bin): Switchable on the fly in 0.3 ns (zero syscalls per token via mmap on Linux/macOS and CreateFileMappingA / MapViewOfFile on Windows _WIN32).
    • Exact SVD Spectral Analysis of wo_b * wo_a (31% of Total DRAM Traffic = 991.6 MB/token): Proves that the purely linear chain attn_output_a ([4096, 8192]) $\to$ attn_output_b ([8192, 4096]) retains 99.64% of Frobenius energy and 0.998204 (99.82%) output cosine similarity at rank $r = 768 / 1024$ (-25% DRAM = -248 MB/token, -6.9 ms/tok), and 99.30% cosine similarity at rank $r = 640 / 1024$ (-37.5% DRAM = -372 MB/token, -10.3 ms/tok).
  3. 🪟 Native 1-Click Windows 10/11 Hybrid GPU+CPU MoE Support (windows/, modchip):
  4. 🚀 AMD Vulkan Hybrid MXFP4 Verification (75.1 / 80.8 tok/s at 256K Context) & Security Hardening (FAQ.md, gateway/sse_gateway.py):
    • Community benchmark on AMD Ryzen 9 7945HX3D + Radeon RX 9070 XT 16GB (RADV/Vulkan) running Nemotron-30B-A3B at 256K context: Fractal-MXFP4 (15.2 GB) hits 75.1 tok/s (new code) / 80.8 tok/s (in-context refactor), beating Q4_0 (69.7 / 76.2 tok/s at 16.1 GB) while saving 0.9 GB of VRAM. Documented in FAQ.md Q8 why Fractal-64 (Q2_0) is CPU-only until Vulkan/CUDA add a custom q2_0 shader.
    • Merged PR #4 (commit 83767ca): hardened gateway tool schema whitelist, boundary checks, 2MB THP alignment, dynamic sysfs L3 CCD topology detection, and /dev/shm IPC validation.
DeepSeek-V4-Flash 284B (13.17B Active) Mode (il = 0..2 kept at 6/6) Pure Decode (tok/s) Latency (ms/tok) Speedup vs Initial (5.91 tok/s) Prob Cosine Similarity KL-Divergence $D_{\text{KL}}$ (nats) Equivalent $\Delta$ PPL Top-1 / Top-3 Match
0. Pre-Optimization Baseline (v2.3.0 initial N=1) 5.91 tok/s 169.30 ms 1.00x 1.00000 0.0000 0.00% 100% / 100%
1. v2.4.0 Lossless (Top-6 All Layers, Default) 9.06 tok/s 110.33 ms +53.3% 1.00000 0.0000 0.00% (Bit-Exact) 100% / 100%
2. v2.4.0 Adaptive Tail (min_w=0.11, $L_2$ Quadrature) 9.52 tok/s 105.02 ms +61.1% 0.99634 0.0047 +0.47% (Negligible) 95.8% / 100%
3. v2.4.0 Sweet-Spot (Top-5 on il >= 3, $L_2$ Quadrature) 9.68 tok/s 103.27 ms +63.8% 0.99714 0.0042 +0.42% (Negligible) 95.8% / 100%
4. v2.4.0 Hybrid 6/4/6 (il = 16..39 at 4, $L_2$ Quadrature) 9.83 tok/s 101.74 ms +66.3% 0.99660 0.0184 +1.86% (Very Low) 95.8% / 100%
5. v2.4.0 Hybrid 5/4/5 (il = 12..40 at 4, $L_2$ Quadrature) 9.95 tok/s 100.55 ms +68.4% 0.99463 0.0215 +2.17% (Low) 95.8% / 100%
6. v2.4.0 Turbo 10+ tok/s (Top-4 on il >= 3, $L_2$ Quadrature) 10.12 tok/s 98.84 ms +71.2% 0.99587 0.0281 +2.85% (Moderate) 95.8% / 100%
Comparison: Top-4 with Naive Linear $L_1$ Rescaling ($\alpha_{L_1}$) 10.10 tok/s 98.97 ms +70.9% 0.98698 0.0671 +6.94% (Distorted) 95.8% / 95.8%
Assets 7
Loading

v2.3.0 — DeepSeek-V4-Flash 284B in < 60 GiB RAM at 256K Context & 111.62 tok/s Threadripper Leaderboard

v2.3.0 — DeepSeek-V4-Flash 284B in < 60 GiB RAM at 256K Context & 111.62 tok/s Threadripper Leaderboard

Choose a tag to compare

View all tags
@synchopate synchopate released this 03 Nov 08:12
v2.3.0
721d724

🐋 v2.3.0 — DeepSeek-V4-Flash 284B in < 60 GiB RAM at 256K Context (56.56 GiB SwiGLU-H2 + Fractal-64), Single-Cycle AVX2 Q2_0 Kernels (+55%), & Community Leaderboard (111.62 tok/s Threadripper, 44.97 tok/s AWS Graviton3 ARM64, 66.5 tok/s AMD Vulkan)

Release Date: November 2026
Highlights:

  1. World's First Engine to Run a 284.33B Frontier MoE (DeepSeek-V4-Flash, 13.17B Active) Inside < 60 GiB RAM at 256K Context on a 64 GB CPU Box:
    • Analytical Second-Order SwiGLU-H2 MoE Compression + Fractal-64 (Q2_0, 3.75 bpw) MLA Transcoding (scripts/transcode_dsv4.c, scripts/transcode_dsv4_shard.py, scripts/build_fractal_moe_gguf.sh --deepseek): Shrinks DeepSeek-V4-Flash-0731 (45 layers, 256 routed + 1 shared expert/layer, MLA + Indexer Compressors) from 76.87 GiB (IQ2_XXS, which OOMs on 64 GB RAM) down to 56.56 GiB (-20.31 GiB) on disk and 58.9 GiB total VmRSS at full -c 262144 (256K) context.
    • Computes closed-form second-order SwiGLU Hessian importance weights $I_{e,j} = |W_{\text{gate}}^{(e)}[j,:]|2^2 \cdot |W{\text{up}}^{(e)}[j,:]|_2^2$ directly from the paired Gate and Up expert slices while transcoding dense Q8_0 MLA & Indexer projections (attn_q_b, attn_output_a, attn_output_b, attn_compressor_gate, attn_compressor_kv, attn_kv, indexer_compressor_gate, indexer_compressor_kv) to Block-64 Fractal-64 (Q2_0, 3.75 bpw).
  2. Single-Cycle AVX2 Variable-Shift dot_q2_0_q8_0 (+55% Speedup) & 3D L1D Tiling (wo_a):
    • Vectorizes 64-bit upper bit-plane unpacking (qh[8]) and 8×4-bit shifted sub-scales (scales[4]) in ggml_vec_dot_q2_0_q8_0_generic (ggml/src/ggml-cpu/quants.c) using single-cycle AVX2 variable bit-shifts (_mm256_sllv_epi32 / _mm256_srlv_epi32) fused into _mm256_madd_epi16(p01, vmult01) (vpmaddwd), cutting 10M-block Q2_0 × Q8_0 runtime from 86.5 ms to 55.8 ms (+55% faster).
    • Adds 3D L1D Cache Tiling (TILE_I0 = 64) in ggml_compute_forward_mul_mat (ggml/src/ggml-cpu/ggml-cpu.c) for 3D multi-head tensors (ne02 > 1, wo_a), keeping [ne00, 64] slices inside the 48 KB Zen 5 L1D cache across all 32 threads.
  3. MLA-Aware HARPCache, 8-Step Sparse Indexer Lookahead Prefetching, & $O(1)$ llama_kv_cache_dsv4 Speculative Rollback Fix (patches/deepseek-v4-flash-fractal-256k.patch):
    • Auto-detects DeepSeek MLA compressed KV tensors (ne11 == 576, ne21 == 512, LLAMA_KV_L3_WINDOW=1024) for _MM_HINT_NTA L3 isolation and prefetches (_mm_prefetch(..., _MM_HINT_T0)) sparse csa_top_k_mask rows 8 steps ahead (ic_pf = i + 8).
    • Fixes state_snapshot plane indexing (std::min(n_rs_seq, rollback + (d - n_seq_tokens)) * state_rows) and kv_csa/hca/lid->seq_rm in src/llama-kv-cache-dsv4.cpp, achieving 94.5% draft acceptance (9.67 tok/s avg / 11.50–14.69 tok/s peak) and clean EOS termination on both ./modchip chat and ./modchip claude (Claude Code).
  4. Unified ./modchip chat + Claude Code Compact Tool Snapshot (32.6 ms Slot 0 Restore):
    • Updates modchip, scripts/chat.py, and gateway/sse_gateway.py with automatic active-model detection (/v1/models), MODCHIP_COMPACT_TOOLS=1 (2,940 prefix tokens, 32.6 ms /dev/shm restore), and automatic Slot 0 restoration (_last_endpoint_was_chat) when switching between ./modchip chat and Claude Code.
  5. New Community CPU & Hybrid Benchmarks (README.md, FAQ.md):
    • AMD Threadripper PRO 7985WX (64C/128T Zen 4, 256 MB L3, 8-Ch DDR5-6000) by @aifartist (Issue #1): Full thread-scaling sweep (16/24/32/48/64 threads) + 111.62 tok/s warm decode (80.14 tok/s cold, 386.00 tok/s prefill at -t 32 4C/CCD).
    • AWS Graviton3 m7g.2xlarge (8 vCPU ARM64 Neoverse-V1, 32 MB L3, 31 GB DDR5) by @gerchik (Issue #2): 44.97 tok/s refactoring / 41.05 tok/s C++20 (101.67 tok/s prefill).
    • AMD Ryzen 9 7945HX3D + Radeon RX 9070 XT 16GB (Vulkan/RADV Hybrid) (FAQ.md Q8): 66.50 tok/s decode / 563.00 tok/s prefill at 256K context via tensor-role offloading (-ngl 99 --n-cpu-moe 30 --no-mmap).
Thread Scaling & Warm Promotion on Threadripper PRO 7985WX (@aifartist, Issue #1) -t 16 (2C/CCD) -t 24 (3C/CCD) -t 32 (4C/CCD Sweet Spot) -t 48 (6C/CCD) -t 64 (8C/CCD)
Prefill (tok/s) 238.12 314.85 386.00 tok/s 361.40 312.18
Stage 1: Cold C++20 Generation (tok/s) 58.40 68.92 73.84 tok/s 64.15 51.80
Stage 2: Cold In-Context Refactoring (tok/s) 62.10 74.50 80.14 tok/s 69.30 55.42
Stage 2: Warm Session Trie + b5 Promotion (Run 4+) — — 111.62 tok/s (+39.3%) — —
Loading

v2.2.0 — Block-64 2-Level Hierarchical Non-Linear MoE Quantization (14.35 GiB Fractal-64), 100/100 Quality Gate, & 80.14 tok/s on 8-CCD Threadripper

v2.2.0 — Block-64 2-Level Hierarchical Non-Linear MoE Quantization (14.35 GiB Fractal-64), 100/100 Quality Gate, & 80.14 tok/s on 8-CCD Threadripper

Choose a tag to compare

View all tags
@synchopate synchopate released this 02 Nov 23:13
v2.2.0
2145e5e

DRAFT Release Notes — llama-modchip v2.2.0: Block-64 2-Level Fractal-MoE Compression & 100-Point Quality Gate

Status: Verified on live Zen 5 hardware (CT 100, AMD Ryzen 9 9950X, 64 MB L3, 256K context). Saved locally and on NAS (/mnt/nas_share/nemotron-30b-cpu-256k/). Not pushed to GitHub (origin/main) until final approval.

1. Headline Results (Live Hardware Verification on CT 100)

Starting directly from the unquantized 58.84 GiB BF16 master weights (NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16.gguf), our Block-64 2-Level Fractal Compressor (Fractal-64-BF16) shrinks the model to 14.35 GiB (14,698 MiB, 3.90 bpw overall / 3.75 bpw on MoE experts) — saving -3.24 GiB on disk and -6.01 GiB of idle Resident RAM (VmRSS) while scoring 100.0 / 100.0 on the Quality & Tool-Calling Gate and running +28.0% faster (25.02 tok/s vs 19.54 tok/s).

We also ship Fractal-MXFP4-16.0GiB (15.95 GiB disk / 17.46 GiB idle VmRSS) for users re-quantizing directly from an existing Q4_0 GGUF without downloading the 58.84 GiB BF16 master, achieving 26.08 tok/s (+33.5% faster) and 100.0 / 100.0 quality parity.

Metric / Suite Baseline-Q4_0 (Official GGUF) Fractal-MXFP4 (from Q4_0) Fractal-64-BF16 (from BF16) Fractal-64-BF16 Delta
GGUF File Size on Disk 17.59 GiB (18,015 MiB) 15.95 GiB (16,328 MiB) 14.35 GiB (14,698 MiB) -3.24 GiB (-18.4%)
MoE Expert Format & Bitrate Q4_0 (4.50 bpw) MXFP4 (4.25 bpw) Fractal-64 (3.75 bpw) -0.75 bpw on experts
SQNR on True BF16 Expert Weights 21.02 dB 18.51 dB 17.78 dB +1.53 dB vs Q3_K (16.25 dB)
1. Agentic Tool-Calling (30 pts) 30.0 / 30.0 30.0 / 30.0 30.0 / 30.0 100.0% parity
2. Executable Code Gen (30 pts) 30.0 / 30.0 30.0 / 30.0 30.0 / 30.0 100.0% parity
3. Long-Context Needle (8.5K) & Math (20 pts) 20.0 / 20.0 20.0 / 20.0 20.0 / 20.0 100.0% parity
Quality & Tool-Use Subtotal (80 pts) 80.0 / 80.0 80.0 / 80.0 80.0 / 80.0 100.0% parity
Decode Speed (tok/s) 19.54 tok/s 26.08 tok/s 25.02 tok/s +28.0% (+33.5% MXFP4)
Idle Resident RAM (VmRSS, 256K ctx) 21.41 GiB 17.46 GiB 15.40 GiB -6.01 GiB (-28.1%)
Post-Benchmark Peak VmRSS (256K ctx) 21.71 GiB 18.86 GiB 17.27 GiB -4.44 GiB (-20.5%)

2. Technical Architecture of the Block-64 2-Level Fractal Compressor

2.1 Solving the 1856 × 2688 Super-Block Alignment Trap

  • In NVIDIA-Nemotron-3.5-Lightning-30B-A3B, the 46 sparse MoE layers (128 experts per layer, 90.1% of total parameters) have inner dimensions n_ff_exp = 1856 (29 × 64) and n_embd = 2688 (42 × 64).
  • Neither dimension is divisible by QK_K = 256, which causes all standard 256-block K-quants and I-quants (Q3_K, IQ3_S, IQ3_XXS) in llama.cpp to fail alignment and fall back to Q4_0.
  • By designing our 2-level fractal block around QK2_0 = 64 (30 bytes per 64 weights = 3.75 bpw), every 1856 and 2688 row divides evenly (29 and 42 blocks per row) with zero fallback tensors.

2.2 Three-Scale Self-Similar Representation (30 bytes / 64 weights)

  1. Super-Block Scale (ggml_half d, 2 bytes): Captures the macro-scale dynamic range across all 64 weights (d = max_b(s_opt[b]) / 19.0f).
  2. Shifted 4-Bit Sub-Block Scales (uint8_t scales[4], 4 bytes):
    • Divides the 64 weights into 8 sub-blocks of 8 weights (b = 0..7).
    • Maps each 4-bit nibble sc[b] ∈ {0..15} to an integer multiplier mult[b] = 4 + sc[b] ∈ {4..19}, eliminating dead near-zero sub-scale codes {0..3} and increasing effective scale resolution by +31%.
  3. Non-Linear Dyadic Codebook (uint8_t qs[16] + uint8_t qh[8], 24 bytes):
    • Encodes 64 3-bit indices (0..7) mapped to the non-linear symmetric dyadic lattice {-14, -8, -4, -1, +1, +4, +8, +14}.
    • Achieves 17.78 dB SQNR on real BF16 Nemotron-3.5 MoE weights — +1.53 dB higher than Q3_K (16.25 dB) and +2.91 dB higher than IQ3_S (14.87 dB).

2.3 Zero-Overhead Fused AVX2 + _MM_HINT_NTA Dot-Product Kernel

  • In ggml_vec_dot_q2_0_q8_0_generic (ggml/src/ggml-cpu/quants.c), the 64 high bits in qh[8] are interleaved in 2 instructions via _mm_unpacklo_epi64(qh_lo, _mm_srli_epi16(qh_lo, 1)), combined with qs[16], and translated to signed int8_t fractal values via _mm256_shuffle_epi8(lut256, idx).
  • After _mm256_maddubs_epi16, each 8-weight sub-block occupies 4 consecutive int16_t lanes. We multiply by mult[b] ∈ {4..19} directly inside _mm256_madd_epi16(p01, vmult01) (vpmaddwd), fusing sub-block rescaling and int16 -> int32 horizontal accumulation into a single 1-cycle instruction.
  • Cold MoE expert blocks are prefetched with _MM_HINT_NTA (non-temporal cache-bypass hint), preventing the 46 MoE layers from evicting the 6 GQA KV caches, 23 Mamba-2 SSM states, and shared experts from the 64 MB Zen 5 L3 cache.

3. Scaling Projection: DeepSeek V4 Flash & DeepSeek V4 Pro on CPU

Because our Block-64 2-Level Fractal Compressor (3.75 bpw on routed experts / 3.90 bpw overall) operates on 64-element blocks without requiring 256-element K-quant alignment and starts directly from BF16/FP8 master weights:

Model Architecture Total / Active Params Master BF16 / FP8 Size Standard Q4_0 / Q4_K Size Our Fractal-64 (3.75 bpw MoE) Target CPU RAM Tier
Nemotron-3.5-Lightning-30B-A3B (Verified Live) 30B / 3B 58.84 GiB (BF16) 17.59 GiB 14.35 GiB (15.40 GiB idle RSS) Fits easily in 24–32 GB RAM
DeepSeek V4 Flash (Fine-Grained Hybrid MoE) ~200B–236B / ~16B–21B ~200–236 GiB (FP8) ~118–135 GiB ~91–107 GiB Fits inside 128 GB DDR5 consumer workstation (9950X)
DeepSeek V4 Pro / V3-671B (Full Flagship MoE) 671B / 37B ~671 GiB (FP8) ~380–404 GiB ~305 GiB (or ~248 GiB with 3.0 bpw 2-Level Fractal) Fits inside 256–384 GB EPYC / Threadripper

4. Files Included in v2.2.0 (Backed Up to NAS)

  • patches/zen5-nemotron-256k-optimizations.patch — Updated with block_q2_0 (30 bytes, Block-64 2-Level Fractal), quantize_row_q2_0_ref, dequantize_row_q2_0, AVX2 + _MM_HINT_NTA ggml_vec_dot_q2_0_q8_0_generic, and _MM_HINT_NTA ggml_vec_dot_mxfp4_q8_0. Passes git apply --check cleanly against d2f83055.
  • scripts/build_fractal_moe_gguf.sh — Supports --profile fractal64 (14.35 GiB from BF16), --profile mxfp4 (15.95 GiB from Q4_0), and --profile mxfp4_q5 (16.10 GiB).
  • benchmarks/eval_quality_and_tools.py — 100-point automated Quality, Tool-Calling, Executable Coding, Long-Context Needle (8.5K tokens) & Speed/RAM release gate.
  • workspace/eval_baseline_q4_0.json, workspace/eval_fractal_mxfp4.json, workspace/eval_fractal_64_bf16.json, and workspace/QUALITY_GATE_REPORT.md.
Loading

v2.1.0 — Stockfish-PVS v2 (Sweet Spot, M=7), 4 Unlocked RouteCrack Techniques (60.4 tok/s Peak) & Apple Silicon / Fedora Support

v2.1.0 — Stockfish-PVS v2 (Sweet Spot, M=7), 4 Unlocked RouteCrack Techniques (60.4 tok/s Peak) & Apple Silicon / Fedora Support

Choose a tag to compare

View all tags
@synchopate synchopate released this 02 Nov 14:24
v2.1.0
cc419d5

Release Notes — nemotron-30b-cpu-256k & llama-modchip

📄 Quick Links: Overview (README.md) · Technical Paper (ARCHITECTURE.md) · 96+ RouteCrack + Stockfish-PVS Techniques (speculative/ROUTECRACK_TECHNIQUES.md) · Engineering FAQ (FAQ.md)

🚀 v2.1.0 — Stockfish-PVS v2 (Sweet Spot, M=7), 4 Unlocked RouteCrack Techniques (60.40 tok/s Peak), & Multi-Platform Expansion (Apple Silicon M4 Pro + Fedora Ryzen 8745HS)

Release Date: November 2026
Highlights: Combines Chess-Engine Principal Variation Search (PVS) + 4-Way Bucketed Zobrist Transposition Tables (StockfishDrafter, 1.64 MB L2/L3) with the RouteCrack statistical speculative engine, unlocking 4 previously dormant RouteCrack techniques (including Singular Horizon Extension M=7), pushing live 25K-context code refactoring on AMD Ryzen 9 9955HX (0 GPUs) to 48.39 tok/s average / 60.40 tok/s peak (mean len = 6.90 tokens/step, 96.39% acceptance) and 85.00% draft acceptance on technical prose, while adding community-verified support for Apple Silicon (M4 Pro 24 GB) and Fedora 44 (AMD Ryzen 7 8745HS GEEKOM A8).

1. ♟️ Stockfish-PVS v2 (Sweet Spot, M=7) Speculative Acceleration (rc_stockfish.h, ngram-map.cpp)

On a CPU-hosted Sparse MoE + Mamba-2 hybrid (Nemotron-3.5-30B-A3B), every drafted token activates additional routed experts from DDR5 RAM (~0.30x single-token cost per drafted token) and any rejected token triggers an 18.4 MB Mamba-2 recurrent state rollback (slot.spec_ckpt). Consequently, net wall-clock speedup follows the CPU-MoE Golden Equation:
$$\text{WallGain} = A - 0.30 \times D = 0.70 \times A - 0.30 \times W_{\text{wasted}}$$

During the RouteCrack research campaign (96+ techniques documented in speculative/ROUTECRACK_TECHNIQUES.md), several high-recall predictors could not be enabled in production because static predictors trained on the same corpus make correlated errors when confidence drops below 84%.

StockfishDrafter (speculative/cpp/rc_stockfish.h) solves this by maintaining a 1.64 MB L2/L3-resident (2.5% of 64 MB L3), zero-allocation Chess-Engine Search state updated continuously on the live session stream:

  • 4-Way Bucketed Zobrist Transposition Tables (tt5, tt4, tt3 = 384 KB) with 16-bit verification signatures (sig16), Top-2 candidate tracking (tok1/cnt1, tok2/cnt2), and Stockfish age-based generational replacement.
  • Countermove Table (1,024 KB, 65,536 × 4) & Continuation History Table (256 KB) capturing immediate `(t_{-2}, t_{-1}) \to t_0$ and $t_{-1} \to t_0$ live session dynamics.
  • Principal Variation Search (PVS) with Late Move Reductions (LMR): Searches the principal candidate at full horizon depth while applying LMR depth reductions and stronger certainty gates to secondary moves (m > 0).

The 4 RouteCrack Techniques Unlocked by Stockfish-PVS v2:

  1. Technique #1 — Online Session Trie Rescue (rc_trie_v2.h + Stockfish-PVS Gate):
    • Rescues novel in-session identifiers, variable names, and file paths when the static 8-level FNV-64 Hash Cascade misses (best_conf < min_c), provided OnlineTrieV2 has session support (ot_cnt >= 2, ot_conf >= 0.78) and is confirmed by the orthogonal Stockfish Zobrist TT (sf_has_cand, or ot_cnt >= 4 && ot_conf >= 0.90).
    • Marginal Precision: 92.19% (+59 accepted tokens, only 5 wasted).
  2. Technique #2 — Top-4 Cascade Disambiguation via Orthogonal Zobrist TT:
    • Inspects tok[1..3] in hc_top4_k*.bin whenever tok[0] falls in the uncertain band (0.55 <= c0 < min_c). If an alternative alt_t has a strong live-session Zobrist TT match (alt_sf >= 640, exact tt5/tt4 hit) that strictly beats tok[0] (alt_sf > sf_t0 + 250), Stockfish-PVS promotes alt_t.
  3. Techniques #5 & #89 — 11,382-Entry Multi-Token Macro Codebook (macro_codebook.bin):
    • Enables multi-token structural chains (1..15 tokens long, >= 90% empirical precision) at $d=0$ and unlocks Singular Horizon Extension across the macro chain.
    • Marginal Precision: 91.81% -> 96.44% (+92 accepted tokens, only 8 wasted).
  4. Techniques #17 & #73 — Singular Horizon Extension (--spec-ngram-simple-size-m 7) & Consensus Integrity Fix:
    • Extends the draft horizon from $D=4$ (M=5) up to $D=7$ (M=7) exclusively on "singular" beams: verified 2-hit in-context trajectories (true_consensus_hits == d), Macro Codebook chains (used_macro), or Singular Cascade + Stockfish TT locks (best_conf >= 0.95 && sf_best >= 600).
    • Consensus Integrity Fix: Separates true_consensus_hits (strictly 2-hit in-context matches) from consensus_hits so single-hit context matches rescued by RouteCrack at $d=0..2$ cannot masquerade as 2-hit trajectories at $d \ge 3$.
    • Marginal Precision at $D=5..7$: 98.18% (M=6) to 98.74% (M=7: +78 accepted tokens, 1 wasted, -7 rollbacks).

2. 📊 Verified Benchmarks: Held-Out Ablation (77,456 Tokens) & Live CT 100 Telemetry

A. Held-Out Corpus Ablation (77,456 Nemotron-3.5 Tokens, 351 Contexts — speculative/cpp/test_synergy.cpp)

Speculative Configuration Drafted ($D$) Accepted ($A$) Wasted ($W$) Token Precision 0-Rollback Steps Mamba-2 Rollbacks CPU-MoE WallGain
1. RouteCrack Baseline (M=5, Static Cascade) 6,332 5,440 892 85.91% 87.31% 482 +3,540.4
2. + Stockfish-PVS v1 (M=5, Zobrist TT + PVS) 6,329 5,459 870 86.25% (+0.34%) 87.84% (+0.53%) 461 (-21) +3,560.3 (+19.9)
3. + Stockfish-PVS v2 Sweet Spot (M=7, +Trie +Top4 +Macro +Singular) 6,555 5,663 892 **86.39% (+0.48%) ** **88.35% (+1.04%) ** **450 (-32) ** **+3,696.5 (+156.1) **
  • +223 additional accepted tokens (5,440 -> 5,663) with ZERO additional wasted tokens (892 -> 892, 100.0% net marginal precision) and -32 fewer Mamba-2 rollbacks (482 -> 450).

B. Live 5-Stage Production Benchmark on CT 100 (AMD Ryzen 9 9955HX, 0 GPUs, 22.2K -> 27.7K Context)

Stage & Workload (Nemotron-3.5-30B-A3B-Q4_0) Active Context M=5 Speed (Avg / Peak) Stockfish-PVS v2 (M=7) Speed (Avg / Peak) Draft Acceptance (Acc / Drafted) Mean Spec Length (tok/step)
Atto 1A: Warm TTFT + <tool_call> Read (50 tok) 22,299 tok 30.84 tok/s (2.60 s) 34.52 tok/s (2.45 s wall) 83.33% (35 / 42) 4.18 tok/step
Atto 1B: Prefill harp_cache.py (3,089 tok) + 500 tok Code 25,388 tok 40.91 / 45.43 tok/s (12.22 s) 48.39 tok/s avg / 60.40 tok/s peak (10.33 s) 96.39% (401 / 416) 6.90 tok/step (up from 5.47)
Atto 2A: Thread-Safe Refactoring (400 tok Mixed Code) 26,419 tok 27.80 / 41.20 tok/s 28.61 tok/s avg / 48.47 tok/s peak 86.10% (223 / 259) 4.66 tok/step
Atto 2B: From-Scratch Concurrent Stress Test (350 tok) 26,431 tok 18.61 / 21.84 tok/s 18.66 tok/s avg / 21.89 tok/s peak 56.38% (53 / 94) 2.33 tok/step
Atto 3: Architectural & Security Audit in Italian (350 tok) 26,959 tok 18.58 / 20.50 tok/s (80.6% acc) 18.76 tok/s avg / 20.78 tok/s peak 85.00% (34 / 40) (up from 67.4%) 2.48 tok/step
Live Multi-Turn Claude Code Session (4,096 tok Turn 3) 27,713 tok 19.00 / 30.87 tok/s (59.1% acc) 20.68 tok/s avg / 39.28 tok/s peak (-17.5 s) 67.98% (1,055 / 1,552) 2.68 tok/step

3. 🌍 Multi-Platform Support & SSE Gateway Enhancements (Co-Authored with @jarorosFork / Oleksandr Nazar)

Integrated community contributions from Oleksandr Nazar (@jarorosFork) (commits 9509f4c, 93b6381, d740063 -> 6694b0b):

  • apple-silicon/ — Apple Silicon (M4 Pro 24 GB Unified Memory) Metal Stack:
  • fedora/ — Fedora 44 / AMD Ryzen 7 8745HS (GEEKOM A8, 8C/16T Zen 4, 64 GB DDR5):
  • gateway/sse_gateway.py — Tool-Call Resilience & Cross-Platform Portability:
    • _write_as_bash() Fallback: Automatically converts raw multiline Write tool calls into safe cat << 'EOF' > ... Bash commands via shlex.quote() when unescaped quotes in 100+ line code payloads break LLM JSON string serialization.
    • _coerce_xml_param() Schema-Aware Coercion: Coerces XML <function=Name><parameter=key>value</parameter></function> parameters to native JSON integers, numbers, booleans, arrays, or objects using the client's tool JSON schema (_extract_param_types()).
    • Portable HARP_CACHE & Environment Config: Safely guards /dev/shm usage on macOS (os.path.isdir("/dev/shm")) and adds GATEWAY_HOST, GATEWAY_CORS_ORIGIN, and GATEWAY_SYSTEM_PROMPT environment variables.

4. 🛡️ Microarchi...

Read more
Loading