Skip to content

v2.5.4 — Bench/Chat UX Fixes, Non-BMI2 VQ3_0 Fix, Cross-Platform Onboarding, & Benchmark Precision

Choose a tag to compare

@synchopate synchopate released this 13 Nov 07:54

v2.5.4 — ./modchip bench & Chat UX Fixes, Non-BMI2 VQ3_0 Scale Unpack & Kernel Test Fixes, Cross-Platform Windows/macOS/Docker Onboarding, & Benchmark Table Precision

Highlights

  1. ./modchip bench Crash Fix & Dynamic Neutral System Prompt in ./modchip chat (scripts/chat.py):

    • Fixed a NameError: name 'prompt_label' is not defined (and remote-endpoint UnboundLocalError on cpu/ram) at the end of run_benchmark_suite() when generating the pre-filled GitHub Leaderboard submission URL in ./modchip bench and /bench.
    • Replaced the hardcoded Italian DeepSeek-V4 / Ryzen 9 9955HX system prompt in ./modchip chat with build_chat_system_prompt(), which dynamically reflects the active served model (Nemotron, DeepSeek-V4, Qwen, etc.) and detected host hardware, responds in the user's language, and refreshes on /model.
  2. Non-BMI2 VQ3_0 Scale Unpack Fix, AVX512F Portability, test-quantize-fns Coverage, & Transcode Layout Parity (vq3/vq3_codebook.h, patches/lut-vq3-64-avx512-vbmi.patch, patches/dsv4-ipc-hardening.patch, vq3/vq3_enc_lib.c, vq3/test_vq3_ggml_kernels.c):

    • Fixed a bit-carry bug (|= -> +=) in the #else non-__BMI2__ fallback of ggml_vq3_unpack_scales (m_all = 0x0202020202020202ULL), eliminating scale unpacking error when compiled without -mbmi2 (such as MSVC or scalar non-BMI2 builds; distributed Linux -avx512 and -avx2 binaries use __BMI2__ _pdep_u64 and were unaffected, and zero GGUF weights changed).
    • Replaced AVX512DQ _mm512_extractf32x8_ps in ops.cpp with pure AVX512F _mm512_extractf64x4_pd, added GGML_TYPE_VQ3_0 thresholds to tests/test-quantize-fns.cpp (verified PASS on both -avx2 and -avx512 GLIBC_2.30 builds), aligned dequantize_row_q2_0_local in vq3/vq3_enc_lib.c (--aggressive optional transcoder) to quants.c's transposed qh layout, and tightened vq3/test_vq3_ggml_kernels.c to validate both near-zero cancellation and positive-signal relative error (exit 0 under both -mbmi2 and -mno-bmi2).
  3. Windows 11 Speculative Flag Detection (Issue #33), -Target nemotron Idempotency, & -Cuda ISA Auto-Detection (windows/modchip_win.ps1, windows/build_patched_engine.ps1, patches/zen5-nemotron-256k-optimizations.patch, windows/modchip_dsv4.ps1):

    • Fixed modchip_win.ps1 matching --spec-ngram-size-n inside llama-server.exe --help removal warnings before checking --spec-ngram-simple-size-n (thanks to @kevinblo in Issue #33), and removed legacy --draft-max.
    • Guarded POSIX <sys/mman.h> / <fcntl.h> / <unistd.h> includes directly with #if defined(__linux__) in zen5-nemotron-256k-optimizations.patch, made build_patched_engine.ps1 -Target nemotron idempotent across repeated runs, and replaced unconditional -DGGML_AVX512=ON under -Cuda with host AVX-512 VBMI auto-detection so AVX2 + RTX PCs never hit STATUS_ILLEGAL_INSTRUCTION.
  4. macOS (Darwin), .modchip_workspace, Docker /dev/shm Guard, & Standalone llama_modchip.py Parity (install.sh, modchip, scripts/build_zen5_llama.sh, apple-silicon/run-server.sh, scripts/llama_modchip.py):

    • Replaced GNU-only realpath -m and bare sha256sum with portable POSIX/Python + shasum -a 256 helpers, persisted custom --workspace paths via .modchip_workspace across ./modchip and ./scripts/chat.py, bypassed /dev/shm slot-snapshot allocation in 64 MiB Docker containers (< 1 GiB tmpfs), routed Darwin dsv4 source builds to patches/apple-silicon and Darwin ./modchip start to ${WORKSPACE_DIR} with unified-memory --load-mode mmap, and added RAM-safe -c 32768 -ub 512, --load-mode, and DSV4_MOE_CFG_PATH defaults to standalone llama_modchip.py --print-cmd (marking boot-time #2/#9 controls as BOOT in --tui).
  5. Benchmark Table Separation, Documentation Precision, & 7-Repo Hugging Face Sync (README.md, benchmarks/QUALITY_GATE_VQ3_64.md, docs/NEMOTRON_OPTIMIZATIONS.md, ARCHITECTURE.md, FAQ.md):

    • Explicitly separated Raw Pure Decode (N=1) from Served Gate Decode (Stockfish-PVS E2E) and In-Context Refactor Burst (M=7), footnoted the 13.08 GiB --all-up-exps eval_nemotron_vq3_64.json telemetry and Abliterated-VQ3-64 (401 total tensors = 76 re-quantized + 325 copied), added @kevinblo's Ryzen 9 9950X3D (9.47 tok/s Top-4 / 8.11 Top-6 284B & 24.92 tok/s Nemotron Q4_0) leaderboard entries, and synchronized all 8 v2.5.4 GLIBC_2.30 binaries + SHA256SUMS + model cards across all 7 Synchopate/* Hugging Face repositories (0 GGUF weight changes).

Portable GLIBC_2.30 Linux x86_64 Binaries (SHA256SUMS)

Built inside Ubuntu 20.04 (GLIBC_2.30 max symbol, -static-libstdc++, zero libssl/libcurl dependency; requires OpenMP libgomp1 / libgomp):

  • *-avx512: AMD Zen 4/5 (Ryzen 7000/8000/9000, EPYC Genoa/Turin, Threadripper 7000) & Intel Ice Lake / Sapphire Rapids (avx512f + avx512vbmi; -DGGML_AVX512_BF16=OFF).
  • *-avx2 (and unsuffixed *-linux-x86_64): Pre-AVX512 x86-64-v3 CPUs (Haswell, Skylake, Alder/Raptor Lake, Zen 1/2/3 — 0 %zmm instructions).
c9fe218d3be566e38c7677c903013ee6d90fa3663aa87d3bbc78c2109e8ac2c4  llama-cli-modchip-linux-x86_64
c9fe218d3be566e38c7677c903013ee6d90fa3663aa87d3bbc78c2109e8ac2c4  llama-cli-modchip-linux-x86_64-avx2
6c9317d68f47d4c9c23740ff597d56cafb83ad816f8581cb5315bc11ebb6e163  llama-cli-modchip-linux-x86_64-avx512
fa214d0f86a37934dcf9c49e32e8d40d5406e6b970d968aac1e093b952c80bdd  llama-perplexity-modchip-linux-x86_64-avx2
282183f33732666ffd34ecbc5258de107267e21797f5205c70ff107bc1d81d84  llama-perplexity-modchip-linux-x86_64-avx512
09f2a1accacb1454aaf0d9a8d75dda8f7cca36df81138eb57bfee45137ef07e4  llama-server-modchip-linux-x86_64
09f2a1accacb1454aaf0d9a8d75dda8f7cca36df81138eb57bfee45137ef07e4  llama-server-modchip-linux-x86_64-avx2
7cf2563ede8670b482ccc7e108d9810023f994a6ce9a3d7bd320f8393dd26304  llama-server-modchip-linux-x86_64-avx512