bins v0.8.1 (Linux)

bins-v0.8.1 — Linux asset (2026-10-01)

Fixes a routing-table hot-swap race that could kill llama-server at runtime.

What changed

Only two files differ from bins-v0.8.0; every other file is byte-identical.

file	v0.8.0	v0.8.1
libggml-base.so.0.22.0	c694817cfc0f45622559b7b4e42ba700	f79a07e4673bcc7addc6739f90be289b
libllama-server-impl.so	8c90e08e8e9b51b3218698b81a1cfaa6	7fc9211d81ee5c091d1c4a079de858cf

Asset: llama_bins.tar.gz — 1,797,377,953 B, md5 b311f302c53e395ab7a4eb9289517272

The bug

POST /moe-set-domain (runtime router-table switch) used to empty the shared router map
in place (clear()), then refill it layer by layer. Inference threads read that same map
(g_router_map[il]) concurrently → out-of-bounds access → the engine process dies.

• Windows: ggml-base.dll / 0xc0000005 / offset 0x175ae (same offset 2026-09-19 and 2026-10-01)
• Linux (AutoDL RTX 3080Ti 12G, this release's test bed): the unpatched release libs died
  on the 2nd /moe-set-domain call.

The fix

• Build the new table in a private draft, then publish it with a single atomic pointer swap:
  ggml_backend_router_commit() (atomic_store, release). Readers take a shared_ptr
  snapshot (moe_router_snapshot()), so a table in use cannot be torn down mid-read.
• server.cpp calls commit() after all set_layer() calls in /moe-set-domain.
• g_prefill_done check-then-insert is now locked (same class of race).

Verification (RTX 3080Ti 12G, Qwen3.6-35B-A3B-UD-IQ2_M, -c 32768)

run	patch state	40 mid-generation table switches
baseline	v0.8.0 libs	crashed on switch #2
this release	v0.8.1 libs	40/40 OK, 40 x router map committed, health 200, VRAM peak 10,368 MiB

Also re-verified with the shipped artifact itself (tarball extracted over a clean install):
40/40 switches, no crash, coherent long generation, 34–37 t/s short-prompt decode.

Windows asset (llama_bins_win.zip) — added 2026-10-01

Same fix for the native-Windows package. Built by swapping the two rebuilt DLLs into the
previously shipped package content:

file	v0.8.0	v0.8.1
ggml-base.dll	b1d4132854ddba790409cf701b1c009d (716,288 B)	b866304e66142c821ee4e43ff69ce1f4 (719,872 B)
llama-server-impl.dll	3073088ec6c197e571f7004808056a45	df660d094d2eb9649fdc0a058cddbb33

Everything else in the zip is byte-identical to the v0.8.0 package as published
(20 entries: engine DLLs + bundled CUDA 12.8 / MSVC runtime DLLs; verified entry-by-entry
against the published asset — ggml.dll, ggml-base.dll, llama.dll, ggml-cpu.dll,
llama-common.dll, cudart64_12.dll, msvcp140.dll all match; the two 10 KB launcher
stubs were taken from the published asset so they match too).

Asset: llama_bins_win.zip — 808,182,800 B, md5 ae8aaa1375bb9ab2b614dfffcb20be65

Windows verification (210 = Win10 + RTX 3060 12G, run_v080f.bat, -c 262144, IQ2_M)

phase	result
probe	9.6 s, garbled 0.000, normal reply
3-round Chinese dialog	24.6 / 27.1 / 27.8 t/s, garbled 0.000-0.012
4.5K-token summary	7,044 prompt tokens, 28.9 t/s, answer "分为 60 段" correct, garbled 0.024
3-way concurrency	14.2-20.1 t/s, wall 82.4 s, no crash
swap storm (earlier session)	25 rounds + 110 mid-generation /moe-set-domain with zero crashes; 40 × router map committed

Note on the matrix's garbled_ratio: it counts \n and CJK quotes as "bad characters",
so a poem (>0) or a multi-line answer can score >0 without any real garbling. Verified by
re-running the flagged request and inspecting raw characters (only \n + “/”).

• MOE_L2_TABLE_PUSH now defaults to on (since PyPI 0.14.1) — the swap path it enables is
  exactly what this release fixes. Set MOE_L2_TABLE_PUSH=0 to keep the old behaviour.
  Domain-driven switching (/moe-set-domain) was already on by default in v0.8.0, and it is
  the code path that could kill the engine before this release.
• The Python side (PyPI 0.14.0) additionally calibrates the A3 expert-cache auto-budget for
  Linux (hard cudaMalloc limit), which fixes CUDA error: out of memory after a few long
  generations on 12 GB cards. That fix needs no new binaries.
• Windows asset (llama_bins_win.zip) carries the same atomic-swap fix — see the Windows asset section above.

bins v0.8.0

moe-l2 bins v0.8.0 (llama.cpp engine update)

What changed

• Engine rebuilt from the current llama.cpp tree (region-42 clean tree; was: stale v070b tree).
• Fixes Windows garbage/slash output: root cause = missing ggml_cuda_get_backend_stream export in the stale tree; stream parse returned null, D2D expert copies fell onto the default stream and raced with consumer kernels under Windows multi-stream scheduling (Linux single-stream happened to dodge it).
• Windows platform fixes for the new tree: backend registry/dl merged into ggml-base for DLL link sanity; dummy ggml export; mincore + /proc diag guard; setenv -> _putenv_s; RSS statm short-circuit; dllexport for llama_mmap_evict / llama_internal_get_tensor_map.
• CUDA architectures: 60;61;70;75;86;89;120a (llama-server + llama-cli).

Verified (Windows, RTX 3060 12G, Qwen3.6-35B-A3B-UD-IQ2_M, expert-cache LRU form)

• Short 6-round session: clean Chinese output, no garbage (was: slash spam)
• 4.3K context, thinking off: correct summary
• 3-way concurrent: all pass
• Cold start: 27.6 t/s (bar >= 21 t/s); warm: 37.2 t/s
• Zero slash/noise characters in every response

Assets

• llama_bins.tar.gz (Linux, 1.79 GB)
• llama_bins_win.zip (Windows exe + DLL, 226 MB)

bins-v0.7.0

bins-v0.7.0 — IQ1_M fix + NCCL multi-GPU + multi-arch (2026-08-28)

Pre-built CUDA llama-server binaries for moe-l2. moe-l2 download-bins fetches this automatically (moe-l2 0.11.0).

What's new

🔥 IQ1_M quantization fix (Qwen4exp / Qwen3.8-Flash-Next support)

• Qwen4exp 125B (512 experts/layer, IQ1_M) previously crashed with GGML_ABORT in MMQ quantize (no IQ1_M case in quantize_mmq_q8_1)
• Fixed by routing IQ1_M to MMVQ (which supports it) + batch-cap split (MMVQ_MAX_BATCH_SIZE chunking)
• Now runs on 4090 at 19.8 t/s (was: crash; no-cache baseline 4.6 t/s)

🚀 NCCL multi-GPU support restored

• libnccl.so.2 bundled in package; --split-mode layer/row/tensor works again
• Multi-card VRAM aggregation for larger models (e.g. dual-GPU Qwen3.8-Flash-Next 74.5G)

🖥️ Multi-arch single binary

• CMAKE_CUDA_ARCHITECTURES="61;75;86;89;120a" — GTX 1080 (sm_61) through RTX 50-series (sm_120a)
• CUDA 12.8, AVX2 (no AVX512 — fixes SIGILL on no-AVX512 CPUs), GGML_BACKEND_DL=ON
• ⚠️ CUDA 12 runtime requires NVIDIA driver ≥ 570

Verified (2026-08-28, full-chain moe-l2 start --gpu, per-domain table switch + A3 cache)

GPU	Model	Speed	Notes
RTX 4090	Qwen4exp 125B IQ1_M	19.8 t/s	cache hit 97.3% (32768 slots), 0 crashes
RTX 4090	Qwen3.6-35B IQ2_M	56.3 t/s	vs v0.6.0 44-48 (+17-28%)
RTX 4090	DS-V2-Lite 16B	143.5 t/s	vs 133.2 (+8%)
RTX 2080 Ti	Qwen3.6-35B IQ2_M	40.3 t/s	cache hit 95.1%, no regression vs 37-39

All outputs verified clean (no garbage). 3 models × 3 rounds on 4090, plus 2080 Ti release validation.

Package contents

• 11 bin/lib (llama-server multi-arch + shared libs)
• cuda-libs (libcublas / libcublasLt / libcudart, real files + symlinks)
• libnccl.so.2.15.5 (multi-GPU runtime dependency)

Install

pip install -U moe-l2
moe-l2 download-bins        # fetches llama_bins.tar.gz from this release
moe-l2 start --model model.gguf --gpu

SHA256: 08db7f828ecb129ccc0f6c4ab09fa036b8111ea8a675e3b72a967c2efa03116b (md5 ae54975f)

bins-v0.6.1

bins-v0.6.1 — AVX2 兼容修复（多架构）

修复

• AVX512 兼容性：v0.6.0 二进制在无 AVX512 的 CPU（如 Intel 桌面 12 代 Alder Lake）上启动即崩（SIGILL，退出码 132）。v0.6.1 改为 AVX2 编译（GGML_AVX512=OFF），兼容所有 x86_64 CPU（2013 年后均支持 AVX2）。
• 多架构 CUDA：sm_61 / sm_75 / sm_86 / sm_89 / sm_120a（覆盖 GTX 10 系 / 20 系 / 30 系 / 40 系 / 50 系）

实测（2026-08-26，Qwen3.6-35B-A3B UD-IQ2_M）

环境	显卡	速度
原生 Linux	RTX 2080 Ti（sm_75）	22.3 t/s（冷启动）
Win10 + WSL2（无 AVX512 CPU）	RTX 3060 12G（sm_86）	19.5 t/s（稳定）

安装

moe-l2 download-bins 自动拉取最新。

bins-v0.6.0

bins-v0.6.0 — llama.cpp binaries (moe-l2)

C++ layer (5 files)

• expert-cache.cuh: EXPERT_CACHE_MAX_SLOTS 16384 -> 32768 (2x cache slot cap)
• expert-cache.cu: soft_resize (retain-hot-experts: domain switch grows cache instead of clearing, old-domain hot experts stay) + per-slot lock optimization (lock-free prescan get + single-slot lock; locked-victim set + double-check; atomic clock — removes global-lock serialization, restores concurrent correctness)
• ggml-backend.cpp: prefill dedup skip (already-present experts not re-copied on domain switch)
• ggml-cuda.cu: get_proc_address whitelist registration for soft_resize
• server.cpp: /moe-set-domain retain param

Python layer (3 files)

• gate.py: v2 multi-domain retain pool (default single-table after 4090 measurement; MOE_L2_POOL_SIZE=3 opt-in)
• vram_adaptive.py (new): VRAM-adaptive main table top-k (<=12G = top-75 / >12G = top-100)
• domain_router_flywheel.py: DEFAULT_TOP_K 75 -> 100

User-visible changes

• Cache slot cap doubled (16384 -> 32768)
• Retain hot experts across domain switches — switch-back first request +30% (measured on 2080Ti, Qwen)
• Default single-table operation (retain pool off by default)
• Concurrent correctness restored — v0.5.0 shipped lock-free and DS concurrent output was garbage; per-slot locks fix it

Measured (2026-08-19)

• Full-chain regression: 2080Ti / 4090 / 5090 — all outputs OK, no garbage, no deadlock
• Same-machine A/B (4090, same script/scenario): Qwen 31.76 t/s vs unlocked 32.42 (≈parity, noise), DS 140.43 t/s vs unlocked 113.15 (+24%)
• Bare-server 4-way concurrency 8/8 OK, no deadlock (unlocked version: DS concurrent = guaranteed garbage)

Release checklist

• md5 (both local + cloud copies): eb82de696074e5f77bf2d7e8f8500953
• Multi-arch build: sm_61/75/86/89/120a (CUDA 12.8, no NCCL)
• Verified: structure/symlinks intact, ldd no missing libs, health OK, output sanity OK

bins-v0.5.0

moe-l2 prebuilt CUDA binaries v0.5.0 (multi-arch sm_61-120a)

Includes: selective pin (router-map driven) + C-scheme per-domain table switch (POST /moe-set-domain) + GPU cache prefill + on-demand pin main path + expert-page eviction v3.1 + layered pin + A3 cache 2048 slots + P0 fix (expert-cache D2D padding + concurrent lock) + DS crash fix (cache slot resize on table switch) + flywheel table persistence (A-fix).

Verified 2026-08-16 on 2080 Ti / 4090 / 5090 (clean outputs): DS-V2-Lite 83-88 / 127-137 / 145-153 t/s; Qwen3.6-A3B 13-25 / 20-34 / 45-51 t/s. Asset: llama_bins.tar.gz (1,564,119,827 bytes).

bins-v0.4.1

bins-v0.4.1 (2026-08-14)

New features

• Per-model flywheel router maps (flywheel B): each model now
  converges its own dynamic routing table
  (domain_router_map_flywheel_{model_id}.json); models no longer
  pollute each other's hot-expert statistics.
• Domain hotness ranking: flywheel tracks per-domain request
  frequency; hot domains stay resident within the VRAM budget,
  cold domains fall back to on-demand pin.
• First-run router auto-collect: models without a router map are
  automatically profiled on first use (~2-3 min, llama-cli EXPERT
  log), generating the model's own table — no manual table needed.
• Qwen router-table fix: Qwen now uses its correct 40-layer table
  (previously driven by the 43-layer V4 table); measured 31.56 t/s
  on 2080 Ti (vs 24-27 t/s before).

P0 fix

• Expert-cache D2D copy now carries es+padding; concurrent
  cache-set protected by a mutex. Cache-hit reads no longer return
  garbage from padding bytes. Verified full-chain on 2080 Ti &
  4090: DeepSeek-V2-Lite 133 t/s, Qwen3.6-A3B 44-48 t/s (4090).

Build

• Multi-arch sm_61/75/86/89/120a, CUDA 12.8, includes llama-cli
  (required for router auto-collect) + cuda-libs.

bins-v0.4.0

Multi-arch prebuilt binaries (sm_61/75/86/89/120a, CUDA 12.8, no NCCL) for moe-l2 v0.8.0.

Includes:

• selective pin (router-map driven): pin only top-K experts per layer — V4 RSS 84 GB → 10.4 GB (↓88%), speed 30.9 t/s zero regression
• GPU cache prefill (phase 2): cold-start round1 10.7 → 19.7 t/s (+84%)
• flywheel router-map auto-learning (--router-map)
• on-demand pin main path + dynamic pin set (low-memory mode, MOE_L2_LRU=1) + expert-page eviction v3.1 + A3 cache 2048
• cuda-libs (libcublas/libcublasLt/libcudart, no libnccl)

New measured baseline: V4 = 30.9 t/s (previous 10.1 t/s was the vanilla llama.cpp build; moe-l2 optimized build was always ~30 t/s). 2080 Ti full-chain: Qwen 52.07 t/s, DS-V2-Lite 94.95 t/s.

asset: llama_bins.tar.gz (2.0 GB, md5 44a2f1c71cd2f71e26297053c0ae5f8e)

bins-v0.3.2

Multi-arch prebuilt binaries (sm_61/75/86/89/120a, CUDA 12.8, no NCCL) for moe-l2 v0.7.2.

Includes:

• on-demand pin main path (whole-tensor registration, fastest: V4 10.1 t/s @ 82 GB RAM)
• dynamic pin set (low-memory mode): per-expert/group registration + LRU eviction + layered pin (MOE_L2_PIN_LAYERS) — V4 RSS 84 GB → 17-24 GB (MOE_L2_LRU_MAX_EXPERTS), 4-5 t/s on V4, full speed on small models
• expert-page eviction v3.1 (MOE_L2_LRU_MAX_EXPERTS=N)
• A3 expert cache 2048 slots
• cuda-libs (libcublas/libcublasLt/libcudart, no libnccl)

asset: llama_bins.tar.gz (1.49 GB, md5 840c05d14c3bd9953aa7feac9ac40c8f)

bins-v0.3.1 — on-demand pin main path

bins-v0.3.1 — on-demand pin main path (2026-08-07)

One multi-architecture binary for all NVIDIA consumer GPUs: sm_61 (GTX 1080) → sm_120a (RTX 50), built with CUDA 12.8. Fetched via moe-l2 download-bins.

What's new

• On-demand pin main path: lazy mmap load + first-touch merge-registration of the whole expert tensor (fixes CUDA 11.8 cross-register-range cudaMemcpyAsync crash)
• A3 expert cache raised to 2048 slots — universal gain: Qwen +7~11%, DS +4%, V4 +6% (GPU util 13% → 86%, now near compute-bound)
• RTX 4090 measured: Qwen 50.2 / DS 37.9 / V4-Flash 10.1 t/s (V4 5x faster than 1.7-2.0); 2080 Ti Qwen 24.5 t/s (2x vs v0.3.0)
• Includes A3 patch + expert-page eviction v3.1 + cuda-libs (built without NCCL — not needed for single-GPU)