cnm-serve v0.4.1

cnm-serve v0.4.1: one release, two zips, and multi-GPU for the NVIDIA kit.

Status

• Built and tested on Linux only.
• Windows: the CUDA kit's first Windows test (RTX 4080, Windows 10, an earlier build) reached the download and found a bug that v0.3 fixed. The server has not run on Windows yet. The Vulkan kit has not run on Windows at all.
• Multi-GPU is untested on hardware. It is planned and unit-tested. The only measured two-card machine (2 x RTX 4060 Ti, Linux) is in use and was not touched for this release.
• All multi-GPU cache sizes and splits are estimates. The single 16 GB card speed is still an estimate too (20-35 tok/s).

Which zip?

• cnm-serve-win64-cuda.zip: one or more NVIDIA cards with 8 GB or more, and at least 64 GB of RAM. Model: Qwen3.8-Flash-Next (all 512 experts) with the GPU expert cache. The engine is unchanged from v0.3.
• cnm-serve-win64-vulkan.zip: AMD or Intel GPUs, laptops with an integrated GPU, or less RAM. Model: gpt-oss-20b. The engine is unchanged from v0.4.
• Both zips carry the same new cnm-serve.exe (v0.4.1). It recognises which engine is in llama\ and shows the models the machine cannot run greyed out with the reason.

New in v0.4.1

• Multi-GPU on CUDA (Qwen).

  • Detection: every NVIDIA card is detected through nvidia-smi: index, name, total, used and free memory, whether it drives the monitor, and its compute capability.
  • Cards used: two or more cards of 8 GB class with compute capability 7.5 or newer, and at least 2 GB of free room. Others are left out with a note. If only one card remains, the start goes to the largest card this build can run (never, for example, a Tesla P40 next to an RTX 3060). With no such card at all, a clear message replaces a crash.
  • Older drivers: if nvidia-smi does not know the new fields (compute_cap), detection falls back to the v0.4 query, so the driver-version warning still works.
  • Split: the layers are split with --device CUDA0,CUDA1 -ts <MiB per card>. Each weight is the card's memory, minus the desktop reserve on the monitor card (or what is in use, if more), minus 0.5 GB headroom, minus 1 GB for the card's own CUDA context and buffers. The largest card goes last: llama.cpp puts the output layer there.
  • Cache: the expert cache spans all cards (upstream ggml-org#30112), shared out by the same weights. Its size is the total usable memory, minus the other weights, the KV cache of the context, and 1 GB for each extra card.
  • Bias: the rung comes from the cache per layer on the tightest card. llama.cpp's layer cut points are replayed. Graded cap −1.
  • MTP: the head goes on the card with the most usable memory (ties: the last card), and that card's share shrinks by the head's size. Auto MTP and the 194560 context only with exactly two 16 GB-class cards, the measured case; other combinations (2 x 8, 16 + 24, 3 x 16, ...) get the single-card defaults for their total VRAM.
  • Anchor: i7's measured 2 x 16 GB setup (-ts 30,19, 17000 MiB cache, MTP on CUDA1, ctx 194560). The formula gives 17250 MiB of cache for it under Windows and 18000 MiB headless. The split is not anchored: the formula puts 21 of 48 layers on the MTP card, while i7 ran with 18 of 48.
  • In the window: in device order, for example "GPU1 RTX 2070 8 GB + GPU0 RTX 4060 Ti 16 GB -> split 7:14, cache 13.2 GB, bias graded PB3-128, cap -1, MTP off".
  • Switch: multi_gpu = off uses the largest card only.

• Less than 96 GB of RAM (for example 64 GB with 16 + 12 GB cards):

  • IQ3_S does not fit (IQ3_S experts are 47962 MiB, plus the 28.8 GB second file, which is mostly per-layer embeddings), so the launcher picks IQ2_XS and never falls back to IQ3_S.
  • The experts are not pinned: the default mmap loading reads them from disk on demand (--load-mode none is left out; --no-repack stays).
  • Because cache misses are slower that way, the context drops to 32K, unless ctx is set, to leave the VRAM to the cache.
  • The window says what it actually did: IQ3_S forced in the ini, pinned = true ignored, context lowered or not.

• Estimated profiles (from the plan output, each backed by a unit test):

  cards (RAM)	model	context	cache	bias	MTP
  16 + 8 GB (128 GB)	IQ3_S	128K	13.2 GB	PB3-128	off
  16 + 8 GB, MTP ticked	IQ3_S	128K	10.5 GB	PB3-96	on, 16 GB card
  16 + 8 GB, ctx 64K	IQ3_S	64K	14.9 GB	PB3-128 (the 8 GB card is the tight one)	off
  16 + 12 GB (128 GB)	IQ3_S	128K	17.1 GB	PB3-160	off
  16 + 12 GB, ctx 64K	IQ3_S	64K	19.0 GB	PB3-192	off
  2 x 16 GB (128 GB)	IQ3_S	194560	16.8 GB	PB3-160	on, second card
  16 + 12 GB (64 GB RAM)	IQ2_XS, mmap	32K	20.5 GB	PB3-256	off
  16 GB alone (128 GB)	IQ3_S	64K	8.3 GB	PB3-64	off
  8 GB alone (128 GB)	IQ2_XS	32K	2.0 GB	PB3-32	off

  Note: the rungs are lower than a simple "total cache / 48" would give, because the tightest card sets the slots per layer. 16 + 12 GB reaches PB3-192 at a 64K context; 16 + 8 GB stays at PB3-128 even at 64K.

• New cnm-serve.ini key: multi_gpu (auto or off).

• Not supported: different precision per card (Q3 hot experts on one card, Q2 on another). The engine loads one quantization for the whole model. This is listed as future work in the README.

Download

• cnm-serve-win64-cuda.zip. Its llama\ folder comes unchanged from Actions run https://github.com/oisee/llama.cpp/actions/runs/40610262501 (commit de985be6c: upstream f3269ef96 + bias patch 8fa13449e + CUDA 12.4 workflow).
• cnm-serve-win64-vulkan.zip. Its llama\ folder comes unchanged from Actions run https://github.com/oisee/llama.cpp/actions/runs/40695404186 (commit 8553cd1d6: the same f3269ef96 + 8fa13449e + Vulkan workflow).
• A .sha256 file for each zip. The packer checked every engine file against its run's SHA256SUMS.
• The tag points at 8fa13449e, the engine source both builds share.
• Models are not in the zips. The launcher downloads them from Hugging Face, pinned to a revision, or uses your copies (Use existing files...).

What was tested (Linux)

• go test -race passes: v0.3 and v0.4 tests plus the new multi-GPU and low-RAM tests.
• New tests:
  • nvidia-smi with several cards and the display column;
  • the 16 + 8, 16 + 12 and 2 x 16 plans: cache, rung from llama.cpp's layer cut points, split, MTP card, context;
  • the llama-server arguments: --device, -ts, -devd, CUDA_VISIBLE_DEVICES (including a left-out middle card);
  • multi_gpu = off; 6 GB and compute-capability 6.1 cards left out; no split on the Vulkan kit; GPU0 assumed to drive the monitor when unknown;
  • 2 x 8, 16 + 8 + 8, 16 + 24 and 3 x 16 not getting the 2 x 16 defaults;
  • a Tesla P40 + RTX 3060 machine starting on the 3060; only old cards giving a message; a busy card left out; the nvidia-smi fallback for old drivers;
  • llama.cpp's layer cut points replayed exactly, including at a boundary;
  • the GPU out-of-memory retry with a split;
  • the 64 GB-RAM path: IQ2_XS with no IQ3_S fallback, no pinning but --no-repack, short context, a warning that matches what happens, no repeated pinned retry.
• New deliberate code mutations of the multi-GPU and low-RAM logic are each caught by a failing test (tools/mutations.py, evidence/mutations-v0.4.1.txt).
• The headless launcher was run in plan-only mode for 16+8, 16+12, 2x16, 16 alone, 8 alone and 16+12 with 64 GB RAM. The commands it would run are recorded in evidence/v0.4.1-plans.txt of the kit's source repo (not public yet).

What is untested

• Any multi-GPU run on real hardware, on Windows or Linux, with this build.
• Whether llama.cpp's cache over several cards spreads by -ts the way the split assumes.
• Everything on Windows after the download (CUDA kit) and everything on Windows (Vulkan kit).
• The speed of any two-card setup other than i7's measured 52 tok/s (research build, Linux).

Source

• Engine: upstream llama.cpp f3269ef96 + 8fa13449e (the bias loader), with the CI workflows on branches cnm/serve-v0.3-rc1 (CUDA) and cnm/serve-v0.4-vulkan (Vulkan) of this fork.
• The launcher source is not on these branches.

cnm-serve v0.4 (Vulkan: AMD/Intel, integrated GPUs, gpt-oss-20b)

cnm-serve v0.4: a Vulkan kit for AMD and Intel GPUs and laptops with an integrated GPU. It serves gpt-oss-20b (MXFP4, 12 GB) on machines with 16-64 GB RAM. The NVIDIA / Qwen3.8-Flash-Next kit stays at v0.3 (cnm-serve-win64-cuda.zip from the v0.3 release).

Status

• Built and tested on Linux only. The Windows parts have not run on Windows: GPU detection through DXGI, the window, file dialogs, and the Vulkan driver on a Radeon 860M.
• The server, the model and tool calling ran once on Linux, with the Linux build of the same engine commit, on a desktop AMD RX 580 8 GB (not the 860M). Results are below.
• The speed on a Radeon 860M is an estimate (12-22 tok/s writing), not a measurement.

What is new

• cnm-serve-win64-vulkan.zip. It is a separate zip because the CUDA runtime is about 500 MB that AMD and Intel machines never use. It is about 41 MB (89 MB unpacked). Its cnm-serve.exe (v0.4) knows both engines; the CUDA zip from v0.3 still carries the v0.3 program.
• gpt-oss-20b from ggml-org/gpt-oss-20b-GGUF, pinned to revision 15dcfc7e. The size and sha256 come from the HF API, and the license is Apache 2.0. The window gives it Download, Use existing files... and Links....
  • Integrated GPU: every layer goes on the GPU, with no expert cache and no router bias.
  • Other settings: KV cache q8_0/q4_0, -ub 512, flash attention, and the model's own chat template (--jinja). The context is 32768, lowered to 16384 when little RAM is free.
  • Small discrete card or little shared memory: llama.cpp --fit keeps part of the experts on the CPU. A GPU out-of-memory at start retries that way automatically, and offload = fit is then saved in cnm-serve.ini (set it back to auto to try all layers again).
• Optional speculative decoding.
  • EAGLE3 draft (eagle3-gpt-oss-20b-Q8_0.gguf, 0.9 GB) with its own Download, Use existing files and Links, and a "Use draft (EAGLE3)" checkbox. It runs as -md ... --spec-type draft-eagle3.
  • n-gram drafting checkbox (ngram-simple).
  • Defaults: EAGLE3 is on when downloaded, n-gram is off. Both can be set in cnm-serve.ini (draft, ngram).
• GPU detection without nvidia-smi.
  • Windows DXGI lists the adapters with vendor id, dedicated and shared memory. Discrete cards are chosen before integrated ones, NVIDIA before other discrete cards. The Basic Render Driver is skipped.
  • An integrated GPU is recognised by its name (Radeon 8xxM/7xxM, Radeon Graphics, Intel UHD/Iris/Arc 1x0V) or by less than 4 GB of dedicated memory.
  • nvidia-smi is asked only for an NVIDIA adapter.
  • The Vulkan kit matches the adapter to llama-server's --list-devices, and falls back to that list when there is nothing else (Linux).
• The window shows what this machine can run. The top line shows the GPU, the RAM and how much is free, and what the plan does with them, e.g. "integrated GPU: all layers on GPU, expert cache off". Models the machine cannot run are greyed out with the reason: "needs >= 64 GB RAM and an NVIDIA GPU", or "needs the CUDA kit".
• Fixed:
  • On a machine without an NVIDIA card, the window showed "no NVIDIA GPU found (nvidia-smi ...)" and a negative "cache -6000 MiB". The cache size is never negative now, in the shared code too.
  • The ini's "out of memory? use N" hint is never negative.
• New cnm-serve.ini keys: draft, draft_path, ngram, offload, kv_cache, backend, model_path_gpt_oss_20b.

Requirements (Vulkan kit)

• Windows 10 or 11, x64.
• A GPU with a current driver that includes Vulkan (any AMD Adrenalin/PRO, Intel or NVIDIA driver).
• RAM: 32 GB recommended, with about 14 GB free at Start. 16 GB is not recommended.
• 13 GB of disk.
• No admin rights needed.

What was tested

• Launcher core, Linux, go test -race.
  • Covered: DXGI adapter choice (the AMD laptop, a hybrid NVIDIA + AMD laptop, the --list-devices fallback, the no-GPU message), integrated-GPU recognition, the Alice-laptop plan and top line, Qwen greyed out, the gpt-oss command line, the Vulkan out-of-memory retry to fit, the new ini keys, the gpt-oss and EAGLE3 downloads from their own repo, and Use existing files.
  • The v0.3 tests still pass.
  • Mutations: 80 deliberate code changes, each caught by a failing test (tools/mutations.py, evidence/mutations-v0.4.txt). 31 of them are new: GPU detection, tier choice, the negative cache fix, the gpt-oss command line and downloads, and the critic-round fixes.
• Real run, Linux on i3, AMD RX 580 8 GB, RADV Vulkan, 4 CPU threads.

  • Engine: llama.cpp f3269ef96 + bias patch, Linux Vulkan build.

  • Model files: the sha256 of both matched HF (checked in place by Use existing files).

  • The headless launcher (same core as the Windows window) started the server. Proved by content:

    • a chat reply via /v1/chat/completions and via /v1/messages;
    • a tool call via /v1/chat/completions (get_weather {"city":"Oslo"}), and its result used in the next answer;
    • a tool_use block via /v1/messages;
    • 401 without the key.

  • A/B, one run per arm, a 512-token code edit at temperature 0, with half of the experts on the CPU (8 GB card):

    arm	decode	draft acceptance
    plain	15.5 tok/s	–
    EAGLE3	12.8 tok/s	74%
    EAGLE3 + n-gram	10.7 tok/s	67%

    The draft arms keep 2 GB more of the card free, so the comparison is confounded. It says little about an integrated GPU.

• Windows binaries.
  • llama\ comes unchanged from Actions run https://github.com/oisee/llama.cpp/actions/runs/40695404186 (workflow cnm-serve-windows-vulkan.yml, commit 8553cd1d6). It is checked against the run's SHA256SUMS.
  • cnm-serve.exe is cross-compiled from Linux.

What is untested

• Anything on Windows: DXGI detection on a real machine, the window and its greyed-out rows, the Vulkan driver and shared-memory limits on the Radeon 860M, -ngl 999 with 12 GB in shared memory, file dialogs.
• Speed on any integrated GPU. Whether EAGLE3 or n-gram help there.
• Answer quality of gpt-oss-20b on ABAP or agent tasks.

Source

• Engine: branch cnm/serve-v0.4-vulkan on this fork, which is upstream f3269ef96 + the bias-loader commit + the Vulkan workflow.
• The launcher source is not on this branch.

cnm-serve v0.3

cnm-serve v0.3: run the full 512-expert Qwen3.8-Flash-Next (IST-DASLab GSQ-RCO GGUF) on one NVIDIA card with 64–128 GB RAM, on Windows.

Status

• Built and tested on Linux: engine build and run, launcher logic, downloads, server start.
• First contact with real Windows (RTX 4080 16 GB / Windows 10 / 128 GB RAM, an earlier internal build): the window opened and the download started. It then hit a bug in cnm-serve: every fresh download failed with "network error, retrying: truncate ... Access is denied", repeated. This is fixed in v0.3, see below. The server itself has not run on Windows yet; treat everything after the download as untested on Windows.
• The speed on a single 16 GB card is an estimate (20–35 tok/s), not a measurement. The same holds for 32 GB.

What it does

cnm-serve.exe is a small window:

• It shows the detected GPU, VRAM and RAM, and the chosen tier.
• Model files.
  • It downloads the model from Hugging Face: IQ3_S (recommended from 16 GB) or IQ2_XS (8 GB). Downloads are resumable, sha256-checked and pinned to a revision.
  • Or Use existing files...: point it at files you already have. They are checked in place by size and sha256, and never moved, renamed or deleted. They are accepted under any name.
  • Links... shows the pinned download pages, exact names, sizes and sha256, with Copy and "Open download page" buttons.
• It starts llama-server on 127.0.0.1 with a GPU expert cache sized to the card and a router bias that matches the cache size.
• It shows the OpenAI-compatible URL, the model name and a per-install API key to paste into RooCode, Cline or Claude Code.
  • The server requires that key, has /slots off, and allows CORS only from localhost, so web pages in your browser cannot use it or read your prompts.
• Optional MTP draft head.
  • A "Download MTP head (optional, 3 GB)" button, or Use existing files, pinned to HF oisee/Qwen3.8-Flash-Next-GSQ-RCO-MoREL-PB3-208-GGUF @ e9894521.
  • A checkbox "Use MTP draft head (experimental on 16 GB)" recomputes the cache and the bias rung, and shows both.
  • Default: off on 8/16 GB (slower on 8 GB: 11–14 vs 17.1 tok/s; unmeasured on 16 GB), on on 32 GB.
• Bias.
  • Default graded, cap −1 (PB3-N.soft-g1-c1). The i7 calibration chose it: there it matched the mean penalty of flat −0.75, gave +3% decode and passed SMOKE-11 11/11. The shipped c1 files average −0.78 to −0.90 per penalized expert across cache sizes (PB3-64: −0.85).
  • graded-c2 (cap −2), flat (−0.75) and off are selectable in cnm-serve.ini.
  • The bias lowers both the selection chance and the mixing weight of rarely used experts.

Fixed from the first Windows test

• Root cause: the partial file was truncated through the handle opened for appending. On Windows, Go opens such a handle without FILE_WRITE_DATA, so the truncate always fails with "Access is denied". (It works on Linux, which is why the Linux tests missed it.) Now:
  • The launcher never truncates through that handle. It skips the truncate when the size is already right (the normal case), and otherwise truncates through its own read-write handle.
  • File operations (truncate, open, rename) are retried with backoff when another program holds the file: Defender scanning the download, a second cnm-serve, a mapped file. The status line shows "file held by another program, retrying", and Pause works during the wait.
  • If access stays denied, it says plainly: "Windows denied access to : an antivirus or another program holds it, or the folder is protected…". The README has the Defender exclusion and Controlled folder access steps.
• Network errors (HTTP status, timeout, TLS) and local file errors are reported separately. The real network error is kept when a file error follows it. Permanent HTTP errors (404, 403) stop at once instead of being retried for minutes.
• If an antivirus quarantines the partial file, that file starts again instead of failing.
• Download events go to logs\download.log.
• Only one cnm-serve can run at a time; a second start says "already running".

Download

• cnm-serve-win64-cuda.zip, containing:
  • cnm-serve.exe;
  • llama/: llama-server.exe, ggml CPU backends, ggml-cuda.dll, CUDA 12.4 runtime DLLs;
  • bias/: 30 files;
  • templates/, README.md, LICENSES/.
• cnm-serve-win64-cuda.zip.sha256.
• llama/ comes unchanged from Actions run https://github.com/oisee/llama.cpp/actions/runs/40610262501 (commit de985be6c, see llama/BUILD-COMMIT.txt). The packer checked every file against the run's SHA256SUMS. The folder also holds the other llama.cpp tools the build makes; only llama-server.exe is used.
• The model (68–84 GB) and the MTP head (3 GB) are not in the zip. The launcher downloads them, or uses your copies. The README lists the exact files, sizes and sha256, and how to check a hash with certutil -hashfile <file> SHA256.

Requirements

• NVIDIA driver 528.33 or newer (551.61+ recommended).
• An 8 GB card or larger.
• 64 GB RAM minimum, 96 GB+ for IQ3_S.
• Windows 10 or 11, x64. No admin rights needed.
• Unpack outside Program Files, to a path with English letters only.
• Set CUDA - Sysmem Fallback Policy = Prefer No Sysmem Fallback for llama-server.exe in the NVIDIA Control Panel. Otherwise an overfull card gets silently slow instead of reporting out-of-memory.

What was tested (all on Linux)

• Engine patch.
  • One commit on upstream llama.cpp f3269ef96, which already has --moe-cache-mib (ggml-org#29887, ggml-org#30112, SLRU ggml-org#30637).
  • It adds the CNM_EXPERT_BIAS / CNM_EXPERT_BIAS_MODEL loader and applies the bias to the router logits before softmax.
  • Each bias vector sits in the buffer type of its router weight.
  • Checks:
    • CPU build on i3.
    • CUDA 12.4 sm_89 build on i3 (compile and link only; i3 has no NVIDIA GPU).
    • Native CPU build on i10 with the real IQ2_XS model: the bias file is logged (48 layers, 48 nonzero vectors) and changes the temp-0 output; a non-matching model path skips it (output byte-identical to unbiased); a bad file stops the server with a clear error.
• Launcher core (Go; shared by the Windows GUI and a Linux headless mode).
  • Unit tests under -race cover:
    • tier, cache and bias sizing;
    • the config file;
    • downloads: resume after a dropped connection, stall recovery, a changed remote file rejected, sha256 mismatch kept aside rather than deleted, shared-shard hard link, and pinned URLs;
    • the MTP head download, the checkbox and its defaults;
    • "Use existing files": in-place check, odd names linked inside the kit, a mismatch named and nothing deleted, changed files no longer trusted, and paths saved and used for -m / -md;
    • server start, the retry with a smaller cache after a GPU out-of-memory, the API key and its log redaction.
  • Held-file handling (on Linux, with an injected "access denied"): it is retried, gives the plain message rather than "network error", Pause interrupts the wait, and a file of the right size is never truncated.
  • A source check guards against truncating through a file handle in the download code again.
  • Windows-only tests (error-code mapping, and the append-handle case) are compiled for Windows but have not run on Windows yet.
  • 49 deliberate code mutations were each caught by a failing test (tools/mutations.py).
  • With real data:
    • The headless launcher started the real llama-server with the real model and served chat completions: 401 without the key, /slots off.
    • It downloaded the real MTP head from the pinned URL; sha256 matched.
    • It adopted the real IQ2_XS shards in place: sha256 matched, files untouched.
• Records of these runs (test log, mutation results, server, download and adoption logs) are in evidence/ of the kit's source repo, which is not public yet.
• Windows binaries.
  • llama-server.exe and the DLLs were built by GitHub Actions on this fork (workflow cnm-serve-windows-cuda.yml, CUDA 12.4, real code for sm_75/86/89 plus PTX compute_89).
  • cnm-serve.exe is cross-compiled from Linux (Go, lxn/walk, no cgo).

What is untested

• On Windows: the server (CUDA, 48 GB of pinned memory), the download fix above (verified on Windows only once Roman retries), the single-instance guard, file dialogs, the job object that stops the server if the launcher crashes, hard links on NTFS.
• The GPU path of the bias patch (bias in VRAM) has not run anywhere yet.
• Single-card speed on 16/32 GB, and MTP on 16 GB.
• Bias quality beyond the i7 smoke test. Cap −1 speed on 8 GB (18.9 tok/s there was cap −2).
• RTX 50 cards: no native code in this build (CUDA 12.4). The driver JIT-compiles PTX on the first start, which can take a long time.

Source

• Branch cnm/serve-v0.3-rc1 on this fork (an internal branch name): upstream f3269ef96 + the bias-loader commit + the workflow. The older branch cnm/serve-v0.3 holds a first draft of the patch and is not what this release was built from.
• The launcher source is not on this branch.

cnm-serve v0.3 rc1 (pre-release, untested on Windows)

cnm-serve v0.3 rc1: pre-release, not yet tested on real Windows.

A Windows kit that runs the full 512-expert Qwen3.8-Flash-Next (IST-DASLab GSQ-RCO GGUF) on one NVIDIA card with 64–128 GB RAM. cnm-serve.exe is a small window: it shows the detected GPU/VRAM/RAM and tier, downloads the model from Hugging Face (resumable, sha256-checked), starts llama-server on 127.0.0.1 with a GPU expert cache and a router bias, and shows the OpenAI-compatible URL and a per-install API key to paste into RooCode, Cline or Claude Code. The server needs that key, has /slots off, and allows CORS only from localhost, so web pages in your browser cannot use it or read your prompts.

The first real test is an RTX 4080 16 GB / Windows 10 / 128 GB RAM machine. Until it reports back, treat everything Windows-specific as untested.

Download

• cnm-serve-win64-cuda.zip: cnm-serve.exe, llama/ (llama-server.exe, ggml CPU backends, ggml-cuda.dll, CUDA 12.4 runtime DLLs), bias/ (20 bias files), templates/, README.md, LICENSES/.
• cnm-serve-win64-cuda.zip.sha256 (sha256 e8943ad2cb223451adbb702136e0eb3433ef792caf0dc51893293c975033f726, 549.5 MB).
• llama/ comes unchanged from Actions run https://github.com/oisee/llama.cpp/actions/runs/40610262501 (commit de985be6c; llama/BUILD-COMMIT.txt). The packer checked every file against the run's SHA256SUMS. The folder also holds the other llama.cpp tools the build makes; only llama-server.exe is used.
• The model (68–84 GB) is not in the zip. The launcher downloads it from https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF (Apache-2.0).
• No MTP head: the 2.98 GB file is over GitHub's per-file limit. The kit runs without it; see README.

Requirements

NVIDIA driver 528.33 or newer (551.61+ recommended), an 8 GB card or larger, 64 GB RAM minimum (96 GB+ for IQ3_S), Windows 10/11 x64. No admin rights needed. Unpack outside Program Files, to a path with English letters only. Set CUDA - Sysmem Fallback Policy = Prefer No Sysmem Fallback for llama-server.exe in the NVIDIA Control Panel. Otherwise an overfull card gets silently slow instead of reporting out-of-memory.

What was tested (all on Linux)

• Engine patch. One commit on upstream llama.cpp f3269ef96, which already has --moe-cache-mib (ggml-org#29887, ggml-org#30112, SLRU ggml-org#30637). It adds the CNM_EXPERT_BIAS / CNM_EXPERT_BIAS_MODEL loader (per-expert object form and cold-ID list form) and applies the bias to the router logits before softmax. The bias therefore lowers both the chance that a cold expert is picked and its mixing weight when it is picked. Each bias vector is allocated in the buffer type of its router weight (VRAM when the router is on the GPU), so the add does not force a CPU round trip. Checks:
  • Linux CPU build on i3.
  • Linux CUDA 12.4 build for sm_89 in a container on i3 (146 sm_89 kernels in libggml-cuda). Compile and link only: i3 has no NVIDIA GPU, so it was not run. The GPU path of the patch (the bias in VRAM) has not run anywhere yet.
  • Native CPU build on i10, run with the real IQ2_XS model. With the PB3-64 graded file the server logs CNM_EXPERT_BIAS: 48 layers, 48 nonzero vectors, and the temp-0 output differs from the unbiased run. A non-matching CNM_EXPERT_BIAS_MODEL skips the bias (output identical to unbiased). A positive bias or a missing file stops the server with a CNM_EXPERT_BIAS: error.
• Launcher core (Go, shared by the Windows GUI and a Linux headless mode). Unit tests cover:
  • tier and cache sizing, bias-file choice following the cache size, and the config file;
  • nvidia-smi parsing;
  • download resume after a dropped connection and from a leftover partial file, restart when the server ignores Range, sha256 rejection, and hard-linking the shard shared between quants;
  • server start, health polling, retry with a smaller cache after a GPU out-of-memory (saved to the settings), and stop;
  • API key required and kept out of the log, --no-slots and localhost-only CORS passed, refusal of non-ASCII paths, a stalled download reconnecting, and a changed remote file rejected.
    Fifteen deliberate mutations of the code were each caught by a failing test. The headless launcher started the real llama-server (CPU-only) on i10 with the real model and served a chat completion.
• Windows binaries. llama-server.exe and the DLLs were built by GitHub Actions on this branch (workflow cnm-serve-windows-cuda.yml, CUDA 12.4, real SASS for sm_75/86/89 plus PTX compute_89). cnm-serve.exe is cross-compiled from Linux (Go, lxn/walk, no cgo).

What is untested

• Any run on Windows: the GUI, CUDA on Windows, pinned memory on Windows, paths with spaces or non-English letters, the Windows job object that stops the server if the launcher crashes.
• Speed on a single 16 GB or 32 GB card: these are estimates. The 16 GB cache (≈8500 MiB) is computed from VRAM minus what is in use at start (at least 0.75 GB), minus measured overheads. Measured reference points (Linux, cnm research build with the same bias loader, not this engine): RTX 2070 8 GB, IQ2_XS, 18.9 tok/s; 2× RTX 4060 Ti 16 GB, IQ3_S, 17 GB cache + MTP, 52 tok/s. The 16 GB estimate is 20–35 tok/s.
• RTX 50 cards: no native code in this build (CUDA 12.4); the driver JIT-compiles PTX on the first start, which can take a long time.
• Graded bias quality: the default graded (cap −2) bias penalizes cold experts more on average than flat −1. Its effect on answer quality is not measured yet; a head-to-head against flat −0.75 is running. bias = flat | off in cnm-serve.ini switches it.

Source

• Branch cnm/serve-v0.3-rc1 on this fork: upstream f3269ef96 + the bias-loader commit + the workflow. The older branch cnm/serve-v0.3 holds a first draft of the same patch and is not what this release was built from.
• The launcher source is not on this branch yet.

cnm-varexp-v0.1

MoREL runtime v0.1 - prerelease

Source: oisee/llama.cpp, cnm/release-v0.1, pinned commit 53f311f39957bf92608753ec2af214fe42ccb08e.

The branch starts at cnm/varexp / 7ed1a0a57e5621073146bca8d5c166e52cfe462b. Bias implementation c31b35f0d and its follow-up 7df19d87 cherry-picked cleanly as 1e4033199 and 53f311f39. The latter alone is not the bias implementation.

MoREL means Mixture of Ragged Experts per Layer. These GGUF models physically retain different expert counts in different layers. They require this fork's variable-expert tensor shapes, source-width router reconstruction, removed-expert masking before softmax, and original-ID to compact-slot mapping. Mainline llama.cpp and Ollama compatibility is not established. The release does not establish new model quality or benchmark results.

Models:

• PB3-208, IQ3 source
• PB3-304, IQ2 source
• Earlier fixed-width K192p-implant16

Download every shard to the same local directory and pass shard 1 to -m. Keep model files on the machine's local disk when running; do not mmap them across the CIFS working share. Use the model card's chat template and optional matching MTP head if serving that configuration. The short checks in MATRIX.md do not validate long-context serving, MTP, task quality, or new GPU throughput. GPU generation on i7/i5 was explicitly excluded from part 1.

Extract a machine-specific tarball and run its binaries from the extracted directory. All project shared libraries are next to the executables, with $ORIGIN runtime paths. Each archive includes llama-server, llama-cli, llama-perplexity, header-check, LICENSE, BUILD.txt, and HEADER-CHECK.txt. Check SHA256SUMS before extraction. Ubuntu 24.04 userspace and the backend's system GPU drivers/runtime are required. These builds disable HTTPS support and the embedded server Web UI. They are for our named machines, not universal Linux binaries. The i3 CPU lacks AVX2; its Vulkan package uses the local CPU configuration. The i10 CPU package explicitly enables AVX2/FMA/F16C with native tuning off. CUDA builds target i7 sm_89/86/75 and i5 sm_75 using CUDA 12.0 and GCC 12.

For the tested 32 GB i3/RX 580 smoke configuration, use --device Vulkan0 -ngl 2 --no-host --no-repack --no-op-offload --load-mode mmap --lazy-mode on -fa off. This offloads one repeating layer plus the output stage. A non-mmap profile allocated a roughly 22 GB Vulkan_Host buffer and failed command submission; the successful flags avoid that model-sized pinned-host allocation. This does not validate full GPU offload or larger contexts. See MATRIX.md and the report for the exact 512-token-context test.

Runtime controls (normally leave both unset for a physically compact MoREL model):

• CNM_EXPERT_MASK=/absolute/keep.json: hard expert selection. Object keys are layer numbers, values arrays of kept expert IDs, for example {"0":[1,4,7]}. A real plan must cover every supported MoE layer, including NextN, and retain at least top-k experts. IDs are in the loaded model's expert index space; do not apply original-width plans blindly to compact weights. Grouped routing has additional safety checks.
• CNM_EXPERT_BIAS=/absolute/plan.json: experimental nonpositive F32 offsets before softmax/top-k, intended for uncut Qwen4Exp with ungrouped softmax routing and no existing expert-selection bias. Scalar example: {"bias":-2,"layers":{"0":[1,4,7]}}; these IDs identify cold experts. Per-expert example: {"layers":{"0":{"1":-1,"4":"-inf"}}}. Omitted entries have zero offset. Positive offsets, malformed IDs and insufficient finite experts are rejected. Cannot combine with CNM_EXPERT_MASK.
• CNM_ROUTING_TRACE=/absolute/trace.bin: optional llama-perplexity routing trace. Its evaluation callback can inhibit fusion; apply it consistently in comparisons.

The source bias documentation specifies validation and numerical limits. A hard mask and matching negative-infinity bias share a graph, but finite penalties and physically compact cuts are not asserted to be numerically identical. These release smoke checks do not repeat the bias equivalence experiments. The optional CUDA top-k determinism override remains at its default in these builds.

Runtime source is MIT (included LICENSE); model weights have their own Qwen community license and quantizer attribution in their HF repositories.

This prerelease publishes the four machine-specific packages verified in part 1. Generic GitHub Actions builds are listed separately when available; they are untested by us.

Release provenance

Tag cnm-varexp-v0.1 points to 45d2652d955542b1666c6fdcb5ed45e87d811637. The runtime source is unchanged from 53f311f39957bf92608753ec2af214fe42ccb08e; subsequent release commits only adapt GitHub workflows for this fork.

Machine package SHA-256

8c40381780f37427c52d7a5b30efe07bf0e0448ad0880f1a7d025ebf2ad0876e  llama-cnm-varexp-v0.1-i10-cpu-linux-x86_64.tar.gz
5420ec8769075c1affe3b011b5a0d7dc92fa3cd688accfb7b8ee4b6e5b07f345  llama-cnm-varexp-v0.1-i3-vulkan-linux-x86_64.tar.gz
7c7390f8193197fb38b8e3aa9ddd5507afa6caa21aaed19672e4c9857a8c502c  llama-cnm-varexp-v0.1-i5-cuda-linux-x86_64.tar.gz
567d67ff6ca26c3046c39247b8efa704f897d981f2fa2671a6edd7ebe8a7e16b  llama-cnm-varexp-v0.1-i7-cuda-linux-x86_64.tar.gz

Test matrix from part 1

v0.1 build and verification matrix

Source commit: 53f311f39957bf92608753ec2af214fe42ccb08e on cnm/release-v0.1. All builds use Release mode and shared project libraries. Detailed flags, compilers, date and CPU information are in each archive's BUILD.txt.

Host / actual hostname	Actual CPU and backend hardware	Build	Tested on that machine	Not tested
i7 / u7	Intel Core i7-10700K; 2 x RTX 4060 Ti 16 GB	CUDA 12.0, GCC 12, fat sm_89/86/75; PASS	All three programs --version; PB3-208 metadata/tensor-shape load with zero weight buffers; extracted-archive relocation/ldd; invalid GGUF rejected; actual library contains all three architectures	GPU generation intentionally excluded; sm_86/sm_75 execution; long context, MTP, serving
i5 / u5	AMD Ryzen 7 5800X; RTX 2070 8 GB	CUDA 12.0, GCC 12, sm_75; PASS	All three programs --version; PB3-208 metadata/tensor-shape load with zero weight buffers; extracted-archive relocation/ldd; invalid GGUF rejected; actual library contains sm_75 code	GPU generation intentionally excluded; long context, MTP, serving
i3 / u3	Intel Core i5-3570K; Radeon RX 580 8 GB, RADV Mesa 25.2.8	Vulkan; local CPU tuning (AVX, no AVX2); GCC 13; PASS	vulkaninfo; shipped CLI lists Radeon Vulkan0; all three programs --version; PB3-208 header load; extracted-archive relocation/ldd; invalid GGUF rejected; 16-token Vulkan completion PASS: exit 0, exact 16-token count, one repeating layer plus output stage offloaded	Full GPU offload, other Vulkan devices, long context, MTP, serving
i10 / truenas	AMD Ryzen 5 5600G, 6 cores / 12 threads	CPU AVX2/FMA/F16C, native OFF, GCC 13; Ubuntu 24.04 Docker; PASS	All three programs --version; PB3-208 header load; extracted-archive relocation/ldd; invalid GGUF rejected; successful -n 16 CPU completion from local ZFS model files, exit 0	GPU backends, long context, MTP, serving

Header checks use the packaged header-check helper linked to that package's libllama, with no_alloc=true, load_mode=NONE, lazy_mode=OFF, and zero GPU layers. They load both split headers, validate tensor shapes and report HEADER_OK layers=48 embedding=2560 with zero-sized weight buffers. This is stronger than a vocabulary-only load, but does not execute the model or check every weight byte. GPU visibility is disabled during CUDA package checks.

The completion model is PB3-208, first shard Qwen3.8-Flash-Next-GSQ-RCO-MoREL-PB3-208-00001-of-00002.gguf, with both shards present. The CPU test runs in the Ubuntu container with a read-only bind of the host's local ZFS dataset. The Vulkan test uses a separate copy on i3's ext4 filesystem and verifies both hashes against the public card. Neither completion reads weights through CIFS. The final Vulkan command uses -ngl 2 (one repeating layer plus the output stage), --no-host --no-repack --no-op-offload --load-mode mmap --lazy-mode on to avoid a model-sized Vulkan pinned-host buffer. Both request 16 output tokens with --ignore-eos, fixed seed 4242, temperature 0, context 512, --no-warmup, and a short arithmetic prompt. Output can consume the budget in reasoning; this is an execution smoke test, not an answer-quality test. Commands and raw output are in the run record.

Every build uses nice 19; i7/i5 compilation uses two parallel jobs. i5 node-check ran before build work. No server or LOCK was stopped or changed by this task. i10 installs its toolchain only inside a new Ubuntu container image; i5 uses a private copied CMake 3.28 runtime under its isolated build directory, with existing GCC/CUDA packages.

Each package contains llama-server, llama-cli, llama-perplexity, project shared libraries, LICENSE, BUILD.txt, HEADER-CHECK.txt, and the header-check helper. System C/C++ libraries and GPU drivers/CUDA system libraries are external dependencies, recorded in ldd logs. HTTPS and the embedded server UI are disabled in these local builds. The tarballs target these machines and Ubuntu 24.04, not arbitrary Linux systems. Version build counts can differ between the full local checkout and shallow remote clones; th...