cnm-serve v0.3

cnm-serve v0.3: run the full 512-expert Qwen3.8-Flash-Next (IST-DASLab GSQ-RCO GGUF) on one NVIDIA card with 64–128 GB RAM, on Windows.

Status

• Built and tested on Linux: engine build and run, launcher logic, downloads, server start.
• First contact with real Windows (RTX 4080 16 GB / Windows 10 / 128 GB RAM, an earlier internal build): the window opened and the download started. It then hit a bug in cnm-serve: every fresh download failed with "network error, retrying: truncate ... Access is denied", repeated. This is fixed in v0.3, see below. The server itself has not run on Windows yet; treat everything after the download as untested on Windows.
• The speed on a single 16 GB card is an estimate (20–35 tok/s), not a measurement. The same holds for 32 GB.

What it does

cnm-serve.exe is a small window:

• It shows the detected GPU, VRAM and RAM, and the chosen tier.
• Model files.
  • It downloads the model from Hugging Face: IQ3_S (recommended from 16 GB) or IQ2_XS (8 GB). Downloads are resumable, sha256-checked and pinned to a revision.
  • Or Use existing files...: point it at files you already have. They are checked in place by size and sha256, and never moved, renamed or deleted. They are accepted under any name.
  • Links... shows the pinned download pages, exact names, sizes and sha256, with Copy and "Open download page" buttons.
• It starts llama-server on 127.0.0.1 with a GPU expert cache sized to the card and a router bias that matches the cache size.
• It shows the OpenAI-compatible URL, the model name and a per-install API key to paste into RooCode, Cline or Claude Code.
  • The server requires that key, has /slots off, and allows CORS only from localhost, so web pages in your browser cannot use it or read your prompts.
• Optional MTP draft head.
  • A "Download MTP head (optional, 3 GB)" button, or Use existing files, pinned to HF oisee/Qwen3.8-Flash-Next-GSQ-RCO-MoREL-PB3-208-GGUF @ e9894521.
  • A checkbox "Use MTP draft head (experimental on 16 GB)" recomputes the cache and the bias rung, and shows both.
  • Default: off on 8/16 GB (slower on 8 GB: 11–14 vs 17.1 tok/s; unmeasured on 16 GB), on on 32 GB.
• Bias.
  • Default graded, cap −1 (PB3-N.soft-g1-c1). The i7 calibration chose it: there it matched the mean penalty of flat −0.75, gave +3% decode and passed SMOKE-11 11/11. The shipped c1 files average −0.78 to −0.90 per penalized expert across cache sizes (PB3-64: −0.85).
  • graded-c2 (cap −2), flat (−0.75) and off are selectable in cnm-serve.ini.
  • The bias lowers both the selection chance and the mixing weight of rarely used experts.

Fixed from the first Windows test

• Root cause: the partial file was truncated through the handle opened for appending. On Windows, Go opens such a handle without FILE_WRITE_DATA, so the truncate always fails with "Access is denied". (It works on Linux, which is why the Linux tests missed it.) Now:
  • The launcher never truncates through that handle. It skips the truncate when the size is already right (the normal case), and otherwise truncates through its own read-write handle.
  • File operations (truncate, open, rename) are retried with backoff when another program holds the file: Defender scanning the download, a second cnm-serve, a mapped file. The status line shows "file held by another program, retrying", and Pause works during the wait.
  • If access stays denied, it says plainly: "Windows denied access to : an antivirus or another program holds it, or the folder is protected…". The README has the Defender exclusion and Controlled folder access steps.
• Network errors (HTTP status, timeout, TLS) and local file errors are reported separately. The real network error is kept when a file error follows it. Permanent HTTP errors (404, 403) stop at once instead of being retried for minutes.
• If an antivirus quarantines the partial file, that file starts again instead of failing.
• Download events go to logs\download.log.
• Only one cnm-serve can run at a time; a second start says "already running".

Download

• cnm-serve-win64-cuda.zip, containing:
  • cnm-serve.exe;
  • llama/: llama-server.exe, ggml CPU backends, ggml-cuda.dll, CUDA 12.4 runtime DLLs;
  • bias/: 30 files;
  • templates/, README.md, LICENSES/.
• cnm-serve-win64-cuda.zip.sha256.
• llama/ comes unchanged from Actions run https://github.com/oisee/llama.cpp/actions/runs/40610262501 (commit de985be6c, see llama/BUILD-COMMIT.txt). The packer checked every file against the run's SHA256SUMS. The folder also holds the other llama.cpp tools the build makes; only llama-server.exe is used.
• The model (68–84 GB) and the MTP head (3 GB) are not in the zip. The launcher downloads them, or uses your copies. The README lists the exact files, sizes and sha256, and how to check a hash with certutil -hashfile <file> SHA256.

Requirements

• NVIDIA driver 528.33 or newer (551.61+ recommended).
• An 8 GB card or larger.
• 64 GB RAM minimum, 96 GB+ for IQ3_S.
• Windows 10 or 11, x64. No admin rights needed.
• Unpack outside Program Files, to a path with English letters only.
• Set CUDA - Sysmem Fallback Policy = Prefer No Sysmem Fallback for llama-server.exe in the NVIDIA Control Panel. Otherwise an overfull card gets silently slow instead of reporting out-of-memory.

What was tested (all on Linux)

• Engine patch.
  • One commit on upstream llama.cpp f3269ef96, which already has --moe-cache-mib (ggml-org#29887, ggml-org#30112, SLRU ggml-org#30637).
  • It adds the CNM_EXPERT_BIAS / CNM_EXPERT_BIAS_MODEL loader and applies the bias to the router logits before softmax.
  • Each bias vector sits in the buffer type of its router weight.
  • Checks:
    • CPU build on i3.
    • CUDA 12.4 sm_89 build on i3 (compile and link only; i3 has no NVIDIA GPU).
    • Native CPU build on i10 with the real IQ2_XS model: the bias file is logged (48 layers, 48 nonzero vectors) and changes the temp-0 output; a non-matching model path skips it (output byte-identical to unbiased); a bad file stops the server with a clear error.
• Launcher core (Go; shared by the Windows GUI and a Linux headless mode).
  • Unit tests under -race cover:
    • tier, cache and bias sizing;
    • the config file;
    • downloads: resume after a dropped connection, stall recovery, a changed remote file rejected, sha256 mismatch kept aside rather than deleted, shared-shard hard link, and pinned URLs;
    • the MTP head download, the checkbox and its defaults;
    • "Use existing files": in-place check, odd names linked inside the kit, a mismatch named and nothing deleted, changed files no longer trusted, and paths saved and used for -m / -md;
    • server start, the retry with a smaller cache after a GPU out-of-memory, the API key and its log redaction.
  • Held-file handling (on Linux, with an injected "access denied"): it is retried, gives the plain message rather than "network error", Pause interrupts the wait, and a file of the right size is never truncated.
  • A source check guards against truncating through a file handle in the download code again.
  • Windows-only tests (error-code mapping, and the append-handle case) are compiled for Windows but have not run on Windows yet.
  • 49 deliberate code mutations were each caught by a failing test (tools/mutations.py).
  • With real data:
    • The headless launcher started the real llama-server with the real model and served chat completions: 401 without the key, /slots off.
    • It downloaded the real MTP head from the pinned URL; sha256 matched.
    • It adopted the real IQ2_XS shards in place: sha256 matched, files untouched.
• Records of these runs (test log, mutation results, server, download and adoption logs) are in evidence/ of the kit's source repo, which is not public yet.
• Windows binaries.
  • llama-server.exe and the DLLs were built by GitHub Actions on this fork (workflow cnm-serve-windows-cuda.yml, CUDA 12.4, real code for sm_75/86/89 plus PTX compute_89).
  • cnm-serve.exe is cross-compiled from Linux (Go, lxn/walk, no cgo).

What is untested

• On Windows: the server (CUDA, 48 GB of pinned memory), the download fix above (verified on Windows only once Roman retries), the single-instance guard, file dialogs, the job object that stops the server if the launcher crashes, hard links on NTFS.
• The GPU path of the bias patch (bias in VRAM) has not run anywhere yet.
• Single-card speed on 16/32 GB, and MTP on 16 GB.
• Bias quality beyond the i7 smoke test. Cap −1 speed on 8 GB (18.9 tok/s there was cap −2).
• RTX 50 cards: no native code in this build (CUDA 12.4). The driver JIT-compiles PTX on the first start, which can take a long time.

Source

• Branch cnm/serve-v0.3-rc1 on this fork (an internal branch name): upstream f3269ef96 + the bias-loader commit + the workflow. The older branch cnm/serve-v0.3 holds a first draft of the patch and is not what this release was built from.
• The launcher source is not on this branch.