Skip to content

cnm-serve v0.3 rc1 (pre-release, untested on Windows)

Pre-release
Pre-release

Choose a tag to compare

@oisee oisee released this 28 Oct 06:38

cnm-serve v0.3 rc1: pre-release, not yet tested on real Windows.

A Windows kit that runs the full 512-expert Qwen3.8-Flash-Next (IST-DASLab GSQ-RCO GGUF) on one NVIDIA card with 64–128 GB RAM. cnm-serve.exe is a small window: it shows the detected GPU/VRAM/RAM and tier, downloads the model from Hugging Face (resumable, sha256-checked), starts llama-server on 127.0.0.1 with a GPU expert cache and a router bias, and shows the OpenAI-compatible URL and a per-install API key to paste into RooCode, Cline or Claude Code. The server needs that key, has /slots off, and allows CORS only from localhost, so web pages in your browser cannot use it or read your prompts.

The first real test is an RTX 4080 16 GB / Windows 10 / 128 GB RAM machine. Until it reports back, treat everything Windows-specific as untested.

Download

  • cnm-serve-win64-cuda.zip: cnm-serve.exe, llama/ (llama-server.exe, ggml CPU backends, ggml-cuda.dll, CUDA 12.4 runtime DLLs), bias/ (20 bias files), templates/, README.md, LICENSES/.
  • cnm-serve-win64-cuda.zip.sha256 (sha256 e8943ad2cb223451adbb702136e0eb3433ef792caf0dc51893293c975033f726, 549.5 MB).
  • llama/ comes unchanged from Actions run https://github.com/oisee/llama.cpp/actions/runs/40610262501 (commit de985be6c; llama/BUILD-COMMIT.txt). The packer checked every file against the run's SHA256SUMS. The folder also holds the other llama.cpp tools the build makes; only llama-server.exe is used.
  • The model (68–84 GB) is not in the zip. The launcher downloads it from https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF (Apache-2.0).
  • No MTP head: the 2.98 GB file is over GitHub's per-file limit. The kit runs without it; see README.

Requirements

NVIDIA driver 528.33 or newer (551.61+ recommended), an 8 GB card or larger, 64 GB RAM minimum (96 GB+ for IQ3_S), Windows 10/11 x64. No admin rights needed. Unpack outside Program Files, to a path with English letters only. Set CUDA - Sysmem Fallback Policy = Prefer No Sysmem Fallback for llama-server.exe in the NVIDIA Control Panel. Otherwise an overfull card gets silently slow instead of reporting out-of-memory.

What was tested (all on Linux)

  • Engine patch. One commit on upstream llama.cpp f3269ef96, which already has --moe-cache-mib (ggml-org#29887, ggml-org#30112, SLRU ggml-org#30637). It adds the CNM_EXPERT_BIAS / CNM_EXPERT_BIAS_MODEL loader (per-expert object form and cold-ID list form) and applies the bias to the router logits before softmax. The bias therefore lowers both the chance that a cold expert is picked and its mixing weight when it is picked. Each bias vector is allocated in the buffer type of its router weight (VRAM when the router is on the GPU), so the add does not force a CPU round trip. Checks:
    • Linux CPU build on i3.
    • Linux CUDA 12.4 build for sm_89 in a container on i3 (146 sm_89 kernels in libggml-cuda). Compile and link only: i3 has no NVIDIA GPU, so it was not run. The GPU path of the patch (the bias in VRAM) has not run anywhere yet.
    • Native CPU build on i10, run with the real IQ2_XS model. With the PB3-64 graded file the server logs CNM_EXPERT_BIAS: 48 layers, 48 nonzero vectors, and the temp-0 output differs from the unbiased run. A non-matching CNM_EXPERT_BIAS_MODEL skips the bias (output identical to unbiased). A positive bias or a missing file stops the server with a CNM_EXPERT_BIAS: error.
  • Launcher core (Go, shared by the Windows GUI and a Linux headless mode). Unit tests cover:
    • tier and cache sizing, bias-file choice following the cache size, and the config file;
    • nvidia-smi parsing;
    • download resume after a dropped connection and from a leftover partial file, restart when the server ignores Range, sha256 rejection, and hard-linking the shard shared between quants;
    • server start, health polling, retry with a smaller cache after a GPU out-of-memory (saved to the settings), and stop;
    • API key required and kept out of the log, --no-slots and localhost-only CORS passed, refusal of non-ASCII paths, a stalled download reconnecting, and a changed remote file rejected.
      Fifteen deliberate mutations of the code were each caught by a failing test. The headless launcher started the real llama-server (CPU-only) on i10 with the real model and served a chat completion.
  • Windows binaries. llama-server.exe and the DLLs were built by GitHub Actions on this branch (workflow cnm-serve-windows-cuda.yml, CUDA 12.4, real SASS for sm_75/86/89 plus PTX compute_89). cnm-serve.exe is cross-compiled from Linux (Go, lxn/walk, no cgo).

What is untested

  • Any run on Windows: the GUI, CUDA on Windows, pinned memory on Windows, paths with spaces or non-English letters, the Windows job object that stops the server if the launcher crashes.
  • Speed on a single 16 GB or 32 GB card: these are estimates. The 16 GB cache (≈8500 MiB) is computed from VRAM minus what is in use at start (at least 0.75 GB), minus measured overheads. Measured reference points (Linux, cnm research build with the same bias loader, not this engine): RTX 2070 8 GB, IQ2_XS, 18.9 tok/s; 2× RTX 4060 Ti 16 GB, IQ3_S, 17 GB cache + MTP, 52 tok/s. The 16 GB estimate is 20–35 tok/s.
  • RTX 50 cards: no native code in this build (CUDA 12.4); the driver JIT-compiles PTX on the first start, which can take a long time.
  • Graded bias quality: the default graded (cap −2) bias penalizes cold experts more on average than flat −1. Its effect on answer quality is not measured yet; a head-to-head against flat −0.75 is running. bias = flat | off in cnm-serve.ini switches it.

Source

  • Branch cnm/serve-v0.3-rc1 on this fork: upstream f3269ef96 + the bias-loader commit + the workflow. The older branch cnm/serve-v0.3 holds a first draft of the same patch and is not what this release was built from.
  • The launcher source is not on this branch yet.