Skip to content

cnm-serve v0.4.1

Latest
Latest

Choose a tag to compare

View all tags
@oisee oisee released this 28 Oct 18:34
cnm-serve-v0.4.1
8fa1344

cnm-serve v0.4.1: one release, two zips, and multi-GPU for the NVIDIA kit.

Status

  • Built and tested on Linux only.
  • Windows: the CUDA kit's first Windows test (RTX 4080, Windows 10, an earlier build) reached the download and found a bug that v0.3 fixed. The server has not run on Windows yet. The Vulkan kit has not run on Windows at all.
  • Multi-GPU is untested on hardware. It is planned and unit-tested. The only measured two-card machine (2 x RTX 4060 Ti, Linux) is in use and was not touched for this release.
  • All multi-GPU cache sizes and splits are estimates. The single 16 GB card speed is still an estimate too (20-35 tok/s).

Which zip?

  • cnm-serve-win64-cuda.zip: one or more NVIDIA cards with 8 GB or more, and at least 64 GB of RAM. Model: Qwen3.8-Flash-Next (all 512 experts) with the GPU expert cache. The engine is unchanged from v0.3.
  • cnm-serve-win64-vulkan.zip: AMD or Intel GPUs, laptops with an integrated GPU, or less RAM. Model: gpt-oss-20b. The engine is unchanged from v0.4.
  • Both zips carry the same new cnm-serve.exe (v0.4.1). It recognises which engine is in llama\ and shows the models the machine cannot run greyed out with the reason.

New in v0.4.1

  • Multi-GPU on CUDA (Qwen).

    • Detection: every NVIDIA card is detected through nvidia-smi: index, name, total, used and free memory, whether it drives the monitor, and its compute capability.
    • Cards used: two or more cards of 8 GB class with compute capability 7.5 or newer, and at least 2 GB of free room. Others are left out with a note. If only one card remains, the start goes to the largest card this build can run (never, for example, a Tesla P40 next to an RTX 3060). With no such card at all, a clear message replaces a crash.
    • Older drivers: if nvidia-smi does not know the new fields (compute_cap), detection falls back to the v0.4 query, so the driver-version warning still works.
    • Split: the layers are split with --device CUDA0,CUDA1 -ts <MiB per card>. Each weight is the card's memory, minus the desktop reserve on the monitor card (or what is in use, if more), minus 0.5 GB headroom, minus 1 GB for the card's own CUDA context and buffers. The largest card goes last: llama.cpp puts the output layer there.
    • Cache: the expert cache spans all cards (upstream ggml-org#30112), shared out by the same weights. Its size is the total usable memory, minus the other weights, the KV cache of the context, and 1 GB for each extra card.
    • Bias: the rung comes from the cache per layer on the tightest card. llama.cpp's layer cut points are replayed. Graded cap −1.
    • MTP: the head goes on the card with the most usable memory (ties: the last card), and that card's share shrinks by the head's size. Auto MTP and the 194560 context only with exactly two 16 GB-class cards, the measured case; other combinations (2 x 8, 16 + 24, 3 x 16, ...) get the single-card defaults for their total VRAM.
    • Anchor: i7's measured 2 x 16 GB setup (-ts 30,19, 17000 MiB cache, MTP on CUDA1, ctx 194560). The formula gives 17250 MiB of cache for it under Windows and 18000 MiB headless. The split is not anchored: the formula puts 21 of 48 layers on the MTP card, while i7 ran with 18 of 48.
    • In the window: in device order, for example "GPU1 RTX 2070 8 GB + GPU0 RTX 4060 Ti 16 GB -> split 7:14, cache 13.2 GB, bias graded PB3-128, cap -1, MTP off".
    • Switch: multi_gpu = off uses the largest card only.

  • Less than 96 GB of RAM (for example 64 GB with 16 + 12 GB cards):

    • IQ3_S does not fit (IQ3_S experts are 47962 MiB, plus the 28.8 GB second file, which is mostly per-layer embeddings), so the launcher picks IQ2_XS and never falls back to IQ3_S.
    • The experts are not pinned: the default mmap loading reads them from disk on demand (--load-mode none is left out; --no-repack stays).
    • Because cache misses are slower that way, the context drops to 32K, unless ctx is set, to leave the VRAM to the cache.
    • The window says what it actually did: IQ3_S forced in the ini, pinned = true ignored, context lowered or not.

  • Estimated profiles (from the plan output, each backed by a unit test):

    cards (RAM) model context cache bias MTP
    16 + 8 GB (128 GB) IQ3_S 128K 13.2 GB PB3-128 off
    16 + 8 GB, MTP ticked IQ3_S 128K 10.5 GB PB3-96 on, 16 GB card
    16 + 8 GB, ctx 64K IQ3_S 64K 14.9 GB PB3-128 (the 8 GB card is the tight one) off
    16 + 12 GB (128 GB) IQ3_S 128K 17.1 GB PB3-160 off
    16 + 12 GB, ctx 64K IQ3_S 64K 19.0 GB PB3-192 off
    2 x 16 GB (128 GB) IQ3_S 194560 16.8 GB PB3-160 on, second card
    16 + 12 GB (64 GB RAM) IQ2_XS, mmap 32K 20.5 GB PB3-256 off
    16 GB alone (128 GB) IQ3_S 64K 8.3 GB PB3-64 off
    8 GB alone (128 GB) IQ2_XS 32K 2.0 GB PB3-32 off

    Note: the rungs are lower than a simple "total cache / 48" would give, because the tightest card sets the slots per layer. 16 + 12 GB reaches PB3-192 at a 64K context; 16 + 8 GB stays at PB3-128 even at 64K.

  • New cnm-serve.ini key: multi_gpu (auto or off).

  • Not supported: different precision per card (Q3 hot experts on one card, Q2 on another). The engine loads one quantization for the whole model. This is listed as future work in the README.

Download

  • cnm-serve-win64-cuda.zip. Its llama\ folder comes unchanged from Actions run https://github.com/oisee/llama.cpp/actions/runs/40610262501 (commit de985be6c: upstream f3269ef96 + bias patch 8fa13449e + CUDA 12.4 workflow).
  • cnm-serve-win64-vulkan.zip. Its llama\ folder comes unchanged from Actions run https://github.com/oisee/llama.cpp/actions/runs/40695404186 (commit 8553cd1d6: the same f3269ef96 + 8fa13449e + Vulkan workflow).
  • A .sha256 file for each zip. The packer checked every engine file against its run's SHA256SUMS.
  • The tag points at 8fa13449e, the engine source both builds share.
  • Models are not in the zips. The launcher downloads them from Hugging Face, pinned to a revision, or uses your copies (Use existing files...).

What was tested (Linux)

  • go test -race passes: v0.3 and v0.4 tests plus the new multi-GPU and low-RAM tests.
  • New tests:
    • nvidia-smi with several cards and the display column;
    • the 16 + 8, 16 + 12 and 2 x 16 plans: cache, rung from llama.cpp's layer cut points, split, MTP card, context;
    • the llama-server arguments: --device, -ts, -devd, CUDA_VISIBLE_DEVICES (including a left-out middle card);
    • multi_gpu = off; 6 GB and compute-capability 6.1 cards left out; no split on the Vulkan kit; GPU0 assumed to drive the monitor when unknown;
    • 2 x 8, 16 + 8 + 8, 16 + 24 and 3 x 16 not getting the 2 x 16 defaults;
    • a Tesla P40 + RTX 3060 machine starting on the 3060; only old cards giving a message; a busy card left out; the nvidia-smi fallback for old drivers;
    • llama.cpp's layer cut points replayed exactly, including at a boundary;
    • the GPU out-of-memory retry with a split;
    • the 64 GB-RAM path: IQ2_XS with no IQ3_S fallback, no pinning but --no-repack, short context, a warning that matches what happens, no repeated pinned retry.
  • New deliberate code mutations of the multi-GPU and low-RAM logic are each caught by a failing test (tools/mutations.py, evidence/mutations-v0.4.1.txt).
  • The headless launcher was run in plan-only mode for 16+8, 16+12, 2x16, 16 alone, 8 alone and 16+12 with 64 GB RAM. The commands it would run are recorded in evidence/v0.4.1-plans.txt of the kit's source repo (not public yet).

What is untested

  • Any multi-GPU run on real hardware, on Windows or Linux, with this build.
  • Whether llama.cpp's cache over several cards spreads by -ts the way the split assumes.
  • Everything on Windows after the download (CUDA kit) and everything on Windows (Vulkan kit).
  • The speed of any two-card setup other than i7's measured 52 tok/s (research build, Linux).

Source

  • Engine: upstream llama.cpp f3269ef96 + 8fa13449e (the bias loader), with the CI workflows on branches cnm/serve-v0.3-rc1 (CUDA) and cnm/serve-v0.4-vulkan (Vulkan) of this fork.
  • The launcher source is not on these branches.
Assets 6
Loading