Skip to content

cnm-serve v0.4 (Vulkan: AMD/Intel, integrated GPUs, gpt-oss-20b)

Choose a tag to compare

View all tags
@oisee oisee released this 28 Oct 15:29
cnm-serve-v0.4
8553cd1

cnm-serve v0.4: a Vulkan kit for AMD and Intel GPUs and laptops with an integrated GPU. It serves gpt-oss-20b (MXFP4, 12 GB) on machines with 16-64 GB RAM. The NVIDIA / Qwen3.8-Flash-Next kit stays at v0.3 (cnm-serve-win64-cuda.zip from the v0.3 release).

Status

  • Built and tested on Linux only. The Windows parts have not run on Windows: GPU detection through DXGI, the window, file dialogs, and the Vulkan driver on a Radeon 860M.
  • The server, the model and tool calling ran once on Linux, with the Linux build of the same engine commit, on a desktop AMD RX 580 8 GB (not the 860M). Results are below.
  • The speed on a Radeon 860M is an estimate (12-22 tok/s writing), not a measurement.

What is new

  • cnm-serve-win64-vulkan.zip. It is a separate zip because the CUDA runtime is about 500 MB that AMD and Intel machines never use. It is about 41 MB (89 MB unpacked). Its cnm-serve.exe (v0.4) knows both engines; the CUDA zip from v0.3 still carries the v0.3 program.
  • gpt-oss-20b from ggml-org/gpt-oss-20b-GGUF, pinned to revision 15dcfc7e. The size and sha256 come from the HF API, and the license is Apache 2.0. The window gives it Download, Use existing files... and Links....
    • Integrated GPU: every layer goes on the GPU, with no expert cache and no router bias.
    • Other settings: KV cache q8_0/q4_0, -ub 512, flash attention, and the model's own chat template (--jinja). The context is 32768, lowered to 16384 when little RAM is free.
    • Small discrete card or little shared memory: llama.cpp --fit keeps part of the experts on the CPU. A GPU out-of-memory at start retries that way automatically, and offload = fit is then saved in cnm-serve.ini (set it back to auto to try all layers again).
  • Optional speculative decoding.
    • EAGLE3 draft (eagle3-gpt-oss-20b-Q8_0.gguf, 0.9 GB) with its own Download, Use existing files and Links, and a "Use draft (EAGLE3)" checkbox. It runs as -md ... --spec-type draft-eagle3.
    • n-gram drafting checkbox (ngram-simple).
    • Defaults: EAGLE3 is on when downloaded, n-gram is off. Both can be set in cnm-serve.ini (draft, ngram).
  • GPU detection without nvidia-smi.
    • Windows DXGI lists the adapters with vendor id, dedicated and shared memory. Discrete cards are chosen before integrated ones, NVIDIA before other discrete cards. The Basic Render Driver is skipped.
    • An integrated GPU is recognised by its name (Radeon 8xxM/7xxM, Radeon Graphics, Intel UHD/Iris/Arc 1x0V) or by less than 4 GB of dedicated memory.
    • nvidia-smi is asked only for an NVIDIA adapter.
    • The Vulkan kit matches the adapter to llama-server's --list-devices, and falls back to that list when there is nothing else (Linux).
  • The window shows what this machine can run. The top line shows the GPU, the RAM and how much is free, and what the plan does with them, e.g. "integrated GPU: all layers on GPU, expert cache off". Models the machine cannot run are greyed out with the reason: "needs >= 64 GB RAM and an NVIDIA GPU", or "needs the CUDA kit".
  • Fixed:
    • On a machine without an NVIDIA card, the window showed "no NVIDIA GPU found (nvidia-smi ...)" and a negative "cache -6000 MiB". The cache size is never negative now, in the shared code too.
    • The ini's "out of memory? use N" hint is never negative.
  • New cnm-serve.ini keys: draft, draft_path, ngram, offload, kv_cache, backend, model_path_gpt_oss_20b.

Requirements (Vulkan kit)

  • Windows 10 or 11, x64.
  • A GPU with a current driver that includes Vulkan (any AMD Adrenalin/PRO, Intel or NVIDIA driver).
  • RAM: 32 GB recommended, with about 14 GB free at Start. 16 GB is not recommended.
  • 13 GB of disk.
  • No admin rights needed.

What was tested

  • Launcher core, Linux, go test -race.
    • Covered: DXGI adapter choice (the AMD laptop, a hybrid NVIDIA + AMD laptop, the --list-devices fallback, the no-GPU message), integrated-GPU recognition, the Alice-laptop plan and top line, Qwen greyed out, the gpt-oss command line, the Vulkan out-of-memory retry to fit, the new ini keys, the gpt-oss and EAGLE3 downloads from their own repo, and Use existing files.
    • The v0.3 tests still pass.
    • Mutations: 80 deliberate code changes, each caught by a failing test (tools/mutations.py, evidence/mutations-v0.4.txt). 31 of them are new: GPU detection, tier choice, the negative cache fix, the gpt-oss command line and downloads, and the critic-round fixes.
  • Real run, Linux on i3, AMD RX 580 8 GB, RADV Vulkan, 4 CPU threads.

    • Engine: llama.cpp f3269ef96 + bias patch, Linux Vulkan build.

    • Model files: the sha256 of both matched HF (checked in place by Use existing files).

    • The headless launcher (same core as the Windows window) started the server. Proved by content:

      • a chat reply via /v1/chat/completions and via /v1/messages;
      • a tool call via /v1/chat/completions (get_weather {"city":"Oslo"}), and its result used in the next answer;
      • a tool_use block via /v1/messages;
      • 401 without the key.

    • A/B, one run per arm, a 512-token code edit at temperature 0, with half of the experts on the CPU (8 GB card):

      arm decode draft acceptance
      plain 15.5 tok/s –
      EAGLE3 12.8 tok/s 74%
      EAGLE3 + n-gram 10.7 tok/s 67%

      The draft arms keep 2 GB more of the card free, so the comparison is confounded. It says little about an integrated GPU.

  • Windows binaries.

What is untested

  • Anything on Windows: DXGI detection on a real machine, the window and its greyed-out rows, the Vulkan driver and shared-memory limits on the Radeon 860M, -ngl 999 with 12 GB in shared memory, file dialogs.
  • Speed on any integrated GPU. Whether EAGLE3 or n-gram help there.
  • Answer quality of gpt-oss-20b on ABAP or agent tasks.

Source

  • Engine: branch cnm/serve-v0.4-vulkan on this fork, which is upstream f3269ef96 + the bias-loader commit + the Vulkan workflow.
  • The launcher source is not on this branch.
Assets 4
Loading