v0.2.1

Highlights

• [Feature] Image input for GGUF models. A llama.cpp mmproj file next to the model, or one passed with --mmproj, adds image input to Qwen3.5 / Qwen3.6 / Qwen3.8, Gemma-4, GLM-5.3-Flash and DeepSeek-V4-Flash-Vision-Exp GGUF files (#803, #810, #816).
• [Feature] More GGUF archs. deepseek4 (DeepSeek-V4 Flash, Flash-0731, Pro, Pro-0813) (#774), qwen3vl and qwen3vlmoe (text only) (#807).
• [Feature] Support Flint, from BF16 safetensors, the ModelOpt NVFP4 export or the official NVFP4 GGUF (#797).
• [Feature] Embedding table in host RAM. --embed-weights host keeps the input token embedding table in pinned host RAM and gives its VRAM to the expert cache and KV: decode is 8.7% faster on Qwen3.6-35B-A3B with 24 GiB of free VRAM (#806).

GGUF Support

• [Feature] Image input from an mmproj file (#803). llama.cpp keeps the vision tower in a separate mmproj-*.gguf. Without --mmproj, FreeToken takes a *mmproj*.gguf that fits the model (output width, tower type, image tokens) from the model file's own folder, BF16 before F32, F16 and quantized files; the log names the file it loads. With none, or with files of several models it cannot tell apart, the model serves text only; a repo that keeps each quant in a subfolder and the mmproj at the top needs --mmproj. An mmproj in the model folder turns image input on; --text-model-only (or --mm-disable vision) serves text only as before. The tower always runs in bf16, so a quantized mmproj is dequantized at load. See Image input.

  Family	GGUF arch	mmproj projector	PR
  Qwen3.5, Qwen3.6, Qwen3.8-27B	qwen35, qwen35moe	qwen3vl_merger	#803
  Qwen3.8-Flash-Next	qwen4exp	qwen3vl_merger	#803
  Gemma-4 26B-A4B, 31B	gemma4	gemma4v	#803
  Gemma-4 12B (audio tensors are skipped)	gemma4	gemma4uv	#803
  GLM-5.3-Flash	glm5-next	glm5v	#810
  DeepSeek-V4-Flash-Vision-Exp	deepseek4	deepseek4v	#816

• [Feature] deepseek4 arch: DeepSeek-V4 Flash, Flash-0731, Pro, Pro-0813, and the Vision-Exp text file. Routed experts keep the ggml types they ship with and run the clamped SwiGLU. A file without a rope.scaling block gets the family's YaRN values with a warning; llama.cpp serves such files unscaled (#774)

• [Feature] qwen3vl, qwen3vlmoe archs: Qwen3-VL dense and MoE, text only (#807)

• [Feature] flint arch: Flint (#797)

• [Feature] NVFP4 token_embd tables load (#801). A table with a .scale sidecar is still refused, as llama.cpp reads no scale for it.

• [Fix] Qwen GGUF files (llama.cpp pre-tokenizer qwen2 or qwen35) apply the NFC normalization of Qwen's tokenizer.json, so text with combining accents tokenizes as with the HF tokenizer (#807)

• Not supported yet: image input on qwen3vl / qwen3vlmoe files; GGUF files that carry the vision tensors themselves (the Ollama layout), which serve text only; image input on DeepSeek-V4-Flash-Vision-Exp text GGUFs converted before llama.cpp #28154 (no exp_probs_b_vl; reconvert); image requests on ggml-org's DeepSeek-V4-Flash-Vision-Exp text GGUF and its re-uploads, whose chat template renders no image placeholder (serve unsloth's release for images); FTWs converted from a GGUF, which stay text-only.

Model Support

• [Feature] Support Flint (FlintForCausalLM) from BF16 safetensors, the ModelOpt NVFP4 export and the official NVFP4 GGUF; the glm47 tool parser and the flint reasoning parser are picked automatically, and ModelOpt NVFP4 experts run on offload, cpu or hybrid (#797)
• [Fix] Take the weight files from model.safetensors.index.json when the checkpoint has one, so shards in a subdirectory (nvidia's *-FP4-Local releases) are found; such checkpoints found no weight files and failed later with an unrelated error (#819)

Engine

• [Feature] --embed-weights gpu|host (default gpu): host keeps the input token embedding table in pinned host RAM, and each step reads its rows over PCIe. It is refused when the table is tied to the LM head or is not a plain table (DF11 GLM-4.x) (#806)
• [Perf] hybrid: the expert-fetch kernel picks cache victims in 8192-slot tiles, so one fetch on a 23,197-slot pool spends 7.7 µs in it instead of 76.9 µs (#778)
• [Fix] hybrid: fix a race in the route rewrite of the expert-fetch kernel (#778)
• [Feature] FREETOKEN_HYBRID_FETCH_ROUND chooses how many misses a hybrid decode step fetches: min-wait (default, the previous rule), stochastic or prefer-fetch (#778)
• [Perf] hybrid with block-FP8 experts: routes that the CPU runs no longer also run on the GPU with weight 0; with 2 of 8 routes on the GPU, the GPU side of a decode MoE layer is about 3x faster (kernel benchmark on an RTX A1000) (#665)
• [Fix] The --moe-prefill-hit-d2d batch-copy probe waits for its buffer fill, so a busy GPU no longer fails the probe and turns --moe-prefill-hit-d2d off (#771)
• [Perf] GLM-5.3-Flash DSA prefill drops redundant activation copies: the attention peak of a 12,288-token chunk goes from 4,473 to 1,749 MiB, which removes the allocator warnings reported in #770 (#795)
• [Fix] DSA sparse attention (GLM-5.2, GLM-5.3-Flash) stops with a message to lower --max-prefill-length when a prefill chunk is past int32 addressing (about 65k tokens), instead of an illegal memory access (#795)
• [Docs] New guide, performance/prefill.md: find whether prefill with offloaded experts is bound by the PCIe copy or by compute, and what to change (#785)

Kernels

• [Perf] Route glm4_moe, glm_moe_dsa, glm5_next, minimax_m2 and minimax_m3 (sigmoid scores with a selection-only bias) through one Triton kernel, fused_topk_sigmoid, with the router logits in fp32 as SGLang and vLLM do: decode is 9.1% faster on MiniMax-M2.5-NVFP4 (H100, offload) (#804)
• [Fix] GLM-5.2 checkpoints and MiniMax-M2 GGUF files no longer round their F32 selection bias to bf16; the fp32 MiniMax-M2 gate takes 93 MiB more GPU memory (#804)
• [Fix] On sm_89 (RTX 40xx, L4, RTX 6000 Ada), quantize fp32 to e4m3 in one rounding. Triton before 3.9 rounds through fp16 there and moved some block-FP8 and per-tensor FP8 activations one e4m3 step toward zero (#805)
• [Fix] Keep fp32 inputs intact in the torch mRoPE fallbacks, which rotated the second half from already-rotated values (#641)

Server and CLI

• [Feature] GET / POST /v1/cache/flush?timeout=<seconds> drops every prefix-cache entry while the server is idle, without reallocating pools or recapturing CUDA graphs, so an eval harness can start each case from an empty cache without a restart. A busy server answers 503; with timeout it waits up to that long for idle and answers 504 if still busy. TP > 1 is not supported yet (#811)
• [Feature] --mmproj FILE picks the mmproj of a GGUF model (#803)

AMD ROCm (WIP, experimental)

• [Perf] --moe-prefill-hit-d2d works on ROCm, through hipMemcpyBatchAsync: cached experts are copied on the device instead of over PCIe. Time to first token at 512 prompt tokens drops 34% on the W7900D and 29% on the R9700 (Qwen3.5-35B-A3B-FP8, 40% of experts cached) (#771)
• [Perf] Disk PLE (Qwen3.8-Flash-Next) and the DeepSeek-V4.1 Engram tables sync with graph replay through HIP stream memory operations (wait-sync) instead of launch gating: decode is 12.7% faster on the R9700 and 0.8% on the W7900D (Qwen3.8-Flash-Next-FP8) (#776)

Breaking Changes & Deprecations

• GLM-5.2 FTWs converted by older builds load without an error but route with a bf16-rounded selection bias: bf16 keeps only 3 to 8 of nvidia/GLM-5.2-NVFP4's 256 bias values per layer in the layers checked. Restore the fp32 values in place with python scripts/ftw_hotfix.py --ftw <FTW dir> --repo nvidia/GLM-5.2-NVFP4 (or --source <checkpoint dir>); see docs/ftw-hotfix.md (#804).
• A GGUF model with a fitting *mmproj*.gguf in its folder now builds the vision tower and serves images; --text-model-only restores the text-only footprint (#803).

Contributors

@jason-fxz, @void-mckenzie, @LZ-QWQ, @taking-lying-flat, @iamanishx, @jesusjodar, @ischencheng, @chrisqianz, @zihaomu

First contributions this release:

• @jesusjodar in #797
• @chrisqianz in #819

Full changelog: v0.2.0...v0.2.1