Skip to content

v0.2.0

Latest
Latest

Choose a tag to compare

View all tags
@github-actions github-actions released this 21 Oct 00:37
· 3 commits to main since this release
v0.2.0
127aa0a
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Highlights

  • GGUF Support. llama.cpp GGUF files serve directly with ft serve --model <local file>.gguf; quantized tensors run on llama.cpp's own kernels (ggml-cuda on the GPU, ggml-cpu for hybrid / cpu experts). Supported GGUF archs:
    • qwen2 (Qwen2 / Qwen2.5), qwen3 and qwen3moe (Qwen3 dense and MoE)
    • qwen35 and qwen35moe (Qwen3.5 / Qwen3.6 dense and MoE, Qwen3.8-27B), qwen4exp (Qwen3.8-Flash-Next)
    • gemma4 (Gemma-4 12B, 26B-A4B, 31B)
    • llama (Llama 3.x, and Mistral files converted before llama.cpp's mistral3 arch)
    • muse-glimmer (Muse-Glimmer)
    • minimax-m2 (MiniMax-M2 / M2.x), minimax-m3 (MiniMax-M3)
    • glm4moe (GLM-4.5, GLM-4.5-Air, GLM-4.6, GLM-4.7), glm5-next (GLM-5.3-Flash)
  • Support new models.
    • DeepSeek-V4.1-Flash, text and images (#603)
    • Image input for DeepSeek-V4-Flash-Vision-Exp (#502)
  • Faster NVFP4 on sm_120 (RTX 50xx, RTX PRO Blackwell). NVFP4 checkpoints with static activation scales run on NVFP4 activations (W4A4): at 128 or more tokens per expert the MoE layer is 2.54x faster on the RTX PRO 6000 and 3.72x on the RTX 5090 (#736), and Qwen3.8-27B-NVFP4 and Gemma-4-31B-IT-NVFP4 reach the first token 1.6x to 2.7x faster at 1k to 8k prompt tokens (#737).

GGUF Support

FreeToken now serves llama.cpp GGUF checkpoints (#690). Quantized tensors keep the ggml type they have in the file and run on llama.cpp's kernels, vendored from llama.cpp 8c2134a8: ggml-cuda MMQ / MMVQ on the GPU, and ggml-cpu row kernels for experts on the CPU (#704). The model config, chat template and stop tokens come from the file's metadata.

  • Weight types (per tensor; a file preset such as Q4_K_M or UD-Q8_K_XL mixes several):
    • Supported: F32, F16, BF16; Q4_0, Q4_1, Q5_0, Q5_1, Q8_0; Q2_K, Q3_K, Q4_K, Q5_K, Q6_K; IQ1_S, IQ2_XXS, IQ2_XS, IQ2_S, IQ3_XXS, IQ3_S, IQ4_NL, IQ4_XS; MXFP4, NVFP4; Q1_0, Q2_0. The layers of one file may use different types, and an expert layer may pair a float projection with a quantized one (#702).
    • NVFP4 GGUFs converted from ModelOpt checkpoints carry per-tensor .scale tensors; the gemma4, llama, qwen3, qwen3moe, qwen35 and qwen35moe loaders apply them on dense linears and on every MoE strategy (#732).
    • Not supported yet: IQ1_M (no MMQ kernel); NVFP4 token embeddings; NVFP4 .scale tensors in qwen2, qwen4exp, muse-glimmer, minimax-m2, minimax-m3 and glm4moe files (refused at load); --dtype float16 with NVFP4 .scale tensors; TQ1_0 and TQ2_0; Q4_0_4_4 / Q4_0_4_8 / Q4_0_8_8, which llama.cpp has removed; types defined only in llama.cpp forks such as ik_llama.cpp; expert layers whose gate and up projections use different types.
  • MoE strategies: fused and offload for every type (#690); hybrid and cpu for every type on x86-64 CPUs, where ggml-cpu picks the best of llama.cpp's x86 CPU builds at startup (#704). Layers of different types get their own slot pools in the expert cache (#689, #700); cpu refuses files whose expert layers use several types. ft bench bw --model <file>.gguf measures the hybrid fetch split for each expert layer kind (#734).
  • scripts/gguf_check/ in the source tree checks whether GGUF files load (config, weights, kernel coverage, chat template) from header-only stubs, without downloading the weights (#756).
  • Weights past the 32-bit block offsets of MMQ / MMVQ raise instead of running (#699).
  • Not supported yet: image input (mmproj files), Gemma-4 E2B / E4B, the mistral3 arch, Mixtral, GLM-4.7-Flash (deepseek2 arch), glm4moe files with a fused attn_qkv, tensor parallelism.

Supported architectures

Family GGUF arch PR
Qwen2 / Qwen2.5 qwen2 #707
Qwen3 dense, Qwen3 MoE qwen3, qwen3moe #707
Qwen3.5 / Qwen3.6 dense and MoE, Qwen3.8-27B qwen35, qwen35moe #701
Qwen3.8-Flash-Next qwen4exp #690
Gemma-4 26B-A4B (every ggml type; was Q4_0 only) gemma4 #690
Gemma-4 12B, 31B gemma4 #706
Llama 3.x, Mistral (files converted before llama.cpp's mistral3 arch) llama #722
Muse-Glimmer muse-glimmer #724
MiniMax-M2 / M2.x, MiniMax-M3 minimax-m2, minimax-m3 #725
GLM-4.5, GLM-4.5-Air, GLM-4.6, GLM-4.7 glm4moe #731
GLM-5.3-Flash glm5-next #721

Model Support

  • Support DeepSeek-V4.1-Flash (DeepseekV41ForCausalLM), text and images (#603)
    • --swa-decoder-replay bounded|exact: bounded (default) is the tech report's decoder SWA replay, exact runs the decoder on every prompt token
    • The two Engram tables stream from the checkpoint shards on demand
    • reasoning_effort takes low / high / max; a 1-100 budget goes through chat_template_kwargs; DSML tool calls are parsed
  • Image input for DeepSeek-V4-Flash-Vision-Exp (DeepseekV4ForCausalLM) (#502)
  • Qwen3.5 / Qwen3.6 / Qwen3.8-27B: add the zero-centred norm weights' +1 in fp32 at runtime, as transformers does, instead of folding it into the bf16 weight (#755)
  • Qwen3.8-Flash-Next:
    • Serve the model card's 1M-token YaRN recipe through --hf-overrides (#573)
    • Reduce QSA block top-k candidates past 16 splits: the per-layer top-k of one decode step at ~985k tokens of context goes from 845 to 45 µs on an RTX 3090 Ti (#571)
    • Load compressed-tensors exports (per-channel fp8 dense layers, NVFP4 experts) and bf16 PLE tables (#759)
    • Fence disk PLE staging against in-flight lookups (#588)
  • Keep the Muse-Glimmer attention gate as its own linear next to qkv_proj (#741)
  • Preserve the GLM-4 router correction bias in fp32 (#469)
  • Raise WeightLoadError when the checkpoint cannot be read (#518)
  • Add scripts/checkpoint_diagnose/ to the source tree: tells whether FreeToken or the checkpoint is at fault when an HF checkpoint does not load, and fixes a non-standard local checkpoint after showing the plan (#762)

Quantization

  • Keep NVFP4 weight global scales in fp32. They were converted to fp16, where many of them fell in the subnormal range (#739).
  • On sm_120 (RTX 50xx, RTX PRO Blackwell), NVFP4 checkpoints with static activation scales (ModelOpt NVFP4, compressed-tensors NVFP4) run on NVFP4 activations (W4A4):
    • MoE prefill from 4 tokens per expert: at 128 or more tokens per expert the MoE layer is 2.54x faster on the RTX PRO 6000 and 3.72x on the RTX 5090 (geomean over 9 model shapes) (#736)
    • Linears: W4A4 prefill and new W4A16 kernels for decode; time to first token at 1k to 8k prompt tokens is 1.6x to 2.7x faster on Qwen3.8-27B-NVFP4 and Gemma-4-31B-IT-NVFP4 (#737)
    • --quant-backend moe.nvfp4=triton and linear.nvfp4=triton keep W4A16

Engine

  • Split the offload expert cache into slot pools: --moe-cache-scope unified|per-layer, --moe-cache-split equal|profile:<path>, and --moe-cache-mem to size the cache in memory (#689)
  • Serve models whose MoE layers use different expert formats, one slot pool per format (#700)
  • --moe-strategy fused serves NVFP4 and MXFP4 experts through the expert banks (#601)
  • Serve block-FP8 experts on the cpu and hybrid strategies (#662)
  • ft bench bw --model <checkpoint> benches every MoE layer kind of a GGUF or HF checkpoint, and hybrid takes its fetch split per layer (#734)
  • Page-locked expert banks: when pinned memory is capped, hybrid with --moe-cpu-layers OS-locks the expert banks past the pin limit, and the locked layers keep GPU slots and fetch part of their misses through a pinned staging buffer. This also works for models whose expert layers need several slot pools, such as GGUF files that mix expert types. Adds --moe-locked-fetch, --moe-locked-max-fetch and --moe-collect-stats (#742)
  • Spin-wait the CPU MoE pool between layers: +20% batch-1 hybrid decode on Qwen3.6-35B-A3B-NVFP4 (H100) (#619)
  • Split hybrid decode routes inside the ensure kernel (#685)
  • Split the CPU MoE executor into weight formats and GPU sync backends (#670)
  • Widen the expert gather on NVLink-C2C hosts (GH200 / GB200 / GB300): 222 → 412 GB/s on GH200 (#610)
  • Upload CUDA graphs at capture so a stream wait cannot block the first launch (#697)
  • Under overlap scheduling, requests now end with max_tokens tokens instead of one fewer (#546)
  • Preserve greedy sampling in mixed batches (#471)
  • Clone inserted radix keys to release request buffers (#559)

Kernels

  • Rewrite the Triton NVFP4 MoE prefill GEMM: 6.6x to 16.1x faster; Gemma-4 26B-A4B NVFP4 prefill with fused goes from 5,571 to 35,732 tok/s at 4.6k tokens on H100 (#723)
  • Speed up the NVFP4 MoE prefill GEMM on RTX GPUs: a warp-specialized Gluon kernel on sm_120, 1.50x faster per MoE layer at 8192 tokens on the RTX PRO 6000 (geomean over 6 model shapes) (#733); pipelined scale loads on GeForce sm_86 / sm_89 (#735)
  • Split Triton decode attention by context length and SM count: batch-1 decode at 128k on H100 goes from 77 to 174 tok/s on Gemma-4 26B-A4B and from 90 to 213 tok/s on gpt-oss-20b (#729)
  • Copy Triton attention metadata without blocking the host (#726)
  • Rework kernels that Triton 3.8 compiles slower: NVFP4 MoE decode (#631), Triton extend attention and Gemma-4 sliding-window decode (#632), DeepSeek-V4 sparse attention and gated pool (#633), the DeepSeek-V4.1 indexer (#634), and block-FP8 MoE decode and the gpt-oss MXFP4 GEMV (#653)
  • Avoid NaN in the Triton sampling softmax when a chunk starts with -inf (#654)
  • Correct JIT tensor matcher checks and PDL arch guards (#638)
  • Pin the tvm-ffi JIT arch to the bound GPU (#521)

Server and CLI

  • Add --api-key authentication to ft serve; ft shell, ft ctl, ft launch and ft daemon send the key (#639)
  • Keep Anthropic inline system messages in place for prefix reuse; --anthropic-inline-system auto|preserve|fold (#608)
  • Apply presence_penalty and frequency_penalty (#527)
  • Add --hf-overrides JSON to override the checkpoint config, with vLLM's semantics (#573)
  • Accept file://localhost/... URIs for local media (#758)
  • Report the backend KV page size in /v1/stats (#599)

AMD ROCm (WIP, experimental)

  • Build from source and run on RDNA3 and RDNA4 GPUs; tested on gfx1100 and gfx1201 (#132, #587)
  • Gate CUDA-only optional backends on ROCm; FlashInfer, sgl_kernel and vLLM kernels fall back to the built-in kernels (#134)
  • cpu and hybrid MoE sync with the GPU through HIP stream memory operations (#670)
  • Clamp the QSA KV tile to the device LDS budget on RDNA (#587)
  • Avoid a Triton 3.8 miscompile of the MoE alignment on ROCm (#691)

Dependencies

  • torch >=2.14,<2.15 (was >=2.11,<2.12), torchvision >=0.29,<0.30, Triton >=3.6,<3.9 (was ==3.6.0), sglang-kernel 0.4.9 (was 0.4.5); source builds compile the C++ extensions as C++20 (#630).
  • transformers >=5.18,<5.19 (was >=5.16,<5.17) (#698).
  • flashlib 0.4.0 (was 0.3.0) and numpy <3 (was <2.5); numba is no longer installed (#669).
  • [fi] pins flashinfer-python 0.6.18.post1 (was >=0.6,<0.7), and install.sh installs flashinfer-python, -cubin and -jit-cache at that one version (#545).
  • The gguf package is no longer a dependency; gguf-py is vendored (#690).
  • sglang-kernel resolves from PyPI; the sglang wheel index is dropped (#524).
  • install.sh writes FREETOKEN_HOME to environment.d instead of FREETOKEN_FT_BIN (#576).

Breaking Changes & Deprecations

  • Some FTW checkpoints converted by older builds must be reconverted with ft checkpoint:
    • experts converted with --moe-backend fused or triton are dense and no longer load; ft checkpoint now always writes experts as banks (--moe-backend is kept for compatibility) (#601)
    • NVFP4 FTWs that store the global scales as fp16 are refused at load (#739)
    • Muse-Glimmer FTWs with the attention gate fused into qkvg_proj no longer load (#741)
    • Qwen3.5 / Qwen3.6 / Qwen3.8-27B FTWs load without an error but give wrong output: their norm weights hold the +1 already (#755)
  • FREETOKEN_API_KEY in the environment of ft serve now turns authentication on (#639).
  • ft bench bw profiles from older builds are ignored; rerun ft bench bw, with --model <checkpoint> to cover every layer kind. A layer without a matching row fetches at most 1 expert per step under hybrid (#662, #734).
  • --moe-strategy auto no longer switches to hybrid from the ft bench bw profile; pass --moe-strategy hybrid (#734).

Contributors

@jason-fxz, @zihaomu, @andy-yang-1, @taking-lying-flat, @LZ-QWQ, @YevheniiKotyrlo, @void-mckenzie, @bouclem, @nekomario28, @yunwei37, @xinze-zheng, @SuperHotDogCat, @shubham3-ucb, @ischencheng, @gdevenyi, @ChenyuHeee, @zhangnju, @bettercallcaleb

First contributions this release:

Full changelog: v0.1.3...v0.2.0

Contributors

gdevenyi, zhangnju, and 16 other contributors
Assets 7
Loading