v0.2.0

Highlights

• GGUF Support. llama.cpp GGUF files serve directly with ft serve --model <local file>.gguf; quantized tensors run on llama.cpp's own kernels (ggml-cuda on the GPU, ggml-cpu for hybrid / cpu experts). Supported GGUF archs:
  • qwen2 (Qwen2 / Qwen2.5), qwen3 and qwen3moe (Qwen3 dense and MoE)
  • qwen35 and qwen35moe (Qwen3.5 / Qwen3.6 dense and MoE, Qwen3.8-27B), qwen4exp (Qwen3.8-Flash-Next)
  • gemma4 (Gemma-4 12B, 26B-A4B, 31B)
  • llama (Llama 3.x, and Mistral files converted before llama.cpp's mistral3 arch)
  • muse-glimmer (Muse-Glimmer)
  • minimax-m2 (MiniMax-M2 / M2.x), minimax-m3 (MiniMax-M3)
  • glm4moe (GLM-4.5, GLM-4.5-Air, GLM-4.6, GLM-4.7), glm5-next (GLM-5.3-Flash)
• Support new models.
  • DeepSeek-V4.1-Flash, text and images (#603)
  • Image input for DeepSeek-V4-Flash-Vision-Exp (#502)
• Faster NVFP4 on sm_120 (RTX 50xx, RTX PRO Blackwell). NVFP4 checkpoints with static activation scales run on NVFP4 activations (W4A4): at 128 or more tokens per expert the MoE layer is 2.54x faster on the RTX PRO 6000 and 3.72x on the RTX 5090 (#736), and Qwen3.8-27B-NVFP4 and Gemma-4-31B-IT-NVFP4 reach the first token 1.6x to 2.7x faster at 1k to 8k prompt tokens (#737).

GGUF Support

FreeToken now serves llama.cpp GGUF checkpoints (#690). Quantized tensors keep the ggml type they have in the file and run on llama.cpp's kernels, vendored from llama.cpp 8c2134a8: ggml-cuda MMQ / MMVQ on the GPU, and ggml-cpu row kernels for experts on the CPU (#704). The model config, chat template and stop tokens come from the file's metadata.

• Weight types (per tensor; a file preset such as Q4_K_M or UD-Q8_K_XL mixes several):
  • Supported: F32, F16, BF16; Q4_0, Q4_1, Q5_0, Q5_1, Q8_0; Q2_K, Q3_K, Q4_K, Q5_K, Q6_K; IQ1_S, IQ2_XXS, IQ2_XS, IQ2_S, IQ3_XXS, IQ3_S, IQ4_NL, IQ4_XS; MXFP4, NVFP4; Q1_0, Q2_0. The layers of one file may use different types, and an expert layer may pair a float projection with a quantized one (#702).
  • NVFP4 GGUFs converted from ModelOpt checkpoints carry per-tensor .scale tensors; the gemma4, llama, qwen3, qwen3moe, qwen35 and qwen35moe loaders apply them on dense linears and on every MoE strategy (#732).
  • Not supported yet: IQ1_M (no MMQ kernel); NVFP4 token embeddings; NVFP4 .scale tensors in qwen2, qwen4exp, muse-glimmer, minimax-m2, minimax-m3 and glm4moe files (refused at load); --dtype float16 with NVFP4 .scale tensors; TQ1_0 and TQ2_0; Q4_0_4_4 / Q4_0_4_8 / Q4_0_8_8, which llama.cpp has removed; types defined only in llama.cpp forks such as ik_llama.cpp; expert layers whose gate and up projections use different types.
• MoE strategies: fused and offload for every type (#690); hybrid and cpu for every type on x86-64 CPUs, where ggml-cpu picks the best of llama.cpp's x86 CPU builds at startup (#704). Layers of different types get their own slot pools in the expert cache (#689, #700); cpu refuses files whose expert layers use several types. ft bench bw --model <file>.gguf measures the hybrid fetch split for each expert layer kind (#734).
• scripts/gguf_check/ in the source tree checks whether GGUF files load (config, weights, kernel coverage, chat template) from header-only stubs, without downloading the weights (#756).
• Weights past the 32-bit block offsets of MMQ / MMVQ raise instead of running (#699).
• Not supported yet: image input (mmproj files), Gemma-4 E2B / E4B, the mistral3 arch, Mixtral, GLM-4.7-Flash (deepseek2 arch), glm4moe files with a fused attn_qkv, tensor parallelism.

Supported architectures

Family	GGUF arch	PR
Qwen2 / Qwen2.5	qwen2	#707
Qwen3 dense, Qwen3 MoE	qwen3, qwen3moe	#707
Qwen3.5 / Qwen3.6 dense and MoE, Qwen3.8-27B	qwen35, qwen35moe	#701
Qwen3.8-Flash-Next	qwen4exp	#690
Gemma-4 26B-A4B (every ggml type; was Q4_0 only)	gemma4	#690
Gemma-4 12B, 31B	gemma4	#706
Llama 3.x, Mistral (files converted before llama.cpp's mistral3 arch)	llama	#722
Muse-Glimmer	muse-glimmer	#724
MiniMax-M2 / M2.x, MiniMax-M3	minimax-m2, minimax-m3	#725
GLM-4.5, GLM-4.5-Air, GLM-4.6, GLM-4.7	glm4moe	#731
GLM-5.3-Flash	glm5-next	#721

Model Support

• Support DeepSeek-V4.1-Flash (DeepseekV41ForCausalLM), text and images (#603)
  • --swa-decoder-replay bounded|exact: bounded (default) is the tech report's decoder SWA replay, exact runs the decoder on every prompt token
  • The two Engram tables stream from the checkpoint shards on demand
  • reasoning_effort takes low / high / max; a 1-100 budget goes through chat_template_kwargs; DSML tool calls are parsed
• Image input for DeepSeek-V4-Flash-Vision-Exp (DeepseekV4ForCausalLM) (#502)
• Qwen3.5 / Qwen3.6 / Qwen3.8-27B: add the zero-centred norm weights' +1 in fp32 at runtime, as transformers does, instead of folding it into the bf16 weight (#755)
• Qwen3.8-Flash-Next:
  • Serve the model card's 1M-token YaRN recipe through --hf-overrides (#573)
  • Reduce QSA block top-k candidates past 16 splits: the per-layer top-k of one decode step at ~985k tokens of context goes from 845 to 45 µs on an RTX 3090 Ti (#571)
  • Load compressed-tensors exports (per-channel fp8 dense layers, NVFP4 experts) and bf16 PLE tables (#759)
  • Fence disk PLE staging against in-flight lookups (#588)
• Keep the Muse-Glimmer attention gate as its own linear next to qkv_proj (#741)
• Preserve the GLM-4 router correction bias in fp32 (#469)
• Raise WeightLoadError when the checkpoint cannot be read (#518)
• Add scripts/checkpoint_diagnose/ to the source tree: tells whether FreeToken or the checkpoint is at fault when an HF checkpoint does not load, and fixes a non-standard local checkpoint after showing the plan (#762)

Quantization

• Keep NVFP4 weight global scales in fp32. They were converted to fp16, where many of them fell in the subnormal range (#739).
• On sm_120 (RTX 50xx, RTX PRO Blackwell), NVFP4 checkpoints with static activation scales (ModelOpt NVFP4, compressed-tensors NVFP4) run on NVFP4 activations (W4A4):
  • MoE prefill from 4 tokens per expert: at 128 or more tokens per expert the MoE layer is 2.54x faster on the RTX PRO 6000 and 3.72x on the RTX 5090 (geomean over 9 model shapes) (#736)
  • Linears: W4A4 prefill and new W4A16 kernels for decode; time to first token at 1k to 8k prompt tokens is 1.6x to 2.7x faster on Qwen3.8-27B-NVFP4 and Gemma-4-31B-IT-NVFP4 (#737)
  • --quant-backend moe.nvfp4=triton and linear.nvfp4=triton keep W4A16

Engine

• Split the offload expert cache into slot pools: --moe-cache-scope unified|per-layer, --moe-cache-split equal|profile:<path>, and --moe-cache-mem to size the cache in memory (#689)
• Serve models whose MoE layers use different expert formats, one slot pool per format (#700)
• --moe-strategy fused serves NVFP4 and MXFP4 experts through the expert banks (#601)
• Serve block-FP8 experts on the cpu and hybrid strategies (#662)
• ft bench bw --model <checkpoint> benches every MoE layer kind of a GGUF or HF checkpoint, and hybrid takes its fetch split per layer (#734)
• Page-locked expert banks: when pinned memory is capped, hybrid with --moe-cpu-layers OS-locks the expert banks past the pin limit, and the locked layers keep GPU slots and fetch part of their misses through a pinned staging buffer. This also works for models whose expert layers need several slot pools, such as GGUF files that mix expert types. Adds --moe-locked-fetch, --moe-locked-max-fetch and --moe-collect-stats (#742)
• Spin-wait the CPU MoE pool between layers: +20% batch-1 hybrid decode on Qwen3.6-35B-A3B-NVFP4 (H100) (#619)
• Split hybrid decode routes inside the ensure kernel (#685)
• Split the CPU MoE executor into weight formats and GPU sync backends (#670)
• Widen the expert gather on NVLink-C2C hosts (GH200 / GB200 / GB300): 222 → 412 GB/s on GH200 (#610)
• Upload CUDA graphs at capture so a stream wait cannot block the first launch (#697)
• Under overlap scheduling, requests now end with max_tokens tokens instead of one fewer (#546)
• Preserve greedy sampling in mixed batches (#471)
• Clone inserted radix keys to release request buffers (#559)

Kernels

• Rewrite the Triton NVFP4 MoE prefill GEMM: 6.6x to 16.1x faster; Gemma-4 26B-A4B NVFP4 prefill with fused goes from 5,571 to 35,732 tok/s at 4.6k tokens on H100 (#723)
• Speed up the NVFP4 MoE prefill GEMM on RTX GPUs: a warp-specialized Gluon kernel on sm_120, 1.50x faster per MoE layer at 8192 tokens on the RTX PRO 6000 (geomean over 6 model shapes) (#733); pipelined scale loads on GeForce sm_86 / sm_89 (#735)
• Split Triton decode attention by context length and SM count: batch-1 decode at 128k on H100 goes from 77 to 174 tok/s on Gemma-4 26B-A4B and from 90 to 213 tok/s on gpt-oss-20b (#729)
• Copy Triton attention metadata without blocking the host (#726)
• Rework kernels that Triton 3.8 compiles slower: NVFP4 MoE decode (#631), Triton extend attention and Gemma-4 sliding-window decode (#632), DeepSeek-V4 sparse attention and gated pool (#633), the DeepSeek-V4.1 indexer (#634), and block-FP8 MoE decode and the gpt-oss MXFP4 GEMV (#653)
• Avoid NaN in the Triton sampling softmax when a chunk starts with -inf (#654)
• Correct JIT tensor matcher checks and PDL arch guards (#638)
• Pin the tvm-ffi JIT arch to the bound GPU (#521)

Server and CLI

• Add --api-key authentication to ft serve; ft shell, ft ctl, ft launch and ft daemon send the key (#639)
• Keep Anthropic inline system messages in place for prefix reuse; --anthropic-inline-system auto|preserve|fold (#608)
• Apply presence_penalty and frequency_penalty (#527)
• Add --hf-overrides JSON to override the checkpoint config, with vLLM's semant...

Nightly (rolling)

FreeToken nightly (rolling)

Automated build of main. This release is overwritten every night: the nightly tag moves to the built commit and the previous wheels are deleted. Do not pin to this tag; pin to a wheel URL below, whose filename embeds the commit it was built from.

Install (Linux x86_64, CPython 3.12, CUDA 13.0):

uv pip install \
  "freetoken[accel] @ https://github.com/FlashML-org/FreeToken/releases/download/nightly/freetoken-0.2.0%2Bgcc4aa085b-cp312-cp312-linux_x86_64.whl" \
  "https://github.com/FlashML-org/FreeToken/releases/download/nightly/freetoken_kernel_cache-0.2.0%2Bcu130.gcc4aa085b-py3-none-linux_x86_64.whl"

commit	built (UTC)	python	cuda	runtime	kernel-cache
cc4aa085b	2026-10-21 11:20	cp312	cu130	4.8 MB	78.1 MB

sha256

c488b17ac0b9f4319b20c25be4b5ee5a7f160bfd0feba93922d6139b0adf91a8  freetoken-0.2.0+gcc4aa085b-cp312-cp312-linux_x86_64.whl
41fba8089abab8c8977c96f77e190ff815eede100e22650ca24b7555b8eb3f6f  freetoken_kernel_cache-0.2.0+cu130.gcc4aa085b-py3-none-linux_x86_64.whl

Machine-readable: engine-linux_x86_64.json next to the wheels (schema 1).
Stable releases: PyPI freetoken and the v* releases here.

v0.1.3

Highlights

• Image input. Qwen3-VL, Qwen3.6, Qwen3.8-Flash-Next, Gemma-4, GLM-5.3-Flash, MiniMax-M3 and Muse-Glimmer-30B accept images through the OpenAI, Anthropic and Responses APIs. Vision towers stream their weights from host memory; --text-model-only serves a multimodal checkpoint as text.
• Support new models.
  • Qwen3.8-Flash-Next (#257)
  • GLM-5.3-Flash (#332).
• Quantization rebuilt (#418). One QuantConfig → QuantScheme → kernel path for every layer; kernels are selected with --quant-backend layer[.kind]=name, and --moe-backend becomes --moe-strategy.

Multimodal

FreeToken now serves vision-language models (#454): an image goes through the checkpoint's own processor and vision tower, then down the same scheduler, radix cache and MoE offload path as text.

• Engine: image spans are content-keyed in the radix cache; encoded embeddings sit in an encoder cache between prefill chunks; the vision tower streams its weights from pinned host memory two blocks at a time.
• Server API: OpenAI image_url, Anthropic image blocks and Responses input_image (URL, base64, file://); tool-result images are carried onto the next user turn (db64879); /v1/stats reports model.input_modalities.
• CLI: --text-model-only, --image-min-tokens / --image-max-tokens, --mm-processor-kwargs, --mm-encoder-weights, --allowed-media-domains, --allowed-local-media-path — see the CLI reference.
• ft launch: declares image input to Codex, OpenCode, OpenClaw and dsh when the server serves it (8ff0cce).

Supported models

Family	Arch	PR
Qwen3-VL	Qwen3VLForConditionalGeneration, Qwen3VLMoeForConditionalGeneration	#454
Qwen3.6 / Qwen3.5	Qwen3_5MoeForConditionalGeneration, Qwen3_5ForConditionalGeneration	#454
Qwen3.8-Flash-Next	Qwen4ExpForConditionalGeneration	#454
Gemma-4	Gemma4ForConditionalGeneration, Gemma4UnifiedForConditionalGeneration	#467
GLM-5.3-Flash	Glm5NextForConditionalGeneration	#479
MiniMax-M3	MiniMaxM3SparseForConditionalGeneration	#480
Muse-Glimmer-30B	MuseGlimmerForConditionalGeneration	#481

Model Support

• Support Qwen3.8-Flash-Next (#257)
  • Stream the PLE n-gram table from disk with --ple-backend disk (#311)
  • Load block-fp8 dense projections natively (#428)
  • Write the PLE table next to the FTW (#420)
• Support GLM-5.3-Flash (#332)
• Add scripts/ftw_hotfix.py to repair FTW checkpoints from older builds (#424)
• Serve and repair FTW checkpoints around the vision encoder (#486)
• A truncated FTW shard raises OSError instead of loading garbage (#25)
• Download the shards the safetensors index names (#336)

Quantization

Checkpoint quantization is rebuilt on a QuantConfig → QuantScheme → QuantMethod path (#418, #427): the checkpoint's quantization_config (ModelOpt, compressed-tensors, block-fp8, mxfp4) decides the scheme of every projection, and each layer declares its tensors and picks its kernel from that, instead of per-family format branches. New formats and kernels plug in once; --quant-backend layer[.kind]=name selects the kernel.

• Qwen3.5 / Qwen3.6 MoE: read every checkpoint layout through the QuantConfig (#438)
• Qwen3.8-Flash-Next: take the expert quant kind from the checkpoint QuantConfig (#426)
• Load ModelOpt exports that ship no input_scale (#462)
• Detect NVFP4 experts behind a mixed-precision compressed-tensors format (#343)

Engine

• Per-layer host-bank residency: split lock-CPU / pin-GPU layers, auto selection under a capped pin quota (#112)
• Resolve --moe-strategy auto to fused on unified-memory GPUs such as GB10 (#445)
• Reserve paged KV at allocation granularity (#367)

Kernels

• Route fused_topk through the in-repo Triton router (#319)
• Make Triton top-k/top-p sampling exact (#329)
• Compute shared experts before in-place routed experts (#463)
• Avoid the row-wise _scaled_mm stall on sm_89 with torch < 2.12 (#243)
• Stop l2norm recompiling per token count (f7c31e9)
• Add sm_80 (A100/A800) to the kernel-cache default arches (#75)

Server and CLI

• Add --gpu to choose the GPU on multi-GPU machines (#117)
• Honor the configured output default across APIs (#411)

Dependencies

• transformers >=5.16,<5.17 (was >=5.5,<6) (#479); torchvision and pillow added for the image processors (#454).

Breaking Changes & Deprecations

• --device is gone from ft checkpoint and ft bench bw; both take --gpu <uuid|index>, as ft serve does (#117).
• --moe-backend is now --moe-strategy; the old spelling still works as a deprecated alias. --nvfp4-backend is deprecated in favour of --quant-backend moe.nvfp4=<kernel> (#418).
• Multimodal checkpoints build their vision tower by default. Qwen3.6, Qwen3.8-Flash-Next, Gemma-4, GLM-5.3-Flash, MiniMax-M3 and Muse-Glimmer-30B now serve images out of the box; --text-model-only restores the text-only footprint (#454).
• Older FTW checkpoints may need a repair before they load. Reconvert with ft checkpoint, or patch in place with scripts/ftw_hotfix.py (see docs/ftw-hotfix.md):
  • NVFP4 dense checkpoints: the FTW must carry the input_scale its scheme declares (#438)
  • Qwen3.8-Flash-Next: the PLE table now lives next to the FTW (#420)
  • any FTW converted before its family served images holds no vision encoder; ft serve refuses it unless started with --text-model-only (#486)

Contributors

@jason-fxz, @andy-yang-1, @taking-lying-flat, @RuixiangMa, @bernimccoy, @earlvanze, @iamanishx

First contributions this release:

• @RuixiangMa in #25
• @bernimccoy in #243
• @taking-lying-flat in #329
• @earlvanze in #411
• @iamanishx in #445

Full changelog: v0.1.2...v0.1.3

v0.1.2

Init release with fixed pypi