v0.1.3

Highlights

• Image input. Qwen3-VL, Qwen3.6, Qwen3.8-Flash-Next, Gemma-4, GLM-5.3-Flash, MiniMax-M3 and Muse-Glimmer-30B accept images through the OpenAI, Anthropic and Responses APIs. Vision towers stream their weights from host memory; --text-model-only serves a multimodal checkpoint as text.
• Support new models.
  • Qwen3.8-Flash-Next (#257)
  • GLM-5.3-Flash (#332).
• Quantization rebuilt (#418). One QuantConfig → QuantScheme → kernel path for every layer; kernels are selected with --quant-backend layer[.kind]=name, and --moe-backend becomes --moe-strategy.

Multimodal

FreeToken now serves vision-language models (#454): an image goes through the checkpoint's own processor and vision tower, then down the same scheduler, radix cache and MoE offload path as text.

• Engine: image spans are content-keyed in the radix cache; encoded embeddings sit in an encoder cache between prefill chunks; the vision tower streams its weights from pinned host memory two blocks at a time.
• Server API: OpenAI image_url, Anthropic image blocks and Responses input_image (URL, base64, file://); tool-result images are carried onto the next user turn (db64879); /v1/stats reports model.input_modalities.
• CLI: --text-model-only, --image-min-tokens / --image-max-tokens, --mm-processor-kwargs, --mm-encoder-weights, --allowed-media-domains, --allowed-local-media-path — see the CLI reference.
• ft launch: declares image input to Codex, OpenCode, OpenClaw and dsh when the server serves it (8ff0cce).

Supported models

Family	Arch	PR
Qwen3-VL	Qwen3VLForConditionalGeneration, Qwen3VLMoeForConditionalGeneration	#454
Qwen3.6 / Qwen3.5	Qwen3_5MoeForConditionalGeneration, Qwen3_5ForConditionalGeneration	#454
Qwen3.8-Flash-Next	Qwen4ExpForConditionalGeneration	#454
Gemma-4	Gemma4ForConditionalGeneration, Gemma4UnifiedForConditionalGeneration	#467
GLM-5.3-Flash	Glm5NextForConditionalGeneration	#479
MiniMax-M3	MiniMaxM3SparseForConditionalGeneration	#480
Muse-Glimmer-30B	MuseGlimmerForConditionalGeneration	#481

Model Support

• Support Qwen3.8-Flash-Next (#257)
  • Stream the PLE n-gram table from disk with --ple-backend disk (#311)
  • Load block-fp8 dense projections natively (#428)
  • Write the PLE table next to the FTW (#420)
• Support GLM-5.3-Flash (#332)
• Add scripts/ftw_hotfix.py to repair FTW checkpoints from older builds (#424)
• Serve and repair FTW checkpoints around the vision encoder (#486)
• A truncated FTW shard raises OSError instead of loading garbage (#25)
• Download the shards the safetensors index names (#336)

Quantization

Checkpoint quantization is rebuilt on a QuantConfig → QuantScheme → QuantMethod path (#418, #427): the checkpoint's quantization_config (ModelOpt, compressed-tensors, block-fp8, mxfp4) decides the scheme of every projection, and each layer declares its tensors and picks its kernel from that, instead of per-family format branches. New formats and kernels plug in once; --quant-backend layer[.kind]=name selects the kernel.

• Qwen3.5 / Qwen3.6 MoE: read every checkpoint layout through the QuantConfig (#438)
• Qwen3.8-Flash-Next: take the expert quant kind from the checkpoint QuantConfig (#426)
• Load ModelOpt exports that ship no input_scale (#462)
• Detect NVFP4 experts behind a mixed-precision compressed-tensors format (#343)

Engine

• Per-layer host-bank residency: split lock-CPU / pin-GPU layers, auto selection under a capped pin quota (#112)
• Resolve --moe-strategy auto to fused on unified-memory GPUs such as GB10 (#445)
• Reserve paged KV at allocation granularity (#367)

Kernels

• Route fused_topk through the in-repo Triton router (#319)
• Make Triton top-k/top-p sampling exact (#329)
• Compute shared experts before in-place routed experts (#463)
• Avoid the row-wise _scaled_mm stall on sm_89 with torch < 2.12 (#243)
• Stop l2norm recompiling per token count (f7c31e9)
• Add sm_80 (A100/A800) to the kernel-cache default arches (#75)

Server and CLI

• Add --gpu to choose the GPU on multi-GPU machines (#117)
• Honor the configured output default across APIs (#411)

Dependencies

• transformers >=5.16,<5.17 (was >=5.5,<6) (#479); torchvision and pillow added for the image processors (#454).

Breaking Changes & Deprecations

• --device is gone from ft checkpoint and ft bench bw; both take --gpu <uuid|index>, as ft serve does (#117).
• --moe-backend is now --moe-strategy; the old spelling still works as a deprecated alias. --nvfp4-backend is deprecated in favour of --quant-backend moe.nvfp4=<kernel> (#418).
• Multimodal checkpoints build their vision tower by default. Qwen3.6, Qwen3.8-Flash-Next, Gemma-4, GLM-5.3-Flash, MiniMax-M3 and Muse-Glimmer-30B now serve images out of the box; --text-model-only restores the text-only footprint (#454).
• Older FTW checkpoints may need a repair before they load. Reconvert with ft checkpoint, or patch in place with scripts/ftw_hotfix.py (see docs/ftw-hotfix.md):
  • NVFP4 dense checkpoints: the FTW must carry the input_scale its scheme declares (#438)
  • Qwen3.8-Flash-Next: the PLE table now lives next to the FTW (#420)
  • any FTW converted before its family served images holds no vision encoder; ft serve refuses it unless started with --text-model-only (#486)

Contributors

@jason-fxz, @andy-yang-1, @taking-lying-flat, @RuixiangMa, @bernimccoy, @earlvanze, @iamanishx

First contributions this release:

• @RuixiangMa in #25
• @bernimccoy in #243
• @taking-lying-flat in #329
• @earlvanze in #411
• @iamanishx in #445

Full changelog: v0.1.2...v0.1.3