KTransformers v0.7.0

KTransformers v0.7.0: Support Full Fine-Tuning and AMD CPU (AVX512) for Fine-Tuning!

KTransformers v0.7.0 brings broader hardware compatibility and new fine-tuning capabilities for large MoE models. This release focuses on five major improvements.

Highlights

1. Full AVX512 Support for LoRA Fine-Tuning

KTransformers LoRA fine-tuning now fully supports compatible AVX512 x86 CPUs without requiring AMX.

The runtime automatically selects the appropriate CPU implementation, allowing MoE expert training to run on a broader range of large-memory servers.

kt_config:
  kt_backend: auto

auto is the recommended setting for both AMX and AVX512 platforms.

Related change: #2141

2. Native FP8 LoRA Support

KTransformers now supports native FP8 LoRA fine-tuning for DeepSeek-V3.1.

The original block-wise E4M3 routed-expert weights and scales are loaded directly from the checkpoint, avoiding the need to materialize a complete BF16 copy before training.

In the tested DeepSeek-V3.1 configuration, host-memory requirements decreased from approximately 1.4 TB with BF16 expansion to about 800 GB.

Related change: #2141

3. Qwen VLM MoE Fine-Tuning Support

KTransformers now supports LoRA fine-tuning for Qwen3-VL MoE models through LLaMA-Factory.

The integration handles Qwen3-VL's nested language-model layers, fused expert weight layout, and vision patch embedding, enabling multimodal data to complete distributed forward and backward passes, optimizer updates, and checkpoint saving.

The compatibility path is scoped to supported Qwen VLM model types and does not change the existing text-only training path.

Related change: #2156

4. DeepSeek V4 Docker Quick Deployment

This release adds and expands the DeepSeek V4 Flash Docker deployment guide.

Users can start a DeepSeek V4 environment through a standardized Docker workflow without manually configuring the full build and runtime stack.

5. CPU Activation Reuse

Advanced users can now retain CPU expert activations during activation checkpoint recomputation, reducing repeated CPU computation and improving training throughput.

Enable it in the training YAML when sufficient host memory is available:

kt_cpu_activation: retain

Leaving this option unset preserves the default recomputation behavior. Activation Reuse trades additional CPU memory for reduced recomputation.

Related change: #2141

Release Validation

The release regression covered:

• Qwen3.5-397B-A17B BF16 LoRA on two GPUs
• DeepSeek-V3.1 native FP8 LoRA on four GPUs
• Qwen3-VL-30B-A3B-Instruct BF16 LoRA with FSDP2, including multimodal preprocessing and checkpoint saving
• Finite training loss
• Non-zero LoRA parameter updates
• Distributed FSDP2 execution
• FP8 loading, TP staging, checkpoint, and adapter lifecycle tests

Installation

pip install "ktransformers[sft]==0.7.0"

Matching SFT integration packages:

• transformers-kt==5.6.0.post2
• accelerate-kt==1.14.0.post2

What's Changed

Fine-Tuning

• Activation Reuse and native block-FP8 routed-expert LoRA SFT by @yyj6666667
• Support Qwen VLM MoE fine-tuning by @Illumination111

Deployment and Documentation

• Add DeepSeek V4 Flash Docker guide by @yyj6666667

Contributors

• @yyj6666667 @Illumination111 @ErvinXie @JimmyPeilinLi

New Contributors

• @Illumination111 made their first contribution in #2156.

Full Changelog: v0.6.4...v0.7.0

KTransformers v0.6.4

KTransformers v0.6.4: High-Performance End-to-End Full-Parameter and LoRA Fine-Tuning

KTransformers v0.6.4 is a major upgrade to the KT fine-tuning path for large sparse MoE models. This release extends KT SFT from end-to-end Full-Parameter Fine-Tuning to LoRA, while improving BF16 CPU-GPU heterogeneous training throughput and memory efficiency across host RAM and GPU VRAM.

🚀 Core Highlights

• End-to-end Full-Parameter and LoRA SFT: train large sparse MoE models with Full Fine-Tuning, LoRA, or Hybrid configurations through the familiar LLaMA-Factory workflow.
• High training throughput: in real end-to-end training runs—not isolated kernel benchmarks—KTransformers reached approximately 400 tokens/s with Full-FT and 600 tokens/s with LoRA on Qwen3-30B-A3B using 2× AMD EPYC 9355 CPUs and 2× RTX 5090 GPUs. A separate LoRA setup using one RTX 4090 and an AMX-enabled CPU sustained more than 700 tokens/s. The resulting checkpoints also showed clear gains in end-to-end fine-tuning validation.
• DeepSeek V4 on Ampere GPUs: BF16 attention fallbacks and an FP8 MoE Marlin path expand DeepSeek V4 Flash support to GPUs without native FP8 Tensor Cores.
• RAWINT4 compatibility and acceleration: broader packed-weight compatibility, AVX-VNNI-256 expert loading, and a faster Kimi K2 prefill path.
• Experimental Intel iGPU backend: new SYCL-based GPTQ INT4 MoE inference support for Intel integrated GPUs.

📌 Full-Parameter, LoRA, and Hybrid SFT

This release completes the KT CPU-GPU heterogeneous training workflow for large sparse MoE models:

• Full Fine-Tuning updates KT-managed CPU expert weights together with the model's regular trainable parameters.
• LoRA covers regular linear layers, routers and gates, and fused MoE experts.
• Hybrid training allows Full-FT and LoRA parameters to be trained together.
• Reliable multi-step training supports gradient accumulation, activation checkpointing, distributed execution, optimizer updates, and profiling.

The new training path primarily targets BF16 KT CPU expert training.

👉 Main implementation: ktransformers PR #2094

📌 DeepSeek V4 and Ampere GPUs

The bundled sglang-kt integration adds broader DeepSeek V4 Flash and FP8 MoE support on Ampere GPUs:

• BF16 KV-cache and attention fallbacks for GPUs without native FP8 Tensor Cores.
• FP8 MoE Marlin support for GPU Experts, GPU Prefill, and dynamic expert updates.
• Stability fixes for long-input scheduling, AMXINT4/8 startup, and FP16 FE8M0 scale dequantization.

Related changes:

• DeepSeek V4 Flash inference on Ampere GPUs
• Fix V4 SWA token-pool sizing
• Fix AMXINT4/8 SwiGLU parameter handling
• FP8 MoE Marlin for GPU Expert and GPU Prefill
• FP16 FE8M0 Marlin scale dequantization

📌 RAWINT4 Compatibility and Prefill Acceleration

KTransformers v0.6.4 improves the RAWINT4 CPU Expert path with:

• Support for compressed-tensors checkpoints that store packed INT4 weights in int32.
• Per-expert RAWINT4 loading on AVX-VNNI-256 systems.
• A blocked matrix-multiplication path for faster Kimi K2 RAWINT4 prefill, while keeping the existing decode path for short sequences.

Related changes:

• Normalize compressed RAWINT4 weights
• AVX-VNNI-256 per-expert RAWINT4 loading
• K2 RAWINT4 prefill mat-mat dispatch

📌 Experimental Intel iGPU SYCL Backend

This release introduces an experimental SYCL backend for GPTQ INT4 MoE inference on Intel integrated GPUs. Compatibility and performance depend on the Intel GPU, driver, SYCL toolchain, and model configuration.

👉 Intel iGPU SYCL GPTQ INT4 MoE backend

🔐 Security

The balance_serve scheduler now binds its ZMQ ROUTER endpoint to 127.0.0.1 instead of all network interfaces, reducing exposure to unauthenticated remote pickle deserialization.

• Security fix PR #2091
• GHSA-83vp-v6wg-x93x

📦 Installation

KT SFT

pip install "ktransformers[sft]"

KT SFT continues to use the LLaMA-Factory training entry and YAML workflow.

KT Inference

pip install kt-kernel sglang-kt

Install from Source

git clone https://github.com/kvcache-ai/ktransformers.git
cd ktransformers
git submodule update --init --recursive
cd kt-kernel && ./install.sh
cd .. && ./install.sh

CUDA 12.0 or newer is required. Individual backends may have additional CPU ISA, GPU architecture, driver, or compiler requirements.

📝 Changelog

ktransformers

• feat: end-to-end Full-Parameter, LoRA, and Hybrid SFT (#2094)
• feat: K2 RAWINT4 prefill mat-mat dispatch (#2080)
• feat: Intel iGPU GPTQ INT4 MoE backend (#2089)
• fix: normalize compressed RAWINT4 packed weights (#2075)
• fix: AVX-VNNI-256 per-expert loading (#2092)
• [security]: bind the balance_serve scheduler ZMQ endpoint to loopback (#2091)
• [build]: enable -mavx512vl for AVX512 multi-variant builds (#2021)
• fix: detect ports that are already bound before launch (#2071)
• [build]: improve PyPI release workflow consistency (#2062)
• [docs]: update Qwen3.5 KT LoRA, FAQ, and DeepSeek V4 documentation (#2057, #2029, #2088)

sglang-kt

• Enable DeepSeek V4 Flash inference on Ampere GPUs (#58)
• Fix kt_ep_wrapper import failure (#59)
• Fix V4 SWA token-pool sizing for chunked prefill (#60)
• Fix AMXINT4/8 startup with non-MXFP SwiGLU parameters (#61)
• Add FP8 MoE Marlin for Ampere GPU Experts and GPU Prefill (#62)
• Add FP16 FE8M0 scale dequantization for Marlin (#63)

🌟 Contributors

Thanks to everyone who contributed code, testing, reviews, documentation, and end-to-end validation for this release.

Full Changelog: v0.6.3...v0.6.4

CC: @ErvinXie @JimmyPeilinLi @hermannklie @Oxygen56 @VectorPeak @lutianshu824 @RockmSockmJesus @callmegaga @Anai-Guo @blazingphoenix7 @harrychk @usrlocalben @jdai0 @Illumination111 @yyj6666667

KTransformers v0.6.3

KTransformers: MiniMax-M3 & GLM-5.2 Day0 Support

We're excited to announce Day0 support for two new large MoE models: MiniMax-M3 and GLM-5.2 in KTransformers + SGLang (sglang-kt). Both models are now available for CPU-GPU heterogeneous inference via kt-kernel. This update also ships end-to-end KT LoRA serving for Qwen3.5 MoE.

🚀 Core Highlights

• MiniMax-M3 Day0 Support: Full inference support for MiniMax's 128-routed-expert sparse MoE architecture (MiniMaxAI/MiniMax-M3-MXFP8). Runs on a single 96 GB GPU or scales to 8×GPU with kt-kernel CPU expert offload.
• GLM-5.2 Day0 Support: Full inference support for Zhipu AI's GLM-5.2 (zai-org/GLM-5.2, zai-org/GLM-5.2-FP8), with FP8 and BF16 precision, NSA attention backend, and dynamic expert updates via kt-kernel.
• KT LoRA Serving: End-to-end Qwen3.5 MoE KT LoRA serving workflow — train with KT SFT, convert once, serve through SGLang with a single merged adapter path.

📌 KT LoRA Serving for Qwen3.5 MoE

A new end-to-end fine-tuning-to-serving workflow for Qwen3.5 MoE with KT expert LoRA:

• Unified adapter pipeline: KT SFT raw output → convert_kt_to_sglang_adapter.py → merged adapter directory → SGLang --lora-paths serving.
• Expert + non-expert LoRA: the server auto-splits expert and non-expert LoRA weights internally; users serve a single merged adapter path.
• SFT kernel improvements: refactored AMX SFT MoE kernels, AVX2 MoE base updates, and tensor-parallel MoE SFT helpers.
• Docs: KT-FT Fine-Tuning and Inference Loop (EN) | 中文

📌 GLM-5.2

GLM-5.2 is Zhipu AI's latest MoE model with NSA (Native Sparse Attention). KTransformers supports it with:

• FP8 and BF16 precision: Both zai-org/GLM-5.2-FP8 and zai-org/GLM-5.2 are supported.
• NSA attention backend: Native sparse attention with skip_topk-gated indexer for DSA layers.
• Dynamic expert updates: --kt-enable-dynamic-expert-update with uniform placement strategy.
• Tool calling & reasoning: glm47 tool-call parser and glm45 reasoning parser.

👉 Full Tutorial

📌 MiniMax-M3

MiniMax-M3 is a 128-expert sparse MoE model shipped natively in MXFP8 (fp8 e4m3 + uint8 ue8m0 1×32 scale). KTransformers supports M3 end-to-end with:

• KT-Kernel hybrid: CPU expert offload with MXFP8 byte-copy and GPU shadow FP8 view; validated on 8×H20 and single H20.
• MXFP8: Direct MXFP8 inference path on Hopper GPUs via Triton MoE backend.
• Tool calling: minimax-m3 function-call parser (native <minimax:tool_call> XML → OpenAI tool_calls array).
• Thinking mode: Request-level control via chat_template_kwargs.thinking_mode — enabled / disabled / adaptive.

👉 Full Tutorial

📥 Installation

# Option A: One-click install (from ktransformers root)
git clone https://github.com/kvcache-ai/ktransformers.git
cd ktransformers
git submodule update --init --recursive
cd kt-kernel && ./install.sh
cd .. && ./install.sh

# Option B: pip install
pip install sglang-kt
pip install kt-kernel

Requires CUDA 12.0+; CUDA 12.8+ recommended for FP8/MXFP8 deployments.

📝 Changelog

ktransformers (kvcache-ai/ktransformers)

• feat: add end-to-end Qwen3.5 MoE KT LoRA serving workflow (#2031) — @jdai0
• [feat] MXFP8 MoE support (#2041) — MXFP8 MoE kernel for MiniMax-M3 Day0.
• fix: CudaGraph replay fix and Add MoE startup log (#2037).
• update glm52 tutorial (#2046) — @ovowei
• [build]: sync sglang submodule to 51032b71 (#2032).
• docs: add MiniMax-M3 SGLang + KT-Kernel tutorial (#2051) — @yyj6666667

sglang-kt (kvcache-ai/sglang)

• feat: support end-to-end KT LoRA serving for Qwen3.5 MoE (#53) — @jdai0
• support glm5.2 — @ovowei
• fix(dsa): wire skip_topk-gated indexer for GlmMoeDsa to unblock GLM-5.2 — @yyj6666667
• fix: kt_ep_wrapper silently fails to import after a2f451315 (#57).
• Feat/minimax m3 (#56): full MiniMax-M3 model port — @yyj6666667

🌟 Contributors

Thanks to all contributors who helped ship this release.

CC: @jdai0 @ovowei @ouqingliang @ErvinXie @chenht2022 @KMSorSMS @SkqLiao @JimmyPeilinLi @mrhaoxx @yyj6666667

KTransformers v0.6.2

ktransformers v0.6.2 Release Notes

🚀 Core Highlights

• Native DeepSeek-V4-Flash support via kt-kernel MXFP4 MoE operator, consuming the model's native E2M1 + ue8m0 weights without offline conversion.
• Hybrid CPU/GPU inference path through SGLang, validated end-to-end on 8× RTX 5090 (consumer Blackwell, SM_120).
• New AVX2 / AVX-VNNI RAWINT4 MoE backend, extending kt-kernel coverage to consumer CPUs without AVX-512 / AMX.

📌 Models, Hardware & Tooling

• Add DeepSeek-V4-Flash model entry, loader, and numerical validation script.
• Bump SGLang submodule to bring in V4-Flash support, SM_120 Triton fallbacks, and a flashinfer guard.
• Repoint sglang extra to post2 for compatibility.

📥 Installation

For most users:

pip install ktransformers==0.6.2

See doc/en/install.md for the general install guide.

From source (recommended for running V4-Flash on SM_120) — pre-built wheels do not ship the Blackwell consumer-GPU fallbacks needed for V4-Flash:

git clone https://github.com/kvcache-ai/ktransformers.git
cd ktransformers
git submodule update --init --recursive
cd kt-kernel && ./install.sh
cd .. && ./install.sh        # builds the kvcache-ai SGLang fork
pip install --upgrade flashinfer-python flashinfer-cubin   # >= 0.6.9 required by V4-Flash MXFP4 MoE

Requires CUDA 12.8+.

Additional notes for running Deepseek v4 flash:
if you encounter with any errors , please check out the DeepSeek-V4-Flash tutorial first. It will update by time to record common errors.

📝 Docs & Community

• Add DeepSeek-V4-Flash tutorial: hardware matrix, full launch command for 8× RTX 5090, OpenAI-compatible API examples, kt chat CLI usage.
• Refresh README entry points and add KT SFT Quick Start.

🌟 Contributors

Thanks to all contributors who helped ship this release.

Full Changelog: v0.6.1...v0.6.2

CC: @JimmyPeilinLi @ouqingliang @ovowei @yyj6666667 @aliez-ren @jdai0

KTransformers v0.6.1

KTransformers v0.6.1 is a full refactor and major upgrade of the existing KT fine-tuning path for large-MoE LoRA SFT. This release rebuilds the KT SFT backend around kt-kernel, packages the SFT stack behind the ktransformers[sft] entry, and keeps the LLaMA-Factory training entry and YAML workflow familiar for users.

In measured benchmark settings, KT SFT reaches 6-12x training performance compared with the ZeRO-Offload baseline. Separately, CPU memory usage is reduced to about 1/2 compared with the previous KT SFT path, with lower GPU memory pressure in the same benchmark context. These results are tied to the benchmark setup and should be read together with the model, hardware, context length, LoRA config, and baseline details.

🚀 Core Highlights

• Refactored large-MoE SFT backend: rebuilt the KT SFT path around AMX MoE kernels with LoRA support, including SFT-specific AMX kernels, LoRA fused add tests, repacking tests, Python SFT wrappers, autograd integration, layer definitions, and weight helpers (#1936).
• Pip-installable SFT stack: ktransformers[sft] now installs the KT SFT stack through regular Python packages: ktransformers, kt-kernel, transformers-kt, and accelerate-kt.
• LLaMA-Factory workflow integration: KT SFT is designed to be used after installing LLaMA-Factory, while preserving the standard accelerate launch / src/train.py / YAML workflow. Companion LLaMA-Factory PR: hiyouga/LlamaFactory#10430.
• Cleaner package boundary: the old kt-sft package has been archived; SFT now uses ktransformers[sft], while KT inference uses kt-kernel + sglang-kt (#1954, #1955).
• Current torch baseline: package metadata and the validated public install path are aligned on torch==2.9.1, with torchaudio==2.9.1 and torchvision==0.24.1 for the current full-stack setup.

📌 Models, Hardware & Tooling

SFT and Packaging

• Prepare v0.6.1 SFT wheel packaging on main, including ktransformers[sft], kt-kernel packaging updates, release workflow updates, and a lightweight top-level package shim (#1945).
• Align kt-kernel and related package metadata with the current release dependency baseline (#1948).
• Flatten the ktransformers package shim for cleaner wheel packaging (#1955).
• Archive the legacy kt-sft package under archive/kt-sft/ so the active package layout is easier to reason about (#1954).

Kernel and Hardware Improvements

• Add AVX512F+BW fallback support for FP8 and BF16 under the AMX backend, improving CPU fallback coverage where AMX-specific paths are unavailable (#1908).
• Add VNNI-256 support for GPTQ INT4 MoE, including a new gptq_int4_avxvnni path and per-commit accuracy test coverage (#1926).
• Add SFT-specific AMX MoE kernels, tensor-parallel MoE SFT helpers, LoRA kernel tests, repacking tests, and Python SFT module wrappers (#1936).

Inference and Runtime Integration

• Sync the bundled sglang submodule for KT layerwise prefill updates and later packaging fixes (#1920).
• Update SGLang-KT release workflow to use a hosted runner, avoiding release blocking on unavailable self-hosted runners.

Model Enablement

• Add GLM-5.1 tutorial and prerequisite notes for the kt-kernel path (#1916, #1932).
• Refresh README and model documentation links for Kimi-K2.5, MiniMax-M2.5, Qwen3.5, DeepSeek-V3.2, and SFT docs.

📦 Installation

KT SFT with LLaMA-Factory

Use a LLaMA-Factory checkout that contains the KT examples and requirements/ktransformers.txt.

cd /path/to/LLaMA-Factory
pip install -e .
pip install -r requirements/ktransformers.txt

For direct package installation outside that requirements file:

pip install "ktransformers[sft]"

KT Inference

KT inference uses the SGLang-KT path:

pip install kt-kernel sglang-kt

Keep kt-kernel in the inference installation path. LLaMA-Factory SFT continues to use ktransformers[sft].

📝 Docs & Community

• Refresh KT installation commands and package boundaries in README / README_ZH and SFT docs (#1958).
• Add GOSIM 2026 announcement and update roadmap link to Q2 (#1937).
• Add and update GLM-5.1 tutorial prerequisites and related docs (#1916, #1932).
• Remove a broken symlink in archive/ktransformers/ (#1906).

🐛 Bug Fixes

• Fix Qwen3 series gibberish output by correcting RoPE write-back in the bundled SGLang integration (#1959).
• Fix kt-kernel CLI environment detection when NUMA node lists are empty (#1929).
• Revert the CPUInfer stream bridge for ROCm after compatibility concerns (#1918, #1925).
• Fix SGLang-KT packaging metadata and point the KTransformers SGLang extra to the corrected SGLang-KT release path (#1964).

🌟 Contributors

Thanks to all contributors who helped ship this release.

Full Changelog: v0.5.3...v0.6.1

CC: @JimmyPeilinLi @mrhaoxx @jdai0 @ouqingliang @ErvinXie @chenht2022 @KMSorSMS @ovowei @SkqLiao @yyj6666667 @james0zan

KTransformers v0.5.3

🚀 Core Highlights

• AVX2-Only MoE Inference Support in kt-kernel: Added AVX2-only inference support for bf16, fp8, and gptq-int4 MoE workloads, expanding deployment coverage to CPUs without AMX while preserving CPU-GPU heterogeneous inference workflows.
• CPU Weight Conversion for GLM-5 & MiniMax-M2.5: Added new tooling to convert model weights for CPU-side deployment, making GLM-5 and MiniMax-M2.5 integration and packaging smoother in production workflows.
• NUMA-Aware Deployment Improvements: Added explicit --numa-nodes support for finer-grained NUMA mapping in multi-socket environments, followed by fixes to improve correctness and deployment stability.
• Lower Idle CPU Overhead & Better Runtime Behavior: Fixed worker pool / task queue idle spinning issues that could cause unnecessary 100% CPU usage when the system was idle, improving runtime efficiency for long-running services.
• Speculative Decode Enhancements: Added more complete speculative decode support across EAGLE-3 / MTP / STANDALONE / NGRAM, with better configuration, observability, and runtime stability.

📌 Models, Hardware & Tooling

• Model & conversion updates

  • Add CPU weight conversion support for GLM-5 and MiniMax-M2.5 (#1853).
  • Add utility script to merge loose layer weights into safetensors for easier packaging and deployment (#1886).
  • Improve deployment readiness for AVX2-only MoE inference on broader CPU hardware.
  • Continue compatibility improvements around expert loading and runtime integration.

• Kernel & hardware improvements

  • Add AVX2-only MoE kernels for bf16, fp8, and gptq-int4 inference (#1892).
  • Add explicit numa_nodes parameter for deployment control on multi-socket systems (#1891).
  • Fix --numa-nodes handling in runtime configuration (#1904).
  • Improve SGLang / kt-kernel detection timing and integration behavior (#1887).

• Speculative decoding & serving acceleration

  • Add more complete speculative decode support across EAGLE-3 / MTP / STANDALONE / NGRAM and related worker paths (#1903, #1876).
  • Improve speculative worker stability and refresh docs/examples for major model families.

• Runtime & stability

  • Fix worker pool idle CPU usage (#1902).
  • Fix TaskQueue worker thread 100% CPU spin when idle (#1899).
  • Improve speculative worker / MoE layer stability in the serving runtime.
  • Improve overall stability for long-lived inference processes.

• Tooling & integration

  • Add utility script to merge loose layer weights to safetensors for easier release packaging (#1886).
  • Sync bundled sglang submodule to the latest compatible revision (#1903, #1876).
  • Keep submodule-based integration aligned with the current KTransformers runtime stack.

📝 Docs & Community

Documentation updates

• Add AVX2 tutorial (EN): doc/en/kt-kernel/AVX2-Tutorial.md
• Add AVX2 tutorial (ZH): doc/zh/AVX2-Tutorial_zh.md
• Refresh speculative decoding docs/examples for newer model families and acceleration paths.
• Update summary / docs navigation for the new AVX2 documentation.
• Small README alignment updates for the latest supported capabilities.

🐛 Bug Fixes

• Fix --numa-nodes handling (#1904).
• Fix worker pool idle CPU overhead (#1902).
• Fix TaskQueue idle spin causing 100% CPU usage (#1899).
• Improve SGLang / kt-kernel detect time duration (#1887).

🌟 Contributors

• Thanks to all contributors who helped ship this release.

Full Changelog: v0.5.2.post1...v0.5.3

CC: @ouqingliang @ErvinXie @chenht2022 @KMSorSMS @ovowei @SkqLiao @JimmyPeilinLi @mrhaoxx @yyj6666667 @james0zan

KTransformers v0.5.2

🚀 Core Highlights

• Simplified Installation with sglang Submodule: The kvcache-ai/sglang fork is now vendored as a git submodule and published to PyPI as sglang-kt. Installation is reduced from a multi-step manual process to a single ./install.sh command or pip install ktransformers (which auto-installs sglang-kt). Added daily CI auto-sync of the sglang submodule and automated PyPI publishing on version bump.
• New Model Support — Qwen3.5, GLM-5, MiniMax-M2.5, Qwen3-Coder-Next: Day-0 support for four new MoE models spanning a wide range of hardware requirements — from Qwen3-Coder-Next (1x RTX 4090, 80B-A3B) to Qwen3.5 (4x RTX 4090, 400B MoE). All models support BF16 and FP8 precision backends with CPU-GPU heterogeneous inference.
• Kimi-K2.5 Support & Mistral MoE Compatibility: Added Kimi-K2.5 deployment guides including SFT fine-tuning integration, fallback expert prefix lookup for robust weight loading, and Mistral MoE loader compatibility for broader model coverage.

📌 Models, Hardware & Tooling

• Model support updates
  • Add Qwen3.5 (MoE-400B) with FP8 VL detection fix.
  • Add GLM-5 with BF16/FP8 precision support.
  • Add MiniMax-M2.5 with FP8 weight optimization.
  • Add Qwen3-Coder-Next (80B-A3B) for code generation.
  • Add Mistral MoE loader compatibility (#1873).
  • Add Kimi-K2.5 with fallback expert prefix lookup (#1822).
• Kernel & hardware improvements
  • Fix k2-moe.hpp weight loading (#1830).
  • Fix wrapper import issue (#1819).
  • Improve CUDA code readability with explicit ele_per_blk variable (#1784).
• Tooling & integration
  • Add top-level install.sh for one-click source installation (sglang + kt-kernel).
  • Publish sglang fork as sglang-kt on PyPI; kt-kernel auto-installs it as dependency.
  • Add CI workflows: daily sglang submodule sync, automated sglang-kt PyPI publishing.
  • Align sglang-kt version with ktransformers (single version.py source of truth).
  • kt-cli enhancements (#1834).
  • Handle unquoted paths and special characters in model scanner (#1840).
  • Update Docker build for submodule-based sglang installation.

📝 Docs & Community

• Add Qwen3.5 tutorial: https://github.com/kvcache-ai/ktransformers/blob/main/doc/en/Qwen3.5.md
• Add GLM-5 tutorial: https://github.com/kvcache-ai/ktransformers/blob/main/doc/en/kt-kernel/GLM-5-Tutorial.md
• Add MiniMax-M2.5 tutorial: https://github.com/kvcache-ai/ktransformers/blob/main/doc/en/MiniMax-M2.5.md
• Add Qwen3-Coder-Next tutorial: https://github.com/kvcache-ai/ktransformers/blob/main/doc/en/kt-kernel/Qwen3-Coder-Next-Tutorial.md
• Add Kimi-K2.5 deployment & SFT guide: https://github.com/kvcache-ai/ktransformers/blob/main/doc/en/Kimi-K2.5.md
• Add maintainers list (#1837).
• Simplify sglang installation instructions across all 13 model tutorials.

🐛 Bug Fixes

• Fix Qwen3.5 FP8 load for VL detection (#1857).
• Fix k2-moe.hpp load weight issue (#1830).
• Fix wrapper import issue (#1819).
• Fix experts-sched-Tutorial.md (#1808).
• Handle unquoted paths and special characters in model scanner (#1840).

🌟 Contributors

• Thanks to all contributors who helped ship this release.

Full Changelog: v0.5.1...v0.5.2

CC: @ouqingliang @ErvinXie @chenht2022 @KMSorSMS @ovowei @SkqLiao @JimmyPeilinLi @mrhaoxx @james0zan

KTransformers v0.5.1

---

🚀 Core Highlights

• Optimized CPU-GPU Expert Scheduling: Introducing a flexible GPU expert mask system that enables intelligent placement of MoE experts across CPU and GPU. The new scheduling system supports multiple placement strategies (frequency-based, uniform, front-loading, random) and dynamic expert updates during inference, significantly improving throughput by up to 30% at lower GPU expert ratios.
• Native Precision MoE Support with CI: Expanded native precision support for FP8 and BF16 MoE models. Run Qwen3-BF16, GLM-4.7, GLM-4.7-FP8 and more models directly in their native precision without conversion overhead, now with comprehensive CI coverage.
• Unified Fine-tuning & Inference Pipeline: New end-to-end tutorial for cost-effective large model fine-tuning and inference using AutoDL cloud infrastructure. Complete the full LoRA fine-tuning and inference loop for models from 14B to 235B with minimal GPU resources.

📌 Models, Hardware & Tooling

• Model support updates
  • Add native precision support for MiniMax-M2, MiniMax-M2.1, MiMo, DeepSeek-V3.2, GLM-4.7-FP8.
  • Extend FP8 and BF16 MoE enablement path with CI validation.
• Kernel & hardware improvements
  • Introduce GPU expert mask system for flexible per-layer expert placement control.
  • Add dual-stream CPU-GPU parallel optimization to hide CPU overhead when experts are fully on GPU.
  • Implement dynamic expert update for runtime adaptive optimization during layerwise prefill.
  • New parameters: --kt-num-gpu-experts (per-layer), --kt-gpu-experts-ratio (global ratio 0.0-1.0).
  • Add expert placement strategies: frequency, uniform, front-loading, random.
• Tooling & integration
  • Add inference statistics and analysis functionality for GPU expert hit rate monitoring.

📝 Docs & Community

• Add CPU-GPU Expert Scheduling https://github.com/kvcache-ai/ktransformers/blob/main/doc/en/kt-kernel/experts-sched-Tutorial.md.
• Add AutoDL unified fine-tuning and inference https://github.com/kvcache-ai/ktransformers/blob/main/doc/zh/【云端低价训推】%20KTransformers+AutoDL+LlamaFactory：随用随租的低成本超大模型「微调+推理」一体化流程.md (Chinese).

🐛 Bug Fixes

• Fix environment mismatch issues in AutoDL community image for fine-tuning and inference.
• Fix various stability issues in kt-kernel.
• Improve error handling and logging for expert distribution recording.

🌟 Contributors

• Thanks to all contributors who helped ship this release.

Full Changelog: v0.5.0...v0.5.1

CC: @ouqingliang @ErvinXie @chenht2022 @KMSorSMS @ovowei @SkqLiao @JimmyPeilinLi @mrhaoxx @james0zan

KTransformers v0.5.0

🚀 Core Highlights

• Native FP8 MoE Kernel: Introducing native FP8 precision support for MoE inference with a new AVX-based kernel. Run FP8 models directly without precision conversion overhead, preserving the original model accuracy while maximizing hardware efficiency.

• kt-cli for Effortless Local Inference: A new CLI tool designed for simplicity and ease of use. Model management, automatic configuration, seamless chat/completions workflows, and built-in SGLang environment detection—get started with local LLM inference in minutes.

• Enhanced Layerwise Prefill: Improved layerwise prefill performance through expert-by-expert pipelining. The layerwise prefill architecture enables efficient memory streaming during prefill, significantly improving throughput and reducing latency for long-context workloads.

📌 Models, Hardware & Tooling

• Model support updates

  • Extend the FP8 enablement path in this release, focusing on native FP8 MoE support and compatibility improvements.
  • Add native MiniMax-M2, MiniMax-M2.1, DeepSeek-V3.2 support and related enablement.

• Kernel & hardware improvements

  • Add AVX-based FP8 MoE kernel.
  • Reduce DRAM requirements for most models during prefill in CPU.
  • Improve layerwise prefill for better throughput.

• Tooling & integration

  • Introduce kt-cli, a new unified CLI for model management, chat, automatic configuration and inference server management.

• Deployment & installation

  • Refactor installation workflows/scripts for the new CLI/tooling path (including cleanup of legacy install steps).
  • Improve CPU instruction set auto detection.

📝 Docs & Community

• Add MiniMax-M2.1 end-to-end tutorial.
• Refine DPO tutorial.

🌟 Contributors

• Thanks to all contributors who helped ship this release.

Full Changelog: v0.4.4...v0.5.0

CC: @ouqingliang @ErvinXie @chenht2022 @KMSorSMS @ovowei @SkqLiao @JimmyPeilinLi @mrhaoxx @james0zan

KTransformers v0.4.4

🚀 Core Highlights

• Add RL-DPO training support to kt-sft, enabling preference-based reinforcement learning fine-tuning on top of KTransformers’ MoE stack.

  • Includes critical PEFT adaptations and bug fixes for RL workflows.
  • Example configurations and end-to-end usage can be found in:
    https://github.com/kvcache-ai/ktransformers/blob/main/doc/en/DPO_tutorial.md

• Improve large-scale MoE stability and efficiency

  • Significantly reduce CPU memory usage during large-chunk prefill.
  • Fix Kimi-K2 MoE decode bugs related to buffer management.
  • Refine NUMA-aware buffer writing and memory handling paths.

📌 Models, Hardware & Tooling

• Model support updates

  • Add GLM-4.6V support via refactored CPU weight conversion utilities.
  • Extend and stabilize Qwen3 / Qwen3-MoE support on NPU (Ascend), including attention, LN, MLP, cache, and expert operators.

• Deployment & installation

  • Add Docker-based deployment support and automatic deployment workflows.
  • Improve CPU instruction set handling (e.g., automatic BLIS detection on AMD CPUs).
  • Polish PyPI release workflows and installation instructions for smoother setup.

📝 Docs & Community

• Update and polish Kimi-K2 / Kimi-K2-Thinking documentation, including installation steps, prefill strategy, and performance metrics.
• Add and refine NPU benchmarks, prerequisites, and Qwen3-NPU guides.
• Fix README assets, image links, path issues, and reorganize documentation structure.

🌟 Contributors

• Thanks to all contributors who helped ship this release.
• Special thanks to @mrhaoxx and @poryfly for enabling RL-DPO support, and to all community members for kernel fixes, model adaptations, documentation, and tooling improvements.

Full Changelog: v0.4.3...v0.4.4

CC: @JimmyPeilinLi @mrhaoxx @ovowei @SkqLiao @KMSorSMS @poryfly @ouqingliang @Azure-Tang @Atream @chenht2022 @qiyuxinlin @ErvinXie @james0zan