🚨FAQs | 常见问题🚨

[KMSorSMS]

Note

Please avoid creating issues regarding the following questions, as they might be closed without a response.
请避免创建与下述问题有关的 issues，这些 issues 可能不会被回复。

Tip

Documentation: https://kvcache-ai.github.io/ktransformers/
中文文档：https://github.com/kvcache-ai/ktransformers/tree/main/doc/zh

---

Most of problems / 大多数问题

How to install kt-kernel/ 怎么安装 kt-kernel

Please update repository and install again following the link below:
请参考文档进行安装，以及常见问题指导：
https://github.com/kvcache-ai/ktransformers/blob/main/kt-kernel/README.md

Inference with kt-kernel+Sglang/ 在 kt-kernel + Sglang 上推理

1. Install SGLang

git clone https://github.com/sgl-project/sglang.git
cd sglang
pip install -e "python[all]"

2. Prepare Weights

You need both GPU weights and CPU weights for heterogeneous inference:

GPU Weights: Use the original / quantized model weights.

CPU Weights: Quantize to AMX-optimized format using the conversion script:

python scripts/convert_cpu_weights.py \
  --input-path /path/to/model \
  --input-type bf16 \  # Depends on your GPU weights type: fp8, fp16, or bf16
  --output /path/to/cpu-weights \
  --quant-method int8  # or int4

Supported input formats: FP8, FP16, BF16 → INT4/INT8.

For more details, see:

• CPU Weights conversion
• GPU Weights quantization

Note: LLAMAFILE backend supports GGUF format directly, but this feature is still in preview.

3. Launch SGLang Server

Start the SGLang server with your normal SGLang parameters, and add the following KT-Kernel specific parameters to enable CPU-GPU heterogeneous inference:

KT-Kernel Parameters to Add:

• --kt-method: Backend method (AMXINT4, AMXINT8, or LLAMAFILE)
• --kt-weight-path: Path to the converted CPU weights
• --kt-cpuinfer: Number of CPU inference threads (set to physical cores)
• --kt-threadpool-count: Number of thread pools (set to NUMA node count)
• --kt-num-gpu-experts: Number of experts to keep on GPU
• --kt-max-deferred-experts-per-token: Deferred experts for pipelined execution

Example:

python -m sglang.launch_server \
  [your normal SGLang parameters...] \
  --kt-method AMXINT8 \
  --kt-weight-path /path/to/cpu-weights \
  --kt-cpuinfer 64 \
  --kt-threadpool-count 2 \
  --kt-num-gpu-experts 32 \
  --kt-max-deferred-experts-per-token 2

---

Support models/ 支持的模型列表

Model_name/ 模型名称	have_tested/ 是否测试过
GLM-4.5-Air	✅
GLM-4.5	✅
Qwen3-30B-A3B	✅
Qwen3-235B-A22B-Thinking-2507	✅
Qwen3-235B-A22B-Instruct-2507	✅
Qwen3-Next-80B-A3B-Thinking	✅
DeepSeek-R1-0528	✅

Note

In principle, we can support any model that SGLang supports. If a model cannot run, then SGLang running purely on GPU likely won't be able to either. So if you test additional models that work, feel free to share them.

---

Known Issues & Tips / 已知问题与提示

"KT fallback" log messages are normal / "KT fallback" 日志是正常的

If you see logs like KT fallback: layer X prepare weight = XXms, compute = XXms, this is expected behavior indicating that Layer-wise Prefill is working correctly. The "fallback" naming is misleading and will be updated in a future release.

如果看到类似 KT fallback: layer X prepare weight = XXms 的日志，这是预期行为，说明 Layer-wise Prefill 已正常启动。"fallback" 这个名称有误导性，后续版本会优化。

Low GPU utilization during prefill / Prefill 阶段 GPU 利用率低

If your GPU utilization is low during prefill, enable Layer-wise Prefill by adding:

--kt-gpu-prefill-token-threshold 2048

This allows the GPU to participate in prefill computation, significantly improving prefill throughput.

如果 prefill 阶段 GPU 利用率很低，请添加 --kt-gpu-prefill-token-threshold 2048 参数开启 Layer-wise Prefill。

GPTQ_INT4 + tensor-parallel-size > 1 / GPTQ_INT4 与多卡 TP 不兼容

GPTQ-Int4 quantized MoE models may fail with IndexError: start out of range when using tensor-parallel-size > 1 with GPU experts. This is an SGLang-side weight loader issue. Workarounds:

• Use --tensor-parallel-size 1
• Use FP8 weights instead of GPTQ-Int4

GPTQ-Int4 量化的 MoE 模型在 tensor-parallel-size > 1 且使用 GPU experts 时可能报错。这是 SGLang 侧的问题。建议降低 TP 或使用 FP8 权重。

---

Tip

If the problems still exist with the latest code, please create an issue.
若使用最新的代码仍然无法解决问题，请创建一个 issue。

[james0zan]

We need a tracking matrix of model support.
For example, the currenct staus of GLM series model, what kind of weight are supported, what are not.

[KMSorSMS]

You are welcome to report new models that are supported in the kt_kernel with your snapshot to prove

[kiankyars]

I'm not too sure what the issue is here. Is this just an FAQ because I'm looking for a good first issue to contribute? I

[KMSorSMS]

@kiankyars Yes, this is just an FAQ. This FAQ contains answers to common questions that may help you. If you can’t find what you’re looking for here, please feel free to open a new issue.

[wqshmzh]

One has to prepare CPU+GPU models, which means he or she is supposed to download the model twice, right ? So that one model occupies the disk for two times. It is very space consuming. Previous versions of ktransformers do not require doing so. Why things changed after updates ?

[KMSorSMS]

I think this is no longer necessary if you use the original-precision mode. Previously, two sets of weights were needed because the INT4/INT8 weights are not the same as the original weights. In low-precision mode, we use INT4/INT8 weights for the MoE experts, while the rest of the model stays unchanged, so both versions had to be kept. If you run in original-precision mode, such as rawint4, then this is no longer needed.

[wqshmzh]

@KMSorSMS I dont have enough resources to run full-precision models. I always use unsloth's UD models, the precisions of moe and other parts are different in UD models. So do I need to download the original full precision model and unsloth's model twice now?

[KMSorSMS]

@wqshmzh I see — yes, in your case you'll need both. The original full-precision model is required (only its non-MoE parts are actually used, but there's no way to download just those, so the full model has to be pulled down), and Unsloth's UD model is needed for the MoE parts. So both downloads are necessary.

[mytbk]

好像没什么关于现在的 SGLang+kt-kernel 架构的文档。我不清楚 kt-kernel 是专用于 CPU 还是也支持 GPU 的，我看 kt-kernel 的 setup.py 和 CMakeLists.txt 里面还提到了 CUDA/MUSA/ROCM，而且在 CUDA 环境的确会调用 nvcc. 另外 llama.cpp 的集成也看得不是很懂，是用来支持 GGUF 相关量化的吗？
我现在的主要部署方式是在自己的机器上先配好 venv 安装，再把 venv 挪到生产环境，所以想知道要构建 kt 有哪些是必须要构建的。