Skip to content

ktransformers-0.2.2rc1+cu120torch26fancy.amd.ubergarm.7a19f3b.flashinfer-cp311-cp311-linux_x86_64.whl

Latest
Latest

Choose a tag to compare

View all tags
@ubergarm ubergarm released this 27 Feb 17:09
· 77 commits to main since this release
7a19f3b
7a19f3b
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

This binary .whl is built from ktransformers@7a19f3b which is also the tagged version 0.2.2rc1 containing API endpoint bug fixes. The only way to know what version you have is to check git rev-parse --short HEAD.

I built ktransformers@7a19f3b following this guide and have it working on a couple different machines with $ nvidia-smi listed below.

It runs about 2x faster than llama.cpp currently, but requires your rig has at least one 16GB+ VRAM GPU. Stick to llama.cpp for pure CPU inferencing for now.

$ sha256sum ktransformers-0.2.2rc1+cu120torch26fancy.amd.ubergarm.7a19f3b.flashinfer-cp311-cp311-linux_x86_64.whl
b46300dcfb7b312d5ebacd01f652edc7d9f63947168275f099b1f21b8973e4f8  ktransformers-0.2.2rc1+cu120torch26fancy.amd.ubergarm.7a19f3b.flashinfer-cp311-cp311-linux_x86_64.whl

Tested Successfully

NVIDIA-SMI 570.86.16 Driver Version: 570.86.16 CUDA Version: 12.8
NVIDIA-SMI 570.86.15 Driver Version: 570.86.15 CUDA Version: 12.8

此二进制文件.whl基于ktransformers@7a19f3b构建，该版本对应标签版本0.2.2rc1，包含API端点修复补丁。唯一可靠版本确认方式为 执行git rev-parse --short HEAD命令。

我们根据此指南成功编译ktransformers@7a19f3b，并已在多台配备下列nvidia-smi配置的机器验证运行：

  • 性能较llama.cpp提升约2倍，但强制要求至少配置16GB+显存的GPU设备。若需纯CPU推理，建议继续使用llama.cpp方案。
$ sha256sum ktransformers-0.2.2rc1+cu120torch26fancy.amd.ubergarm.7a19f3b.flashinfer-cp311-cp311-linux_x86_64.whl
b46300dcfb7b312d5ebacd01f652edc7d9f63947168275f099b1f21b8973e4f8  ktransformers-0.2.2rc1+cu120torch26fancy.amd.ubergarm.7a19f3b.flashinfer-cp311-cp311-linux_x86_64.whl

已验证兼容环境

NVIDIA-SMI 570.86.16 驱动版本: 570.86.16 CUDA版本: 12.8
NVIDIA-SMI 570.86.15 驱动版本: 570.86.15 CUDA版本: 12.8

Assets 3
Loading