Repository navigation
Kimi k3.1 pro full #2795
CyberRSR
started this conversation in
Show and tell
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Kimi K3.1 Pro, the full checkpoint (2.8T parameters, 1.56 TB of MXFP4 experts and BF16, vision tower included), runs on one desktop with colibri's
kimi_k3engine. Fork: https://github.com/CyberRSR/colibri, latest install: kimi-k31-pro-4090-r10.Machine: Ryzen 9 7950X3D (16 cores), 192 GB DDR5-5400, RTX 4090 24 GB, two NVMe drives (C: ~9.4 GB/s and D: ~6.3 GB/s on random 17.5 MB reads; the experts are split across them), Windows 11.
The GPU computes a prompt step in the CPU's own arithmetic, so the logits are the same bytes as a CPU-only run (checked at every position of 5.9k- and 16.5k-token prompts). The vision tower's rows are within 1.3e-5 of Moonshot's own
modeling_kimi_k3.py.What went in: K3.1's DSA lightning indexer on the GPU, the routed experts of a prompt step on the GPU beside the CPU (tile kernels, 18-21 TMAC/s), host buffers registered so copies run beside kernels, a photo of the state at the end of the system prompt and of every turn, conversations kept in files, the MoonViT tower and patchmergerv2 projector (several pictures a request, prefix reuse over them), and gateway fixes so agent clients such as Kimi Code with Computer Use screenshots keep going. Details and raw logs: docs/kimi_k3.md, measurements.
Where a decode token's 1.16 s go
During decode the drives average 9.1 GB/s at ~70% busy, the CPU ~78%, the GPU ~3%. Measured and slower, all with the same greedy tokens:
Consecutive tokens share only ~5 of 16 experts a layer, so speculative decoding does not pay either; the MXFP4 experts do not compress (zlib 0.90).
Question: does anyone have ideas for faster decode on this kind of machine (one consumer GPU, RAM much smaller than the experts, two NVMe drives) without changing the model's output? We are at roughly max(DRAM side ~0.9 s, drives ~0.7 s) plus imperfect overlap. Ideas about better expert prediction from earlier layers, cache policies, NUMA/memory tricks on Zen 4, or a smarter split between CPU and GPU are all welcome. Lossy options (fewer routed experts, top-p) exist but change the answers.
All reactions