Kimi K3 support — weights are now live (MXFP4, ~594 GB)

[fermionoid]

Continuing from #344 as requested (moving idea threads to Discussions).

Update (edited): the K3 weights are live — moonshotai/Kimi-K3, ungated, 118 files, ~1.56 TB total. (An earlier version of this post quoted a ~594 GB MXFP4 figure from pre-release coverage — that was wrong; thanks @MichaelFomenko for the correction. ~1.56 TB is consistent with ~2.8T params at ~4.25 bits/param.) So external-NVMe territory after all.

Recap of why K3 looks like a natural target for the streaming architecture (from #344):

• Sparsity ratio is more extreme than GLM-5.2: 16-of-896 (~1.8%) routed activation vs 8-of-256. (Edit: the tech report revises the per-token picture — active params are much larger in absolute terms; see my comment below for the corrected math.)
• The non-trivial part is the architecture: Kimi Delta Attention + attention residuals need genuine engine work on the attention/KV path — per the maintainer, the plan is DeepSeek-V4-Pro first (shared MLA lineage), then Kimi as the second large-scale target, with a token-exact tiny oracle as the gate for merging.

Standing offer from #344 still holds: happy to run the oracle and any Apple Silicon testing on my M4 Max (same box as #308) whenever the KDA work starts.

[brad-evony]

The size of the MXFP4 repo on hugging face is 1.56TB, just as one would expect from a model of nearly 3 trillion parameters quantized to around 4 bits. To run this on a personal computer would require significant SSD space. However, I would still like to have that option even if I don't use it all the time and store the weights on a hard drive when I'm not using it. I am also willing to help with this project and I will try to do some benchmarking with GLM5.2 soon.

[MichaelFomenko]

the Information are Incorrect. The storage is 1.56TB

[fermionoid]

@MichaelFomenko you're right — the ~594 GB figure I quoted from pre-release coverage was wrong, and I've corrected the post. Confirmed first-hand: 118 files, ~1.56 TB, ungated.

The tech report (https://arxiv.org/abs/2607.24653) also corrects my earlier napkin math in a more important way: active params are 104B, so per-token routed-expert reads work out to roughly ~26 GB/token at MXFP4 (92 MoE layers × 16 experts × ~33M params each) — about 2.3× GLM-5.2's ~11 GB/token, not "comparable" as I speculated. Architecturally it still fits the streaming thesis — only 24 of 93 layers carry a growing KV cache; the 69 KDA layers keep fixed-size recurrent state, and the released weights are natively MXFP4 (QAT from SFT onward), so no community re-quantization step is needed — but the disk-bandwidth bar is meaningfully higher than GLM-5.2. KDA math: §2.1.1 pp.4–5; config table: p.11.

@brad-evony more GLM-5.2 datapoints would be great — the community benchmark table is the backbone of this project (I contributed the 64 GB M4 Max row).

Edit: one more thing the report confirms (answering my own open question from #344) — K3 does ship an MTP layer, officially fine-tuned EAGLE-3-style and released in MXFP4 (§4.1.4), so colibri's existing MTP speculative path carries over conceptually; though as the README already documents, speculation buys less when disk-bound.

[JustVugg]

We support Kimi 3 now!

[fermionoid]

Impressive turnaround — congrats @JustVugg and steve-m, "on the roadmap" to merged inside three days is quite the pace. Streaming the QAT-native MXFP4 shards with zero conversion is a lovely property too.

I'll try for an Apple Silicon datapoint: M4 Max / 64 GB, with the ~1.6 TB snapshot on a 2 TB USB 3.2 Gen 2 external SSD (~1 GB/s real-world), since it doesn't fit my internal disk. Given the ~26 GB/token expert traffic, I expect that to land firmly disk-bound — well below my GLM-5.2 numbers on the same machine — but it should be a useful datapoint for the external-drive configuration. Will report here.