Kimi K3 (July 27): pre-release analysis and groundwork for llama.cpp support

[Hudabey]

Moonshot's Kimi K3, a 2.8T MoE with Kimi Delta Attention, a new “Attention Residuals” mechanism, and native-MXFP4 release weights is announced for release by July 27.

I have been conducting pre-release gap analysis against current llama.cpp, Kimi-Linear, and the flash-linear-attention reference implementation. I am sharing the work now to reduce speculation and avoid duplicating anyone else's effort.

This is not a claim that K3 already runs in llama.cpp. No working K3 port exists yet.

Repository, with commit-pinned file-and-line citations throughout:

https://github.com/Hudabey/theseus

What appears reusable in current llama.cpp mainline

Based on pinned source inspection:

• ggml_gated_delta_net already supports the per-channel gate mode used by KDA across backends.
• A complete LLM_ARCH_KIMI_LINEAR graph and converter already exist.
• Hybrid recurrent/attention memory infrastructure is present.
• GGML_TYPE_MXFP4 is already a loadable and runnable type.

If K3 matches Kimi-Linear's recurrence, tensor shapes, and gate conventions which remains unknown until release. KDA itself may require substantially less new kernel work than expected.

One visible optimization gap is the chunked-prefill TODO in the fused CUDA path, although the released model may expose additional gaps.

Attention Residuals

I traced the reference implementation in fla/ops/attnres and mapped a correctness-first implementation using existing ggml primitives:

• rms_norm
• mul_mat
• soft_max
• row-view multiplication
• addition

In fla's wiring, Attention Residuals replaces the read of the residual stream with a per-token softmax mixture over depth. The block's ordinary prenorm is folded into the operation.

Its depth state is local to one forward pass, so this reference design does not imply changes to the KV cache or persistent memory system.

Whether K3 follows fla's implementation exactly remains checkpoint-dependent. The full analysis is in recon/04-attnres-analysis.md.

Native MXFP4 preservation

Because the release weights are MXFP4, the correct conversion path if K3 uses a compatible serialization is a lossless byte repack into block_mxfp4, not dequantization followed by re-quantization.

This follows the same general pattern already used by the gpt-oss and DeepSeek-V4 converters.

The repository includes a working repacker and oracle test suite covering:

• byte-exact round trips;
• float32 bit equality across normal and edge-case scales;
• rejection of invalid 0xFF E8M0 scales;
• equivalence with llama.cpp's existing gpt-oss MXFP4 transformation.

Run:

python -m pytest tests/test_repack_mxfp4.py