[Request] Support Kimi K3 on Apple Silicon clusters with RDMA

[zhwangsir]

Request: Support Kimi K3 on Apple Silicon clusters with RDMA

Model

• Name: Kimi K3
• HuggingFace: https://huggingface.co/moonshotai/Kimi-K3
• Parameters: 32B active / 1T total (Mixture-of-Experts)
• Key architecture components:
  • Kimi Delta Attention (KDA)
  • Attention Residuals
  • Stable LatentMoE
  • MoonViT-V2 (for multimodal inputs)
• Quantization: MXFP4 / MXFP8 weights

Hardware setup

I have a cluster of 4× Mac Studio (M3 Ultra, 512 GB unified memory each) connected via RDMA over Thunderbolt, providing ~2 TB of addressable memory and very high peer-to-peer bandwidth. This is exactly the kind of setup EXO is designed for, and I would love to run Kimi K3 on it.

Current blocker

Kimi K3 was released with first-party support for vLLM / SGLang / TokenSpeed, all of which require NVIDIA/AMD GPUs. EXO is currently the only practical path to run a model of this size on Apple Silicon with unified memory + RDMA, but the following pieces appear to be missing:

1. MXFP4 / MXFP8 weight dequantization in EXO/MLX.
2. Kimi Delta Attention (KDA) implementation.
3. Attention Residuals routing.
4. Stable LatentMoE expert mapping.
5. MoonViT-V2 vision encoder (for multimodal use cases).

Ask

Could the maintainers/community share:

• Whether there is already a plan or branch to support Kimi K3?
• The estimated effort / priority?
• How contributors with Apple Silicon hardware can help test or debug?

I am happy to provide remote access to this 4-node M3 Ultra RDMA cluster for validation if that helps accelerate the work.

Related references

• Kimi K3 model card: https://huggingface.co/moonshotai/Kimi-K3
• Kimi K3 technical report: https://github.com/MoonshotAI/Kimi-K3

Thank you for the amazing work on EXO — it is the most promising inference stack for local Apple Silicon clusters today.

[EternaPeptix]

Feel free to try: https://github.com/EternaPeptix/exo/tree/experiment/kimi-k3-tp2-long-context
I'm getting 130 tok/s prefill 13 tok/s decode on 2x 512M3s

[zizip60]

Interesting, but 130 tok/s is painful for agentic tasks with compactation. I'm working on adding 5090 cards to prefill large models on the cheap with nvme streaming on epyc/am5 boards. 4x reaktive cheap 9100 pro 1tb nvme drives can move alot on the cheap. Ram would be better but not until cxmt gets in the game. My benchmarks estimate around 2500 tok/s (single card) past load floor on glm 5.2. AM5 works on single card with the right board that gives 2 4x 5.0 on cpu lanes.

[mightyC1]

You can try https://github.com/mightyC1/exo. Tensor RDMA works on 4 nodes, but performance is currently poor (3.3 t/s decode, 60 t/s prefill). It hasn't been optimized yet, so feel free to contribute!

[marksmeltzer]

@zizip60 any links or documentation about what you're envisioning? I'm curious if your goal is to have the 5090s cooperating with EXO somehow. In my setup, I have a host windows PC with dual 5090s and 4x Mac Studios on a TB5 mesh (256 GB each)... would have loved to get 512 GB Macs, but came to the party too late.

[zizip60]

@marksmeltzer The idea is here: ggml-org/llama.cpp#26564. It´s a evolution of EXO´s own https://blog.exolabs.net/nvidia-dgx-spark. The idea is the same bring M5 compute to the M3, the issue with their idea it works on small models that are already fast. My solution is to use a blackwell gpu with little more go like the 5090/5080 etc. Issue is that these don´t have the vram to hold model, so instead I repackage prefill task, 1 forward pass layer by layer so I give the 5090 layer to compute and then the next. The hard limit is the pcie buss of the 5090 that caps at x16 pcie 5.0. So cant be faster than 8s for small prompts on glm 5.2. 8s sounds bad but feed the m3 ultra anything above 2k and it will bog down. Second issue is ram is so expensive but when you think of it we only need to feed 50GB/s and nvme 5.0 drives loves this task. I have a modified Vllm for running with nvme and the 5090 is destroying m3 ultra, 150 -> 2000+ tok/s on prefill for glm 5.2 at a 300k prompt! Thats 150 sec (single card) instead of a lunch break :)