Repository navigation
PagedMoE: run 500+ GB MoEs on a 48 GB Mac, efficiently enough to support local agentic work. #4524
Replies: 1 comment 1 reply
|
Really cool work — the memory-ceiling numbers here (380 GB on 48 GB unified memory) are wild, and the framing of "can it load" vs. "can it run at agent-usable speed" is the right one; that gap is where most local-MoE demos quietly fall apart. This is a nice complement to something I've been building: VeloxQuant-MLX — 43 KV cache compression methods (quantizers, token-eviction caches, cross-layer merging) for any A few things that might be relevant to what you're doing:
Links if you want to try it or dig in:
Would genuinely love it if people tried it out against their own MoE/agent stacks and filed issues — especially interested in how the compression methods behave stacked with weight-paging setups like this one, since that combination isn't something I've benchmarked yet. |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
A few months ago I bought a 48 GB MacBook Pro M5 to push my research into machine learning and how these models actually work. The machine has been far better than I expected. I can develop, research, train, and build locally with more headroom than I need.
But as I got deeper into ML, I kept wanting a 128 GB MacBook, which is not in the cards right now. That led me down a rabbit hole: what is the real ceiling for running huge models locally? After weeks of research and smaller builds, PagedMoE (originally expert-stream) came together and got wired into a custom local agent inference stack I have been extending for a while. As it got more useful, I decided it was time to share it.
MoE models like Qwen3-235B, GLM, DeepSeek, and similar do not have to fit in unified memory. Prediction, routing, and caching, tuned and timed correctly, stretch usable memory by a wide margin. Early on it worked, but it was not usable for agent tasks: decode was too slow, context blew up, and fidelity took hits. So the focus shifted from “can it load” to “can it run like the machine actually supports it.”
Measured peaks on this machine (without PagedMoE these do not load):
Those decode numbers are after warmup, with the optional prune/wait recipe on the disk-bound models. Defaults leave prune off and stay bit-identical to a fully resident run. Prune is an explicit fidelity knob (e.g. GLM ~6.4 tok/s vs ~1.8 full mixture, ~96% token agree on the A/B I ran).
Not limited to these models. They were the test set. No special format: plain MLX MoE checkpoints that the current implementation supports.
I will not go into every moving part here. The public repo is the core engine only. The private tree is full of notes, experiments, and benches. What is public is the working system without that overhead.
The ceiling is not done yet. There is room to grow almost everywhere, and it is still experimental, so know which knobs you are turning before you lean on it.
Repo: https://github.com/noahclark556/paged-moe
Open to suggestions. I have enjoyed building this, and the research is far from over.
All reactions