Edge0-145B running on an M3 Max (36 GB) via Metal

[somoore]

I got Edge0-145B (Q3_K) running on my MacBook through Metal and put it up as a fork in case it's useful to anyone:

https://github.com/somoore/edge0-macos-metal

I know you have your own Metal work going (the M2 Ultra numbers in the README), so this is just a community port for people who want to try it on a Mac now.

What it does:

• edge0-infer serve and chat work on macOS, both API styles.
• Experts stream from SSD, so it fits in 36 GB.
• M3 Max, 36 GB: 11 to 13 tok/s generation, about 160 tok/s on long prompts, about 3 s to first token.
• Works with OpenCode if you start it with --context 32768.

What it doesn't:

• Only tested on my machine.
• Never compared against a CUDA run. Parity is against a NumPy reference only.
• Much slower than your M2 Ultra numbers, since only about a fifth of the experts fit in memory.

Two things I ran into that might be useful to you either way:

• macOS compresses the Metal buffers when the engine sits idle, which cost about 2 s on the next request. Pinning the arena and weights with mlock fixed it.
• A small ridge-fitted correction on the read-ahead guess took top-16 recall from 80% to 92% and gave about 16% more decode speed. Output doesn't change.

Details are in docs/macos-metal.md.

Questions:

1. Would you want any of this as a PR, or should it stay a fork?
2. Could someone with a CUDA box share reference tokens for a fixed prompt so I can check parity?
3. Are you OK with me hosting the small predictor file on the fork's release page?

If you try it on another Mac, especially 64 GB or more, I'd love to hear how it goes.

[shuk777]

Thanks for your interest in our work! We are currently working on macOS support, will release in next week. We would be happy to have you helping us to build the community.
I think the best way is to keep your fork, wait until our next release on macOS, and then you could PR on this repo.

[corysus]

Sharing numbers from another machine, in case it helps — M2 Ultra, 64 GB,
macOS 27.2, running somoore's Metal fork of edge0-strata:

• decode: 28.7 tok/s (chat with thinking, MTP accept 51/71)
• prefill: 420 tok/s at ~16K tokens, 403 at ~64K, 329 at ~128K
• time to first token at 128K prompt: 229 s
• serve with --context 131072 loads and runs clean on 64 GB

Numbers come from a small custom bench script (fixed-size prompts against /v1/chat/completions, timings read from the response's timings field), happy to share it or re-run anything against the official macOS build when it ships — good to hear that's about a week away.