Adding external compute. Glm 5.2 2500 tok/s prefill

[zizip60]

I have been running GLM 5.2 and it´s first time i felt local AI could work in production. Issue is M3 has very weak compute and no FP4 in hardware. So i was thinking how we could add a compute to an M3, TB5 is to slow to stream weights on a bigger model like GLM 5.2 and 10gbe is not even close. But if the gpu had a local cache for weights and just streamed tokens the connection would matter less. A 5090/5080 has the compute I would need but AM5 (AMD) cant hold that much ram and its very expensive. I think Nvme is great for this 2 or 4x 9100 pro from Samsung would saturate pcie 5.0. If streamed by layer this would hide under compute anyway. It would take 3s load Deepseek V4 Flash weights to vram and you would anyway do compute while you load. And a external GPU could help in so many ways. 1 Let mac batch higher as it would only do decode. 2 Running specprefill with small model in the fast vram a 5090 has and then prefill it and hand KV to mac prepackaged. Run smaller subagents to classify etc for bigger model. 5 when you are tired of AI workloads you start a game and remote stream it from the 5090. Deepseek V4 Flash would prefill faster much faster than M5 around 8000 tok/s. I could even prefill a Kimi K3 without a problem.

I know this is more of an EXO project, but I think adding external compute to omlx would make big difference.

Happy to hear your thoughts