I built something that lets you run models larger than your Mac's memory with llama.cpp

[t8]

Hey all,

Wanted to share Hypura: NVMe tensor streaming for llama.cpp. It places model tensors across GPU, RAM, and NVMe tiers based on access patterns, bandwidth costs, and hardware capabilities. It works especially well with MoE models since not all experts need to be loaded into memory at the same time, enabling offloading others to NVMe when not in use.

I've been running models larger than what I should be able to on my Mac with it and wanted to OSS it and share with the community here: https://github.com/t8/hypura

Thanks!

[lsb]

This is particularly interesting with larger MoE models like GLM 5.2