Prerequisites
Feature Description
I am experiencing system-wide crashes (kernel panics) and aggressive page faults when running large MoE models on an M4 Pro Mac with 24GB RAM. Currently, llama.cpp relies on the macOS system swap for memory overflow, which the OS cannot handle when the "router" jumps between experts.
The Problem:
System Failure: The OS becomes extremely sluggish, memory faults occur, and even the system speakers stop working/distort (likely due to coreaudiod being paged out) before a crash.
Failure of Current Flags: Even using `-n-cpu-moe` and offloading layers to the CPU, the system still enters a swap death spiral once physical RAM is exceeded.
Attempted Fixes: I have tried various quantizations (IQ4_KSS, Q4_K_M), reducing -ngl to 0, and even forcing a higher memory limit for the GPU via:
`sudo sysctl iogpu.wired_limit_mb=22000`
Motivation
If we can move away from letting the OS handle the memory overflow and instead implement a managed SSD-to-GPU pipeline, we unlock a massive tier of models for the entire Mac community. We have the hardware power to run these SOTA models; we just need a stable way to bridge the gap between the SSD and the RAM.
Possible Implementation
In my testing with the ik_llama.cpp fork, I can successfully run these massive models using their --cpu-moe implementation. It proves that by managing the SSD-to-RAM pipeline internally, I can achieve usable speeds on "low" RAM hardware:
Qwen 30b IQ4_KSS: 35+ t/s
Qwen 30b Q_4_M: 25+ t/s
Qwen Next 80b IQ4_KSS: 10+ t/s
(Note: While generation is fast in that fork, prompt processing remains a bottleneck, which is why I am requesting a Metal-optimized version here.)
Hardware Environment
- Chip: M4 Pro (14 core CPU 20 core GPU)
- RAM: 24GB
- OS: mac OS Tahoe 26.2
Build command: cmake -B build -DGGML_NATIVE=ON && cmake --build build --config Release -j$(nproc)
Prerequisites
Feature Description
I am experiencing system-wide crashes (kernel panics) and aggressive page faults when running large MoE models on an M4 Pro Mac with 24GB RAM. Currently, llama.cpp relies on the macOS system swap for memory overflow, which the OS cannot handle when the "router" jumps between experts.
The Problem:
Motivation
If we can move away from letting the OS handle the memory overflow and instead implement a managed SSD-to-GPU pipeline, we unlock a massive tier of models for the entire Mac community. We have the hardware power to run these SOTA models; we just need a stable way to bridge the gap between the SSD and the RAM.
Possible Implementation
In my testing with the ik_llama.cpp fork, I can successfully run these massive models using their --cpu-moe implementation. It proves that by managing the SSD-to-RAM pipeline internally, I can achieve usable speeds on "low" RAM hardware:
(Note: While generation is fast in that fork, prompt processing remains a bottleneck, which is why I am requesting a Metal-optimized version here.)
Hardware Environment
- Chip: M4 Pro (14 core CPU 20 core GPU)
- RAM: 24GB
- OS: mac OS Tahoe 26.2
Build command:
cmake -B build -DGGML_NATIVE=ON && cmake --build build --config Release -j$(nproc)