HotPin: Solving the MoE RAM problem – selective expert pinning for llama.cpp

[LozzKappa]

Hi everyone,

I've been working on a project called HotPin that solves a specific problem with MoE models: when the model is bigger than available RAM, --mlock fails because it tries to lock the entire model.

HotPin takes a different approach. Instead of locking everything, it:

1. Analyzes routing statistics to identify frequently used experts
2. Uses mlock (Linux) / VirtualLock (Windows) only on those hot expert tensors
3. Leaves the rest on NVMe via mmap

The result is lossless (SHA-256 bit-identical to a full-RAM run) with drastically reduced RAM usage.

Tested on AMD Ryzen AI 9 HX 370 (23.6GB RAM, Zen5, AVX512):

Model	Disk	Min RAM	tok/s	RAM saved
qwen3:30b-a3b	18.0 GB	10.4 GB	19.7	−42%
gemma4:26b-a4b	16.9 GB	10.6 GB	11.5	−37%
GLM-4.7-Flash	19.0 GB	13.3 GB	12.4	−30%
gpt-oss:120b	58.5 GB	19.1 GB	3.84	−67%

Boundary condition I found:

• Disk > RAM: pinning is highly effective (gpt-oss:120b 2.64 → 3.84 tok/s, +45.5%)
• Model fits in RAM: OS page cache already does the job, pinning adds noise

The patches are ~50 lines of C++ in llama-model.cpp. There's also an autoconfig.py that reads free RAM and applies the right config.

📄 Full paper + code: github.com/LozzKappa/hotpin-llm (includes paper PDF + LaTeX source)

📄 Looking for arXiv endorsement in cs.LG – if any researcher with an arXiv account can help, I'd really appreciate it!

Would love to hear feedback from the maintainers and the community!