Skip to content

Releases

moe-stream 0.1.0

moe-stream 0.1.0 Latest
Latest

Choose a tag to compare

View all tags
@Wok3b Wok3b released this 13 Oct 23:18
v0.1.0
aa87515

First tagged release. Mixture-of-Experts models run from an SSD through an explicit expert cache inside llama.cpp, with exactly the same tokens as its mmap path.

Highlights

  • Qwen3.6-35B-A3B (20.6 GiB) on a 15.4 GB laptop: 6.4 tok/s in an 8 GB memory budget and 6.8 with no limit, against 1.6 and 3.5 memory-mapped.
  • Gemma 4 26B-A4B: 10.1 tok/s against mmap's 6.3, and 9.3 in 8 GB.
  • --cache-slots auto / MOE_CACHE_SLOTS=auto sizes the cache from the cgroup limit or free RAM.
  • The next layer's experts are predicted with its router on a separate thread, evicted by decayed use counts, and prompt processing reads only the experts the prompt picks.
  • llama-server runs with the cache, and run.sh downloads a model (SHA-256 checked) and serves it in one command: --mem, --no-limit, --ctx, --no-think.

Quickstart

git clone --recursive https://github.com/Wok3b/moe-stream && cd moe-stream
./build.sh
./run.sh qwen        # or: ./run.sh gemma --no-think --mem 8

Tested on

Ryzen 5 7430U, 15.4 GB, Solidigm NVMe, CPU only: Arch Linux with GCC 16.2 (bit-exact output re-checked for this release on Gemma 4), and Windows 11 with MSVC (docs/NOTES.md, Results 1-10). The build is checked on GitHub Actions for Linux and Windows.

There are no prebuilt binaries: the build is tuned to the CPU it is compiled on (GGML_NATIVE=ON).

Every number is in docs/NOTES.md, Results 1 to 17, with the raw logs in bench/results/.

Assets 2
Loading