Pure-C MoE engine that streams experts from disk β run a 744B model on a ~25 GB box. Highlights since v1.1.1:
Hardware & correctness
- Fix OOM on integrated / unified-memory GPUs β GB10 / DGX Spark (#653)
- Recognize AMD/ROCm in
doctorandresource_planβ a HIP build is no longer reported CPU-only (#662, #663) - Chat stop-set fix (#633 / #381); async packed-int4 parity (#632); stable KV slot per conversation (#634, #639)
Performance
- AVX2
matmul_e8β fmt=6 (E8/IQ3) was 92% of decode on the scalar kernel (#654) - Native SIMD fmt=6 encoder, 15Γ over numpy, byte-identical (#655)
- AVX-512
matmul_i3for fmt=5 int3-g64 (#661)
Web & docs
- Web dashboard with the live Expert Atlas (#641); new landing page (#645)
- README / quickstart point to the gs64 model container (#642)
Verify your download against SHA256SUMS.txt.