Repository navigation
v0.37.1
A patch for every DeepSeek V4 Flash file: on 0.37.0, --auto and bmoe-server crashed at the real-layer probe before the model loaded.
- The real-layer probe leaves integer tables out of the layer's matmuls (#416). A block of DeepSeek V4 Flash holds a token-to-expert index map, an
I32tensor that is 2-D and named.weight; the probe priced it as a dense matrix, and the CPU backend accepted a matmul on it and then called a kernel that does not exist. A weight the probe prices is now one a matmul reads as floats, by ggml's own type traits.
Measured with the fix on a rented 24 GB card (RTX 4090, 89 GiB memory control group, NVMe): DeepSeek V4 Flash 0731 UD-IQ2_M, 85 GB of experts, streamed from the drive with a 13 GB expert cache on the card and a 66 GB one in RAM, 9.3 tok/s at 94% hits, --auto only. On the same file and card, llama.cpp's own -ngl 99 --n-cpu-moe 43 with the file in the page cache decoded at 18.4 tok/s: the planner's choice there is being looked at (#418 is one of the reasons).
The full list is in CHANGELOG.md.