Skip to content

v0.37.1

Choose a tag to compare

@Helldez Helldez released this 26 Oct 15:41
· 102 commits to main since this release
eebd2cd

A patch for every DeepSeek V4 Flash file: on 0.37.0, --auto and bmoe-server crashed at the real-layer probe before the model loaded.

  • The real-layer probe leaves integer tables out of the layer's matmuls (#416). A block of DeepSeek V4 Flash holds a token-to-expert index map, an I32 tensor that is 2-D and named .weight; the probe priced it as a dense matrix, and the CPU backend accepted a matmul on it and then called a kernel that does not exist. A weight the probe prices is now one a matmul reads as floats, by ggml's own type traits.

Measured with the fix on a rented 24 GB card (RTX 4090, 89 GiB memory control group, NVMe): DeepSeek V4 Flash 0731 UD-IQ2_M, 85 GB of experts, streamed from the drive with a 13 GB expert cache on the card and a 66 GB one in RAM, 9.3 tok/s at 94% hits, --auto only. On the same file and card, llama.cpp's own -ngl 99 --n-cpu-moe 43 with the file in the page cache decoded at 18.4 tok/s: the planner's choice there is being looked at (#418 is one of the reasons).

The full list is in CHANGELOG.md.