Inference time on Linux with openBLAS

[Scriptwonder]

I am running on Linux with OpenBLAS, but it is significantly slower than running via Python (256X256 with 2 steps). Is this intentional as the tradeoff of not using a GPU?

[antirez]

Hi! The problem is that the optimization effort was spent on MPS, but indeed OpenBLAS needs more love. May you tell me how much faster PyTorch is?

[antirez]

I added some BLAS speedup and the ability to tun the threads usage, maybe this will help.

[Scriptwonder]

Hi! The problem is that the optimization effort was spent on MPS, but indeed, OpenBLAS needs more love. May you tell me how much faster PyTorch is?

Sorry for the late reply! PyTorch costs around 10-12 seconds for a loaded 4B model from my usage.

I added some BLAS speedup and the ability to tun the threads usage, maybe this will help.

Sounds great! I will get back to it to try it when I'm free. Have you tested on linux machines with Python for the speedup?