-
Notifications
You must be signed in to change notification settings - Fork 326
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Consider adding Windows on ARM build of llama.dll to LLamaSharp.Backend.Cpu #600
Comments
Hi, we're certainly willing to add such a backend. Could you please test whether it works well with current LLamaSharp? Though we could compile and test it with github actions, the period of debugging it will be much longer since we don't have arm windows device. So if you'd like to help it'll save a lot of time. :) Here're some steps for it:
If all things above go well, what we need to do is just adding it to CI and release next version. :) |
Unfortunately it looks like Github Actions doesn't have Windows+ARM available (docs) :( Edit: Note that's not a total blocker for this. I think we could cross compile the DLLs from Windows x64 Actions, it just means we couldn't test it. |
And using the ARM DLL from llama.cpp's official releases isn't a viable workaround? |
Using dlls from llama.cpp's release will bring some troubles for auto binary-updating in CI, but I think we could make it as long as you've confirmed that LLamaSharp works well with that dll. |
I have a Windows Arm device. The latest llama.cpp releases have two options for Windows arm64:
I can't really say why the llvm build works fine but the msvc build fails to load. The differences between various c++ compilers/toolsets is a bit outside my wheelhouse (which is why I really love LlamaSharp), but I can say that the win-llvm-arm64 build works fine on my Windows ARM laptop. |
The current LLamaSharp version (0.15.0) is compatible with llama.cpp b3479. You need to make sure you're using that version if you're loading custom binaries. |
Good to know, thank you. |
The llama.cpp project added a native binary build of
llama.dll
for the Windows on ARM platform in release b2399 (with the rather misleading filename llama-b2399-bin-win-arm64-x64.zip). Please consider adding this to the LlamaSharp.Backend.Cpu NuGet package. 🙂The text was updated successfully, but these errors were encountered: