LLamaSharp, Unity and GPU

[footh]

Using NuGetForUnity, I've installed the LLamaSharp package and the Cuda12 Windows backend. This backend has the CPU backend as a dependency so that installs too. Clearly all the native libraries are recognized by Unity as evidenced by all the "Multiple plugins with the same name 'llama'", etc. errors. I can run my project and get LLamaSharp to work fine with the CPU. I can run LLamaSharp on a GPU fine outside of Unity. But I can't seem to run LLamaSharp on Unity with the GPU.

I've scoured this project's discussion board and issue list for help but it mostly seems to be outdated due to the fast-moving nature of this project. I know there is a LLamaSharpForUnity git example, but that is 2 years old. Despite that, I tried using the method described in the readme and still no luck. I've tried removing all the native libraries but the cuda 12 windows dlls, and while that gets rid of the "Multiple plugins" error messages, LLamaSharp give me the native backend error and does not work. I looked into the NativeLibaryConfig methods but WithLibrary and WithCuda is not supported in Unity's .NET version (and I see there is a feature request about those being supported).

I've basically tried everything I could on my own and I'm stuck hence the reason for this post. Am I missing something simple? How does LLamaSharp determine what backend to use upon bootstrapping? Any help is greatly appreciated.

[Lyrcaxis]

I got it working by doing the following steps:

1. Install the NuGetForUnity package via git link
2. Use it to install LLamaSharp and the backend of my choice
3. Copy the ggml-cpu.dll over from the Backend.Cpu folder to the actual backend's folder
4. Delete the Backend.Cpu folder completely
5. (Optional) uninstall NuGetForUnity to prevent it from re-downloading the Cpu backend as dependency.

And this'll use the backend you downloaded -- the only one available. No WithLibary() nor WithCuda() needed!
If you wanna ship to multiple platforms, just use different configs before building and ship with the appropriate DLLs.
It's only a little problematic if other people in your team use different OS's.

So yeah if you use git with multiple people with multiple OS's, consider the following:

1. Download all backends and place them OUTSIDE the assets folder. Make sure each includes ggml-cpu.dll.
2. Make a script (e.g. bash) that copies over the desired backend into the Assets folder -- based on the dev's machine.
3. Add all possible final paths to .gitignore so they won't be pushed.

And, either do some Unity magic for 100% automation, or just let your teammates know they need to run the bash script first.

[footh]

Nice, thanks for this! This works great!

Also note, for your step 5, you can actually keep NuGetForUnity by opening up packages.config at the root of the Assets folder and manually removing the line of the CPU dependency which should look like this:

<package id="LLamaSharp.Backend.Cpu" version="0.21.0" />

Thanks a ton for your help.