Fatal error. System.Runtime.InteropServices.SEHException

[aDL1n]

When trying to run a model on CUDA 12, a fatal error occurs

[SanftMonster]

Hi, which version of LLamaSharp and dotnet runtime are you using?

[aDL1n]

[SanftMonster]

Could you please check your model path again to confirm it's the right path?

[aDL1n]

The model path is definitely correct since everything is running on the processor back end.

[SanftMonster]

OK, then could you please take a try with the master branch code? There've been many commits ahead of v0.10.0 now. If the error still appears, please give us a minimal example to reproduce it and we'll dig on this problem.

[aDL1n]

Okay, I'll try now

[aDL1n]

I tried updating net core to 8.0, but nothing changed. The code was taken from the quick start example. Here are the specs of the PC it runs on. Maybe this will help find the problem
Processor: AMD FX 8320
GPU: GTX 1660 super
RAM: 16 GB
OS: Windows 10 pro

[aDL1n]

The same error in the example from master

[SanftMonster]

Hi, I think I've found the problem. Your cpu is a bit old (released in 2012) so I think it has no avx support, while our cuda backend is compiled with avx2 by default.

Firstly, you could confirm this point by installing cpu backend and add this line to the beginning of your code. (with v0.10, not master)

using LLama.Native;
NativeLibraryConfig.WithLogs();

Then, you'll see which level avx of the native library is selected by default.

If it's confirmed that your PC has no avx support or does not support avx, please follow the steps to fix it:

1. clone llama.cpp repo and enter its directory.
2. mkdir build && cd build
3. cmake -DBUILD_SHARED_LIBS=ON -DLLAMA_BUILD_TESTS=OFF -DLLAMA_BUILD_EXAMPLES=ON -DLLAMA_AVX=OFF -DLLAMA_AVX2=OFF -DLLAMA_FMA=OFF -DLLAMA_BUILD_SERVER=OFF ..
4. cmake --build . --config Release -j4 (you can change the number due to your cpu cores)
5. Now, you will see a file named llama.dll in the folder of build/bin/Release. Please put it somewhere of your device and copy its path.
6. Add this line to the beginning of your code: NativeLibraryConfig.WithLibrary(<path>), in which the <path> is the path of your compiled llama.dll.

If you can't compile it, please feel free to ask us for help.

Finally, since your cpu is not of high performance, please consider set GpuLayerCount as large as you can, otherwise the inference speed may be slow.

[aDL1n]

Okay, I'll check now

[SanftMonster]

Note: there's already a discussion about supporting different levels of avx in cuda backend #605. If that hits your case, it indicates that it makes sense to support cuda with different avx level. :)

[aDL1n]

Yes, this processor does not support avx 2.

[aDL1n]

Note: there's already a discussion about supporting different levels of avx in cuda backend #605. If that hits your case, it indicates that it makes sense to support cuda with different avx level. :)

I think this can help many users

[aDL1n]

Unfortunately it didn't help