NativeApi: TryLoadLibrary() can fail for some systems

[swharden]

My system does not have a cuda-supported graphics card, but LLamaSharp's native API class detects cuda version 12.

The result is that the first time I try to interact with a LLM, I get an exception:

ggml_backend_cuda_init: error: invalid device 0
llama_new_context_with_model: failed to initialize CUDA0 backend
Fatal error. System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
Repeat 2 times:
--------------------------------
   at LLama.Native.NativeApi.llama_n_ctx(LLama.Native.SafeLLamaContextHandle)
--------------------------------
   at LLama.Native.SafeLLamaContextHandle.get_ContextSize()
   at LLama.LLamaContext.get_ContextSize()
   at LLama.StatefulExecutorBase..ctor(LLama.LLamaContext, Microsoft.Extensions.Logging.ILogger)
   at LLama.InteractiveExecutor..ctor(LLama.LLamaContext, Microsoft.Extensions.Logging.ILogger)
   at LLama.Examples.Examples.ChatSessionWithHistory+<Run>d__0.MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[System.__Canon, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](System.__Canon ByRef)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[System.__Canon, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](System.__Canon ByRef)
   at LLama.Examples.Examples.ChatSessionWithHistory.Run()
   at LLama.Examples.Examples.Runner+<Run>d__1.MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[System.__Canon, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](System.__Canon ByRef)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[System.__Canon, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](System.__Canon ByRef)
   at LLama.Examples.Examples.Runner.Run()
   at Program+<<Main>$>d__0.MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[System.__Canon, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](System.__Canon ByRef)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[System.__Canon, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](System.__Canon ByRef)
   at Program.<Main>$(System.String[])
   at Program.<Main>(System.String[])

Details

Note that this file is present in my build folder (which is why cuda 12 is "detected"), but the only graphics card on my system is a GeForce GT 710 which I don't think supports cuda 11 or 12

./runtimes/win-x64/native/cuda12/llama.dll

This is the code that runs "successfully", deciding that I should be running cuda 12 because it found the DLL and loaded it

LLamaSharp/LLama/Native/NativeApi.Load.cs

Lines 321 to 331 in 3d7bf42

	// Try to load a DLL from the path if supported. Returns null if nothing is loaded.
	static IntPtr? TryLoad(string path, bool supported = true)
	{
	if (!supported)
	return null;
	if (NativeLibrary.TryLoad(path, out var handle))
	return handle;
	return null;
	}

Possible Solution: Interact with cuda before "succeeding"?

Perhaps TryLoad() shouldn't blindly return the pointer, but rather actually load the library and interact with it (e.g., getting ContextSize without crashing) before indicating that the loading was successful. I tried to figure out how to do this without loading a heavy model (maybe there's a dummy model or something used for testing?) but haven't been able to figure that out yet. I'll keep poking at this issue, but I welcome feedback in case anyone has suggestions!

[martindevans]

What is this method returning for your system?

[swharden]

I have a little more info.... my environment variables are signaling to use cuda 12

LLamaSharp/LLama/Native/NativeApi.Load.cs

Lines 66 to 74 in 3d7bf42

	if (RuntimeInformation.IsOSPlatform(OSPlatform.Windows))
	{
	cudaPath = Environment.GetEnvironmentVariable("CUDA_PATH");
	if (cudaPath is null)
	{
	return -1;
	}
	version = GetCudaVersionFromPath(cudaPath);
	}

What is this method returning for your system?

ha! we converged on the same thing at the same time

I'm going to uninstall this and see what happens ...

[swharden]

Success! 🚀

Uninstalling the NVIDIA CUDA toolkit automatically deleted the CUDA_PATH environment variable and the example runner works as expected.

Thanks so much for your input along the way!

I wonder if there's something we can do to detect this issue and show a more helpful error message to the user when the cuda DLL is detected but it can't be used because the available hardware doesn't support it. I'd love to add that logic right here:

LLamaSharp/LLama/Native/NativeApi.Load.cs

Lines 321 to 331 in 3d7bf42

	// Try to load a DLL from the path if supported. Returns null if nothing is loaded.
	static IntPtr? TryLoad(string path, bool supported = true)
	{
	if (!supported)
	return null;
	if (NativeLibrary.TryLoad(path, out var handle))
	return handle;
	return null;
	}