DLL not found

[sgjohnson1981]

Trying to run with cublas for an RTX 2070 Super. Updated graphics driver and CUDA 11.4 is installed. That subfolder doesn't exist in the Temp folder. Is that the correct version of CUDA for my card?

C:\code\LLMs>koboldcpp.exe
***
Welcome to KoboldCpp - Version 1.54
For command line arguments, please refer to --help
***
Attempting to use CuBLAS library for faster prompt ingestion. A compatible CuBLAS will be required.
Initializing dynamic library: koboldcpp_cublas.dll
Traceback (most recent call last):
  File "PyInstaller\loader\pyimod03_ctypes.py", line 53, in __init__
  File "ctypes\__init__.py", line 373, in __init__
OSError: [WinError 1114] A dynamic link library (DLL) initialization routine failed

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "koboldcpp.py", line 2515, in <module>
  File "koboldcpp.py", line 2330, in main
  File "koboldcpp.py", line 216, in init_library
  File "PyInstaller\loader\pyimod03_ctypes.py", line 55, in __init__
pyimod03_ctypes.PyInstallerImportError: Failed to load dynlib/dll 'C:\\Users\\Dell T3500\\AppData\\Local\\Temp\\_MEI170722\\koboldcpp_cublas.dll'. Most likely this dynlib/dll was not found when the application was frozen.
[16180] Failed to execute script 'koboldcpp' due to unhandled exception!

[LostRuins]

It seems like the DLL file was not able to be loaded, the error says file not found, so I'm wondering if you might have any software that could be blocking or removing it. Does this happen with the other backends such as clblast/openblas too, or is it only the cublas one that fails?

The next time it fails, try navigating to the extracted temp directory (e.g. C:\\Users\\Dell T3500\\AppData\\Local\\Temp\\_MEI170722\\) and take note of what files were found.

[Edobois]

I've been getting the same issue, fresh install on 1.54 but it spazzes out on those dlls. I get the same message as sgjohnson, regarding koboldcpp_cublas.dll

The next time it fails, try navigating to the extracted temp directory (e.g. C:\Users\Dell T3500\AppData\Local\Temp\_MEI170722\) and take note of what files were found.

The executable seems to wipe the temp folder in question, so you can briefly see it show up in temp, but it vanishes in about a second. GPU and drive are nice and new (RTX 4070 and an EVO 980), but my CPU and motherboard are from 2012, not sure if that's an issue.

[sgjohnson1981]

@LostRuins I don't think it's merely an issue of file not found, from the output at least. It says the file not found was caused by failed initialization. The output isn't verbose enough to understand why. It seems something similar to @Edobois is occurring. The folder shows up when the executable starts, along with all the files, including the one it's looking for. But it gets deleted when the executable errors out.

Output for openblas:

C:\code\LLMs>koboldcpp.exe        
***
Welcome to KoboldCpp - Version 1.54
For command line arguments, please refer to --help
***
Attempting to use OpenBLAS library for faster prompt ingestion. A compatible libopenblas will be required.
Initializing dynamic library: koboldcpp_openblas.dll
==========
Namespace(bantokens=None, blasbatchsize=512, blasthreads=5, config=None, contextsize=2048, debugmode=0, forceversion=0, foreground=False, gpulayers=200, highpriority=False, hordeconfig=None, host='', launch=False, lora=None, model=None, model_param='C:/code/LLMs/models/phi-2_q8_0.gguf', multiuser=1, noavx2=False, noblas=False, nommap=False, noshift=False, onready='', port=5001, port_param=5001, preloadstory=None, quiet=False, remotetunnel=False, ropeconfig=[0.0, 10000.0], skiplauncher=False, smartcontext=False, ssl=None, tensor_split=None, threads=5, useclblast=None, usecublas=None, usemlock=False)
==========
Loading model: C:\code\LLMs\models\phi-2_q8_0.gguf
[Threads: 5, BlasThreads: 5, SmartContext: False, ContextShift: True]

Output for clblast:

C:\code\LLMs>koboldcpp.exe
***
Welcome to KoboldCpp - Version 1.54
For command line arguments, please refer to --help
***
Attempting to use CLBlast library for faster prompt ingestion. A compatible clblast will be required.
Initializing dynamic library: koboldcpp_clblast.dll
==========
Namespace(bantokens=None, blasbatchsize=512, blasthreads=5, config=None, contextsize=2048, debugmode=0, forceversion=0, foreground=False, gpulayers=200, highpriority=False, hordeconfig=None, host='', launch=False, lora=None, model=None, model_param='C:/code/LLMs/models/phi-2_q8_0.gguf', multiuser=1, noavx2=False, noblas=False, nommap=False, noshift=False, onready='', port=5001, port_param=5001, preloadstory=None, quiet=False, remotetunnel=False, ropeconfig=[0.0, 10000.0], skiplauncher=False, smartcontext=False, ssl=None, tensor_split=None, threads=5, useclblast=[0, 0], usecublas=None, usemlock=False)
==========
Loading model: C:\code\LLMs\models\phi-2_q8_0.gguf
[Threads: 5, BlasThreads: 5, SmartContext: False, ContextShift: True]
Traceback (most recent call last):
  File "koboldcpp.py", line 2515, in <module>
  File "koboldcpp.py", line 2362, in main
  File "koboldcpp.py", line 309, in load_model
OSError: [WinError -1073741795] Windows Error 0xc000001d
[15476] Failed to execute script 'koboldcpp' due to unhandled exception!

I have an older CPU as well. CLBlast with noavx2 doesn't work either. Get the same Windows Error as the other clblast option.

[LostRuins]

@sgjohnson1981 Yeah if the DLL file shows up when it starts, but gets DELETED and disappears when executed, it sounds like a windows defender thing - or some other false positive from an antimalware software. What antivirus/antimalware software does your PC have?

@henk717 any ideas?

[sgjohnson1981]

@sgjohnson1981 Yeah if the DLL file shows up when it starts, but gets DELETED and disappears when executed, it sounds like a windows defender thing - or some other false positive from an antimalware software. What antivirus/antimalware software does your PC have?

Just windows defender. I've built llama.cpp successfully. It worked for cpu-only inference, even though there was no AVX support. Nothing got deleted out of my llama.cpp/build folders. From what i can tell, nothing gets deleted in this case until AFTER the exe crashes with the unhandled exception. You know your code better than I do. Do the tracebacks indicate anything? Seems there's a different line of code that errors out depending on which support I choose. It appears the cublas support wouldn't even initialize but the other two at least got to loading the models. The model I'm trying is a quantized phi-2 model, quantized with llama.cpp's quantize.exe to int8.

[henk717]

The binary cleans up once its done executing so its correct the files are gone after the crash. They were there while the crash happens I do not suspect an AV here.

I do suspect the CPU is to old, which one is it?

[henk717]

Looked up the dell machine from the path. Doesn't have AVX support so you will need to use the failsave mode.
KoboldAI united can be more interesting for your system.

[sgjohnson1981]

The binary cleans up once its done executing so its correct the files are gone after the crash. They were there while the crash happens I do not suspect an AV here.

I do suspect the CPU is to old, which one is it?

Xeon X5690. 2011. Is the CPU an issue if I just want to use the GPU? What is Kobold United? I was interested in Kobold.cpp because it was compatible with MemGPT.

[henk717]

The CPU is an issue because the dll's require AVX. KoboldAI United is the koboldai in my account (not koboldcpp) which is huggingface based. Its CPU requirements are milder.

[henk717]

@LostRuins knows why we can't provide GPU support without AVX for Koboldcpp. From memory the dll's that interact with the GPU were also AVX.

[sgjohnson1981]

@LostRuins knows why we can't provide GPU support without AVX for Koboldcpp. From memory the dll's that interact with the GPU were also AVX.

Ah. Going to go ahead and close the issue. For my reference, why were those DLLs chosen? Were they required for modern CUDA? Might help me in understanding what inference libraries will or won't work for my PC.

[LostRuins]

Yeah if its just without AVX2 you can still use CLBlast (select noavx2 mode).

But without AVX(1), you can only use the failsafe mode - although you can try self-compiling in GPU support but it doesn't come prebuilt (this is untested).

The prebuilt DLLs are chosen for the widest range of supported devices, with the reasoning that any device modern enough to want to run with compatible GPU support would also have at minimum AVX1 support.