cuBLAS, GPU compile instructions not working

[Free-Radical]

          > @Free-Radical Try with `CMAKE_ARGS="-DLLAMA_CUBLAS=on" FORCE_CMAKE=1 pip install llama-cpp-python`.

Did not work, BLAS = 0

Originally posted by @Free-Radical in #113 (comment)

[abetlen]

@Free-Radical can you also add the --verbose flag to pip, it may be that it can't find cuda during the build (in which case llama.cpp just silently skips it).

[gjmulder]

Any update?

[akfhsueh]

Hello, just following up on this issue in case others were wondering about the same thing.

I'm using a virtual environment through Anaconda3. When runing the complie instructions from #182 , CMake's find_package() instruction will not look at the correct location where my CUADToolkit is installed. I got to this realization thanks to abetlen's hint above that llama.cpp might be silently skipping CUDA. The relevant code lines are around https://github.com/ggerganov/llama.cpp/blob/7552ac586380f202b75b18aa216ecfefbd438d94/CMakeLists.txt#L180 .

To solve the above issue, I added the path to an environment variable called CUDAToolkit_ROOT which seemed to have done the trick. From cmake's documentation https://cmake.org/cmake/help/latest/command/find_package.html?highlight=_ROOT%20environment%20variable , search "root environment variable" for the source of my inspiration.

	python -m pip install -U pip wheel setuptools
	git clone https://github.com/abetlen/llama-cpp-python.git
	cd llama-cpp-python
	cd vendor
	git clone https://github.com/ggerganov/llama.cpp.git 
	cd .. 
	SET LLAMA_CUBLAS=1&& SET CMAKE_ARGS=-DLLAMA_CUBLAS=on&& SET FORCE_CMAKE=1&&SET "CUDAToolkit_ROOT=C:\Users\<Username>\anaconda3\envs\<venv name>\pkgs\cuda-toolkit\nvcc"
	python setup.py bdist_wheel

UPDATE: might want to try installing the right CUDAToolkit also.. I noticed that running the commands from https://anaconda.org/nvidia/cuda-toolkit vs the one from https://anaconda.org/anaconda/cudatoolkit put my headerfiles in different places. The former put them in the location as anticipated by the setup script and the latter required me to set the CUDAToolkit_ROOT env variable. Not sure if anyone else can confirm this finding.