Configuring Code-Llama to use an NVIDIA GPU on Windows

[jordanbtucker]

Open Interpreter can be used with local language models, however these can be rather taxing on your computer's resources. If you have an NVIDIA GPU, you may benefit from offloading some of the work to your GPU.

The set this up, follow the steps in Local Language Models with GPU Support.

If you still run into problems, leave a comment here or ask on the Discord server.

[mathpopo]

@jordanbtucker thanks first, i use CUDA_PATH=/usr/local/cuda FORCE_CMAKE=1 CMAKE_ARGS='-DLLAMA_CUBLAS=on' pip3 install llama-cpp-python --force-reinstall --upgrade --no-cache-dir -vv --->

error: subprocess-exited-with-error

× Building wheel for llama-cpp-python (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
full command: /home/chenxin/disk1/anaconda3/envs/open-interpreter/bin/python /home/chenxin/disk1/anaconda3/envs/open-interpreter/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py build_wheel /tmp/tmpg1mx83i7
cwd: /tmp/pip-install-08qxbi8m/llama-cpp-python_1a811c2cc36641a287946db95a7dc51f
Building wheel for llama-cpp-python (pyproject.toml) ... error
ERROR: Failed building wheel for llama-cpp-python
Failed to build llama-cpp-python
ERROR: Could not build wheels for llama-cpp-python, which is required to install pyproject.toml-based projects
Exception information:
Traceback (most recent call last):
File "/home/chenxin/disk1/anaconda3/envs/open-interpreter/lib/python3.11/site-packages/pip/_internal/cli/base_command.py", line 180, in exc_logging_wrapper
status = run_func(*args)
^^^^^^^^^^^^^^^
File "/home/chenxin/disk1/anaconda3/envs/open-interpreter/lib/python3.11/site-packages/pip/_internal/cli/req_command.py", line 248, in wrapper
return func(self, options, args)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/chenxin/disk1/anaconda3/envs/open-interpreter/lib/python3.11/site-packages/pip/_internal/commands/install.py", line 429, in run
raise InstallationError(
pip._internal.exceptions.InstallationError: Could not build wheels for llama-cpp-python, which is required to install pyproject.toml-based projects

(open-interpreter) chenxin@chenxin-Nitro-AN515-52:$ gcc --version
gcc (Ubuntu 9.4.0-1ubuntu120.04.2) 9.4.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

(open-interpreter) chenxin@chenxin-Nitro-AN515-52:$ g++ --version
g++ (Ubuntu 9.4.0-1ubuntu120.04.2) 9.4.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

[gnivler]

Hi - thanks for all your work on this, very exciting!

The link in your OP is 404 - I think it should go to here maybe (missing docs path)? https://github.com/KillianLucas/open-interpreter/blob/main/docs/GPU.md

[raptor-bot]

I installed CUDA and verified GPU support set up properly using all instructions from https://github.com/KillianLucas/open-interpreter/blob/main/docs/GPU.md. But CPU still pegged and barely using GPU and open-interpreter too slow to use.

Am I still missing a step or is my Win laptop just too slow? GPU is RTX3050.

Thank you.

[jordanbtucker]

@raptor-bot Thanks for reporting. Please make sure you are on the latest version of open-interpreter by running the following.

pip install --upgrade open-interpreter

Unfortunately, running a local model is going to be a degraded experience at this early stage. Ensuring you are on the latest version of open-interpreter will help to a degree, but unless you run a 30B model on an RTX 3090 or 4090, you will not likely be able to get anything close to GPT-3.5, at least for now.

We are of course working on improving this.

[raptor-bot]

Thanks Jordan, I appreciate the response. I also have a Mac mini with attached specs. Is the performance likely to be any better than my windows laptop?

[jordanbtucker]

@raptor-bot It will likely be the same or slower based on your mac's GPU.

[ghost]

@raptor-bot you are constrained by your GPUs memory. If the model is larger than the onboard memory (keeping at least 1gb of headroom for stability) it will be shifted to CPU and RAM.

Your best bet is probably to use smaller, heavily quantized models. Of course this has direct negative implications for performance but that's the cost of being on the bleeding edge of tech. Financially probably better off subbing to OpenAi and utilizing the API if you are looking for performance. Otherwise you are looking at dropping a couple stacks on a new rig.

Honestly with the pace of things the best bet for most is probably a combo of gpu rental (Llamba, vast etc) and OpenAi api. Gives the best performance with very little upfront buy-in cost.