How to run the web UI only using CPU and without an NVIDIA graphic card

[alessandro-violante-dsgroup]

Hi,

I'm new to Code Llama and I'm trying to use the web UI with the model "codellama_CodeLlama-7b-Instruct-hf" I have found on Hugging Face.

I've followed these steps:

1. I've created a Conda environment (after having installed Anaconda)
2. I've activated the environment
3. I've installed the CPU only version of PyTorch for Windows
4. I've installed the web UI
5. I've downloaded the model from Hugging Face and I've put it in the models folder
6. I've started the web UI
7. I've tried to load the model using the "Load" button

After having clicked on "Load", after some time, on the command prompt I get the error visible in the attached print screen:

I would like to use only my CPU just to make some tests, as I don't have an NVIDIA graphic card.

If this could help in any way, when I clone the repository https://github.com/oobabooga/text-generation-webui, I get this error, although the web UI can be started:

ERROR: auto_gptq-0.4.2+cu117-cp310-cp310-win_amd64.whl is not a supported wheel on this platform.

I've also tried running this command, but when I try to load the model, I still have the same error:

pip install -r requirements_nocuda.txt

What should I do in order to use only my CPU and to be able to make a test? Thank you so much!

[lucasuniverse]

Conseguiu resolver seu problema? Utilizando apenas a CPU? Se sim! Poderia informar o que foi feito?

[TheLounger]

Use a model and loader that are primarily designed for CPU, such as codellama-7b-instruct.Q6_K.gguf.
Use these settings under the Model tab:

• Model: The GGUF model you downloaded
• Model loader: llama.cpp
• n-gpu-layers: 0
• threads: the amount of physical cores on your CPU

Then press Load.

Edit: Oof, necropost 😑