How to launch alpaca-13b with 6gb of VRAM ?

[vivelespatess]

Describe the bug

I want to run gpt4-x-alpaca-13b with only 6gb of VRAM, but i've this error :

Is there an existing issue for this?

• I have searched the existing issues

Reproduction

i use the last version of oobabooga

Screenshot

No response

Logs

Gradio HTTP request redirected to localhost :)
bin D:\oobabooga_windows\installer_files\env\lib\site-packages\bitsandbytes\libbitsandbytes_cuda117_nocublaslt.dll
Loading anon8231489123_gpt4-x-alpaca-13b-native-4bit-128g...
Warning: more than one .pt model has been found. The last one will be selected. It could be wrong.
Found the following quantized model: models\anon8231489123_gpt4-x-alpaca-13b-native-4bit-128g\gpt-x-alpaca-13b-native-4bit-128g.pt
Loading model ...
Done.
Traceback (most recent call last):
  File "D:\oobabooga_windows\text-generation-webui\server.py", line 914, in <module>
    shared.model, shared.tokenizer = load_model(shared.model_name)
  File "D:\oobabooga_windows\text-generation-webui\modules\models.py", line 158, in load_model
    model = load_quantized(model_name)
  File "D:\oobabooga_windows\text-generation-webui\modules\GPTQ_loader.py", line 197, in load_quantized
    model = model.to(torch.device('cuda:0'))
  File "D:\oobabooga_windows\installer_files\env\lib\site-packages\transformers\modeling_utils.py", line 1896, in to
    return super().to(*args, **kwargs)
  File "D:\oobabooga_windows\installer_files\env\lib\site-packages\torch\nn\modules\module.py", line 1145, in to
    return self._apply(convert)
  File "D:\oobabooga_windows\installer_files\env\lib\site-packages\torch\nn\modules\module.py", line 797, in _apply
    module._apply(fn)
  File "D:\oobabooga_windows\installer_files\env\lib\site-packages\torch\nn\modules\module.py", line 797, in _apply
    module._apply(fn)
  File "D:\oobabooga_windows\installer_files\env\lib\site-packages\torch\nn\modules\module.py", line 797, in _apply
    module._apply(fn)
  [Previous line repeated 2 more times]
  File "D:\oobabooga_windows\installer_files\env\lib\site-packages\torch\nn\modules\module.py", line 844, in _apply
    self._buffers[key] = fn(buf)
  File "D:\oobabooga_windows\installer_files\env\lib\site-packages\torch\nn\modules\module.py", line 1143, in convert
    return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 34.00 MiB (GPU 0; 6.00 GiB total capacity; 5.19 GiB already allocated; 0 bytes free; 5.38 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

Done!
Appuyez sur une touche pour continuer...

System Info

i use the last version of oobabooga with :
Ryzen 7 3800x
GTX 1060 6GB vram
32GB ram ddr4

[LaaZa]

You don't. Not on gpu. This isn't magic, even if 4bit appears so at first glance. Try GGML that will run on your cpu and ram(not vram)

[xrd]

Ggml 7b should work, I've got a 6GB card and that runs within 4GB.

[ClayShoaf]

@vivelespatess could you please close this issue?