Windows not supported #28

mottosso · 2023-04-18T08:59:52Z

To save fellow Windows users some time and download bandwidth, this project only runs on Linux due to (at least) one of the dependencies not supporting Windows.

https://github.com/TimDettmers/bitsandbytes#requirements--installation

You'll receive an error upon running it.

python.exe .\demo.py --cfg-path .\eval_configs\minigpt4_eval.yaml
# Required library version not found: libsbitsandbytes_cpu.so. Maybe you need to compile it from source?

The .so extension is for Linux only.

The text was updated successfully, but these errors were encountered:

cibernicola · 2023-04-18T10:23:49Z

Trying with bitsandbytes version from https://github.com/oobabooga/text-generation-webui

I'm getting this:

(minigpt4) J:\gpt\MiniGPT-4>python demo.py --cfg-path eval_configs/minigpt4_eval.yaml
Initializing Chat
Loading VIT
Loading VIT Done
Loading Q-Former
Loading Q-Former Done
Loading LLAMA


===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please submit your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
================================================================================
CUDA SETUP: Loading binary J:\conda\envs\minigpt4\lib\site-packages\bitsandbytes\libbitsandbytes_cpu.dll...
J:\conda\envs\minigpt4\lib\site-packages\bitsandbytes\cextension.py:31: UserWarning: The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers and GPU quantization are unavailable.
  warn("The installed version of bitsandbytes was compiled without GPU support. "
Traceback (most recent call last):
  File "J:\gpt\MiniGPT-4\demo.py", line 57, in <module>
    model = model_cls.from_config(model_config).to('cuda:0')
  File "J:\gpt\MiniGPT-4\minigpt4\models\mini_gpt4.py", line 241, in from_config
    model = cls(
  File "J:\gpt\MiniGPT-4\minigpt4\models\mini_gpt4.py", line 89, in __init__
    self.llama_model = LlamaForCausalLM.from_pretrained(
  File "J:\conda\envs\minigpt4\lib\site-packages\transformers\modeling_utils.py", line 2740, in from_pretrained
    raise ValueError(
ValueError:
                        Some modules are dispatched on the CPU or the disk. Make sure you have enough GPU RAM to fit
                        the quantized model. If you want to dispatch the model on the CPU or the disk while keeping
                        these modules in 32-bit, you need to set `load_in_8bit_fp32_cpu_offload=True` and pass a custom
                        `device_map` to `from_pretrained`. Check
                        https://huggingface.co/docs/transformers/main/en/main_classes/quantization#offload-between-cpu-and-gpu

mottosso · 2023-04-18T10:26:09Z

One step closer! However..

The installed version of bitsandbytes was compiled without GPU support

I would be weary of this; I expect a CPU version to take minutes to hours to days, compared to seconds.

cibernicola · 2023-04-18T10:40:24Z

Still searching and remixing :S
Don't know why its saying that having in mind that https://github.com/oobabooga/text-generation-webui is working in CUDA

¬¬'

cibernicola · 2023-04-18T10:49:37Z

I guess is this: #4

hufuzhipeng · 2023-04-18T11:38:45Z

===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please submit your error trace to: https://github.com/TimDettmers/bitsandbytes/issues

CUDA SETUP: Required library version not found: libsbitsandbytes_cpu.so. Maybe you need to compile it from source?
CUDA SETUP: Defaulting to libbitsandbytes_cpu.so...
argument of type 'WindowsPath' is not iterable
C:\Users\zphu.conda\envs\minigpt4\lib\site-packages\bitsandbytes\cextension.py:31: UserWarning: The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers and GPU quantization are unavailable.
warn("The installed version of bitsandbytes was compiled without GPU support. "
This model has some weights that should be kept in higher precision, you need to upgrade accelerate to properly deal with them (pip install --upgrade accelerate).
Traceback (most recent call last):
File "E:\code\ai\MiniGPT-4\demo.py", line 57, in
model = model_cls.from_config(model_config).to('cuda:0')
File "E:\code\ai\MiniGPT-4\minigpt4\models\mini_gpt4.py", line 241, in from_config
model = cls(
File "E:\code\ai\MiniGPT-4\minigpt4\models\mini_gpt4.py", line 89, in init
self.llama_model = LlamaForCausalLM.from_pretrained(
File "C:\Users\zphu.conda\envs\minigpt4\lib\site-packages\transformers\modeling_utils.py", line 2740, in from_pretrained
raise ValueError(
ValueError:
Some modules are dispatched on the CPU or the disk. Make sure you have enough GPU RAM to fit
the quantized model. If you want to dispatch the model on the CPU or the disk while keeping
these modules in 32-bit, you need to set load_in_8bit_fp32_cpu_offload=True and pass a custom
device_map to from_pretrained. Check
https://huggingface.co/docs/transformers/main/en/main_classes/quantization#offload-between-cpu-and-gpu
for more details.

(minigpt4) E:\code\ai\MiniGPT-4>
(minigpt4) E:\code\ai\MiniGPT-4>
(minigpt4) E:\code\ai\MiniGPT-4>python -m torch.utils.collect_env
Collecting environment information...
PyTorch version: 2.0.0+cpu
Is debug build: False
CUDA used to build PyTorch: Could not collect
ROCM used to build PyTorch: N/A

OS: Microsoft Windows 10 企业版
GCC version: Could not collect
Clang version: Could not collect
CMake version: version 3.24.0-rc4
Libc version: N/A

Python version: 3.9.16 (main, Mar 8 2023, 10:39:24) [MSC v.1916 64 bit (AMD64)] (64-bit runtime)
Python platform: Windows-10-10.0.19044-SP0
Is CUDA available: False
CUDA runtime version: 11.7.64
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 3090
Nvidia driver version: 516.01
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture=9
CurrentClockSpeed=4008
DeviceID=CPU0
Family=207
L2CacheSize=2048
L2CacheSpeed=
Manufacturer=GenuineIntel
MaxClockSpeed=4008
Name=Intel(R) Core(TM) i9-9900KS CPU @ 4.00GHz
ProcessorType=3
Revision=

Versions of relevant libraries:
[pip3] numpy==1.21.0
[pip3] torch==2.0.0
[pip3] torchaudio==0.12.1
[pip3] torchvision==0.15.1
[conda] blas 1.0 mkl
[conda] cudatoolkit 11.3.1 h59b6b97_2
[conda] mkl 2021.4.0 haa95532_640
[conda] mkl-service 2.4.0 py39h2bbff1b_0
[conda] mkl_fft 1.3.1 py39h277e83a_0
[conda] mkl_random 1.2.2 py39hf11a4ad_0
[conda] numpy 1.23.5 py39h3b20f71_0
[conda] numpy-base 1.23.5 py39h4da318b_0
[conda] pytorch-mutex 1.0 cuda pytorch
[conda] torch 2.0.0 pypi_0 pypi
[conda] torchaudio 0.12.1 py39_cu113 pytorch
[conda] torchvision 0.15.1 pypi_0 pypi

thibaudart · 2023-04-18T16:20:10Z

Still searching and remixing :S Don't know why its saying that having in mind that https://github.com/oobabooga/text-generation-webui is working in CUDA

¬¬'

Yes with that, we can have GPU support but still some error:

Any idea how to solve this:

Some modules are dispatched on the CPU or the disk. Make sure you have enough GPU RAM to fit
the quantized model. If you want to dispatch the model on the CPU or the disk while keeping
these modules in 32-bit, you need to set load_in_8bit_fp32_cpu_offload=True and pass a custom
device_map to from_pretrained. Check
https://huggingface.co/docs/transformers/main/en/main_classes/quantization#offload-between-cpu-and-gpu
for more details.

I have 48gb of vram the GPU RAM must be enough!

BANGBUP · 2023-04-19T06:09:02Z

use windows precompiled version dlls from below.
https://github.com/DeXtmL/bitsandbytes-win-prebuilt

and, you have to change some source code....

C:\ProgramData\Anaconda3\envs\minigpt4\Lib\site-packages\bitsandbytes\cuda_setup\main.py (my case...)
about 370 line:
if not torch.cuda.is_available(): return 'libsbitsandbytes_cpu.so', None, None, None, None
to
if torch.cuda.is_available(): return 'libbitsandbytes_cuda116.dll', None, None, None, None

and, about 126 line:
self.lib = ct.cdll.LoadLibrary(binary_path)
to
self.lib = ct.cdll.LoadLibrary(str(binary_path))

then you can see no errors.

but my case, I still get an OOM problem... although i have 3090ti 24g...

(minigpt4) PS D:\pythonworkspace\MiniGPT-4> python demo.py --cfg-path eval_configs/minigpt4_eval.yaml
Initializing Chat
Loading VIT
Loading VIT Done
Loading Q-Former
Loading Q-Former Done
Loading LLAMA

===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please submit your error trace to: https://github.com/TimDettmers/bitsandbytes/issues

binary_path = C:\ProgramData\Anaconda3\envs\minigpt4\lib\site-packages\bitsandbytes\libbitsandbytes_cuda116.dll
This model has some weights that should be kept in higher precision, you need to upgrade accelerate to properly deal with them (pip install --upgrade accelerate).
Loading checkpoint shards: 33%|███████████████████ | 1/3 [00:09<00:19, 9.92s/it]
Traceback (most recent call last):
File "D:\pythonworkspace\MiniGPT-4\demo.py", line 57, in
model = model_cls.from_config(model_config).to('cuda:0')
File "D:\pythonworkspace\MiniGPT-4\minigpt4\models\mini_gpt4.py", line 241, in from_config
model = cls(
File "D:\pythonworkspace\MiniGPT-4\minigpt4\models\mini_gpt4.py", line 89, in init
self.llama_model = LlamaForCausalLM.from_pretrained(
File "C:\ProgramData\Anaconda3\envs\minigpt4\lib\site-packages\transformers\modeling_utils.py", line 2795, in from_pretrained
) = cls._load_pretrained_model(
File "C:\ProgramData\Anaconda3\envs\minigpt4\lib\site-packages\transformers\modeling_utils.py", line 3123, in _load_pretrained_model
new_error_msgs, offload_index, state_dict_index = _load_state_dict_into_meta_model(
File "C:\ProgramData\Anaconda3\envs\minigpt4\lib\site-packages\transformers\modeling_utils.py", line 706, in _load_state_dict_into_meta_model
set_module_8bit_tensor_to_device(
File "C:\ProgramData\Anaconda3\envs\minigpt4\lib\site-packages\transformers\utils\bitsandbytes.py", line 78, in set_module_8bit_tensor_to_device
new_value = bnb.nn.Int8Params(new_value, requires_grad=False, has_fp16_weights=has_fp16_weights).to(device)
File "C:\ProgramData\Anaconda3\envs\minigpt4\lib\site-packages\bitsandbytes\nn\modules.py", line 196, in to
return self.cuda(device)
File "C:\ProgramData\Anaconda3\envs\minigpt4\lib\site-packages\bitsandbytes\nn\modules.py", line 159, in cuda
B = self.data.contiguous().half().cuda(device)
RuntimeError: CUDA out of memory. Tried to allocate 136.00 MiB (GPU 0; 23.99 GiB total capacity; 7.83 GiB already allocated; 14.09 GiB free; 8.25 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
(minigpt4) PS D:\pythonworkspace\MiniGPT-4>

cibernicola · 2023-04-19T11:55:37Z

with the last version I'm reaching this point (without touching code):

`(miniGPT4) J:\gpt\MiniGPT-4>python demo.py --cfg-path eval_configs/minigpt4_eval.yaml
Initializing Chat
Loading VIT
Loading VIT Done
Loading Q-Former
Loading Q-Former Done
Loading LLAMA

===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please submit your error trace to: https://github.com/TimDettmers/bitsandbytes/issues

J:\conda\envs\miniGPT4\lib\site-packages\bitsandbytes\cuda_setup\main.py:141: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {WindowsPath('J:/conda/envs/miniGPT4/bin')}
warn(msg)
J:\conda\envs\miniGPT4\lib\site-packages\bitsandbytes\cuda_setup\main.py:141: UserWarning: J:\conda\envs\miniGPT4 did not contain cudart64_110.dll as expected! Searching further paths...
warn(msg)
J:\conda\envs\miniGPT4\lib\site-packages\bitsandbytes\cuda_setup\main.py:141: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {WindowsPath('C:/Program Files (x86)/Java/jre1.8.0_321/bin'), WindowsPath('C:/Python38'), WindowsPath('L:/COLMAP-3.7-windows-cuda'), WindowsPath('C:/xampp/php'), WindowsPath('C:/Python38/Scripts'), WindowsPath('J:/conda/envs/miniGPT4/Library/mingw-w64/bin'), WindowsPath('J:/conda/envs/miniGPT4/bin'), WindowsPath('J:/conda/envs/miniGPT4/Library/usr/bin'), WindowsPath('J:/conda/envs/minigpt4/Lib/site-packages/cv2/../../x64/vc14/bin'), WindowsPath('C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.34.31933/bin/Hostx64/x64')}
warn(msg)
CUDA SETUP: CUDA runtime path found: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.7\bin\cudart64_110.dll
CUDA SETUP: Highest compute capability among GPUs detected: 8.6
CUDA SETUP: Detected CUDA version 117
CUDA SETUP: Loading binary J:\conda\envs\miniGPT4\lib\site-packages\bitsandbytes\libbitsandbytes_cuda117.dll...
Traceback (most recent call last):
File "J:\gpt\MiniGPT-4\demo.py", line 57, in
model = model_cls.from_config(model_config).to('cuda:0')
File "J:\gpt\MiniGPT-4\minigpt4\models\mini_gpt4.py", line 241, in from_config
model = cls(
File "J:\gpt\MiniGPT-4\minigpt4\models\mini_gpt4.py", line 89, in init
self.llama_model = LlamaForCausalLM.from_pretrained(
File "J:\conda\envs\miniGPT4\lib\site-packages\transformers\modeling_utils.py", line 2740, in from_pretrained
raise ValueError(
ValueError:
Some modules are dispatched on the CPU or the disk. Make sure you have enough GPU RAM to fit
the quantized model. If you want to dispatch the model on the CPU or the disk while keeping
these modules in 32-bit, you need to set load_in_8bit_fp32_cpu_offload=True and pass a custom
device_map to from_pretrained. Check
https://huggingface.co/docs/transformers/main/en/main_classes/quantization#offload-between-cpu-and-gpu
for more details.

(miniGPT4) J:\gpt\MiniGPT-4>`

Korner83 · 2023-04-21T05:03:11Z

It is working me on Win 10 GPU, just use this version for Bitsandbytes: https://github.com/Keith-Hon/bitsandbytes-windows.git

cibernicola · 2023-04-21T09:10:46Z

can you please describe a little bit about what you did?

Korner83 · 2023-04-21T12:43:32Z

First I uninstalled the none-windows package of bitsandbytes and after that I installed this version in Anaconda: https://github.com/Keith-Hon/bitsandbytes-windows.git

You just need to have installed Git on your Windows

zxcvbn114514 · 2023-04-21T12:56:28Z

First I uninstalled the none-windows package of bitsandbytes and after that I installed this version in Anaconda: https://github.com/Keith-Hon/bitsandbytes-windows.git

You just need to have installed Git on your Windows

I did it and it just showed
Unable to create process using 'C:\Users\Ge Yunxiang.conda\envs\minigpt4\python.exe "C:\Users\Ge Yunxiang.conda\envs\minigpt4\Scripts\pip-script.py" install bitsandbytes'

Korner83 · 2023-04-21T13:13:03Z

Check out this post: https://bobbyhadz.com/blog/python-fatal-error-in-launcher-unable-to-create-process-using-pip-exe

zxcvbn114514 · 2023-04-21T13:22:40Z

Check out this post: https://bobbyhadz.com/blog/python-fatal-error-in-launcher-unable-to-create-process-using-pip-exe

I see.But I still dont know how to install bitsandbytes-windows with 'pip install bitsandbytes'.Shouldn't it become installing bitsandbytes from the source on the Internet rather than from the keith-hon's source?

Korner83 · 2023-04-21T13:27:09Z

The official version is only for Linux. Only keith-hon's version of bitsandbyte supports Windows as far as I know. I had the same issue and was not working, because as a default it's installing wrong package (Linux version onto Windows) by running the command: pip install bitsandbyte

zxcvbn114514 · 2023-04-21T13:31:40Z

The official version is only for Linux. Only keith-hon's version of bitsandbyte supports Windows as far as I know. I had the same issue and was not working, because as a default it's installing wrong package (Linux version onto Windows) by running the command: pip install bitsandbyte

OH,I understand .Thousand thanks man.But now I've encounted the same problem after successfully installed bitsandbytes-0.37.5-py3.9.egg.The picture below shows the problem.

oldtreee · 2023-04-21T15:46:28Z

正式版仅适用于 Linux。据我所知，只有 keith-hon 的 bitsandbyte 版本支持 Windows。我有同样的问题并且没有工作，因为默认情况下它通过运行命令安装错误的包（Linux 版本到 Windows）：pip install bitsandbyte

哦，我明白了。非常感谢。但是现在我在成功安装 bitsandbytes-0.37.5-py3.9.egg 后遇到了同样的问题。下图显示了问题。

我也遇到了同样的问题，你的解决了吗？

zxcvbn114514 · 2023-04-22T02:54:47Z

正式版仅适用于 Linux。据我所知，只有 keith-hon 的 bitsandbyte 版本支持 Windows。我有同样的问题并且没有工作，因为默认情况下它通过运行命令安装错误的包（Linux 版本到 Windows）：pip install bitsandbyte

哦，我明白了。非常感谢。但是现在我在成功安装 bitsandbytes-0.37.5-py3.9.egg 后遇到了同样的问题。下图显示了问题。

我也遇到了同样的问题，你的解决了吗？

看看是不是虚拟环境的torch版本装成cpu了。
还有能加一下qq讨论一下吗，我是571130091

latorc · 2023-04-26T19:31:23Z

I got it working on Windows 10 (CUDA 118, RTX 4090).
Install this version compatible with windows:
conda install git pip
pip uninstall bitsandbytes
pip install git+https://github.com/Keith-Hon/bitsandbytes-windows.git

===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please submit your error trace to: https://github.com/TimDettmers/bitsandbytes/issues

binary_path: C:\ProgramData\miniconda3\envs\minigpt4\lib\site-packages\bitsandbytes\cuda_setup\libbitsandbytes_cuda116.dll
CUDA SETUP: Loading binary C:\ProgramData\miniconda3\envs\minigpt4\lib\site-packages\bitsandbytes\cuda_setup\libbitsandbytes_cuda116.dll...
Loading checkpoint shards: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:29<00:00, 9.74s/it]
Loading LLAMA Done
Load 4 training prompts
Prompt Example
###Human: Describe this image in detail. ###Assistant:
Load BLIP2-LLM Checkpoint: ./model/pretrained_minigpt4.pth
Initialization Finished
Running on local URL: http://127.0.0.1:7860

First I uninstalled the none-windows package of bitsandbytes and after that I installed this version in Anaconda: https://github.com/Keith-Hon/bitsandbytes-windows.git

You just need to have installed Git on your Windows

huangzhongzhong mentioned this issue Apr 19, 2023

Mistake in the preparation of vicuna weights (error when loading delta weights) #52

Open

This was referenced Apr 20, 2023

Bitsandbytes cant find CUDA #63

Open

need help have a 3090 wont work #66

Open

Korner83 mentioned this issue May 12, 2023

Can this model run on Windows? #217

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Windows not supported #28

Windows not supported #28

mottosso commented Apr 18, 2023

cibernicola commented Apr 18, 2023 •

edited

mottosso commented Apr 18, 2023

cibernicola commented Apr 18, 2023

cibernicola commented Apr 18, 2023

hufuzhipeng commented Apr 18, 2023

thibaudart commented Apr 18, 2023

BANGBUP commented Apr 19, 2023

cibernicola commented Apr 19, 2023

Korner83 commented Apr 21, 2023

cibernicola commented Apr 21, 2023

Korner83 commented Apr 21, 2023 •

edited

zxcvbn114514 commented Apr 21, 2023

Korner83 commented Apr 21, 2023

zxcvbn114514 commented Apr 21, 2023

Korner83 commented Apr 21, 2023 •

edited

zxcvbn114514 commented Apr 21, 2023

oldtreee commented Apr 21, 2023

zxcvbn114514 commented Apr 22, 2023

latorc commented Apr 26, 2023

Windows not supported #28

Windows not supported #28

Comments

mottosso commented Apr 18, 2023

cibernicola commented Apr 18, 2023 • edited

mottosso commented Apr 18, 2023

cibernicola commented Apr 18, 2023

cibernicola commented Apr 18, 2023

hufuzhipeng commented Apr 18, 2023

===================================BUG REPORT=================================== Welcome to bitsandbytes. For bug reports, please submit your error trace to: https://github.com/TimDettmers/bitsandbytes/issues

thibaudart commented Apr 18, 2023

BANGBUP commented Apr 19, 2023

===================================BUG REPORT=================================== Welcome to bitsandbytes. For bug reports, please submit your error trace to: https://github.com/TimDettmers/bitsandbytes/issues

cibernicola commented Apr 19, 2023

===================================BUG REPORT=================================== Welcome to bitsandbytes. For bug reports, please submit your error trace to: https://github.com/TimDettmers/bitsandbytes/issues

Korner83 commented Apr 21, 2023

cibernicola commented Apr 21, 2023

Korner83 commented Apr 21, 2023 • edited

zxcvbn114514 commented Apr 21, 2023

Korner83 commented Apr 21, 2023

zxcvbn114514 commented Apr 21, 2023

Korner83 commented Apr 21, 2023 • edited

zxcvbn114514 commented Apr 21, 2023

oldtreee commented Apr 21, 2023

zxcvbn114514 commented Apr 22, 2023

latorc commented Apr 26, 2023

===================================BUG REPORT=================================== Welcome to bitsandbytes. For bug reports, please submit your error trace to: https://github.com/TimDettmers/bitsandbytes/issues

cibernicola commented Apr 18, 2023 •

edited

===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please submit your error trace to: https://github.com/TimDettmers/bitsandbytes/issues

===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please submit your error trace to: https://github.com/TimDettmers/bitsandbytes/issues

===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please submit your error trace to: https://github.com/TimDettmers/bitsandbytes/issues

Korner83 commented Apr 21, 2023 •

edited

Korner83 commented Apr 21, 2023 •

edited

===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please submit your error trace to: https://github.com/TimDettmers/bitsandbytes/issues