[Bug]: 2080ti 22G * 4, vllm=0.4.0 startup error #3764
Labels
bug
Something isn't working
Comments
|
Can you try to execute some small piece of code to verify cuda/torch is working? e.g. import torch
data = torch.randn(555).cuda()
print(data.sum().cpu().item()) |
|
In addition, you can try to build from source: https://docs.vllm.ai/en/latest/getting_started/installation.html#build-from-source . See if it works. |
|
In case you met errors, you can change the following line: Line 16 in 563c1d7 to |
|
cuda runs normally. The key issue is that it runs normally in vllm==0.3.3. The problem only arises after upgrading to vllm==0.4.0. |
|
Yes, the released binary wheel might not contain the cuda kernel for your GPU model. If you can help to build from source to verify it, it would be very helpful. |
|
Please try this. |
|
7.5 has already been set. Reporting this error. |
|
7.5 is supported. There is a bug in our new cmake build system which leads to your failure. Hopefully it will be solved in this week. Please stay tuned. |
|
Eagerly anticipating. Thank you for your support. |
|
same error, happy to see this! but my device is GTX1060, which compute capability is 6.1, is that be supported? both vllm 0.3.3 and 0.4.0 not work. |
|
I test on GTX1660 supper, which compute capability is 7.5, version 0.3.3 works, version 0.4.0 don't work and get same error. |
|
@lzcchl there is one problem: your installation of pytorch is for cuda 12.1, but your cuda version is 11.5 .
Please update your cuda toolkit version at https://developer.nvidia.com/cuda-toolkit-archive . |
|
It is recommended to have cuda 12.1, because then you can install torch seamlessly. Otherwise you have to install a specific torch version, which is again very difficult. |
|
It is recommended to have cuda 12.1, because then you can install torch seamlessly. Otherwise you have to install a specific torch version, which is again very difficult. Thank you for pointing out the issue, I have already upgraded the cuda-toolkit version to 12.1. However, it seems unrelated to the current compilation error, and the same error is still being reported. |
|
Same error here with version 0.4.0, using cuda 12.1 and a T4 GPU. It worked with v0.3.3. |
The moon rises over the sea, from afar we share this moment. 海上生明月,天涯共此时,很高兴看到老外也报错. |
|
@chuanzhubin @GennVa we just release |
|
I'm glad to receive the new tag, but it seems that there is still an error when running |
|
You can download and use the attached wheel, no need to install from source. Your build fails because your environment is not set up correctly. See https://docs.vllm.ai/en/latest/getting_started/installation.html#build-from-source :
|
|
The installed |
I guess that need to downgrade CUDA to 11.8 |
|
After the experiment, inference under CUDA 11.8 and CUDA 12.1 will result in garbled text. There is a little bit of frustration. |
|
When you are installing # Install vLLM with CUDA 11.8.
export VLLM_VERSION=0.4.0
export PYTHON_VERSION=39
pip install https://github.com/vllm-project/vllm/releases/download/v${VLLM_VERSION}/vllm-${VLLM_VERSION}+cu118-cp${PYTHON_VERSION}-cp${PYTHON_VERSION}-manylinux1_x86_64.whl --extra-index-url https://download.pytorch.org/whl/cu118Note that the last line is very long, and there is If your model's output is not as expected, that should be a separate issue, and you need to file another issue with details on your environment and model. |
|
Today I did not give up, and finally compiled and installed successfully, and got the fragrant qwen-moe. |
安装NVIDIA驱动和CUDA Toolkit1. 安装NVIDIA驱动
2. 安装CUDA Toolkit
配置环境变量添加CUDA Toolkit到PATH为了确保 export PATH=$PATH:/usr/local/cuda-12.1/bin请确保将 安装VLLM1. 切换到VLLM的特定版本
2. 安装VLLM使用 pip install -e .可选步骤：修改VLLM的CMakeLists.txt如果在安装过程中遇到问题，可能需要修改 set(CUDA_SUPPORTED_ARCHS "7.5")请注意，这个修改可能不是必要的，具体取决于您的系统配置和需求。 |
|
@youkaichao Thank you for your professional and patient help |
@youkaichao v0.4.0.post1 working for me, thanks! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Your current environment
🐛 Describe the bug
The text was updated successfully, but these errors were encountered: