-
Notifications
You must be signed in to change notification settings - Fork 211
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
import error, undefined symbol: ncclBroadcast #18
Comments
To ensure that the CUDA version used to compile your Torch C++ plugin matches the runtime version of your current CUDA Toolkit, you can use the following Python command:
This command will print the CUDA version that was used to compile PyTorch. Please ensure that this version matches the version of your installed CUDA Toolkit. In addition, please note that PyTorch version 2.0.0 and above are not yet supported. You should ensure that your installed version of PyTorch is less than 2.0.0. You can check the PyTorch version with the following Python command:
If your PyTorch version is not compatible, please downgrade PyTorch to a compatible version using pip or conda, depending on how you initially installed PyTorch. |
I tried downgrade to torch.version.cuda=11.7 and touch__version__=1.13.1+cu117, still the same error. |
torch.version.cuda=11.7 and torch__version__=1.13.1+cu117 only means the cuda version used to compile torch is 11.7.You need to make sure that the CUDA Toolkit version matches the version used to compile torch. |
cuda version:11.3 still the same error |
please ensure that you have tried
|
@MayDomine hi, my server environment, also had the errors.
|
这个环境我测试过不会出错,请检查cuda runtime的路径,pip安装是否使用cache、以及本地nccl版本是否有冲突等等 |
我在用 transformers 进行训练的时候:
@Fword4u 你好,我这边检查的环境是这样,实在看不出来哪里环境配置有冲突; |
我执行这个 |
I'm trying the demo code, here is the information: with CUDA 12.1
the command
!python -c "import torch;print(torch.cuda.nccl.version())"
, can return(2, 14, 3)
below is the original import error stack:
The text was updated successfully, but these errors were encountered: