Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PyTorch CUDA extensions break during compilation with CUDA10.2.89 #33203

Closed
ptrblck opened this issue Feb 11, 2020 · 3 comments
Closed

PyTorch CUDA extensions break during compilation with CUDA10.2.89 #33203

ptrblck opened this issue Feb 11, 2020 · 3 comments
Labels
high priority module: build Build system issues module: cpp-extensions Related to torch.utils.cpp_extension module: cuda Related to torch.cuda, and CUDA support in general triage review triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module

Comments

@ptrblck
Copy link
Collaborator

ptrblck commented Feb 11, 2020

🐛 Bug

Currently CUDA extensions in PyTorch break using nvcc 10.2.89 with the following error:

/workspace/src/pytorch/torch/include/c10/util/variant.h: In functionconstexpr decltype(auto) c10::visit(Visitor&&, Vs&& ...)’:
/workspace/src/pytorch/torch/include/c10/util/variant.h:2718:97: error: parameter packs not expanded with ‘...’:
     return (detail_::all({!vs.valueless_by_exception()...})

After debugging this issue, it appears to be related to an nvcc issue in CUDA version 10.2.89. As we are working on fixing this issue, it is recommended for those who compile CUDA extensions to use CUDA 10.1.168.

To Reproduce

Steps to reproduce the behavior:

# PyTorch compiled with CUDA10.2.89
git clone https://github.com/pytorch/extension-cpp
cd extension-cpp/cuda
python setup.py install

CC @csarofeen @ngimel

cc @ezyang @gchanan @zou3519 @yf225 @ngimel

@csverma610
Copy link

Hello,

I am confused. Should we install only Cuda10.1 until the issues mentioned are resolved. Somehow
installing pytorch from the source is non-trivial.

@ptrblck
Copy link
Collaborator Author

ptrblck commented Feb 12, 2020

@csverma610
If you are planning on building custom CUDA extensions, then we would recommend to stick to CUDA10.1 until the PR gets merged.
Otherwise you could build PyTorch from source using CUDA10.2.

If you are stuck in the process of building, I would recommend to create a topic in our discussion board and tag me there.

@zhangguanheng66 zhangguanheng66 added module: build Build system issues module: cuda Related to torch.cuda, and CUDA support in general high priority labels Feb 12, 2020
@zhangguanheng66 zhangguanheng66 added the triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module label Feb 12, 2020
@zhangguanheng66
Copy link
Contributor

Thanks for the post. Will keep monitoring.

@zou3519 zou3519 added the module: cpp-extensions Related to torch.utils.cpp_extension label Feb 12, 2020
ttumiel pushed a commit to ttumiel/pytorch that referenced this issue Mar 4, 2020
…33230)

Summary:
Fixes: pytorch#33203
PR based on mpark/variant#73

Verified locally on CUDA10.2.89 and 10.1.243

Thanks ngimel for the hint and gridley for the initial fix in the variant repo! :)
Pull Request resolved: pytorch#33230

Differential Revision: D19858083

Pulled By: ngimel

fbshipit-source-id: b9438084f5688712c6aa6b17813c68ccde237bbb
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
high priority module: build Build system issues module: cpp-extensions Related to torch.utils.cpp_extension module: cuda Related to torch.cuda, and CUDA support in general triage review triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants