Thank you for reporting! Building on a machine without a GPU is a typical way how CI and binaries are built, so it is strange that it is not working.
cc @seemethere

It might be an easy problem to miss in CI / test. I can only find 4 test binaries (test_api, test_dist_autograd, test_jit, and torch_shm_manager) and one library (libcaffe2_nvrtc.so) that end up with the stubs/ path hard-coded.

The test binaries tolerate the CUDA failure caused by the stub library, and run to successful completion with just an info message that the CUDA tests are skipped. test_api, for example:

CUDA not available. Disabling CUDA and MultiCUDA tests
Note: Google Test filter = *-*_CUDA:*_MultiCUDA
[==========] Running 744 tests from 31 test cases.
...
[==========] 744 tests from 31 test cases ran. (56442 ms total)
[  PASSED  ] 744 tests.

The libcaffe2_nvrtc.so library case is more interesting... That library is only ever dynamically loaded (i.e. dlopen(), not "normal" dynamic linking) by libtorch.so. But libtorch depends on libcudart which dynamically loads libcuda itself. So I think anthing that would use libcaffe2_nvrtc will, in practice, end up with libcuda already being loaded before libcaffe2_nvrtc's RPATH has any force. So stuff like:

import torch

x = torch.randn(1, device='cuda:0')                                                                                                                                         
print(torch._C._cuda_hasPrimaryContext(0))

Will work correctly despite the stubs/ RPATH in libcaffe2_nvrtc.

So I think the most visible symptom of this will be for binaries built against libtorch, where the bad RPATH entry is incorporated into the new binary itself and so will be in force before libtorch is loaded.

In general, though, the libcuda in stubs/ isn't functional for runtime, so I think the risk of ever having stubs/ in RPATH outweighs the benefits.

Yeah, this seems like a major problem with the way we're doing builds right now.

When I build on a system with no cuda drivers but with the cuda toolkit installed, I see a difference between libcaffe2_nvrtc.so and the various binaries. The so has (my cmake uses RUNPATH, and the install cmake command adds $ORIGIN)

./torch/lib/libcaffe2_nvrtc.so:   RUNPATH  /
        $ORIGIN:/usr/local/cuda-10.2/lib64/stubs:/usr/local/cuda-10.2/lib64

where the binaries change the order (only showing one, all the files in ./torch/bin are the same):

./torch/bin/test_api:   RUNPATH /
        $ORIGIN:/usr/local/cuda-10.2/lib64:/usr/local/cuda-10.2/lib64/stubs

I wonder why the order changes. If I understand correctly, these executable should find the real drivers before the fake stub one because of the search order.

I don't have an explanation for the ordering difference there, but one observation...

When installed from rpm (at least), the CUDA Toolkit libraries will live in /usr/local/cuda-xxx, but the CUDA driver interface library (libcuda.so) will only be found in in the stubs/ directory and in /usr/lib64/

So I think the 2nd case, with the main cuda-xxx/lib64 directory ahead of stubs/ doesn't actually help with libcuda.so

Reopening because #37737 was reverted

In #37737 I tried to remove RPATH linking outside of the pytorch directory. This broke conda, since it depends on RPATH linking to conda-env/lib where all the mkl, cuda, and other DLLs live. Perhaps there is a way to be more selective when telling CMake what directories to insert into RPATH: LINK_PATH is too broad.

Actually, this RPATH linking also affects me in #40829 since it means that the shm manager is not relocatable.

I think conda is supposed to be able to identify situations like this an automatically update the rpath in executables when it installs conda packages; c.f., https://docs.conda.io/projects/conda-build/en/latest/resources/define-metadata.html#detect-binary-files-with-prefix So maybe we just need to figure out how to make this play ball