cmsenv breaks local CUDA toolchain

[felicepantaleo]

reproduce:
$ nvcc aCUDAprogram.cu
it works
$ cd CMSSW_10_0_1/src/
CMSSW_10_0_1/src$ cmsenv
$ cd -
$ nvcc aCUDAprogram.cu
sh: cicc: command not found

[fwyzard]

Because SCRAM.

[fwyzard]

nvcc relies on a file called nvcc.profile to be found in the same directory as the nvcc binary, to find its components and libraries (cicc, libcuda, libdevice, etc.)

The nvcc.profile file provided with nvcc assumes that such components can be found using relative paths from its location (e.g., cicc is supposed to be at ../nvvm/bin/cicc).

When we link the externals in the CMSSW release area, nvcc.profile ends up at $CMSSW_BASE/external/$SCRAM_ARCH/bin/nvcc.profile; now the relative paths do not work any more: e.g. nvcc looks for cicc at $CMSSW_BASE/external/$SCRAM_ARCH/bin/../nvvm/bin/cicc, which does not exist.

When compiling a .cu file with "scram build" this is not an issue, because scram calls nvcc with its full path (not the symlinked one), so the nvcc.profile file works as intended.

However, running nvcc directly from the command line fails because it finds the symlinked nvcc.profile, for which the relative paths are not valid.

My preferred solution would be to write the CUDA installation path in the nvcc.profile file; however this path needs to be adjusted to the CMSSW installation directory.

I've asked on the "Software Development Tools" hypernews, hopefully some expert can help me figure out how to do that.

[fwyzard]

For the moment, a workaround is to run

cmsenv
eval $(scram tool info cuda | grep ^CUDA_BASE)
sed -e's/^<tool name="cuda" version="9.1.85.*">/<tool name="cuda" version="9.1.85-workaround">/' -i $CMSSW_BASE/config/toolbox/$SCRAM_ARCH/tools/selected/cuda.xml
scram setup $CMSSW_BASE/config/toolbox/$SCRAM_ARCH/tools/selected/cuda.xml
rm -f $CMSSW_BASE/external/$SCRAM_ARCH/bin/nvcc.profile
cat > $CMSSW_BASE/external/$SCRAM_ARCH/bin/nvcc.profile << @EOF

TOP              = $CUDA_BASE

NVVMIR_LIBRARY_DIR = \$(TOP)/nvvm/libdevice

LD_LIBRARY_PATH += \$(TOP)/lib:
PATH            += \$(TOP)/nvvm/bin:\$(TOP)/bin:

INCLUDES        +=  "-I\$(TOP)/include" \$(_SPACE_)

LIBRARIES        =+ \$(_SPACE_) "-L\$(TOP)/lib\$(_TARGET_SIZE_)/stubs" "-L\$(TOP)/lib\$(_TARGET_SIZE_)"

CUDAFE_FLAGS    +=
PTXAS_FLAGS     +=
@EOF

[fwyzard]

Should be fixed by cms-sw/cmsdist#3740 (10.1.x) and cms-sw/cmsdist#3741 (10.0.x).

[fwyzard]

This is now included in CMSSW 10.1.X and 10.1.0-pre2.

[cmsbot]

A new Issue was created by @felicepantaleo Felice Pantaleo.

can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here