fsycl-targets ordering interfere with cuda-gpu-arch flag

[dosarudaniel]

Description
When I am trying to compile this sample code (make test12) , with this CMakelist file
which contains these flags:

"-fsycl-targets=nvptx64-nvidia-cuda-sycldevice,spir64_x86_64-unknown-unknown-sycldevice"
"-Xsycl-target-backend=nvptx64-nvidia-cuda-sycldevice"
"--cuda-gpu-arch=sm_61"

I get this error : const: Assertion !GpuArch.empty() && "Must have an explicit GPU arch."' failed.

If I swap the fsycl-targets (to fsycl-targets=spir64_x86_64-unknown-unknown-sycldevice,nvptx64-nvidia-cuda-sycldevice) I do not have this.

I need to use the "--cuda-gpu-arch=sm_61" flag because some of my libraries were compiled for this arch.

Details:

dadosaru@pcphsft106:~/oneAdePT/debug$ make test12
/usr/local/bin/cmake -S/home/dadosaru/oneAdePT -B/home/dadosaru/oneAdePT/debug --check-build-system CMakeFiles/Makefile.cmake 0
-- Using VecCore version 0.7.0
-- Using VecGeom version 1.1.12
-- Using Geant4 version 10.7.1 from /home/dadosaru/local/include/Geant4
-- Using VecCore version 0.7.0
-- Using VecGeom version 1.1.12
-- Using Geant4 version 10.7.1 from /home/dadosaru/local/include/Geant4
-- Configuring done
-- Generating done
-- Build files have been written to: /home/dadosaru/oneAdePT/debug
make  -f CMakeFiles/Makefile2 test12
make[1]: Entering directory '/home/dadosaru/oneAdePT/debug'
/usr/local/bin/cmake -S/home/dadosaru/oneAdePT -B/home/dadosaru/oneAdePT/debug --check-build-system CMakeFiles/Makefile.cmake 0
/usr/local/bin/cmake -E cmake_progress_start /home/dadosaru/oneAdePT/debug/CMakeFiles 2
make  -f CMakeFiles/Makefile2 tests/CMakeFiles/test12.dir/all
make[2]: Entering directory '/home/dadosaru/oneAdePT/debug'
make  -f tests/CMakeFiles/test12.dir/build.make tests/CMakeFiles/test12.dir/depend
make[3]: Entering directory '/home/dadosaru/oneAdePT/debug'
cd /home/dadosaru/oneAdePT/debug && /usr/local/bin/cmake -E cmake_depends "Unix Makefiles" /home/dadosaru/oneAdePT /home/dadosaru/oneAdePT/tests /home/dadosaru/oneAdePT/debug /home/dadosaru/oneAdePT/debug/tests /home/dadosaru/oneAdePT/debug/tests/CMakeFiles/test12.dir/DependInfo.cmake --color=
make[3]: Leaving directory '/home/dadosaru/oneAdePT/debug'
make  -f tests/CMakeFiles/test12.dir/build.make tests/CMakeFiles/test12.dir/build
make[3]: Entering directory '/home/dadosaru/oneAdePT/debug'
[ 50%] Building CXX object tests/CMakeFiles/test12.dir/test12.cpp.o
cd /home/dadosaru/oneAdePT/debug/tests && /home/dadosaru/sycl_workspace_may_2/llvm/build//bin/clang++ -DVECCORE_ENABLE_CUDA -I/home/dadosaru/oneAdePT/base/inc -isystem /home/dadosaru/sycl_workspace_may_2/llvm/build/include/sycl -isystem /home/dadosaru/local/include -isystem /usr/local/cuda-11.1/include -O3 -DNDEBUG -fsycl -fsycl-unnamed-lambda -fsycl-targets=nvptx64-nvidia-cuda-sycldevice,spir64_x86_64-unknown-unknown-sycldevice -Xsycl-target-backend=nvptx64-nvidia-cuda-sycldevice --cuda-gpu-arch=sm_61 -std=c++20 -stdlib=libstdc++ -Wno-unknown-cuda-version -D_ONEADEPT_ -DNDEBUG -DROOT_NO_INT128 -DG4HepEm_CUDA_BUILD -I/home/dadosaru/oneAdePT/../local/include -std=c++20 -o CMakeFiles/test12.dir/test12.cpp.o -c /home/dadosaru/oneAdePT/tests/test12.cpp
clang++: /home/dadosaru/sycl_workspace_may_2/llvm/clang/lib/Driver/ToolChains/Cuda.cpp:683: virtual void clang::driver::toolchains::CudaToolChain::addClangTargetOptions(const llvm::opt::ArgList&, llvm::opt::ArgStringList&, clang::driver::Action::OffloadKind) const: Assertion `!GpuArch.empty() && "Must have an explicit GPU arch."' failed.
PLEASE submit a bug report to https://bugs.llvm.org/ and include the crash backtrace, preprocessed source, and associated run script.
Stack dump:
0.      Program arguments: /home/dadosaru/sycl_workspace_may_2/llvm/build//bin/clang++ -DVECCORE_ENABLE_CUDA -I/home/dadosaru/oneAdePT/base/inc -isystem /home/dadosaru/sycl_workspace_may_2/llvm/build/include/sycl -isystem /home/dadosaru/local/include -isystem /usr/local/cuda-11.1/include -O3 -DNDEBUG -fsycl -fsycl-unnamed-lambda -fsycl-targets=nvptx64-nvidia-cuda-sycldevice,spir64_x86_64-unknown-unknown-sycldevice -Xsycl-target-backend=nvptx64-nvidia-cuda-sycldevice --cuda-gpu-arch=sm_61 -std=c++20 -stdlib=libstdc++ -Wno-unknown-cuda-version -D_ONEADEPT_ -DNDEBUG -DROOT_NO_INT128 -DG4HepEm_CUDA_BUILD -I/home/dadosaru/oneAdePT/../local/include -std=c++20 -o CMakeFiles/test12.dir/test12.cpp.o -c /home/dadosaru/oneAdePT/tests/test12.cpp
1.      Compilation construction
2.      Building compilation jobs
3.      Building compilation jobs
4.      Building compilation jobs
5.      Building compilation jobs
Stack dump without symbol names (ensure you have llvm-symbolizer in your PATH or set the environment var `LLVM_SYMBOLIZER_PATH` to point to it):
/home/dadosaru/sycl_workspace_may_2/llvm/build//bin/clang++(_ZN4llvm3sys15PrintStackTraceERNS_11raw_ostreamEi+0x2c)[0x555769dcd31c]
/home/dadosaru/sycl_workspace_may_2/llvm/build//bin/clang++(_ZN4llvm3sys17RunSignalHandlersEv+0x34)[0x555769dcb124]
/home/dadosaru/sycl_workspace_may_2/llvm/build//bin/clang++(+0x21bc293)[0x555769dcb293]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x128a0)[0x7f0420d508a0]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x7f041f9b6f47]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x7f041f9b88b1]
/lib/x86_64-linux-gnu/libc.so.6(+0x3042a)[0x7f041f9a842a]
/lib/x86_64-linux-gnu/libc.so.6(+0x304a2)[0x7f041f9a84a2]
/home/dadosaru/sycl_workspace_may_2/llvm/build//bin/clang++(+0x29d1516)[0x55576a5e0516]
/home/dadosaru/sycl_workspace_may_2/llvm/build//bin/clang++(+0x29ae92b)[0x55576a5bd92b]
/home/dadosaru/sycl_workspace_may_2/llvm/build//bin/clang++(_ZNK5clang6driver6Driver25BuildJobsForActionNoCacheERNS0_11CompilationEPKNS0_6ActionEPKNS0_9ToolChainEN4llvm9StringRefEbbPKcRSt3mapISt4pairIS6_NSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEENS0_9InputInfoESt4lessISM_ESaISF_IKSM_SN_EEENS4_11OffloadKindE+0x1300)[0x55576a5524f0]
/home/dadosaru/sycl_workspace_may_2/llvm/build//bin/clang++(_ZNK5clang6driver6Driver18BuildJobsForActionERNS0_11CompilationEPKNS0_6ActionEPKNS0_9ToolChainEN4llvm9StringRefEbbPKcRSt3mapISt4pairIS6_NSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEENS0_9InputInfoESt4lessISM_ESaISF_IKSM_SN_EEENS4_11OffloadKindE+0x193)[0x55576a5542e3]
/home/dadosaru/sycl_workspace_may_2/llvm/build//bin/clang++(+0x2947706)[0x55576a556706]
/home/dadosaru/sycl_workspace_may_2/llvm/build//bin/clang++(_ZNK5clang6driver13OffloadAction24doOnEachDeviceDependenceERKN4llvm12function_refIFvPNS0_6ActionEPKNS0_9ToolChainEPKcEEE+0x5c)[0x55576a68e71c]
/home/dadosaru/sycl_workspace_may_2/llvm/build//bin/clang++(_ZNK5clang6driver6Driver25BuildJobsForActionNoCacheERNS0_11CompilationEPKNS0_6ActionEPKNS0_9ToolChainEN4llvm9StringRefEbbPKcRSt3mapISt4pairIS6_NSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEENS0_9InputInfoESt4lessISM_ESaISF_IKSM_SN_EEENS4_11OffloadKindE+0xb97)[0x55576a551d87]
/home/dadosaru/sycl_workspace_may_2/llvm/build//bin/clang++(_ZNK5clang6driver6Driver18BuildJobsForActionERNS0_11CompilationEPKNS0_6ActionEPKNS0_9ToolChainEN4llvm9StringRefEbbPKcRSt3mapISt4pairIS6_NSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEENS0_9InputInfoESt4lessISM_ESaISF_IKSM_SN_EEENS4_11OffloadKindE+0x193)[0x55576a5542e3]
/home/dadosaru/sycl_workspace_may_2/llvm/build//bin/clang++(_ZNK5clang6driver6Driver25BuildJobsForActionNoCacheERNS0_11CompilationEPKNS0_6ActionEPKNS0_9ToolChainEN4llvm9StringRefEbbPKcRSt3mapISt4pairIS6_NSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEENS0_9InputInfoESt4lessISM_ESaISF_IKSM_SN_EEENS4_11OffloadKindE+0xd97)[0x55576a551f87]
/home/dadosaru/sycl_workspace_may_2/llvm/build//bin/clang++(_ZNK5clang6driver6Driver18BuildJobsForActionERNS0_11CompilationEPKNS0_6ActionEPKNS0_9ToolChainEN4llvm9StringRefEbbPKcRSt3mapISt4pairIS6_NSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEENS0_9InputInfoESt4lessISM_ESaISF_IKSM_SN_EEENS4_11OffloadKindE+0x193)[0x55576a5542e3]
/home/dadosaru/sycl_workspace_may_2/llvm/build//bin/clang++(_ZNK5clang6driver6Driver9BuildJobsERNS0_11CompilationE+0x1ea)[0x55576a55451a]
/home/dadosaru/sycl_workspace_may_2/llvm/build//bin/clang++(_ZN5clang6driver6Driver16BuildCompilationEN4llvm8ArrayRefIPKcEE+0xf07)[0x55576a555d67]
/home/dadosaru/sycl_workspace_may_2/llvm/build//bin/clang++(main+0xf11)[0x5557688fb061]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7)[0x7f041f999b97]
/home/dadosaru/sycl_workspace_may_2/llvm/build//bin/clang++(_start+0x2a)[0x555768983eca]
Aborted (core dumped)
tests/CMakeFiles/test12.dir/build.make:84: recipe for target 'tests/CMakeFiles/test12.dir/test12.cpp.o' failed
make[3]: *** [tests/CMakeFiles/test12.dir/test12.cpp.o] Error 134
make[3]: Leaving directory '/home/dadosaru/oneAdePT/debug'
CMakeFiles/Makefile2:613: recipe for target 'tests/CMakeFiles/test12.dir/all' failed
make[2]: *** [tests/CMakeFiles/test12.dir/all] Error 2
make[2]: Leaving directory '/home/dadosaru/oneAdePT/debug'
CMakeFiles/Makefile2:620: recipe for target 'tests/CMakeFiles/test12.dir/rule' failed
make[1]: *** [tests/CMakeFiles/test12.dir/rule] Error 2
make[1]: Leaving directory '/home/dadosaru/oneAdePT/debug'
Makefile:316: recipe for target 'test12' failed
make: *** [test12] Error 2

It might be related to this bug as well.

Environment:

OS: Ubuntu 18.04.5 LTS
Target device and vendor: NVIDIA GPU: GeForce RTX 2080 SUPER
DPC++ version: clang version 13.0.0 (https://github.com/intel/llvm c2d6cfa)

[alexbatashev]

Tagging @AGindinson @mdtoguchi and @steffenlarsen

[keryell]

@Ralender is this the same bug you are working on with Nvidia GPU + Xilinx FPGA in https://github.com/triSYCL/sycl ?

[Ralender]

I dodged this issue via:

     Action::OffloadKind DeviceOffloadingKind) const {
   HostTC.addClangTargetOptions(DriverArgs, CC1Args, DeviceOffloadingKind);
 
-  StringRef GpuArch = DriverArgs.getLastArgValue(options::OPT_march_EQ);
+  StringRef GpuArch = DriverArgs.getLastArgValue(options::OPT_march_EQ, "sm_50");
   assert(!GpuArch.empty() && "Must have an explicit GPU arch.");
   assert((DeviceOffloadingKind == Action::OFK_OpenMP ||
           DeviceOffloadingKind == Action::OFK_SYCL ||

but this is more a way to bypass the issue than to fix it as it will also affect the cuda workflow.

[AidanBeltonS]

@dosarudaniel I believe this is a duplicate of #3631 which has now been resolved. Can you test if the error still occurs with the most recent build? Many Thanks.

BTW: I accidentality closed this, then reopened it. Just a miss click, I apologize if there was any confusion

[AerialMantis]

We believe this issue is resolved now, addressed by #4673 so we are closing this ticket. If you are sill seeing any issues with this please leave a comment and we can re-open it.