building with cuda 11.2 fails

[cwsmith]

The following compilation error is repeatable on SCOREC rhel7 and Aimos rhel8 using omegah d56a86e. The build instructions are listed below.

Thrust version is 1.10.0 ( packaged with CUDA Toolkit 11.2) according to https://github.com/NVIDIA/thrust.

Dan reported it already! NVIDIA/thrust#1332

error

[  0%] Building CUDA object src/CMakeFiles/omega_h.dir/Omega_h_int_scan.cpp.o                                                                                                            
cd /space/cwsmith/omegahCuda11/buildCuda11/src && /usr/local/cuda-11.2/bin/nvcc -forward-unknown-to-host-compiler -Domega_h_EXPORTS -I/space/cwsmith/omegahCuda11/omega_h/src -I/space/cw
/usr/local/cuda-11.2/bin/../targets/x86_64-linux/include/thrust/system/cuda/detail/scan.h(578): error: array of reference is not allowed                                                 
          detected during:                                                                                                                                                               
            instantiation of class "thrust::cuda_cub::__scan::DoNothing<T> [with T=const Omega_h::LO &]"                                                                                 
(784): here                                                                                                                                                                              
            instantiation of "OutputIt thrust::cuda_cub::inclusive_scan_n(thrust::cuda_cub::execution_policy<Derived> &, InputIt, Size, OutputIt, ScanOp) [with Derived=thrust::cuda_cub:
/usr/local/cuda-11.2/bin/../targets/x86_64-linux/include/thrust/system/cuda/detail/transform_scan.h(72): here                                                                            
            instantiation of "OutputIt thrust::cuda_cub::transform_inclusive_scan(thrust::cuda_cub::execution_policy<Derived> &, InputIt, InputIt, OutputIt, TransformOp, ScanOp) [with D
/usr/local/cuda-11.2/bin/../targets/x86_64-linux/include/thrust/detail/transform_scan.inl(47): here                                                                                      
            instantiation of "OutputIterator thrust::transform_inclusive_scan(const thrust::detail::execution_policy_base<DerivedPolicy> &, InputIterator, InputIterator, OutputIterator,
/space/cwsmith/omegahCuda11/omega_h/src/Omega_h_scan.hpp(83): here                                                                                                                       
            instantiation of "OutputIterator Omega_h::transform_inclusive_scan(InputIterator, InputIterator, OutputIterator, BinaryOp, UnaryOp) [with InputIterator=Omega_h::LO *, Output
/space/cwsmith/omegahCuda11/omega_h/src/Omega_h_int_scan.cpp(32): here

SCOREC Build

The following was tested on cranium.

env

module use /opt/scorec/spack/dev/lmod/linux-rhel7-x86_64/Core
module unuse /opt/scorec/spack/lmod/linux-rhel7-x86_64/Core
module load gcc/7.4.0-c5aaloy cuda/11.2
module load mpich/3.3.1-bfezl2l
module load cmake

cmake

cmake ../omega_h \
  -DCMAKE_INSTALL_PREFIX=$PWD/install \
  -DBUILD_TESTING=on  \
  -DOmega_h_USE_CUDA=on \
  -DOmega_h_CUDA_ARCH=75 \
  -DOmega_h_USE_MPI=on  \
  -DBUILD_SHARED_LIBS=ON

AiMOS Build

The following was tested on dcs217 (a rhel8 fen).

env

module load spectrum-mpi/10.4
module load cmake/3.20.0
module load cuda/11.2

cmake

cmake ../omega_h \
  -DCMAKE_INSTALL_PREFIX=$oh \
  -DBUILD_SHARED_LIBS=OFF \
  -DOmega_h_USE_Kokkos=ON \
  -DOmega_h_USE_CUDA=on \
  -DOmega_h_CUDA_ARCH=70 \
  -DOmega_h_USE_MPI=on  \
  -DBUILD_TESTING=on  \
  -DCMAKE_CXX_COMPILER=g++ \
  -DKokkos_PREFIX=$kk/lib64/cmake

[cwsmith]

This is fixed in 11.3. Don't use 11.2.

[cwsmith]

cmake checks for cuda 11.2:

omega_h/CMakeLists.txt

Lines 113 to 118 in 9a95ddb

	#check for versions of cuda/thrust that dont't work with omegah
	if(CMAKE_CUDA_COMPILER_VERSION VERSION_GREATER_EQUAL 11.2
	AND CMAKE_CUDA_COMPILER_VERSION VERSION_LESS 11.3)
	message(FATAL_ERROR "CUDA 11.2 does not support Omega_h, "
	"use an older or newer version")
	endif()