RULE_LAUNCH_COMPILE and RULE_LAUNCH_LINK system for nvcc_wrapper #3136

jrmadsen · 2020-06-29T21:08:23Z

This enables Kokkos_ENABLE_CUDA=ON with a CMAKE_CXX_COMPILER to be set to a non-clang C++ compiler instead of nvcc_wrapper
This provides a way for Kokkos + CUDA to be compiled without forcing CMAKE_CXX_COMPILER to be nvcc_wrapper or clang

Proposed

kokkos_compilation macro
- This will enable setting the launch rule globally and on projects, directories, targets, and source files
- By default, find_package(Kokkos) will invoke kokkos_compilation(GLOBAL)
find_package(Kokkos COMPONENTS separable_compilation)
- This will disable kokkos_compilation(GLOBAL) and the user will be responsible for doing:
  - kokkos_compilation(PROJECT)
  - kokkos_compilation(DIRECTORY <DIR>)
  - kokkos_compilation(TARGET <TARGETS...>)
  - kokkos_compilation(SOURCE <SOURCES...>)

jrmadsen · 2020-07-01T06:40:17Z

@crtrott @jjwilke This is good to go as long unless the CI fails. It works based my various compilation tests locally.

@dalg24 we need to add not setting CXX to nvcc_wrapper somewhere in the Jenkins testing I think.

jrmadsen · 2020-07-01T06:45:26Z

Oh and in all the CUDA tests, it will still be using the kokkos_launch_compiler command even if CXX=nvcc_wrapper.

jrmadsen · 2020-07-02T04:46:58Z

@dalg24 This should now provide this exact same behavior if CMAKE_CXX_COMPILER=nvcc_wrapper. Thus, all the current changes should have zero effect on any of the testing.

Before this is merged, Jenkins needs to have new instances of all tests with -DKokkos_ENABLE_CUDA=ON that do not set the compiler to nvcc_wrapper.

masterleinad · 2020-07-02T12:27:27Z

There still is

clang-8: error: unsupported option '--diag_suppress=esa_on_defaulted_function_ignored'

jrmadsen · 2020-07-02T14:27:59Z

@masterleinad I really need to finished that PR for CDash... if only @dalg24 wasn't so damn picky... I have no idea where that error is in the log. Did that happen in the build tree or in the install tree?

masterleinad · 2020-07-02T14:35:59Z

It's in the build step for CUDA-9.2-Clang (https://cloud.cees.ornl.gov/jenkins-ci/blue/organizations/jenkins/Kokkos/detail/Kokkos/2152/pipeline).

jrmadsen · 2020-07-02T14:38:14Z

Thanks, I think just needed logic for Clang

cmake/KokkosConfig.cmake.in

masterleinad · 2020-07-06T19:11:33Z

The architecture auto-detection fails (maybe unsurprisingly) with

/usr/bin/c++  -DSM_ONLY  -Werror    -std=c++11 -o CMakeFiles/cmTC_35a1f.dir/cuda_compute_capability.cc.o -c /tmp/ArborX/my_docker/kokkos/cmake/compile_tests/cuda_compute_capability.cc
/tmp/ArborX/my_docker/kokkos/cmake/compile_tests/cuda_compute_capability.cc: In function 'int main()':
/tmp/ArborX/my_docker/kokkos/cmake/compile_tests/cuda_compute_capability.cc:49:3: error: 'cudaDeviceProp' was not declared in this scope
   cudaDeviceProp device_properties;
   ^~~~~~~~~~~~~~
/tmp/ArborX/my_docker/kokkos/cmake/compile_tests/cuda_compute_capability.cc:50:9: error: 'cudaError_t' does not name a type; did you mean 'error_t'?
   const cudaError_t error = cudaGetDeviceProperties(&device_properties,
         ^~~~~~~~~~~
         error_t
/tmp/ArborX/my_docker/kokkos/cmake/compile_tests/cuda_compute_capability.cc:52:7: error: 'error' was not declared in this scope
   if (error != cudaSuccess) {
       ^~~~~
/tmp/ArborX/my_docker/kokkos/cmake/compile_tests/cuda_compute_capability.cc:52:7: note: suggested alternative: 'perror'
   if (error != cudaSuccess) {
       ^~~~~
       perror
/tmp/ArborX/my_docker/kokkos/cmake/compile_tests/cuda_compute_capability.cc:52:16: error: 'cudaSuccess' was not declared in this scope
   if (error != cudaSuccess) {
                ^~~~~~~~~~~
/tmp/ArborX/my_docker/kokkos/cmake/compile_tests/cuda_compute_capability.cc:53:36: error: 'cudaGetErrorString' was not declared in this scope
     std::cout << "CUDA error: " << cudaGetErrorString(error) << '\n';
                                    ^~~~~~~~~~~~~~~~~~
/tmp/ArborX/my_docker/kokkos/cmake/compile_tests/cuda_compute_capability.cc:57:7: error: 'device_properties' was not declared in this scope
       device_properties.major * 10 + device_properties.minor;
       ^~~~~~~~~~~~~~~~~

jrmadsen · 2020-07-06T21:36:16Z

Oh, well this isn't too complicated of a fix.

IF(KOKKOS_ENABLE_CUDA AND NOT CUDA_ARCH_ALREADY_SPECIFIED)
  TRY_RUN(
    _RESULT
    _COMPILE_RESULT
    ${_BINARY_TEST_DIR}
    ${CMAKE_CURRENT_SOURCE_DIR}/cmake/compile_tests/cuda_compute_capability.cc
    COMPILE_DEFINITIONS -DSM_ONLY
    RUN_OUTPUT_VARIABLE _CUDA_COMPUTE_CAPABILITY)
ENDIF()

I don't see a way to tell cmake to use the launcher so just need to either:

add an if block around using try_run and then do it manually via execute_process
enable_language(CUDA) + CMAKE_TRY_COMPILE_TARGET_TYPE CUDA

@masterleinad @jjwilke Thoughts? Preferences?

masterleinad · 2020-07-07T18:58:37Z

Currently, the auto-detection doesn't work with clang as a compiler either. It would be nice to come up with a solution that works for compilers different from nvcc_wrapper but that could be done separately and I don't think it's crucial for this pull request.

AFAICT, CMAKE_TRY_COMPILE_TARGET_TYPE can only differentiate between executables and libraries (https://cmake.org/cmake/help/latest/variable/CMAKE_TRY_COMPILE_TARGET_TYPE.html#variable:CMAKE_TRY_COMPILE_TARGET_TYPE) but can't be used to specify the language resp. the compiler to use. I guess the only way really is execute_process.

jrmadsen · 2020-07-14T19:19:44Z

@masterleinad Been too busy lately to figure out the auto-detection but I agree it would be nice to have in the future.

.jenkins

masterleinad · 2020-07-15T15:14:44Z

Retest this please.

jrmadsen · 2020-07-27T17:50:02Z

@masterleinad Hey just got back from vacation this week, is this ready for merging?

masterleinad · 2020-07-28T17:27:46Z

I think we should wait for this until after the release to have sufficient time to test it out.

jjwilke

Iniital comments

bin/kokkos_launch_compiler

cmake/KokkosConfig.cmake.in

cmake/KokkosConfigCommon.cmake.in

cmake/kokkos_functions.cmake

jrmadsen · 2020-07-28T23:20:06Z

@masterleinad I'm not sure I see the merit in delaying it. It doesn't change anything at all if CXX is set to nvcc_wrapper, it can be explicitly disabled, and merging it will help find edge cases much easier.

jjwilke

Just add comment in launch script. This all seems fine to me. Merging this sooner than later might be good. It solves nasty problems with FetchContent and ExternalProject that we've been having.

bin/kokkos_launch_compiler

cmake/KokkosConfigCommon.cmake.in

bin/kokkos_launch_compiler

jjwilke

I'm happy with my tests. A bit of cleanup on error messages/status messages and I'm basically ready to approve.

Curious about thoughts on changing default for the launcher script to be globally applied.

bin/kokkos_launch_compiler

cmake/KokkosConfig.cmake.in

masterleinad · 2020-07-29T22:35:35Z

Retest this please.

jjwilke · 2020-08-02T22:46:03Z

Still thinking about this. My concern now is if 2 projects import Kokkos. One of them asks for separable_compilation. The other does not. You now won't get separable compilation because the global property has been set.

A counter-proposal would be this:

Keep everything you have, except that you can now ONLY point to nvcc_wrapper from your Kokkos installation. These are not allowed to be inconsistent.
Have Kokkos add a transitive flag like --kokkos-dependence to everything. The compiler launcher can actually look for this. If this flag does not appear, the compiler launcher can actually just pass the args through. If the flag does appear, use nvcc_wrapper.

This PR would be a better place to add the --kokkos-dependence flag.

Now we have the separable compilation problem fixed. No one needs to ask for it. If you depend on Kokkos, you go to the wrapper. If not, you're good to go. And you don't need to worry about two different Kokkos projects thrashing each other's global properties (of course, I could be misunderstanding global scope rules in find_package).

jrmadsen · 2020-08-02T23:27:51Z

@jjwilke I like it. What about using a compiler definition instead of a compiler flag? E.g. -DKOKKOS_DEPENDENCE. The benefit is that it wouldn't have to be removed from the args in the launcher and it could actually help downstream projects (in the 2 project scenario you mentioned) detect whether Kokkos is being used by parent project.

…ee integration

…tional behavior - Added some info in cmake about what kokkos_launch_compiler does and provides - Added ability to set Kokkos_LAUNCH_COMPILER=OFF in downstream projects - Added kokkos_compiler_is_nvcc to KokkosConfigCommon.cmake.in to avoid using kokkos_launch_compiler if CMAKE_CXX_COMPILER=nvcc_wrapper

- Fixed AND NOT AND NOT

- Created new KCL (Kokkos-Compiler-Launcher) tests

- specify CUDA arch

- Fixed environment to set CXX instead of NVCC_WRAPPER_DEFAULT_COMPILER

crtrott · 2020-09-22T15:48:17Z

The CUDA Arch detection doesn't work without a CUDA compiler. And it looks like downstream you have to set CMAKE_POLICY CMP0057 explicitly down stream? We just worked with some folks in Germany which compile some other raw CUDA as well and this made it possible relatively straight forward. Just those two things didn't work (i.e. we had to set the architecture explicitly - which I think is fine for now and can be subsequently fixed) and we had to explicitly set CMAKE_POLICY CMP0057 in the application cmake file.

crtrott · 2020-09-23T03:38:25Z

Retest this please.

I think this was resolved since we did change the default for the launcher script to not be globally applied

bin/kokkos_launch_compiler

Co-Authored-By: Christian Trott <crtrott@sandia.gov>

…h_compiler

dalg24

I think this looks good

I moved the testing into the CUDA 11 build

kokkos/.jenkins

Lines 184 to 245 in 6bf9c87

    
           stage('CUDA-11.0-NVCC-C++17-RDC') { 
        
               agent { 
        
                   dockerfile { 
        
                       filename 'Dockerfile.nvcc' 
        
                       dir 'scripts/docker' 
        
                       additionalBuildArgs '--pull --build-arg BASE=nvidia/cuda:11.0-devel --build-arg ADDITIONAL_PACKAGES="g++-8 gfortran" --build-arg CMAKE_VERSION=3.17.3' 
        
                       label 'nvidia-docker && volta' 
        
                       args '-v /tmp/ccache.kokkos:/tmp/ccache' 
        
                   } 
        
               } 
        
               environment { 
        
                   OMP_NUM_THREADS = 8 
        
                   OMP_PLACES = 'threads' 
        
                   OMP_PROC_BIND = 'spread' 
        
                   NVCC_WRAPPER_DEFAULT_COMPILER = 'g++-8' 
        
               } 
        
               steps { 
        
                   sh 'ccache --zero-stats' 
        
                   sh '''rm -rf install && mkdir -p install && \ 
        
                         rm -rf build && mkdir -p build && cd build && \ 
        
                         cmake \ 
        
                           -DCMAKE_BUILD_TYPE=Release \ 
        
                           -DCMAKE_CXX_COMPILER=g++-8 \ 
        
                           -DCMAKE_CXX_FLAGS=-Werror \ 
        
                           -DCMAKE_CXX_STANDARD=17 \ 
        
                           -DKokkos_ENABLE_COMPILER_WARNINGS=ON \ 
        
                           -DKokkos_ENABLE_OPENMP=ON \ 
        
                           -DKokkos_ENABLE_CUDA=ON \ 
        
                           -DKokkos_ENABLE_CUDA_LAMBDA=OFF \ 
        
                           -DKokkos_ENABLE_CUDA_UVM=ON \ 
        
                           -DKokkos_ENABLE_CUDA_RELOCATABLE_DEVICE_CODE=ON \ 
        
                           -DKokkos_ARCH_VOLTA70=ON \ 
        
                           -DCMAKE_INSTALL_PREFIX=${PWD}/../install \ 
        
                         .. && \ 
        
                         make -j8 install && \ 
        
                         cd .. && \ 
        
                         rm -rf build-tests && mkdir -p build-tests && cd build-tests && \ 
        
                         export CMAKE_PREFIX_PATH=${PWD}/../install && \ 
        
                         cmake \ 
        
                           -DCMAKE_BUILD_TYPE=Release \ 
        
                           -DCMAKE_CXX_COMPILER_LAUNCHER=ccache \ 
        
                           -DCMAKE_CXX_COMPILER=$WORKSPACE/bin/nvcc_wrapper \ 
        
                           -DCMAKE_CXX_FLAGS=-Werror \ 
        
                           -DCMAKE_CXX_STANDARD=17 \ 
        
                           -DKokkos_INSTALL_TESTING=ON \ 
        
                         .. && \ 
        
                         make -j8 && ctest --output-on-failure && \ 
        
                         cd ../example/build_cmake_installed && \ 
        
                         rm -rf build && mkdir -p build && cd build && \ 
        
                         cmake \ 
        
                           -DCMAKE_CXX_COMPILER=g++-8 \ 
        
                           -DCMAKE_CXX_FLAGS=-Werror \ 
        
                           -DCMAKE_CXX_STANDARD=17 \ 
        
                         .. && \ 
        
                         make -j8 && ctest --output-on-failure''' 
        
               } 
        
               post { 
        
                   always { 
        
                       sh 'ccache --show-stats' 
        
                   } 
        
               } 
        
           }

I purposefully use nvcc_wrapper directly in the "install testing" step.

jrmadsen added the [WIP] label Jun 29, 2020

jrmadsen requested review from jjwilke and crtrott June 29, 2020 21:08

jrmadsen force-pushed the nvcc-wrapper-rule-launch branch from 617b330 to 96dc78c Compare July 1, 2020 06:34

jrmadsen removed the [WIP] label Jul 1, 2020

jrmadsen force-pushed the nvcc-wrapper-rule-launch branch from c677aba to d732854 Compare July 2, 2020 04:42

masterleinad reviewed Jul 2, 2020

View reviewed changes

cmake/KokkosConfig.cmake.in Outdated Show resolved Hide resolved

masterleinad reviewed Jul 14, 2020

View reviewed changes

.jenkins Outdated Show resolved Hide resolved

jjwilke reviewed Jul 28, 2020

View reviewed changes

jjwilke suggested changes Jul 29, 2020

View reviewed changes

bin/kokkos_launch_compiler Show resolved Hide resolved

cmake/KokkosConfigCommon.cmake.in Show resolved Hide resolved

cmake/KokkosConfigCommon.cmake.in Show resolved Hide resolved

jjwilke suggested changes Jul 29, 2020

View reviewed changes

bin/kokkos_launch_compiler Show resolved Hide resolved

jjwilke previously requested changes Jul 29, 2020

View reviewed changes

bin/kokkos_launch_compiler Outdated Show resolved Hide resolved

bin/kokkos_launch_compiler Outdated Show resolved Hide resolved

cmake/KokkosConfig.cmake.in Show resolved Hide resolved

cmake/KokkosConfig.cmake.in Show resolved Hide resolved

jrmadsen added 14 commits September 21, 2020 22:14

kokkos_launch_compiler handles being set to nvcc_wrapper + install tr…

0ff5a05

…ee integration

Update KokkosConfigCommon.cmake.in

179375f

Added logic for Clang

e2a165a

Update KokkosConfig.cmake.in

d5688a3

- Fixed AND NOT AND NOT

Update .jenkins

160f7eb

- Created new KCL (Kokkos-Compiler-Launcher) tests

Update .jenkins

a76df43

- specify CUDA arch

Update .jenkins

517215d

- Fixed environment to set CXX instead of NVCC_WRAPPER_DEFAULT_COMPILER

Fixed double-negative typo in comment

6731a91

Additional comments in the kokkos_launch_compiler

9d655e0

Status message and piping to /dev/null when missing nvcc_wrapper

f17a887

Re-direct which nvcc_wrapper to /dev/null

842698d

Support KOKKOS_DEPENDENCE compiler definition

88a58e4

Update kokkos_functions.cmake

fe84432

jrmadsen force-pushed the nvcc-wrapper-rule-launch branch from 5d036ee to fe84432 Compare September 22, 2020 05:17

jrmadsen requested a review from jjwilke September 22, 2020 17:58

Set CMP0057 to NEW in KokkosConfig.cmake

91e0c3f

crtrott mentioned this pull request Sep 23, 2020

Experimental: Change default behavior to host_only for nvcc_wrapper #2654

Closed

crtrott reviewed Sep 24, 2020

View reviewed changes

bin/kokkos_launch_compiler Outdated Show resolved Hide resolved

dalg24 and others added 5 commits September 23, 2020 22:51

Fixing missing -e option for echo in kokkos_launch_compiler

9740866

Co-Authored-By: Christian Trott <crtrott@sandia.gov>

Drop the two "KCL" extra CI builds on Jenkins

1bdcc3c

Use kokkos_launch_compiler in CUDA 11 build on Jenkins

c618288

Explicitly set NVIDIA GPU architcture in build that uses kokkos_launc…

741ac33

…h_compiler

Weak attempt to resolve linktime error cannot find gfortran in CI

6bf9c87

dalg24 approved these changes Sep 24, 2020

View reviewed changes

crtrott merged commit 7a4dd39 into kokkos:develop Sep 24, 2020

jrmadsen mentioned this pull request Dec 16, 2020

Add NVCC_WRAPPER_DEFAULT_ARCH Variable #3667

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RULE_LAUNCH_COMPILE and RULE_LAUNCH_LINK system for nvcc_wrapper #3136

RULE_LAUNCH_COMPILE and RULE_LAUNCH_LINK system for nvcc_wrapper #3136

jrmadsen commented Jun 29, 2020

jrmadsen commented Jul 1, 2020

jrmadsen commented Jul 1, 2020

jrmadsen commented Jul 2, 2020

masterleinad commented Jul 2, 2020

jrmadsen commented Jul 2, 2020

masterleinad commented Jul 2, 2020

jrmadsen commented Jul 2, 2020

masterleinad commented Jul 6, 2020

jrmadsen commented Jul 6, 2020

masterleinad commented Jul 7, 2020

jrmadsen commented Jul 14, 2020

masterleinad commented Jul 15, 2020

jrmadsen commented Jul 27, 2020

masterleinad commented Jul 28, 2020

jjwilke left a comment

jrmadsen commented Jul 28, 2020

jjwilke left a comment •

edited

jjwilke left a comment

masterleinad commented Jul 29, 2020

jjwilke commented Aug 2, 2020

jrmadsen commented Aug 2, 2020

crtrott commented Sep 22, 2020 •

edited

crtrott commented Sep 23, 2020

dalg24 left a comment

	stage('CUDA-11.0-NVCC-C++17-RDC') {
	agent {
	dockerfile {
	filename 'Dockerfile.nvcc'
	dir 'scripts/docker'
	additionalBuildArgs '--pull --build-arg BASE=nvidia/cuda:11.0-devel --build-arg ADDITIONAL_PACKAGES="g++-8 gfortran" --build-arg CMAKE_VERSION=3.17.3'
	label 'nvidia-docker && volta'
	args '-v /tmp/ccache.kokkos:/tmp/ccache'
	}
	}
	environment {
	OMP_NUM_THREADS = 8
	OMP_PLACES = 'threads'
	OMP_PROC_BIND = 'spread'
	NVCC_WRAPPER_DEFAULT_COMPILER = 'g++-8'
	}
	steps {
	sh 'ccache --zero-stats'
	sh '''rm -rf install && mkdir -p install && \
	rm -rf build && mkdir -p build && cd build && \
	cmake \
	-DCMAKE_BUILD_TYPE=Release \
	-DCMAKE_CXX_COMPILER=g++-8 \
	-DCMAKE_CXX_FLAGS=-Werror \
	-DCMAKE_CXX_STANDARD=17 \
	-DKokkos_ENABLE_COMPILER_WARNINGS=ON \
	-DKokkos_ENABLE_OPENMP=ON \
	-DKokkos_ENABLE_CUDA=ON \
	-DKokkos_ENABLE_CUDA_LAMBDA=OFF \
	-DKokkos_ENABLE_CUDA_UVM=ON \
	-DKokkos_ENABLE_CUDA_RELOCATABLE_DEVICE_CODE=ON \
	-DKokkos_ARCH_VOLTA70=ON \
	-DCMAKE_INSTALL_PREFIX=${PWD}/../install \
	.. && \
	make -j8 install && \
	cd .. && \
	rm -rf build-tests && mkdir -p build-tests && cd build-tests && \
	export CMAKE_PREFIX_PATH=${PWD}/../install && \
	cmake \
	-DCMAKE_BUILD_TYPE=Release \
	-DCMAKE_CXX_COMPILER_LAUNCHER=ccache \
	-DCMAKE_CXX_COMPILER=$WORKSPACE/bin/nvcc_wrapper \
	-DCMAKE_CXX_FLAGS=-Werror \
	-DCMAKE_CXX_STANDARD=17 \
	-DKokkos_INSTALL_TESTING=ON \
	.. && \
	make -j8 && ctest --output-on-failure && \
	cd ../example/build_cmake_installed && \
	rm -rf build && mkdir -p build && cd build && \
	cmake \
	-DCMAKE_CXX_COMPILER=g++-8 \
	-DCMAKE_CXX_FLAGS=-Werror \
	-DCMAKE_CXX_STANDARD=17 \
	.. && \
	make -j8 && ctest --output-on-failure'''
	}
	post {
	always {
	sh 'ccache --show-stats'
	}
	}
	}

RULE_LAUNCH_COMPILE and RULE_LAUNCH_LINK system for nvcc_wrapper #3136

RULE_LAUNCH_COMPILE and RULE_LAUNCH_LINK system for nvcc_wrapper #3136

Conversation

jrmadsen commented Jun 29, 2020

Proposed

jrmadsen commented Jul 1, 2020

jrmadsen commented Jul 1, 2020

jrmadsen commented Jul 2, 2020

masterleinad commented Jul 2, 2020

jrmadsen commented Jul 2, 2020

masterleinad commented Jul 2, 2020

jrmadsen commented Jul 2, 2020

masterleinad commented Jul 6, 2020

jrmadsen commented Jul 6, 2020

masterleinad commented Jul 7, 2020

jrmadsen commented Jul 14, 2020

masterleinad commented Jul 15, 2020

jrmadsen commented Jul 27, 2020

masterleinad commented Jul 28, 2020

jjwilke left a comment

Choose a reason for hiding this comment

jrmadsen commented Jul 28, 2020

jjwilke left a comment • edited

Choose a reason for hiding this comment

jjwilke left a comment

Choose a reason for hiding this comment

masterleinad commented Jul 29, 2020

jjwilke commented Aug 2, 2020

jrmadsen commented Aug 2, 2020

crtrott commented Sep 22, 2020 • edited

crtrott commented Sep 23, 2020

dalg24 left a comment

Choose a reason for hiding this comment

jjwilke left a comment •

edited

crtrott commented Sep 22, 2020 •

edited