Complete earlier TMPDIR change

[sebrowne]

I forgot to actually set TMPDIR on the invocation of the compiler (the
other change just set it for the nvcc_wrapper temp_dir that it uses for
some preprocessing).

See #3479

[dalg24-jenkins]

Can one of the admins verify this patch?

[dalg24]

OK to test

[crtrott]

This terribly failed any idea why?

[sebrowne]

Hmmm I'm unable to replicate the failure, though it does appear to be my fault. I'll try harder :(

[sebrowne]

Is there any way to re-trigger the build?

[masterleinad]

Is there any way to re-trigger the build?

I would not expect simply retriggering would fix all the CI errors since the error messages look similar to:

CMake Error at /opt/cmake/share/cmake-3.15/Modules/CMakeTestCXXCompiler.cmake:53 (message):
  The C++ compiler

    "/var/jenkins/workspace/Kokkos/bin/nvcc_wrapper"

  is not able to compile a simple test program.

  It fails with the following output:

    Change Dir: /var/jenkins/workspace/Kokkos/build/CMakeFiles/CMakeTmp
    
    Run Build Command(s):/usr/bin/make cmTC_02bcc/fast && /usr/bin/make -f CMakeFiles/cmTC_02bcc.dir/build.make CMakeFiles/cmTC_02bcc.dir/build
    make[1]: Entering directory '/var/jenkins/workspace/Kokkos/build/CMakeFiles/CMakeTmp'
    Building CXX object CMakeFiles/cmTC_02bcc.dir/testCXXCompiler.cxx.o
    /var/jenkins/workspace/Kokkos/bin/nvcc_wrapper    -Werror    -o CMakeFiles/cmTC_02bcc.dir/testCXXCompiler.cxx.o -c /var/jenkins/workspace/Kokkos/build/CMakeFiles/CMakeTmp/testCXXCompiler.cxx
    g++: error: /tmpxft_000000ad_00000000-5_testCXXCompiler.cudafe1.cpp: No such file or directory
    g++: warning: '-x c++' after last input file has no effect
    g++: fatal error: no input files
    compilation terminated.
    CMakeFiles/cmTC_02bcc.dir/build.make:65: recipe for target 'CMakeFiles/cmTC_02bcc.dir/testCXXCompiler.cxx.o' failed
    make[1]: *** [CMakeFiles/cmTC_02bcc.dir/testCXXCompiler.cxx.o] Error 1
    make[1]: Leaving directory '/var/jenkins/workspace/Kokkos/build/CMakeFiles/CMakeTmp'
    Makefile:121: recipe for target 'cmTC_02bcc/fast' failed
    make: *** [cmTC_02bcc/fast] Error 2

Of course, you can just push another potential fix to retrigger.
Is this rebased on top of a somewhat recent version of develop?

[sebrowne]

I am aware that I likely caused the error (and that it's a real error), but I'm unable to replicate it in my local testing environment. Is there any way for me to get to the CMakeFiles for this CI build, or is the workspace cleaned/inaccessible?

I'll rebase it to pull in any other changes as well.

[masterleinad]

No, the build directories are not available but you can see one of the failing configurations in

kokkos/.jenkins

Lines 293 to 331 in 897b79d

	stage('CUDA-10.1-NVCC-DEBUG') {
	agent {
	dockerfile {
	filename 'Dockerfile.nvcc'
	dir 'scripts/docker'
	additionalBuildArgs '--build-arg BASE=nvidia/cuda:10.1-devel'
	label 'nvidia-docker && volta'
	args '-v /tmp/ccache.kokkos:/tmp/ccache --env NVIDIA_VISIBLE_DEVICES=$NVIDIA_VISIBLE_DEVICES'
	}
	}
	steps {
	sh 'ccache --zero-stats'
	sh '''rm -rf build && \
	cmake \
	-DCMAKE_BUILD_TYPE=Debug \
	-DCMAKE_CXX_COMPILER_LAUNCHER=ccache \
	-DCMAKE_CXX_COMPILER=$WORKSPACE/bin/nvcc_wrapper \
	-DCMAKE_CXX_FLAGS=-Werror \
	-DCMAKE_CXX_STANDARD=14 \
	-DKokkos_ENABLE_COMPILER_WARNINGS=ON \
	-DKokkos_ENABLE_DEBUG=ON \
	-DKokkos_ENABLE_DEBUG_BOUNDS_CHECK=ON \
	-DKokkos_ENABLE_TESTS=ON \
	-DKokkos_ENABLE_CUDA=ON \
	-DKokkos_ENABLE_CUDA_LAMBDA=ON \
	-DKokkos_ENABLE_LIBDL=OFF \
	-DBUILD_NAME=${STAGE_NAME} \
	-DTARGET=install \
	-P cmake/KokkosCI.cmake && \
	cd example/build_cmake_in_tree && \
	rm -rf build && mkdir -p build && cd build && \
	cmake -DCMAKE_CXX_STANDARD=14 .. && make -j8 && ctest --verbose'''
	}
	post {
	always {
	sh 'ccache --show-stats'
	}
	}
	}

where the corresponding Docker file is https://github.com/kokkos/kokkos/blob/develop/scripts/docker/Dockerfile.nvcc.

[sebrowne]

Awesome, that's helpful. I'll keep trying to get this fixed up.

[jrmadsen]

So I think the issue here lies in:

kokkos/bin/nvcc_wrapper

Line 94 in 1fb0c28

if [[ ! -z ${NVCC_WRAPPER_TMPDIR+x} ]]; then

because if I run:

if [[ ! -z ${NVCC_WRAPPER_TMPDIR+x} ]]; then
  temp_dir=${TMPDIR:-/tmp}
else
  temp_dir=${NVCC_WRAPPER_TMPDIR+x}
fi

in my terminal with no env variables set, echo $temp_dir is empty. I think it is supposed to be if [[ -z ${NVCC_WRAPPER_TMPDIR+x} ]] instead of if [[ ! -z ${NVCC_WRAPPER_TMPDIR+x} ]] because if I change the ! -z (which really should be -n) to -z, then I have $temp_dir set to $TMPDIR in my environment and it looks like you are expecting $temp_dir to always be set.

[masterleinad]

Retest this please.

[masterleinad]

Retest this please.

[jrmadsen]

This needs to target the develop branch

[Rombur]

This PR needs to be rebased on develop. That should fix the conflicts.

[masterleinad]

This PR needs to be rebased on develop. That should fix the conflicts.

I took care of that.