Help CMake find CUDA even if nvcc is not in $PATH

[maxhgerlach]

Checklist before submitting

• Did you read the contributor guide?
• Did you update the docs?
• Did you write any tests to validate this change?
• Did you update the CHANGELOG, if this change affects users?

Description

If nvcc is not in $PATH, CMake's check_language(CUDA) will not find it although it may be installed in a default location like /usr/local/cuda. In the latter case find_package(CUDAToolkit) can still find it, so with this PR we use the result of that to guide check_language(CUDA) a second time.

Note: This would align the behavior with that of Horovod 0.23, which used find_package(CUDA) and did not always require nvcc to be in $PATH.

Here are a couple of log excerpts from my system:

Output if neither PATH, nor HOROVOD_CUDA_HOME is set (new with this PR)

    -- Looking for a CUDA compiler
    -- Looking for a CUDA compiler - NOTFOUND
    -- Looking for a CUDA host compiler - /usr/bin/c++
    -- Found CUDAToolkit: /usr/local/cuda/include (found version "11.2.152")
    -- Looking for C++ include pthread.h
    -- Looking for C++ include pthread.h - found
    -- Performing Test CMAKE_HAVE_LIBC_PTHREAD
    -- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
    -- Found Threads: TRUE
    CUDA compiler was not found in $PATH, but searching again in CUDA Toolkit binary directory
    -- The CUDA compiler identification is NVIDIA 11.2.152
    -- Detecting CUDA compiler ABI info
    -- Detecting CUDA compiler ABI info - done
    -- Check for working CUDA compiler: /usr/local/cuda/bin/nvcc - skipped
    -- Detecting CUDA compile features
    -- Detecting CUDA compile features - done

Output if PATH is not set, but HOROVOD_CUDA_HOME=/usr/local/cuda-11.2

    -- The CUDA compiler identification is NVIDIA 11.2.152
    -- Detecting CUDA compiler ABI info
    -- Detecting CUDA compiler ABI info - done
    -- Check for working CUDA compiler: /usr/local/cuda-11.2/bin/nvcc - skipped
    -- Detecting CUDA compile features
    -- Detecting CUDA compile features - done
    -- Found CUDAToolkit: /usr/local/cuda-11.2/include (found version "11.2.152")

Output if PATH=/usr/local/cuda/bin:$PATH and HOROVOD_CUDA_HOME is not set

    -- Looking for a CUDA compiler
    -- Looking for a CUDA compiler - /usr/local/cuda/bin/nvcc
    -- The CUDA compiler identification is NVIDIA 11.2.152
    -- Detecting CUDA compiler ABI info
    -- Detecting CUDA compiler ABI info - done
    -- Check for working CUDA compiler: /usr/local/cuda/bin/nvcc - skipped
    -- Detecting CUDA compile features
    -- Detecting CUDA compile features - done
    -- Found CUDAToolkit: /usr/local/cuda/include (found version "11.2.152")

Review process to land

1. All tests and other checks must succeed.
2. At least one member of the technical steering committee must review and approve.
3. If any member of the technical steering committee requests changes, they must be addressed.

[EnricoMi]

I'll cut release 0.24.1 when this is in master.

[maxhgerlach]

I'll cut release 0.24.1 when this is in master.

👍 I was about to suggest that

[github-actions]

Unit Test Results

     802 files   -   28       802 suites   - 28   9h 10m 49s ⏱️ - 48m 13s
     722 tests ±    0       679 ✔️ ±    0       43 💤 ±    0  0 ❌ ±0 
17 373 runs   - 664  12 410 ✔️  - 406  4 963 💤  - 258  0 ❌ ±0 

Results for commit 85d63b3. ± Comparison against base commit 2df6e1e.

[github-actions]

Unit Test Results (with flaky tests)

     886 files   -   34       886 suites   - 34   9h 31m 17s ⏱️ - 53m 43s
     722 tests ±    0       679 ✔️ +    1       43 💤 ±    0  0 ❌  - 1 
19 365 runs   - 625  13 646 ✔️  - 377  5 719 💤  - 247  0 ❌  - 1 

Results for commit 85d63b3. ± Comparison against base commit 2df6e1e.