Set c++17 standard in CMake for recent torch/cuda versions #109

RaulPPelaez · 2023-08-08T11:40:19Z

Compiling with CUDA 12 and a very recent pytorch version (such as v2.1.0 from the nightly) will make compilation fail because C++17 is required to compile pytorch:

(test7) $ Torch_DIR=$(python -c 'import torch;print(torch.utils.cmake_prefix_path)')  cmake -DCMAKE_BUILD_TYPE=Release ..                                 
make -j15                                                                                                                                                                    
-- The CXX compiler identification is GNU 12.3.0                                                                                                                             
-- Detecting CXX compiler ABI info                                                                                                                                           
-- Detecting CXX compiler ABI info - done                                                                                                                                    
-- Check for working CXX compiler: /shared/raul/mambaforge/envs/test7/bin/x86_64-conda-linux-gnu-c++ - skipped                                                               
-- Detecting CXX compile features                                                                                                                                            
-- Detecting CXX compile features - done                                                                                                                                     
-- The CUDA compiler identification is NVIDIA 12.1.105                                                                                                                       
-- Detecting CUDA compiler ABI info                                                                                                                                          
-- Detecting CUDA compiler ABI info - done                                                                                                                                   
-- Check for working CUDA compiler: /shared/raul/mambaforge/envs/test7/bin/nvcc - skipped                                                                                    
-- Detecting CUDA compile features                                                                                                                                           
-- Detecting CUDA compile features - done                                                                                                                                    
-- Found Python3: /shared/raul/mambaforge/envs/test7/bin/python3.11 (found version "3.11.0") found components: Interpreter Development Development.Module Development.Embed  
-- Found CUDA: /shared/raul/mambaforge/envs/test7 (found version "12.1")                                                                                                     
-- Found CUDAToolkit: /shared/raul/mambaforge/envs/test7/include (found version "12.1.105") 
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD                                            
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed                                   
-- Looking for pthread_create in pthreads                                             
-- Looking for pthread_create in pthreads - not found                                 
-- Looking for pthread_create in pthread                                              
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE  
-- Caffe2: CUDA detected: 12.1
-- Caffe2: CUDA nvcc is: /shared/raul/mambaforge/envs/test7/bin/nvcc
-- Caffe2: CUDA toolkit directory: /shared/raul/mambaforge/envs/test7
-- Caffe2: Header version is: 12.1
-- /shared/raul/mambaforge/envs/test7/lib/libnvrtc.so shorthash is 8144a3bc      
-- USE_CUDNN is set to 0. Compiling without cuDNN support                          
-- USE_CUSPARSELT is set to 0. Compiling without cuSPARSELt support                                                                                                          -- Autodetected CUDA architecture(s):  8.9 8.9 8.9 8.9                                                                                                                       
-- Added CUDA NVCC flags for: -gencode;arch=compute_89,code=sm_89                                                                                                            
-- MKL_ARCH: intel64                                                                                                                                                         
-- MKL_ROOT /shared/raul/mambaforge/envs/test7                                                                                                                               
-- MKL_LINK: dynamic                                                                                                                                                         
-- MKL_INTERFACE_FULL: intel_ilp64                                                                                                                                           
-- MKL_THREADING: intel_thread                                                                                                                                               
-- MKL_MPI: intelmpi                                                                                                                                                         
CMake Warning at /shared/raul/mambaforge/envs/test7/lib/python3.11/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:22 (message):                 
  static library kineto_LIBRARY-NOTFOUND not found.                                                                                                                          
Call Stack (most recent call first):                                                                                                                                         
  /shared/raul/mambaforge/envs/test7/lib/python3.11/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:127 (append_torchlib_if_found)                                     CMakeLists.txt:13 (find_package)                                                                                                                                           
                                                                                                                                                                             
                                                                                                                                                                             
-- Configuring done (1.6s)                                                                                                                                                   
-- Generating done (0.1s)                                                                                                                                                    
-- Build files have been written to: /shared/raul/NNPOps/build                                                                                                               
(test7) $ make -j15                                                                                                                             [140/1551]
[ 21%] Building CXX object CMakeFiles/NNPOpsPyTorch.dir/src/pytorch/BatchedNN.cpp.o                                                                                          
[ 21%] Building CXX object CMakeFiles/NNPOpsPyTorch.dir/src/ani/CpuANISymmetryFunctions.cpp.o                            
[ 26%] Building CXX object CMakeFiles/NNPOpsPyTorch.dir/src/pytorch/CFConv.cpp.o                                                                                             
[ 26%] Building CUDA object CMakeFiles/NNPOpsPyTorch.dir/src/ani/CudaANISymmetryFunctions.cu.o                                            
[ 34%] Building CXX object CMakeFiles/NNPOpsPyTorch.dir/src/pytorch/SymmetryFunctions.cpp.o                                          
[ 34%] Building CXX object CMakeFiles/NNPOpsPyTorch.dir/src/pytorch/CFConvNeighbors.cpp.o                                                       
[ 43%] Building CXX object CMakeFiles/NNPOpsPyTorch.dir/src/pytorch/neighbors/getNeighborPairsCPU.cpp.o                                                                      [ 43%] Building CUDA object CMakeFiles/NNPOpsPyTorch.dir/src/pytorch/neighbors/getNeighborPairsCUDA.cu.o                                                                     
[ 52%] Building CXX object CMakeFiles/NNPOpsPyTorch.dir/src/pytorch/neighbors/neighbors.cpp.o                                                                                
[ 60%] Building CXX object CMakeFiles/NNPOpsPyTorch.dir/src/pytorch/pme/pmeCPU.cpp.o                                                                                         
[ 60%] Building CXX object CMakeFiles/NNPOpsPyTorch.dir/src/pytorch/pme/pme.cpp.o                                                                                            
[ 60%] Building CUDA object CMakeFiles/NNPOpsPyTorch.dir/src/schnet/CudaCFConv.cu.o                                                                                          
[ 60%] Building CUDA object CMakeFiles/NNPOpsPyTorch.dir/src/pytorch/pme/pmeCUDA.cu.o                                                                                        
[ 60%] Building CXX object CMakeFiles/NNPOpsPyTorch.dir/src/schnet/CpuCFConv.cpp.o                                                                                           
In file included from /shared/raul/mambaforge/envs/test7/lib/python3.11/site-packages/torch/include/torch/extension.h:4,                                                     
                 from /shared/raul/NNPOps/src/pytorch/pme/pmeCUDA.cu:1:                                                                                                      
/shared/raul/mambaforge/envs/test7/lib/python3.11/site-packages/torch/include/torch/csrc/api/include/torch/all.h:4:2: error: #error C++17 or later compatible compiler is req
uired to use PyTorch.                                                                                                                                                        
    4 | #error C++17 or later compatible compiler is required to use PyTorch.                                                                                                      |  ^~~~~                                                                                                                                                               
[ 60%] Built target copy_test

Simply setting the standard from 14 to 17 in CMakeLists.txt fixes it.
CUDA 11 also supports C++17, but CUDA 10.2 does not. I check for this and leave it at C++14 in that case.
GCC supports C++17 since version 7, so I default it to it.

RaulPPelaez · 2023-08-17T13:04:09Z

This is ready to merge.

RaulPPelaez · 2023-08-17T14:12:31Z

CUDA 11.8 build tends to fail due to some form of disk access error when installing CUDA. Must be a bug in the Jimver thingy. There is a new version, lets try with that...

raimis · 2023-08-17T14:15:57Z

I have purged the GA cache. If it fails, try to rerun.

RaulPPelaez · 2023-08-17T14:46:43Z

I am not sure if I do not have rights to do so or just do not know how, but I cannot rerun the CI. I will just make a spurious commit.

RaulPPelaez · 2023-08-17T15:16:51Z

11.8 Still refuses to download it seems.

raimis · 2023-08-17T15:19:13Z

[Linux (CUDA 11.8, Python 3.10, PyTorch 2.0)](https://github.com/openmm/NNPOps/actions/runs/5892449251/job/15981745203#step:1:39)
You are running out of disk space. The runner will stop working when the machine runs out of disk space. Free space left: 0 MB

RaulPPelaez · 2023-08-17T15:29:08Z

Do you know if this disk limit is per action or per individual check?
If it is the former maybe we can do something, for the latter I do not really know why cuda 11.2 takes more space than 11.8 as to go over the threshold.

This reverts commit f70c859.

RaulPPelaez · 2023-11-17T11:28:34Z

This is ready for review.
With the changes in conda-forge regarding CUDA, from version 12 there is no need to install cuda at the OS level in the CI (so no Jimver/cuda github action). This is good news here because the current CI is constantly running out of space.
However, the workflow is different enough that I decided to move it to a different CI. The idea being that eventually the old one will be dropped (when CUDA 12 is the oldest version supported I guess).

I had to deal with a couple of quicks in the compilation process for pytorch 2.1 and CUDA 12. In particular:

torch is autodetecting wrongly the cuda archs, sending sm_35 to nvcc 12, which is deprecated. To fix it I just set TORCH_CUDA_ARCHS=8.9 to give it an example
conda-forge installs cuda headers to a non standard directory $CONDA_PREFIX/$targetsDir/include. For some reason this is preventing torch from finding the cuda headers. I had to set CUDA_INC_PATH manually to that directory.

RaulPPelaez · 2023-11-17T11:29:03Z

I am using the changes to CMakeLists.txt as a patch to build this conda-forge/nnpops-feedstock#29

RaulPPelaez · 2024-03-04T11:30:09Z

@mikemhenry I would like to merge this, but I believe the self hosted runner is not working.

RaulPPelaez added 2 commits August 8, 2023 13:26

Set c++17 standard in CMake

acc626e

Set C++17 only with CUDA>10

0e16c49

raimis self-requested a review August 17, 2023 14:02

Update Jimver/cuda-toolkit to v0.2.11

8ac9c6f

RaulPPelaez added 2 commits August 17, 2023 16:47

Spurious commit

e8559a3

Undo spurious commit to trigger rerun

1c1a504

RaulPPelaez added 17 commits August 18, 2023 08:06

Try network CUDA method

f70c859

Revert "Try network CUDA method"

56e1c72

This reverts commit f70c859.

Detect pytorch 2.1 to set cpp 17

a09f4c3

Set CUDA standard too

573ba63

Update CMakeLists

a0ab926

Update CI

888fc0e

Update CI

d2973f7

Add CI for cuda 12

7e958a5

Update ci

0deca16

Update ci

81a5e2e

Update CI

e9e6a6d

Update CI

d7075e8

Update ci

262dfd6

Update ci

170a266

Add nvtx

a7002bf

typo

256fb00

Update ci

2b7e61f

RaulPPelaez added 2 commits November 17, 2023 11:47

Set TORCH_CUDA_ARCH_LIST

ed48fcb

Use targetsDir

f2bf8cc

Empty commit to trigger github actions

e4e2971

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Set c++17 standard in CMake for recent torch/cuda versions #109

Set c++17 standard in CMake for recent torch/cuda versions #109

RaulPPelaez commented Aug 8, 2023

RaulPPelaez commented Aug 17, 2023

RaulPPelaez commented Aug 17, 2023

raimis commented Aug 17, 2023

RaulPPelaez commented Aug 17, 2023

RaulPPelaez commented Aug 17, 2023

raimis commented Aug 17, 2023

RaulPPelaez commented Aug 17, 2023

RaulPPelaez commented Nov 17, 2023

RaulPPelaez commented Nov 17, 2023

RaulPPelaez commented Mar 4, 2024

Set c++17 standard in CMake for recent torch/cuda versions #109

Are you sure you want to change the base?

Set c++17 standard in CMake for recent torch/cuda versions #109

Conversation

RaulPPelaez commented Aug 8, 2023

RaulPPelaez commented Aug 17, 2023

RaulPPelaez commented Aug 17, 2023

raimis commented Aug 17, 2023

RaulPPelaez commented Aug 17, 2023

RaulPPelaez commented Aug 17, 2023

raimis commented Aug 17, 2023

RaulPPelaez commented Aug 17, 2023

RaulPPelaez commented Nov 17, 2023

RaulPPelaez commented Nov 17, 2023

RaulPPelaez commented Mar 4, 2024