Missing cuda in docker for submission

[srama2512]

My code requires this library for running: https://github.com/rusty1s/pytorch_scatter . When I try installing it, it throws out the following error:

  No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda'
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-3.6
  creating build/lib.linux-x86_64-3.6/torch_scatter
  copying torch_scatter/sub.py -> build/lib.linux-x86_64-3.6/torch_scatter
  copying torch_scatter/add.py -> build/lib.linux-x86_64-3.6/torch_scatter
  copying torch_scatter/div.py -> build/lib.linux-x86_64-3.6/torch_scatter
  copying torch_scatter/mul.py -> build/lib.linux-x86_64-3.6/torch_scatter
  copying torch_scatter/std.py -> build/lib.linux-x86_64-3.6/torch_scatter
  copying torch_scatter/__init__.py -> build/lib.linux-x86_64-3.6/torch_scatter
  copying torch_scatter/mean.py -> build/lib.linux-x86_64-3.6/torch_scatter
  copying torch_scatter/logsumexp.py -> build/lib.linux-x86_64-3.6/torch_scatter
  copying torch_scatter/min.py -> build/lib.linux-x86_64-3.6/torch_scatter
  copying torch_scatter/max.py -> build/lib.linux-x86_64-3.6/torch_scatter
  creating build/lib.linux-x86_64-3.6/test
  copying test/test_backward.py -> build/lib.linux-x86_64-3.6/test
  copying test/__init__.py -> build/lib.linux-x86_64-3.6/test
  copying test/test_logsumexp.py -> build/lib.linux-x86_64-3.6/test
  copying test/test_max_min.py -> build/lib.linux-x86_64-3.6/test
  copying test/utils.py -> build/lib.linux-x86_64-3.6/test
  copying test/test_std.py -> build/lib.linux-x86_64-3.6/test
  copying test/test_multi_gpu.py -> build/lib.linux-x86_64-3.6/test
  copying test/test_forward.py -> build/lib.linux-x86_64-3.6/test
  copying test/test_broadcasting.py -> build/lib.linux-x86_64-3.6/test
  creating build/lib.linux-x86_64-3.6/torch_scatter/composite
  copying torch_scatter/composite/__init__.py -> build/lib.linux-x86_64-3.6/torch_scatter/composite
  copying torch_scatter/composite/softmax.py -> build/lib.linux-x86_64-3.6/torch_scatter/composite
  creating build/lib.linux-x86_64-3.6/torch_scatter/utils
  copying torch_scatter/utils/ext.py -> build/lib.linux-x86_64-3.6/torch_scatter/utils
  copying torch_scatter/utils/__init__.py -> build/lib.linux-x86_64-3.6/torch_scatter/utils
  copying torch_scatter/utils/gen.py -> build/lib.linux-x86_64-3.6/torch_scatter/utils
  running build_ext
  building 'torch_scatter.scatter_cpu' extension
  creating build/temp.linux-x86_64-3.6
  creating build/temp.linux-x86_64-3.6/cpu
  gcc -pthread -B /opt/conda/envs/habitat/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/opt/conda/envs/habitat/lib/python3.6/site-packages/torch/include -I/opt/conda/envs/habitat/lib/python3.6/site-packages/torch/include/torch/csrc/api/include -I/opt/conda/envs/habitat/lib/python3.6/site-packages/torch/include/TH -I/opt/conda/envs/habitat/lib/python3.6/site-packages/torch/include/THC -I/opt/conda/envs/habitat/include/python3.6m -c cpu/scatter.cpp -o build/temp.linux-x86_64-3.6/cpu/scatter.o -Wno-unused-variable -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=scatter_cpu -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++11
  cc1plus: warning: command line option '-Wstrict-prototypes' is valid for C/ObjC but not for C++
  g++ -pthread -shared -B /opt/conda/envs/habitat/compiler_compat -L/opt/conda/envs/habitat/lib -Wl,-rpath=/opt/conda/envs/habitat/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-3.6/cpu/scatter.o -o build/lib.linux-x86_64-3.6/torch_scatter/scatter_cpu.cpython-36m-x86_64-linux-gnu.so
  building 'torch_scatter.scatter_cuda' extension
  creating build/temp.linux-x86_64-3.6/cuda
  gcc -pthread -B /opt/conda/envs/habitat/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/opt/conda/envs/habitat/lib/python3.6/site-packages/torch/include -I/opt/conda/envs/habitat/lib/python3.6/site-packages/torch/include/torch/csrc/api/include -I/opt/conda/envs/habitat/lib/python3.6/site-packages/torch/include/TH -I/opt/conda/envs/habitat/lib/python3.6/site-packages/torch/include/THC -I/usr/local/cuda/include -I/opt/conda/envs/habitat/include/python3.6m -c cuda/scatter.cpp -o build/temp.linux-x86_64-3.6/cuda/scatter.o -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=scatter_cuda -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++11
  cc1plus: warning: command line option '-Wstrict-prototypes' is valid for C/ObjC but not for C++
  /usr/local/cuda/bin/nvcc -I/opt/conda/envs/habitat/lib/python3.6/site-packages/torch/include -I/opt/conda/envs/habitat/lib/python3.6/site-packages/torch/include/torch/csrc/api/include -I/opt/conda/envs/habitat/lib/python3.6/site-packages/torch/include/TH -I/opt/conda/envs/habitat/lib/python3.6/site-packages/torch/include/THC -I/usr/local/cuda/include -I/opt/conda/envs/habitat/include/python3.6m -c cuda/scatter_kernel.cu -o build/temp.linux-x86_64-3.6/cuda/scatter_kernel.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options '-fPIC' -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=scatter_cuda -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++11
  unable to execute '/usr/local/cuda/bin/nvcc': No such file or directory
  error: command '/usr/local/cuda/bin/nvcc' failed with exit status 1
  ----------------------------------------
  ERROR: Failed building wheel for torch-scatter

Looks like nvcc is missing. I tried manually looking at the base docker. I couldn't find nvcc. Should I manually install it in that case?

[srama2512]

I was able to fix this by installing the cuda development libraries:

RUN apt-get update && apt-get install -y --allow-unauthenticated --no-install-recommends \
        cuda-libraries-dev-$CUDA_PKG_VERSION \
        cuda-nvml-dev-$CUDA_PKG_VERSION \
        cuda-minimal-build-$CUDA_PKG_VERSION \
        cuda-command-line-tools-$CUDA_PKG_VERSION \
        libnccl-dev=$NCCL_VERSION-1+cuda10.1 \
        libcublas-dev=10.2.1.243-1 \
        && \
    rm -rf /var/lib/apt/lists/*