Fails with CUDA driver version is insufficient for CUDA runtime version

[AlexeyMatskevich]

1. Issue or feature description

I get this error when I try to run the code.

terminate called after throwing an instance of 'thrust::system::detail::bad_alloc'
  what():  std::bad_alloc: cudaErrorInsufficientDriver: CUDA driver version is insufficient for CUDA runtime version

2. Steps to reproduce the issue

Get code from first example https://docs.nvidia.com/cuda/thrust/index.html#vectors and run
My example https://github.com/AlexeyMatskevich/cuda_test

3. Information to attach (optional if deemed irrelevant)

• Some nvidia-container information: nvidia-container-cli -k -d /dev/tty info
• Kernel version from uname -a
Linux workstation 5.15.90.1-microsoft-standard-WSL2 #1 SMP Fri Jan 27 02:56:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
• Any relevant kernel output lines from dmesg
• Driver information from nvidia-smi -a
• Docker version from docker version
• NVIDIA packages version from dpkg -l '*nvidia*' or rpm -qa '*nvidia*'
• NVIDIA container library version from nvidia-container-cli -V
• NVIDIA container library logs (see troubleshooting)
• Docker command, image and tag used

      #Dockerfile
      FROM nvcr.io/nvidia/cuda:12.2.0-devel-ubuntu20.04
      
      WORKDIR /usr/src/app
      RUN  apt-get update \
        && apt-get install -y wget libssl-dev \
        && rm -rf /var/lib/apt/lists/*
      RUN wget https://github.com/Kitware/CMake/releases/download/v3.27.4/cmake-3.27.4.tar.gz && tar -zxvf cmake-3.27.4.tar.gz && cd cmake-3.27.4 && ./bootstrap && make && make install
      
      CMD ["/usr/bin/bash"]

     #docker-compose.yml
     services:
       cuda:
         build:
           context: .
           dockerfile: Dockerfile
           network: host
         deploy:
           resources:
             reservations:
               devices:
                 - driver: nvidia
                   count: 1
                   capabilities: [ gpu ]
         ulimits:
           stack: 67108864
           memlock: -1
         volumes:
           - .:/usr/src/app
  

nvidia-smi.txt
dmesq.txt
docker-version.txt
findmnt.txt
nvidia-smi-in-docker.txt
nvidia-container-cli.txt
dpkq.txt

[AlexeyMatskevich]

I found out that this is a problem with the way clion calls cmake and nvcc when compiling remotely (as docker).