Debian 12: incomplete repository

[marxin]

With the release of CUDA 12.3, the new Linux distribution Debian 12 has been added to the table of supported distros:
https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#id12

However, one can't effectively use the repository for the following reasons:

1. cuda-archive-keyring.gpg is missing for debian12: https://developer.download.nvidia.com/compute/cuda/repos/debian12/x86_64/
2. there are missing dependencies that block installation of cuda-12-3:

 => ERROR [10/16] RUN apt update && apt-get install -y cuda-12-3                                                                                                                                                                                                                                                                                                      1.3s
------                                                                                                                                                                                                                                                                                                                                                                     
 > [10/16] RUN apt update && apt-get install -y cuda-12-3:                                                                                                                                                                                                                                                                                                                 
0.363                                                                                                                                                                                                                                                                                                                                                                      
0.363 WARNING: apt does not have a stable CLI interface. Use with caution in scripts.                                                                                                                                                                                                                                                                                      
0.363                                                                                                                                                                                                                                                                                                                                                                      
0.412 Hit:1 http://deb.debian.org/debian bookworm InRelease                                                                                                                                                                                                                                                                                                                
0.413 Hit:2 http://deb.debian.org/debian bookworm-updates InRelease
0.420 Hit:3 http://deb.debian.org/debian-security bookworm-security InRelease
0.506 Get:4 https://developer.download.nvidia.com/compute/cuda/repos/debian12/x86_64  InRelease [1581 B]
0.599 Ign:4 https://developer.download.nvidia.com/compute/cuda/repos/debian12/x86_64  InRelease
0.638 Get:5 https://developer.download.nvidia.com/compute/cuda/repos/debian12/x86_64  Packages [45.3 kB]
0.666 Fetched 46.8 kB in 0s (161 kB/s)
0.666 Reading package lists...
0.890 Building dependency tree...
0.925 Reading state information...
0.929 All packages are up to date.
0.931 W: GPG error: https://developer.download.nvidia.com/compute/cuda/repos/debian12/x86_64  InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY A4B469963BF863CC
0.936 Reading package lists...
1.157 Building dependency tree...
1.191 Reading state information...
1.216 Some packages could not be installed. This may mean that you have
1.216 requested an impossible situation or if you are using the unstable
1.216 distribution that some required packages have not yet been created
1.216 or been moved out of Incoming.
1.216 The following information may help to resolve the situation:
1.216 
1.216 The following packages have unmet dependencies:
1.244  cuda-drivers-545 : Depends: nvidia-opencl-icd (>= 545.23.06) but it is not installable
1.244  libcuda1 : PreDepends: nvidia-legacy-check (>= 396) but it is not going to be installed
1.244             Depends: nvidia-support but it is not installable
1.244             Depends: nvidia-alternative (= 545.23.06-1) but it is not installable
1.244             Recommends: nvidia-kernel-dkms (= 545.23.06-1) but it is not installable or
1.244                         nvidia-kernel-545.23.06
1.244             Recommends: libnvidia-cfg1 (= 545.23.06-1) but it is not going to be installed
1.244             Recommends: nvidia-persistenced but it is not going to be installed
1.244             Recommends: libcuda1-i386 (= 545.23.06-1) but it is not installable
1.244  libnvidia-ml1 : Depends: nvidia-alternative (= 545.23.06-1) but it is not installable
1.244  nvidia-driver : PreDepends: nvidia-installer-cleanup but it is not installable
1.244                  PreDepends: nvidia-legacy-check (>= 396) but it is not going to be installed
1.244                  Depends: nvidia-driver-libs (= 545.23.06-1) but it is not going to be installed
1.244                  Depends: nvidia-driver-bin (= 545.23.06-1) but it is not going to be installed
1.244                  Depends: xserver-xorg-video-nvidia (= 545.23.06-1) but it is not going to be installed
1.244                  Depends: nvidia-vdpau-driver (= 545.23.06-1) but it is not going to be installed
1.244                  Depends: nvidia-alternative (= 545.23.06-1) but it is not installable
1.244                  Depends: nvidia-kernel-dkms (= 545.23.06-1) but it is not installable or
1.244                           nvidia-kernel-545.23.06 or
1.244                           nvidia-kernel-open-dkms (= 545.23.06-1) but it is not installable
1.244                  Depends: nvidia-support but it is not installable
1.244                  Recommends: libnvidia-cfg1 (= 545.23.06-1) but it is not going to be installed
1.244                  Recommends: nvidia-persistenced but it is not going to be installed
1.244  nvidia-settings : PreDepends: nvidia-installer-cleanup but it is not installable
1.244                    Depends: nvidia-alternative but it is not installable
1.244                    Recommends: libgl1-nvidia-glvnd-glx but it is not going to be installed or
1.244                                libgl1-nvidia-glx but it is not installable
1.244                    Recommends: nvidia-vdpau-driver but it is not going to be installed
1.244  nvidia-smi : Depends: nvidia-alternative (= 545.23.06-1) but it is not installable
1.244               Recommends: nvidia-kernel-dkms (= 545.23.06-1) but it is not installable or
1.244                           nvidia-kernel-545.23.06
1.244  nvidia-xconfig : PreDepends: nvidia-installer-cleanup but it is not installable
1.244                   Recommends: libgl1-nvidia-glx-any
1.246 E: Unable to correct problems, you have held broken packages.

[moetayuko]

I just successfully upgraded to the new repo, so these issues should be fixed now.

[marxin]

Can you please share the steps and output of how you made the update?

[moetayuko]

Can you please share the steps and output of how you made the update?

I migrated from debian11 to debian12 repo, hence:

wget https://developer.download.nvidia.com/compute/cuda/repos/debian12/x86_64/cuda-keyring_1.1-1_all.deb
sudo apt purge cuda-keyring  # must manually remove old keyring, otherwise /etc/apt/sources.list.d/cuda-debian11-x86_64.list will be kept
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt update && sudo apt dist-upgrade -y
sudo apt autoremove --purge

[marxin]

Ok, but that's something different as you've updated the system from Debian 11.
Can you pleas try installing CUDA on the vanilla Debian 12 system?

[pauleonix]

@marxin I did not install the whole toolkit (using spack/runfile to do that in user-space), but apt-get install cuda-drivers seems to work for me and it includes e.g. nvidia-driver which you seem to have had issues with.

[marxin]

Still not working for me. I might be doing something wrong:

$ cat bookworm.Dockerfile
FROM debian:bookworm
RUN apt update && apt-get install -y wget
RUN wget https://developer.download.nvidia.com/compute/cuda/repos/debian12/x86_64/cuda-keyring_1.1-1_all.deb
RUN dpkg -i cuda-keyring_1.1-1_all.deb
RUN apt update && apt-get install -y cuda-drivers
$ docker build -t bookworm -f bookworm.Dockerfile .
...
1.168 The following packages have unmet dependencies:
1.196  cuda-drivers-545 : Depends: nvidia-opencl-icd (>= 545.23.06) but it is not installable
1.196  libcuda1 : PreDepends: nvidia-legacy-check (>= 396) but it is not going to be installed
1.196             Depends: nvidia-support but it is not installable
1.196             Depends: nvidia-alternative (= 545.23.06-1) but it is not installable
1.196             Recommends: nvidia-kernel-dkms (= 545.23.06-1) but it is not installable or
1.196                         nvidia-kernel-545.23.06
1.196             Recommends: libnvidia-cfg1 (= 545.23.06-1) but it is not going to be installed
1.196             Recommends: nvidia-persistenced but it is not going to be installed
1.196             Recommends: libcuda1-i386 (= 545.23.06-1) but it is not installable

[pauleonix]

Looks to me like you forgot (among other things)

3. Enable the contrib repository:

sudo add-apt-repository contrib

from the preparation steps in the Installation Guide.

[marxin]

You are correct, with the fixed:

$ cat bookworm.Dockerfile && docker build -t bookworm -f bookworm.Dockerfile .
FROM debian:bookworm
RUN apt update && apt-get install -y wget software-properties-common
RUN wget https://developer.download.nvidia.com/compute/cuda/repos/debian12/x86_64/cuda-keyring_1.1-1_all.deb
RUN dpkg -i cuda-keyring_1.1-1_all.deb
RUN add-apt-repository contrib
RUN apt update
RUN apt-get install -y cuda-drivers

I still get the very same problem :(

[pauleonix]

Hmm, I cited that step b/c it seemed like the most important one to me given your output. That being said, there are more steps like sudo apt-get install linux-headers-$(uname -r).

In the end I'm not sure if you should install the drivers in a container either way (or if it will work at all). I have more experience with Apptainer where drivers etc. are mounted from the container host into the container.

You could try using NVIDIA HPC Container Maker to generate a sensible Dockerfile for you. As the repository works for me on bare metal, this seems like an issue with the Dockerfile rather than with the repo.

[elezar]

Note that this issue seems related to the installation of the CUDA driver (and possibly the CUDA toolkit) on Debian 12-based systems. This should not affect the installation of the NVIDIA Container Toolkit packages on these systems.

Note however, that a working CUDA Driver is required for the NVIDIA Container Toolkit to function.

I would recommend transferring this issue to https://github.com/NVIDIA/cuda-repo-management