The following packages have unmet dependencies: cuda-cudart-dev-10-1 : Depends: cuda-cudart-10-1 (>= 10.1.243) but 10.1.105-1 is to be installed

[terryyizhong]

Thanks for your help and Looking forward your reply first.

1. Issue or feature description

when I try to install nvidia docker by:
sudo apt-get install -y nvidia-container-toolkit
I get following messages:

The following packages have unmet dependencies:
cuda-cudart-dev-10-1 : Depends: cuda-cudart-10-1 (>= 10.1.243) but 10.1.105-1 is to be installed
Depends: cuda-driver-dev-10-1 (>= 10.1.243) but 10.1.105-1 is to be installed
cuda-cufft-dev-10-1 : Depends: cuda-cufft-10-1 (>= 10.1.243) but 10.1.105-1 is to be installed
cuda-curand-dev-10-1 : Depends: cuda-curand-10-1 (>= 10.1.243) but 10.1.105-1 is to be installed
cuda-cusolver-dev-10-1 : Depends: cuda-cusolver-10-1 (>= 10.1.243) but 10.1.105-1 is to be installed
cuda-cusparse-dev-10-1 : Depends: cuda-cusparse-10-1 (>= 10.1.243) but 10.1.105-1 is to be installed
cuda-libraries-dev-10-1 : Depends: cuda-driver-dev-10-1 (>= 10.1.243) but 10.1.105-1 is to be installed
cuda-npp-dev-10-1 : Depends: cuda-npp-10-1 (>= 10.1.243) but 10.1.105-1 is to be installed
cuda-nvgraph-dev-10-1 : Depends: cuda-nvgraph-10-1 (>= 10.1.243) but 10.1.105-1 is to be installed
cuda-nvjpeg-dev-10-1 : Depends: cuda-nvjpeg-10-1 (>= 10.1.243) but 10.1.105-1 is to be installed
cuda-nvrtc-dev-10-1 : Depends: cuda-nvrtc-10-1 (>= 10.1.243) but 10.1.105-1 is to be installed
cuda-toolkit-10-1 : Depends: cuda-nvml-dev-10-1 (>= 10.1.243) but 10.1.105-1 is to be installed
cuda-visual-tools-10-1 : Depends: cuda-driver-dev-10-1 (>= 10.1.243) but 10.1.105-1 is to be installed
Depends: cuda-nvml-dev-10-1 (>= 10.1.243) but 10.1.105-1 is to be installed
E: Unmet dependencies. Try 'apt-get -f install' with no packages (or specify a solution).

2. Steps to reproduce the issue

I first install docker, nvidia-docker and pull pytorch20.03 images successfully.
When I run the image, it shows need nvidia driver > 440.XX . But my driver is 418.xx with cuda 10.1 (shows in nvidia-smi)
So I follow a instruction to update nvidia driver.
I first remove the old driver by: sudo apt-get remove –purge nvidia
And then install newest driver with cuda10.2 successfully. nvidia-smi shows fine.*

3. Information to attach (optional if deemed irrelevant)

• Some nvidia-container information: nvidia-container-cli -k -d /dev/tty info
  nvidia-container-cli -k -d /dev/tty info

-- WARNING, the following logs are for debugging purposes only --

I0520 11:25:25.451478 22383 nvc.c:281] initializing library context (version=1.0.7, build=b71f87c04b8eca8a16bf60995506c35c937347d9)
I0520 11:25:25.451517 22383 nvc.c:255] using root /
I0520 11:25:25.451525 22383 nvc.c:256] using ldcache /etc/ld.so.cache
I0520 11:25:25.451527 22383 nvc.c:257] using unprivileged user 65534:65534
I0520 11:25:25.452542 22384 nvc.c:191] loading kernel module nvidia
I0520 11:25:25.452675 22384 nvc.c:203] loading kernel module nvidia_uvm
I0520 11:25:25.452737 22384 nvc.c:211] loading kernel module nvidia_modeset
I0520 11:25:25.452924 22385 driver.c:133] starting driver service
I0520 11:25:25.861005 22383 nvc_info.c:438] requesting driver information with ''
I0520 11:25:25.863231 22383 nvc_info.c:152] selecting /usr/lib/x86_64-linux-gnu/vdpau/libvdpau_nvidia.so.440.64.00
I0520 11:25:25.865159 22383 nvc_info.c:152] selecting /usr/lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so.440.64.00
I0520 11:25:25.866234 22383 nvc_info.c:152] selecting /usr/lib/x86_64-linux-gnu/libnvidia-opticalflow.so.440.64.00
I0520 11:25:25.867288 22383 nvc_info.c:152] selecting /usr/lib/x86_64-linux-gnu/libnvidia-opencl.so.440.64.00
I0520 11:25:25.867329 22383 nvc_info.c:152] selecting /usr/lib/x86_64-linux-gnu/libnvidia-ml.so.440.64.00
I0520 11:25:25.869982 22383 nvc_info.c:152] selecting /usr/lib/x86_64-linux-gnu/libnvidia-ifr.so.440.64.00
I0520 11:25:25.871621 22383 nvc_info.c:152] selecting /usr/lib/x86_64-linux-gnu/libnvidia-fbc.so.440.64.00
I0520 11:25:25.873062 22383 nvc_info.c:152] selecting /usr/lib/x86_64-linux-gnu/libnvidia-fatbinaryloader.so.440.64.00
I0520 11:25:25.874196 22383 nvc_info.c:152] selecting /usr/lib/x86_64-linux-gnu/libnvidia-encode.so.440.64.00
I0520 11:25:25.875667 22383 nvc_info.c:152] selecting /usr/lib/x86_64-linux-gnu/libnvidia-compiler.so.440.64.00
I0520 11:25:25.877114 22383 nvc_info.c:152] selecting /usr/lib/x86_64-linux-gnu/libnvidia-cfg.so.440.64.00
I0520 11:25:25.878765 22383 nvc_info.c:152] selecting /usr/lib/x86_64-linux-gnu/libnvcuvid.so.440.64.00
I0520 11:25:25.878962 22383 nvc_info.c:152] selecting /usr/lib/x86_64-linux-gnu/libcuda.so.440.64.00
W0520 11:25:25.879067 22383 nvc_info.c:303] missing library libnvidia-eglcore.so
W0520 11:25:25.879070 22383 nvc_info.c:303] missing library libnvidia-glcore.so
W0520 11:25:25.879073 22383 nvc_info.c:303] missing library libnvidia-tls.so
W0520 11:25:25.879075 22383 nvc_info.c:303] missing library libnvidia-glsi.so
W0520 11:25:25.879077 22383 nvc_info.c:303] missing library libnvidia-rtcore.so
W0520 11:25:25.879079 22383 nvc_info.c:303] missing library libnvoptix.so
W0520 11:25:25.879081 22383 nvc_info.c:303] missing library libGLX_nvidia.so
W0520 11:25:25.879083 22383 nvc_info.c:303] missing library libEGL_nvidia.so
W0520 11:25:25.879085 22383 nvc_info.c:303] missing library libGLESv2_nvidia.so
W0520 11:25:25.879087 22383 nvc_info.c:303] missing library libGLESv1_CM_nvidia.so
W0520 11:25:25.879090 22383 nvc_info.c:303] missing library libnvidia-glvkspirv.so
W0520 11:25:25.879092 22383 nvc_info.c:303] missing library libnvidia-cbl.so
W0520 11:25:25.879094 22383 nvc_info.c:307] missing compat32 library libnvidia-ml.so
W0520 11:25:25.879096 22383 nvc_info.c:307] missing compat32 library libnvidia-cfg.so
W0520 11:25:25.879098 22383 nvc_info.c:307] missing compat32 library libcuda.so
W0520 11:25:25.879101 22383 nvc_info.c:307] missing compat32 library libnvidia-opencl.so
W0520 11:25:25.879103 22383 nvc_info.c:307] missing compat32 library libnvidia-ptxjitcompiler.so
W0520 11:25:25.879105 22383 nvc_info.c:307] missing compat32 library libnvidia-fatbinaryloader.so
W0520 11:25:25.879122 22383 nvc_info.c:307] missing compat32 library libnvidia-compiler.so
W0520 11:25:25.879125 22383 nvc_info.c:307] missing compat32 library libvdpau_nvidia.so
W0520 11:25:25.879127 22383 nvc_info.c:307] missing compat32 library libnvidia-encode.so
W0520 11:25:25.879129 22383 nvc_info.c:307] missing compat32 library libnvidia-opticalflow.so
W0520 11:25:25.879131 22383 nvc_info.c:307] missing compat32 library libnvcuvid.so
W0520 11:25:25.879133 22383 nvc_info.c:307] missing compat32 library libnvidia-eglcore.so
W0520 11:25:25.879135 22383 nvc_info.c:307] missing compat32 library libnvidia-glcore.so
W0520 11:25:25.879137 22383 nvc_info.c:307] missing compat32 library libnvidia-tls.so
W0520 11:25:25.879139 22383 nvc_info.c:307] missing compat32 library libnvidia-glsi.so
W0520 11:25:25.879141 22383 nvc_info.c:307] missing compat32 library libnvidia-fbc.so
W0520 11:25:25.879144 22383 nvc_info.c:307] missing compat32 library libnvidia-ifr.so
W0520 11:25:25.879146 22383 nvc_info.c:307] missing compat32 library libnvidia-rtcore.so
W0520 11:25:25.879148 22383 nvc_info.c:307] missing compat32 library libnvoptix.so
W0520 11:25:25.879150 22383 nvc_info.c:307] missing compat32 library libGLX_nvidia.so
W0520 11:25:25.879152 22383 nvc_info.c:307] missing compat32 library libEGL_nvidia.so
W0520 11:25:25.879154 22383 nvc_info.c:307] missing compat32 library libGLESv2_nvidia.so
W0520 11:25:25.879156 22383 nvc_info.c:307] missing compat32 library libGLESv1_CM_nvidia.so
W0520 11:25:25.879218 22383 nvc_info.c:307] missing compat32 library libnvidia-glvkspirv.so
W0520 11:25:25.879221 22383 nvc_info.c:307] missing compat32 library libnvidia-cbl.so
I0520 11:25:25.879408 22383 nvc_info.c:233] selecting /usr/bin/nvidia-smi
I0520 11:25:25.879418 22383 nvc_info.c:233] selecting /usr/bin/nvidia-debugdump
I0520 11:25:25.879428 22383 nvc_info.c:233] selecting /usr/bin/nvidia-persistenced
I0520 11:25:25.879441 22383 nvc_info.c:233] selecting /usr/bin/nvidia-cuda-mps-control
I0520 11:25:25.879451 22383 nvc_info.c:233] selecting /usr/bin/nvidia-cuda-mps-server
I0520 11:25:25.879476 22383 nvc_info.c:370] listing device /dev/nvidiactl
I0520 11:25:25.879479 22383 nvc_info.c:370] listing device /dev/nvidia-uvm
I0520 11:25:25.879481 22383 nvc_info.c:370] listing device /dev/nvidia-uvm-tools
I0520 11:25:25.879483 22383 nvc_info.c:370] listing device /dev/nvidia-modeset
W0520 11:25:25.879509 22383 nvc_info.c:278] missing ipc /var/run/nvidia-persistenced/socket
W0520 11:25:25.879523 22383 nvc_info.c:278] missing ipc /tmp/nvidia-mps
I0520 11:25:25.879525 22383 nvc_info.c:494] requesting device information with ''
I0520 11:25:25.885196 22383 nvc_info.c:524] listing device /dev/nvidia0 (GPU-2ecae5d2-3a27-6851-52a1-d11046a65791 at 00000000:00:07.0)
NVRM version: 440.64.00
CUDA version: 10.2

Device Index: 0
Device Minor: 0
Model: Tesla V100-PCIE-16GB
Brand: Tesla
GPU UUID: GPU-2ecae5d2-3a27-6851-52a1-d11046a65791
Bus Location: 00000000:00:07.0
Architecture: 7.0
I0520 11:25:25.885251 22383 nvc.c:318] shutting down library context
I0520 11:25:25.885556 22385 driver.c:192] terminating driver service
I0520 11:25:26.128060 22383 driver.c:233] driver service terminated successfully

• Kernel version from uname -a
  Linux KS-10-101-8-85 4.4.0-62-generic Plugin.Activate, 400 Bad Request: malformed Host header #83-Ubuntu SMP Wed Jan 18 14:10:15 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

• Any relevant kernel output lines from dmesg

• Driver information from nvidia-smi -a
  ==============NVSMI LOG==============

Timestamp : Wed May 20 19:26:14 2020
Driver Version : 440.64.00
CUDA Version : 10.2

Attached GPUs : 1
GPU 00000000:00:07.0
Product Name : Tesla V100-PCIE-16GB
Product Brand : Tesla
Display Mode : Enabled
Display Active : Disabled
Persistence Mode : Disabled
Accounting Mode : Disabled
Accounting Mode Buffer Size : 4000
Driver Model
Current : N/A
Pending : N/A
Serial Number : 0320118119095
GPU UUID : GPU-2ecae5d2-3a27-6851-52a1-d11046a65791
Minor Number : 0
VBIOS Version : 88.00.1A.00.03
MultiGPU Board : No
Board ID : 0x7
GPU Part Number : 900-2G500-0000-000
Inforom Version
Image Version : G500.0200.00.03
OEM Object : 1.1
ECC Object : 5.0
Power Management Object : N/A
GPU Operation Mode
Current : N/A
Pending : N/A
GPU Virtualization Mode
Virtualization Mode : Pass-Through
Host VGPU Mode : N/A
IBMNPU
Relaxed Ordering Mode : N/A
PCI
Bus : 0x00
Device : 0x07
Domain : 0x0000
Device Id : 0x1DB410DE
Bus Id : 00000000:00:07.0
Sub System Id : 0x121410DE
GPU Link Info
PCIe Generation
Max : 3
Current : 3
Link Width
Max : 16x
Current : 16x
Bridge Chip
Type : N/A
Firmware : N/A
Replays Since Reset : 0
Replay Number Rollovers : 0
Tx Throughput : 0 KB/s
Rx Throughput : 0 KB/s
Fan Speed : N/A
Performance State : P0
Clocks Throttle Reasons
Idle : Not Active
Applications Clocks Setting : Not Active
SW Power Cap : Not Active
HW Slowdown : Not Active
HW Thermal Slowdown : Not Active
HW Power Brake Slowdown : Not Active
Sync Boost : Not Active
SW Thermal Slowdown : Not Active
Display Clock Setting : Not Active
FB Memory Usage
Total : 16160 MiB
Used : 0 MiB
Free : 16160 MiB
BAR1 Memory Usage
Total : 16384 MiB
Used : 2 MiB
Free : 16382 MiB
Compute Mode : Default
Utilization
Gpu : 3 %
Memory : 0 %
Encoder : 0 %
Decoder : 0 %
Encoder Stats
Active Sessions : 0
Average FPS : 0
Average Latency : 0
FBC Stats
Active Sessions : 0
Average FPS : 0
Average Latency : 0
Ecc Mode
Current : Enabled
Pending : Enabled
ECC Errors
Volatile
Single Bit
Device Memory : 0
Register File : 0
L1 Cache : 0
L2 Cache : 0
Texture Memory : N/A
Texture Shared : N/A
CBU : N/A
Total : 0
Double Bit
Device Memory : 0
Register File : 0
L1 Cache : 0
L2 Cache : 0
Texture Memory : N/A
Texture Shared : N/A
CBU : 0
Total : 0
Aggregate
Single Bit
Device Memory : 0
Register File : 0
L1 Cache : 0
L2 Cache : 0
Texture Memory : N/A
Texture Shared : N/A
CBU : N/A
Total : 0
Double Bit
Device Memory : 0
Register File : 0
L1 Cache : 0
L2 Cache : 0
Texture Memory : N/A
Texture Shared : N/A
CBU : 0
Total : 0
Retired Pages
Single Bit ECC : 0
Double Bit ECC : 0
Pending Page Blacklist : No
Temperature
GPU Current Temp : 29 C
GPU Shutdown Temp : 90 C
GPU Slowdown Temp : 87 C
GPU Max Operating Temp : 83 C
Memory Current Temp : 26 C
Memory Max Operating Temp : 85 C
Power Readings
Power Management : Supported
Power Draw : 35.39 W
Power Limit : 250.00 W
Default Power Limit : 250.00 W
Enforced Power Limit : 250.00 W
Min Power Limit : 100.00 W
Max Power Limit : 250.00 W
Clocks
Graphics : 1245 MHz
SM : 1245 MHz
Memory : 877 MHz
Video : 1132 MHz
Applications Clocks
Graphics : 1245 MHz
Memory : 877 MHz
Default Applications Clocks
Graphics : 1245 MHz
Memory : 877 MHz
Max Clocks
Graphics : 1380 MHz
SM : 1380 MHz
Memory : 877 MHz
Video : 1237 MHz
Max Customer Boost Clocks
Graphics : 1380 MHz
Clock Policy
Auto Boost : N/A
Auto Boost Default : N/A
Processes : None

• Docker version from docker version
  Client: Docker Engine - Community
  Version: 19.03.8
  API version: 1.40
  Go version: go1.12.17
  Git commit: afacb8b7f0
  Built: Wed Mar 11 01:25:58 2020
  OS/Arch: linux/amd64
  Experimental: false

Server: Docker Engine - Community
Engine:
Version: 19.03.9
API version: 1.40 (minimum version 1.12)
Go version: go1.13.10
Git commit: 9d988398e7
Built: Fri May 15 00:24:07 2020
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.2.13
GitCommit: 7ad184331fa3e55e52b890ea95e65ba581ae3429
runc:
Version: 1.0.0-rc10
GitCommit: dc9208a3303feef5b3839f4323d9beb36df0a9dd
docker-init:
Version: 0.18.0
GitCommit: fec3683

• NVIDIA packages version from dpkg -l '*nvidia*' or rpm -qa '*nvidia*'
  or: command not found
  Desired=Unknown/Install/Remove/Purge/Hold
  | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
  |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
  ||/ Name Version Architecture Description
  +++-============================-===================-===================-=============================================================
  ii libnvidia-container-tools 1.0.7-1 amd64 NVIDIA container runtime library (command-line tools)
  ii libnvidia-container1:amd64 1.0.7-1 amd64 NVIDIA container runtime library
  un nvidia-legacy-340xx-vdpau-dr (no description available)
  un nvidia-libopencl1-dev (no description available)
  un nvidia-vdpau-driver (no description available)
  dpkg-query: no packages found matching nvidiarpm
  dpkg-query: no packages found matching -qa

• NVIDIA container library version from nvidia-container-cli -V
  version: 1.0.7
  build date: 2020-01-21T19:00+00:00
  build revision: b71f87c04b8eca8a16bf60995506c35c937347d9
  build compiler: gcc-5 5.4.0 20160609
  build platform: x86_64
  build flags: -D_GNU_SOURCE -D_FORTIFY_SOURCE=2 -DNDEBUG -std=gnu11 -O2 -g -fdata-sections -ffunction-sections -fstack-protector -fno-strict-aliasing -fvisibility=hidden -Wall -Wextra -Wcast-align -Wpointer-arith -Wmissing-prototypes -Wnonnull -Wwrite-strings -Wlogical-op -Wformat=2 -Wmissing-format-attribute -Winit-self -Wshadow -Wstrict-prototypes -Wunreachable-code -Wconversion -Wsign-conversion -Wno-unknown-warning-option -Wno-format-extra-args -Wno-gnu-alignof-expression -Wl,-zrelro -Wl,-znow -Wl,-zdefs -Wl,--gc-sections

• NVIDIA container library logs (see troubleshooting)

• Docker command, image and tag used
  https://ngc.nvidia.com/catalog/containers/nvidia:pytorch

[klueska]

This doesn't look specific to nvidia-docker or nvidia-container-toolkit. It looks like you have broken dependencies in some of the other packages on your system (that also happen to be related to nvidia/cuda, but are outside of the dependency graph for nvidia-docker).

My guess is you would see these errors for any package you were trying to install, not just nvidia-container-toolkit.

I would recommend filing an issue with the maintainer of the repository where you tried to install the nvidia/cuda drivers from. As far as I know, we (NVIDIA) don't provide packages for the driver, and only recommend installing it from https://www.nvidia.com/Download/index.aspx?lang=en-us

[terryyizhong]

I installed the driver just from the official website you provided. Yes, I see these errors for any package you were trying to install.
I try to fix using apt-get -f install. It seems try to installing CUDA10.1, But also got error when installing dependencies