nvidia-docker | 2016/01/21 20:42:37 Error: link /usr/bin/nvidia-cuda-mps-control /home/docker/volumes/nvidia_driver_352.68/_data/bin/nvidia-cuda-mps-control: invalid cross-device link

[carlodelmundo-zz]

Hi folks,

Been using nvidia-docker and noticed a new upgrade. I'm using a CentOS 7 box and I followed the instructions as follows:

542 sudo make install
543 sudo nvidia-docker volume setup

When I run volume setup, I'm met with this error:

nvidia-docker | 2016/01/21 20:42:37 Error: link /usr/bin/nvidia-cuda-mps-control /home/docker/volumes/nvidia_driver_352.68/_data/bin/nvidia-cuda-mps-control: invalid cross-device link

Any ideas?

[flx42]

Please look at the known limitations:

NVIDIA driver installation needs to exist on the same partition as the root of the Docker runtime.
nvidia-docker internally needs to create hard links to some driver files. Because of this, you need the NVIDIA driver (usually found under /usr) to be on the same partition as the Docker root (/var by default). Possible workarounds includes installing your NVIDIA driver at a different location (see --advanced-options of the installer) or changing your Docker root directory (see docker daemon -g)

Or, you can use nvidia-docker-plugin, it has the same limitation but you can fix it with nvidia-docker-plugin -d.

Please confirm this is indeed the problem you are having!

[japrogramer]

@flx42 is the -v supposed to be the -d symbol for nvidia-docker-plugin? if so i don't know to what i should set it, would /var/lib/docker work?

--- ~ » tail -f /tmp/nvidia-docker.log                  
NVIDIA Docker plugin: 1.0.1 
^[a^C                       
--- ~ » whereis docker                                                                                    130 ↵ 
docker: /usr/bin/docker /etc/docker /usr/share/man/man1/docker.1.gz                                             
--- ~ » lsblk               
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT             
sda      8:0    0 149.1G  0 disk                        
├─sda1   8:1    0 108.5G  0 part                        
└─sda2   8:2    0  40.6G  0 part                        
sdb      8:16   0 223.6G  0 disk                        
├─sdb1   8:17   0   511M  0 part /boot                  
└─sdb2   8:18   0 223.1G  0 part /                      
sdc      8:32   0 931.5G  0 disk                        
├─sdc1   8:33   0    30G  0 part /var                   
├─sdc2   8:34   0   500G  0 part /home/archangel/Video  
└─sdc3   8:35   0 401.5G  0 part                        
sr0     11:0    1  1024M  0 rom                         
--- ~ » pacman -Ql nvidia   
nvidia /usr/                
nvidia /usr/lib/            
nvidia /usr/lib/modprobe.d/ 
nvidia /usr/lib/modprobe.d/nvidia.conf                  
nvidia /usr/lib/modules/    
nvidia /usr/lib/modules/extramodules-4.11-ARCH/         
nvidia /usr/lib/modules/extramodules-4.11-ARCH/nvidia-drm.ko.gz                                                 
nvidia /usr/lib/modules/extramodules-4.11-ARCH/nvidia-modeset.ko.gz                                             
nvidia /usr/lib/modules/extramodules-4.11-ARCH/nvidia-uvm.ko.gz                                                 
nvidia /usr/lib/modules/extramodules-4.11-ARCH/nvidia.ko.gz                                                     
--- ~ » cd /var/lib/docker 
--- lib/docker » ls
ls: cannot open directory '.': Permission denied
--- lib/docker » sudo sh -c ls                                                                                                                                                                                                2 ↵
[sudo] password for archangel: 
containers  image  network  overlay2  plugins  swarm  tmp  trust  volumes
--- lib/docker » 

EDIT: ok, i understand now;
ended up using dockers -g in daemon.json and the -d in the nvidia-docker-plugin set them both to /docker which falls in the same partition as /usr

[flx42]

Yes, sorry, I meant -d, I edited my post above.