Problem with ./run_docker

[hrishi16]

[hrishikesh@gulgula DeepCutAnalysis]$ ./run_docker.sh 
+ set -e
+ GPU=1
+ nvidia-docker run -v /home1/hrishikesh/Work:/root/Work --net=host --env=DISPLAY --volume=/home1/hrishikesh/.Xauthority:/root/.Xauthority:rw -v /tmp/.X11-unix:/tmp/.X11-unix -it bhallalab/deeplabcut2:latest /bin/bash
docker: Error response from daemon: OCI runtime create failed: container_linux.go:344: starting container process caused "process_linux.go:424: container init caused \"process_linux.go:407: running prestart hook 0 caused \\\"error running hook: exit status 1, stdout: , stderr: exec command: [/usr/bin/nvidia-container-cli --load-kmods configure --ldconfig=@/sbin/ldconfig --device=all --compute --utility --require=cuda>=9.0 --pid=478 /var/lib/docker/overlay2/a876935e7e60b8785b34fc882660f2d574a8dbc51446c84ea9047bca28752453/merged]\\\\nnvidia-container-cli: initialization error: nvml error: driver/library version mismatch\\\\n\\\"\"": unknown.

[dilawar]

Related to https://stackoverflow.com/questions/43022843/nvidia-nvml-driver-library-version-mismatch and NVIDIA/nvidia-docker#365

[dilawar]

I have upgraded the system but it needs to rebooted. Some users have suggested that a reboot is must after upgrade. When no one is using the system, please reboot and check. If it does not work, update the ticket.

[dilawar]

$ dmesg | tail 
NVRM: this kernel module has the version 410.48.  Please
NVRM: make sure that this kernel module and all NVIDIA driver
NVRM: components have the same version.
[6828552.840331] SELinux: 2048 avtab hash slots, 112532 rules.
[6828552.891132] SELinux: 2048 avtab hash slots, 112532 rules.
[6828552.917294] SELinux:  8 users, 14 roles, 5036 types, 317 bools, 1 sens, 1024 cats
[6828552.917297] SELinux:  129 classes, 112532 rules
[6830178.461824] NVRM: API mismatch: the client has the version 418.39, but
NVRM: this kernel module has the version 410.48.  Please
NVRM: make sure that this kernel module and all NVIDIA driver
NVRM: components have the same version.
[6830202.950174] NVRM: API mismatch: the client has the version 418.39, but
NVRM: this kernel module has the version 410.48.  Please
NVRM: make sure that this kernel module and all NVIDIA driver
NVRM: components have the same version.
[6830432.255374] NVRM: API mismatch: the client has the version 418.39, but
NVRM: this kernel module has the version 410.48.  Please
NVRM: make sure that this kernel module and all NVIDIA driver
NVRM: components have the same version.
[6830434.835355] NVRM: API mismatch: the client has the version 418.39, but
NVRM: this kernel module has the version 410.48.  Please
NVRM: make sure that this kernel module and all NVIDIA driver
NVRM: components have the same version.

So we need a reboot so the new kernel and load. Ask Bhanu/Others if they are using the machine.

[hrishi16]

it worked