Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

VolumeDriver.Create: internal error on Ubuntu 16.04 #148

Closed
dpatschke opened this issue Jul 21, 2016 · 6 comments

Comments

Projects
None yet
5 participants
@dpatschke
Copy link

commented Jul 21, 2016

I am unable to run the example given on the front page:
sudo nvidia-docker run --rm nvidia/cuda nvidia-smi

I receive the following error:
docker: Error response from daemon: create nvidia_driver_367.35: VolumeDriver.Create: internal error, check logs for details.

Details surrounding the attempted problem resolution are documented in issue #133. My nvidia drivers are installed in '/usr' but my '/var' directory is sitting on a separate partition. I have reinstalled nvidia-docker, created an override.conf file as recommended in the comments of the above issue, and rebooted.

Below is the output from the journalctl command:

Jul 21 08:52:56 Precision-Tower-7910 systemd[1]: Starting NVIDIA Docker plugin...
Jul 21 08:52:56 Precision-Tower-7910 systemd[1]: Started NVIDIA Docker plugin.
Jul 21 08:52:56 Precision-Tower-7910 nvidia-docker-plugin[1142]: /usr/bin/nvidia-docker-plugin | 2016/07/21 08:52:56 Loading NVIDIA unified memory
Jul 21 08:52:56 Precision-Tower-7910 nvidia-docker-plugin[1142]: /usr/bin/nvidia-docker-plugin | 2016/07/21 08:52:56 Loading NVIDIA management library
Jul 21 08:52:56 Precision-Tower-7910 nvidia-docker-plugin[1142]: /usr/bin/nvidia-docker-plugin | 2016/07/21 08:52:56 Discovering GPU devices
Jul 21 08:52:57 Precision-Tower-7910 nvidia-docker-plugin[1142]: /usr/bin/nvidia-docker-plugin | 2016/07/21 08:52:57 Provisioning volumes at /usr/local/nvidia-driver
Jul 21 08:52:57 Precision-Tower-7910 nvidia-docker-plugin[1142]: /usr/bin/nvidia-docker-plugin | 2016/07/21 08:52:57 Serving plugin API at /var/lib/nvidia-docker
Jul 21 08:52:57 Precision-Tower-7910 nvidia-docker-plugin[1142]: /usr/bin/nvidia-docker-plugin | 2016/07/21 08:52:57 Serving remote API at localhost:3476
@flx42

This comment has been minimized.

Copy link
Member

commented Jul 21, 2016

After the nvidia-docker run command, you have nothing else showing up in journalctl? That would be surprising.

@dpatschke

This comment has been minimized.

Copy link
Author

commented Jul 21, 2016

Yes!!! That did it ... well ... your suggestion led me to figure out what was going on.

You are correct ... I did not execute the 'run' command. However, when I did, I noticed it was failing on the 'mkdir' command. Once I gave the folder the correct permission, it worked liked a charm.

Thanks again for your help! I will close this issue and attempt to remember all the various commands to help diagnose potential issues in the future :-).

@dpatschke dpatschke closed this Jul 21, 2016

@flx42

This comment has been minimized.

Copy link
Member

commented Jul 21, 2016

Ah yes, it's a little tricky since the plugin runs under user nvidia-docker. Sorry for the trouble!

@jingpengw

This comment has been minimized.

Copy link

commented Sep 14, 2016

This works for me too, Thanks 💯

To be more clear:

systemctl edit nvidia-docker

add following:

[Service]
ExecStart=
ExecStart=/usr/bin/nvidia-docker-plugin -s $SOCK_DIR -d /usr/local/nvidia-docker-test
sudo mkdir /usr/local/nvidia-docker-test
sudo chown nvidia-docker /usr/local/nvidia-docker-test
sudo systemctl restart docker nvidia-docker
@ferrouswheel

This comment has been minimized.

Copy link

commented Feb 5, 2017

I got this after an update from 14.04 to 16.04, and after purging and reinstalling nvidia-docker. To fix I just had to do:

sudo chown -R nvidia-docker:nvidia-docker /var/lib/nvidia-docker/volumes/

@ljf1239848066

This comment has been minimized.

Copy link

commented Sep 11, 2017

create the volume mamually
nvidia-docker volume create nvidia_driver_375.66

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.