Verify NFS mount fails #22

Closed
huggsboson opened this Issue Nov 27, 2015 · 36 comments

Projects

None yet

7 participants

@huggsboson

In the newest version (as of today, though ) using the 1.9.0a packages I'm getting a verifying NFS mount error on first run through. Though it works on the second invocation.

Here's the output of docker-machine-nfs default:
[INFO] machine presence ... OK
[INFO] machine running ... OK
[INFO] Lookup mandatory properties ... OK

- Machine IP: 192.168.99.100
- Network ID: vboxnet1
- NFSHost IP: 192.168.99.1

[INFO] Configure NFS ...

!!! Sudo will be necessary for editing /etc/exports !!!
Password:
OK
[INFO] Configure Docker Machine ... OK
[INFO] Restart Docker Machine ... OK
[INFO] Verify NFS mount ... FAIL

Cannot detect the NFS mount :(

@dimasch
dimasch commented Nov 29, 2015

Excellent! Fixed in last pull request #19.

1448789032791

@huggsboson

Hi @gdemad. I believe I was on the latest pull request. Can you try tearing down your docker-machine, starting a new one and try running again? In my original report I stated:

I'm getting a verifying NFS mount error on first run through. Though it works on the second invocation.

Which based on your message "NFS already mounted". Looks like you are on the second run through.

@dimasch
dimasch commented Nov 29, 2015

Hi @box-jhuffaker. Sorry, yes, looks like on the second run.
This issue is reproduce in first run.

Thx.

@huggsboson

@gdemad No problem, thanks for confirming!

@huggsboson

Few things, I spent the morning debugging this. I somehow got to a point where it eventually wouldn't ever mount the drive on one of my VMs (I cleared /etc/exports and re created and things came together). But generally if I keep restarting the VM it will eventually mount it. (I verify by running the df command like the script). I'm not sure if there's a race between nfsd and restarts or some other issue.

@huggsboson

I tried moving the restart into the verify loop, but it didn't fix my problem. Which debunks my theory about restarts.

@huggsboson

I should note: I'm on OSX El Capitan

@huggsboson

I tried moving the nfsd restart into the verify loop. Still not working.

@huggsboson

I tried screwing around. Adding an nfsd update... moving boot local to /opt/bootlocal.sh (which seems to be the preferred location on boot2docker 1.9.1 which docker-machine 0.5.1a seems go come with. Also explicitly cleard ~/.docker/machine/cache to remove the boot2docker.iso and caused a re-download.

Basically I'm shooting blindly at things. I'm happy to try things for people that can actually aim fixes, as my machine is reproing.

@huggsboson

I notice this in /var/log/bootlocal.log:
Starting nfs client utilities.
mount: RPC: Remote system error - Connection timed out
mount: mounting 192.168.99.1:/Users on /Users failed: Bad file descriptor

I'm not sure what state my nfsd was when I started the VM. But this verifies that the bootlocal.sh is in the right place.

@dimasch
dimasch commented Nov 29, 2015

I try remove docker machine and create new instance. See the same result, after start docker-machine-nfs:
1448822486543

Tested with last release updates. Thx @box-jhuffaker โœ…

@dimasch
dimasch commented Nov 29, 2015

Possible workaround:

I try open new terminal session and see message:

Error running connection boilerplate: Error checking and/or regenerating the certs: There was an error validating certificates for host "192.168.99.100:2376": dial tcp 192.168.99.100:2376: i/o timeout
You can attempt to regenerate them using 'docker-machine regenerate-certs name'.
Be advised that this will trigger a Docker daemon restart which will stop running containers.
  1. docker-machine regenerate-certs dev
  2. docker-machine-nfs dev

Nice mounted! ๐Ÿ”ฎ
d632e508f7

@huggsboson

I assume this is just because it's the second run. If you remove the
regenerate certs step and do the process. I bet it will also work.
On Nov 29, 2015 12:02 PM, "Dmitry Schegolihin" notifications@github.com
wrote:

Possible workaround:

I try open new terminal session and see message:

Error running connection boilerplate: Error checking and/or regenerating the certs: There was an error validating certificates for host "192.168.99.100:2376": dial tcp 192.168.99.100:2376: i/o timeout
You can attempt to regenerate them using 'docker-machine regenerate-certs name'.
Be advised that this will trigger a Docker daemon restart which will stop running containers.

  1. docker-machine regenerate-certs dev
  2. docker-machine-nfs dev

Nice mounted! [image: ๐Ÿ”ฎ]
[image: d632e508f7]
https://cloud.githubusercontent.com/assets/414067/11459302/9114085e-96e4-11e5-99c9-21effc6e61a5.jpg

โ€”
Reply to this email directly or view it on GitHub
#22 (comment)
.

@tonivdv
Member
tonivdv commented Nov 30, 2015

Hey guys,

I upgraded to latest docker engine, with latest docker machine, latest virtualbox on OS X El Capitan and it all works without issues ...

$ dm create --driver virtualbox tst
Running pre-create checks...
Creating machine...
(tst) OUT | Creating VirtualBox VM...
(tst) OUT | Creating SSH key...
(tst) OUT | Starting VirtualBox VM...
(tst) OUT | Starting VM...
Waiting for machine to be running, this may take a few minutes...
Machine is running, waiting for SSH to be available...
Detecting operating system of created instance...
Detecting the provisioner...
Provisioning created instance...
Copying certs to the local machine directory...
Copying certs to the remote machine...
Setting Docker configuration on the remote daemon...
To see how to connect Docker to this machine, run: docker-machine env tst
$ docker-machine-nfs tst
[INFO] machine presence ...             OK 
[INFO] machine running ...          OK 
[INFO] Lookup mandatory properties ...      OK 

    - Machine IP: 192.168.99.101 
    - Network ID: vboxnet4 
    - NFSHost IP: 192.168.99.1 

[INFO] Configure NFS ... 

 !!! Sudo will be necessary for editing /etc/exports !!! 
Password:
                        OK 
[INFO] Configure Docker Machine ...         OK 
[INFO] Restart Docker Machine ...       OK 
[INFO] Verify NFS mount ...             OK 

--------------------------------------------

 The docker-machine 'tst'
 is now mounted with NFS!

 ENJOY high speed mounts :D

--------------------------------------------

Are you sure the docker-machine-nfs script is up to date on your machine?

@dimasch
dimasch commented Nov 30, 2015

Really, after upgrade docker engine, see nice work!

โžœ  ~  docker -v
Docker version 1.9.1, build a34a1d5
โžœ  ~  docker-machine -v
docker-machine version 0.5.1 (7e8e38e)
@tonivdv
Member
tonivdv commented Nov 30, 2015

@gdemad and it still doesn't work?

@dimasch
dimasch commented Nov 30, 2015

@tonivdv no, It works fine! ๐Ÿ‘

Can be closed this issue.

Thx,

@tonivdv
Member
tonivdv commented Nov 30, 2015

Okay great :)

@tonivdv tonivdv closed this Nov 30, 2015
@huggsboson

Broken for me...

Macbook pro, retina.
VirtualBox 5.0.10 r104061. I installed the extensions.

jhuffaker@mbp-003138 ~ $ docker -v
Docker version 1.9.1, build a34a1d5
jhuffaker@mbp-003138 ~ $ docker-machine -v
docker-machine version 0.5.1 (7e8e38e)
jhuffaker@mbp-003138 ~ $ curl https://raw.githubusercontent.com/adlogix/docker-machine-nfs/master/docker-machine-nfs.sh | sudo tee /usr/local/bin/docker-machine-nfs > /dev/null && sudo chmod +x /usr/local/bin/docker-machine-nfs
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 9229 100 9229 0 0 11397 0 --:--:-- --:--:-- --:--:-- 11393
jhuffaker@mbp-003138 ~ $ docker-machine create --driver=virtualbox tst && /usr/local/bin/docker-machine-nfs tst
Running pre-create checks...
Creating machine...
(tst) OUT | Creating VirtualBox VM...
(tst) OUT | Creating SSH key...
(tst) OUT | Starting VirtualBox VM...
(tst) OUT | Starting VM...
Waiting for machine to be running, this may take a few minutes...
Machine is running, waiting for SSH to be available...
Detecting operating system of created instance...
Detecting the provisioner...
Provisioning created instance...
Copying certs to the local machine directory...
Copying certs to the remote machine...
Setting Docker configuration on the remote daemon...
To see how to connect Docker to this machine, run: docker-machine env tst
[INFO] Configuration:

- Machine Name: tst
- Shared Folder: /Users
- Force: false

[INFO] machine presence ... OK
[INFO] machine running ... OK
[INFO] Lookup mandatory properties ... OK

- Machine IP: 192.168.99.114
- Network ID: vboxnet1
- NFSHost IP: 192.168.99.1

[INFO] Configure NFS ...

!!! Sudo will be necessary for editing /etc/exports !!!
OK
[INFO] Configure Docker Machine ... OK
[INFO] Restart Docker Machine ... OK
[INFO] Verify NFS mount ... FAIL

Cannot detect the NFS mount :(

@huggsboson

Here's /etc/exports:
jhuffaker@mbp-003138 ~ $ cat /etc/exports

"/Users" 192.168.99.112 -alldirs -mapall=563785388:1221986466
"/Users" 192.168.99.113 -alldirs -mapall=563785388:1221986466
/Users 192.168.99.114 -alldirs -mapall=563785388:1221986466

@huggsboson

Here's the tst bootlocal log:
jhuffaker@mbp-003138 ~ $ docker-machine ssh tst "cat /var/log/bootlocal.log"
Starting nfs client utilities.

@huggsboson

I ran this manually from the command line:
jhuffaker@mbp-003138 ~ $ docker-machine ssh tst "sudo mount -t nfs -o noacl,async 192.168.99.1:/Users /Users"

Just seems to hang.

@huggsboson

boot2docker info:
Boot2Docker version 1.9.1, build master : cef800b - Fri Nov 20 19:33:59 UTC 2015
Docker version 1.9.1, build a34a1d5

@huggsboson

I'm starting to think my VM just can't reach my host laptop for some reason.

@huggsboson

It just takes a while. Both bootlocal.log and my manually run mount command ended with:
jhuffaker@mbp-003138 ~ $ docker-machine ssh tst "cat /var/log/bootlocal.log"
Starting nfs client utilities.
mount: RPC: Remote system error - Connection timed out
mount: mounting 192.168.99.1:/Users on /Users failed: Bad file descriptor

@tonivdv
Member
tonivdv commented Nov 30, 2015

@box-jhuffaker Did you already reboot your mac? When nfsd was hanging this somehow unblocked the situation and never happened again.

@huggsboson

Oh I really haven't! I'll throw in a reboot and get back to you.

jhuffaker@mbp-003138 ~ $ uptime
8:12 up 13 days, 21:05, 5 users, load averages: 1.75 1.91 1.96

On Mon, Nov 30, 2015 at 7:11 AM, Toni Van de Voorde <
notifications@github.com> wrote:

@box-jhuffaker https://github.com/box-jhuffaker Did you already reboot
your mac? When nfsd was hanging this somehow unblocked the situation and
never happened again.

โ€”
Reply to this email directly or view it on GitHub
#22 (comment)
.

@huggsboson

A restart fixed it, thanks for the tip and all of your work on this script. Glad I got a pretty good root cause in the end.

@tonivdv
Member
tonivdv commented Dec 1, 2015

@box-jhuffaker glad it works. Cheers

@nhooey
Contributor
nhooey commented Dec 30, 2015

Rebooting fixed the issue for me as well, @huggsboson.

@huggsboson

Glad to hear that, @nhooey.

@madyanov

Reboot not worked for me =/

$ docker-machine create --driver virtualbox dev
...
$ docker-machine-nfs dev
[INFO] Configuration:

    - Machine Name: dev
    - Shared Folder: /Users
    - Force: false

[INFO] machine presence ...             OK
[INFO] machine running ...          OK
[INFO] Lookup mandatory properties ... OK

    - Machine IP: 192.168.99.100
    - Network ID: vboxnet0
    - NFSHost IP: 192.168.99.1

[INFO] Configure NFS ...

 !!! Sudo will be necessary for editing /etc/exports !!!
Password:
The nfsd service does not appear to be running.
Starting the nfsd service
                        OK
[INFO] Configure Docker Machine ...         OK
[INFO] Restart Docker Machine ...       OK
[INFO] Verify NFS mount ...             FAIL

Cannot detect the NFS mount :(

The docker-machine-nfs script stucks here [INFO] Restart Docker Machine ..., so I check docker-machine ls:

$ docker-machine ls
NAME   ACTIVE   URL          STATE     URL   SWARM   DOCKER    ERRORS
dev    *        virtualbox   Running                 Unknown   No IP address found 4: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 08:00:27:4d:fe:0f brd ff:ff:ff:ff:ff:ff
    inet6 fe80::a00:27ff:fe4d:fe0f/64 scope link tentative
       valid_lft forever preferred_lft forever

Additional info:

$ cat /etc/exports
/Users 192.168.99.100 -alldirs -mapall=501:20

$ docker-machine --version
docker-machine version 0.5.6, build

$ docker --version
Docker version 1.9.1, build a34a1d5
@madyanov

Second (!) run of docker-machine-nfs with patch from issue #31 (sleep 20 after create bootlocal.sh) seems to work for me.

docker@dev:~$ cat /var/lib/boot2docker/bootlocal.sh
#!/bin/sh
  sudo umount /Users
    sudo mkdir -p /Users
  sudo /usr/local/etc/init.d/nfs-client start
    sudo mount -t nfs -o noacl,async 192.168.99.1:/Users /Users
@tonivdv
Member
tonivdv commented Jan 13, 2016

@nenemoianemoi Ok thanks for sharing. I guess I will have to put those 20s until docker-machine answers my question about a potential bug on their side.

Cheers

@jmreicha

Just ran in to this issue, the sleep 20 statement mentioned in #31 fixed this. It would be nice to have this as a workaround until docker-machine folks fix the bug.

@mxl
mxl commented Apr 5, 2016

I also had this issue and the reason was "Block incoming connections" for nfsd in Preferences -> Security & Privacy" -> Firewall -> Firewall Options.... I have also unchecked option "Automatically allow signed software to receive incoming connections" there and I think that because of that OS X automatically blocks incoming connections for all software by setting "Block incoming connections", even signed. So you need to manually select "Allow incoming connections" for each blocked software.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment