New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NFS Performance on boot2docker is much worse than on other VMs #1022

Open
jsingle opened this Issue Aug 5, 2015 · 7 comments

Comments

Projects
None yet
5 participants
@jsingle
Copy link

jsingle commented Aug 5, 2015

I've been trying to rig up a local dev setup using an NFS client on my boot2docker VM, but I've noticed that read performance is far slower than is should be for some workloads. An example of one such access pattern is scanning many smallish files, like a grep through a code repo.

I've run the same file access patterns on some other linux VMs that I've set up manually, and the performance on those VMs has generally been an order of magnitude better than on boot2docker. One of these VMs was Tiny Core Linux v6.3 (on which boot2docker is based), so this does indeed seem to be a product of boot2docker's specific setup.

To reproduce the issue:

  • Run the script I made here to create a directory at ~/nfs_test for mounting
  • Add the directory to your /etc/exports; an example entry would be /Users/<username>/nfs_test/ 192.168.59.103 -alldirs -maproot=501:20
  • On your VM (boot2docker or otherwise), mount the directory, for example with sudo mount -t nfs 192.168.59.3:/Users/<mac-username>/nfs_test/ nfs_test/
  • On your VM, run time grep -r bacon * from inside nfs_test.

Typical results for me:

  • boot2docker: 90 seconds first time, 60 seconds subsequent times
  • Other VMs: 7 seconds first time, 4 seconds subsequent times

You can also run nfsstat -w 1 on your host machine to see what NFS is doing. The types of operations performed is the same for each VM, but for boot2docker VMs the number of operations per second is much smaller. This makes me wonder if a network issue is the culprit.

Some setup details (happy to provide more):
host OSX 10.10.4 with SSD
boot2docker v1.7.1

I'm going to investigate this issue a bit more, since it makes NFS unusable for us, and I'll definitely follow up here if I find out more.

@jsingle

This comment has been minimized.

Copy link

jsingle commented Aug 6, 2015

It seems like switching the Virtualbox network "Adapter Type" from Paravirtualized to PCnet improves this performance greatly. This might be because of missing virtio kernel modules

@jsingle

This comment has been minimized.

Copy link

jsingle commented Aug 6, 2015

Another theory is that it's due to this Virtualbox bug:
https://www.virtualbox.org/ticket/10157

@bastnic

This comment has been minimized.

Copy link

bastnic commented Aug 10, 2015

@jsingle nice catch! On my team we have three last generation of macbook pro with last version of everything on it. On our current Symfony2 dev project, using "PCnet-FAST |||" instead of "Paravirtualized", our homepage on "app_dev.php" environment just go from ~220ms to ~95ms.

FYI, our settings now:

Our settings

@thieman

This comment has been minimized.

Copy link

thieman commented Aug 10, 2015

I've been working with @jsingle on investigating this problem. We're seeing really significant network performance gains after moving our NICs to PCnet-FAST III, including about an order of magnitude improvement on NFS performance.

It'd be great if other people could try this out and see if they're noticing the same improvements. If this bug is widespread, which we believe it is, we should look into changing the defaults to PCnet-FAST III.

To re-state the issue: there is a known VBox bug where having multiple virtual CPUs and using Paravirtualized networking (virtio) results in network performance degradation. Both conditions need to hold for the bug to present itself, virtio with only 1 CPU works just fine, as does multiple CPUs with a different network interface. Since multiple CPUs is necessary for compute performance, we prefer moving the network interface to PCnet-FAST III as a solution.

@thieman

This comment has been minimized.

Copy link

thieman commented Aug 10, 2015

Some additional information regarding Docker Machine since the two projects are linked: the default Docker Machine setup does not seem to exhibit this problem. Starting a new box with the VirtualBox driver in Machine gives you 1 CPU and one of the Intel NIC drivers, so it should be sidestepping the problem entirely.

@thieman

This comment has been minimized.

Copy link

thieman commented Aug 10, 2015

Pinging @steeve who merged the original PR moving to virtio: #71

Do you recall the performance tests you ran to reach your conclusion in that PR? Do you remember if b2d was defaulting to multiple CPUs at the time? I'm curious whether you see improvements now from moving to a different adapter.

@willdady

This comment has been minimized.

Copy link

willdady commented Oct 6, 2015

Just want to chime in to confirm that I've seen significant performance gain by switching Adapter 2 from the default Intel PRO/1000 MT Desktop to PCnet-FAST III on OSX. VM is set to 4 cpus + 8GB RAM. I've enabled nfs by using docker-machine-nfs.

My particular use-case is I'm trying to run prospector (code analysis tool for Python). Before the switch it took a ridiculous 11 minutes to complete vs 3.5 minutes after changing adapter. When running directly on the host it takes 74 seconds.

cezarsa added a commit to tsuru/tsuru-client that referenced this issue Feb 15, 2017

@wglambert wglambert added the Issue label Jul 9, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment