Docker Desktop 2.3.02 host volume mount performance

[ghost]

Expected behavior

Windows 10 host mounted volumes performance which is comparable with previous versions of Docker Desktop.

Actual behavior

performance of file i/o against Windows 10 host mounted volumes appears to have taken an extreme turn for the worse.

In my case I'm performing a "build" which involves 70k+ files and some tarring and untarring. The results of this build in various environments are as follows:
12 minutes - Straight Windows 10 (git bash)
19 minutes - Docker Desktop 2.1.0.5 against host mounted volume (Windows /c drive)
71 minutes - Docker Desktop 2.3.0.2 against host mounted volume (Windows /c drive)
2 minutes - Docker Desktop 2.1.0.5 against container file system
2 minutes - Docker Desktop 2.3.0.2 against container file system

Information

Please, help us understand the problem. For instance:

• Is it reproducible?
  Yes
• Is the problem new?
  Yes
• Did the problem appear with an update?
  Yes
• A reproducible case if this is a bug, Dockerfiles with reproduction inside is best.
  I've not attemped to simplify this yet. But a large zip/tar file being unzipped and then rezipped would likely show the issue

-->

• Windows Version:
  10
• Docker Desktop Version:
  2.3.0.2
• Are you running inside a virtualized Windows e.g. on a cloud server or on a mac VM:
  No

[Geekimo]

Got the same problem using a php container to run Symfony framework.
Displaying application homepage previously took ~600ms, now it takes about 11s.
Running wsl2 containers with wsl2 update as required.

[mat007]

Thanks for the report, is this with the WSL 2 or the Hyper-V backend?
Could you provide diagnostics (Troubleshoot window from the systray menu)?

[Geekimo]

Hi,
mine is using WSL2, and here is the diagnosis id -> 947B4C0B-BE38-4B2B-89B7-8E9ABD5D4989/20200516103831

[ghost]

I don't have diagnostics handy but will try and pull that together on Monday.

My original numbers were Hyper-V

I've since switched to WSL2 and the numbers when using Windows volume mount are still what I described originally.

Not surprisingly if I use a mount to file system in WSL then the numbers are quite good. Essentially the same as if I had everything directly in the container itself.

[Geekimo]

Disabling WSL2 and going back to the Hyper-V backend made me recover decent performances.

[simonferquel]

Bind mounting files from Windows into wsl2 is very costly indeed. The best practice about bind mounts with the WSL 2 backend is to fully embrace WSL 2 and host your source files in your WSL distro instead of on the Windows file system. In this scenario, bind mounts performance is much better than with the Hyper-V backend.

[bashess]

Same problem here with Hyper-V backend. Since update from 2.2.0.0 build time raised from 12 minutes to 35 minutes.

[simonferquel]

Hi all,

Can you please describe the reason why you could not put your source files inside you wsl 2 distro filesystem ? Doing this bring you:

• Linux-native-like IO performance and features (including kernel caching, inotify support, uid/gid and permissions capabilities)
• Running your build scripts and tools from a real Linux environment instead of a Windows environment.

I know that some IDEs have limitations with working with the \wsl$ file share, and would really like to author IDE-specific workarounds and guidance for having the best development experience as well as running performances.

[Geekimo]

I'm doing PHP, I do not need to build my code every time to test it.

The problem is about the fact that we always worked like this under Windows, then suddenly we have to do it an other way without any clear information or complete documentation.

At this point, I load my code from a volume specified in my docker-compose.yaml. How should I declare it now to use the advantages of WSL2 ?

[simonferquel]

is it a named volume, or a mount from the Windows host ?

If it is a named volume, you can continue like this.
If it is a mount from the windows host, you should:

• move your sources to a wsl2 distro (e.g. from Ubuntu: copy /mnt/c/sources ~/sources)
• use the docker and docker-compose from your wsl2 distro (cd ~/sources && docker-compose up

If you use Visual Studio Code, you can run it in "Remote WSL mode" (just run code . from your wsl distro, and it will run vs code UI connecting to the WSL instance)

Otherwise, other editors can access your WSL 2 files via the network share \\wsl$\<distro name\

[Geekimo]

Second case, mount from Windows.
I'll try later and let you know.

[ghost]

@simonferquel Couldn't agree more that keeping source in WSL 2 and using docker/docker-compose from within that space is a great performance improvement. Even over native Windows.

I did go down that WSL 2 path as indicated by the numbers I posted. However Windows build 2004 is not even generally available yet. I had to go WIndows Insider Preview to use WSL 2 and provide those numbers.

I work in a large enterprise and had to get special permission to even try the Insider Preview and get WSL 2. Even though Windows build 2004 will be generally available on May 28th that doesn't mean developers in large corporations will have access to it. Before they drop it in place for general usage it will go through a wide array of security and compatibility testing internally which could put it another 6 months off.

But it is pretty hard to stomach that without encouraging thousands of developers to ask for Insider Preview permissions and changing up the way everyone works that DD 2.3.02 is ~6 times slower than native Windows and ~3 times slower than DD 2.1.0.5.

[simonferquel]

@jtoe using the Hyper-V backend, you should see performance improvements compared to DD 2.1.0.5. It is only the WSL 2 backend that suffers this performance issue when mounting windows files.

This is because our hyper-v backend comes with GRPC Fuse, which we cannot enable on WSL 2 backend, due to dependencies on some Linux Kernel patches.

In term of performance, you should see things in this order:

• 2.3.0.2 WSL 2 backend mounting files from Linux file system
• 2.3.0.2 Hyper-V backend mounting files from Windows
• 2.1.0.5 Hyper-V backend mounting files from Windows
• 2.3.0.2 WSL 2 backend mounting files from Windows

[ghost]

@simonferquel - The 2.3.0.2 Hyper-v backend is where we don't appear to be on the same page.

• 1 minute 32 seconds - 2.3.0.2 WSL 2 backend mounting files from Linux file system
• 19 minutes 20 seconds - 2.1.0.5 Hyper-V backend mounting files from Windows
• 1 hour 11 minutes - 2.3.0.2 Hyper-V backend mounting files from Windows

This has been prevalent with many people in my organization as we try and move to 2.2+

[Telokis]

I can confirm that I experience a similar issue on my end: ugprading to 2.3.0.3 completely destroyed the performance of my bind mounts on my Windows server. (I can't migrate at all).
My diagnostic ID is 727C28F9-6E10-4A01-B7AD-759A812C5EAB/20200701092809.

I have opened a forum post explaining the details of what happened.
https://forums.docker.com/t/extremely-slow-bind-mounts-with-docker-desktop-2-3-0-3-on-windows-server-2019-1809/96002

If you can't read, I'll copy/paste the content here:

I have a big issue after upgrading Docker Desktop to 2.3.0.3.

Earlier today, my Docker for Windows crashed on a production instance running Windows Server 2019 (1809).
I had absolutely no idea what the issue was so I decided to shutdown Docker completely and restart it.
Once it restarted, it told me an update was available. I noticed a lot of bugfixes and decided it was probably worth it since I hadn't updated for at least eight months.

After updating, I had a lot of issues with bind mounts and my containers complaining they couldn't access the files. I added the whole drives to the Docker Desktop "resources/File sharing" setting and rebooted. It still didn't work and after looking around on the internet, I saw someone mention something about Windows permissions. I decided to simply allow full control to the group "users" for my whole drives. It didn't fix the issue even after rebooting but after some time it started working again (no idea what happened).

Once all my containers worked, I noticed one of them was extremely slow.
In our setup, we are progressively migrating from a Windows setup to a linux/containerized one. We are using a Windows Server because our CMS only runs on Windows. We use a bind mount in order for our containers to access the 500GB (around 1 000 000 files) of data managed by the CMS.

One of these containers in particular, will try to calculate the size of some directories among those files. For example, a directory containing 3 868 files organized into 8 folders for a total of arount 800MB of data. Nothing too insane there.

Before the Docker upgrade happened, the container did its thing for every directory in about 2 minutes. Handling the specific directory I mentioned above took less than one second overall.

Now that we upgraded Docker, it takes around one hour for the exact same data. For the small directory it takes around 2 minutes. Just performing a readdir (in nodejs) takes 12 seconds. And we only do 20 stats per second at most. Extremely slow, we were very surprised.

I'd like to insist on the fact that we absolutely didn't change the files at all. The only difference was that we applied "Full control" permission to all of them and updated Docker Desktop to the most recent version (2.3.0.3).

Would someone more experienced have any idea what is causing this odd behavior?
Is there a way to revert to a previous Docker version if no other solution is possible?

I thought that Docker might be performing some checks in order to make sure we access the host's files properly. If it is the case, is there a way to disable these?

I know my architecture might sound a bit silly but please keep in mind that we are transitioning and I'm doing the best I can given the requirements I need to meet.
I can't move the CMS into a linux container and the containers need access to those files.

Thanks in advance for reading my message.

Sincerely,
Telokis

I don't use WSL, I'm still on the Hyper-V backend (and I don't want to migrate for now, I'm too scared of the consequences).

[elliotm00]

I'm having a similar problem. I'm trying to run Nextcloud in Docker containers using a Windows host and WSL2 backend. I need to put the data on a separate drive, and there is no way to mount a second VHDX drive or a raw ext4 drive to WSL2. My only option is to mount a Windows volume as the data drive. I hope there is something that can be done in a future version of DD to improve the performance across the Windows mount.

[kmmizouri]

suffering the same issue after Upgrading to WSL2...... export process is taking more than 40min instead of 5min with the Legacy (Hyper-V).... this is really too annoying

[docker-robott]

Issues go stale after 90 days of inactivity.
Mark the issue as fresh with /remove-lifecycle stale comment.
Stale issues will be closed after an additional 30 days of inactivity.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle stale

[ghost]

/remove-lifecycle stale

[stephen-turner]

Unfortunately we (Docker) can't fix the mount speed from files on the Windows side to the WSL 2 backend, because it's not our code, it's wholly managed by Windows. We do document the issues, see https://docs.docker.com/docker-for-windows/wsl/. There are tickets for it at microsoft/WSL#873 and microsoft/WSL#4197.

[stephen-turner]

Sorry, I closed this without reading the whole ticket: it looks like there is more than one issue here. I've reopened this ticket for the Hyper-V backend.

[ghost]

Thanks @stephen-turner. My originally reported issue was entirely about Hyper-V.

[docker-robott]

Issues go stale after 90 days of inactivity.
Mark the issue as fresh with /remove-lifecycle stale comment.
Stale issues will be closed after an additional 30 days of inactivity.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle stale

[docker-robott]

Closed issues are locked after 30 days of inactivity.
This helps our team focus on active issues.

If you have found a problem that seems similar to this, please open a new issue.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle locked