Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Shared Volumes Slow #188

Open
dimasnake opened this issue Nov 1, 2016 · 166 comments
Open

Shared Volumes Slow #188

dimasnake opened this issue Nov 1, 2016 · 166 comments

Comments

@dimasnake
Copy link

@dimasnake dimasnake commented Nov 1, 2016

Expected behavior

File access in volumes should be comparable to access times in non-volumes, similarly to Linux installations of docker

Actual behavior

File access in volumes is many times slower than on non-volumes.

Information

Version: 1.12.3-beta29.2 (8280)
Channel: Beta
Sha1: 902414df0cea7fdc85b87f0077b0106c3af9f64c
Started on: 2016/11/01 21:19:46.408
Resources: C:\Program Files\Docker\Docker\Resources
OS: Windows 10 Pro
Edition: Professional
Id: 1607
Build: 14393
BuildLabName: 14393.351.amd64fre.rs1_release_inmarket.161014-1755

Steps to reproduce the behavior

Get on the commandline of a lightweight docker container

root@a6b2e82c167b:/# dd if=/dev/zero of=test.dat bs=1024 count=100000
100000+0 records in
100000+0 records out
102400000 bytes (102 MB) copied, 0.569183 s, 180 MB/s

and mount a volume:

root@a6b2e82c167b:/var/www# dd if=/dev/zero of=test.dat bs=1024 count=100000
100000+0 records in
100000+0 records out
102400000 bytes (102 MB) copied, 5.11662 s, 20.0 MB/s

In 9 times slower.

@friism
Copy link

@friism friism commented Nov 1, 2016

Thanks for reporting. The volume mounts are implemented using a SMB share mounted over the guest/host network.

Out of curiosity, what's your development use-case that requires greater than 20 MB/s transfer-rate when using volumes?

@dimasnake
Copy link
Author

@dimasnake dimasnake commented Nov 1, 2016

I use docker for local web development. I have nginx, php-fpm, mysql containers. Speed website pages load very slow 5-10 sec.

It is possible to encrease volumes speed?

@xdesbieys
Copy link

@xdesbieys xdesbieys commented Nov 2, 2016

I have same problem.

Inside container :

root@63d3c3f00862:/# dd if=/dev/zero of=test.dat bs=1024 count=100000
100000+0 records in
100000+0 records out
102400000 bytes (102 MB) copied, 0.208075 s, 492 MB/s

Insidde container share folder :

root@63d3c3f00862:/var/lib/mysql# dd if=/dev/zero of=test.dat bs=1024 count=100000
100000+0 records in
100000+0 records out
102400000 bytes (102 MB) copied, 11.572 s, 8.8 MB/s

Docker info :

PS C:\Users\????\Docker> docker info
Containers: 4
 Running: 4
 Paused: 0
 Stopped: 0
Images: 11
Server Version: 1.12.3
Storage Driver: overlay2
 Backing Filesystem: extfs
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 4.4.27-moby
Operating System: Alpine Linux v3.4
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 1.919 GiB
Name: moby
ID: DBIQ:SFE4:ZUSS:AEDJ:4KJC:ODIJ:W4L7:33D7:QUB7:OE4R:YEIE:UANK
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 41
 Goroutines: 78
 System Time: 2016-11-02T12:20:15.4120124Z
 EventsListeners: 1
Registry: https://index.docker.io/v1/
WARNING: No kernel memory limit support
Experimental: true
Insecure Registries:
 127.0.0.0/8

Docker volume inspect :

PS C:\Users\????\Docker> docker volume inspect 7c27e585d8d4f55c34a34f6a47d5c0687f0851fc55765b096183f6ee327ea609
[
    {
        "Name": "7c27e585d8d4f55c34a34f6a47d5c0687f0851fc55765b096183f6ee327ea609",
        "Driver": "local",
        "Mountpoint": "/var/lib/docker/volumes/7c27e585d8d4f55c34a34f6a47d5c0687f0851fc55765b096183f6ee327ea609/_data",
        "Labels": null,
        "Scope": "local"
    }
]

Docker compose :

version: "2"
services:
    mysql:
        image: mysql:latest
        volumes:
            - ./mysql:/var/lib/mysql
        environment:
            - MYSQL_ROOT_PASSWORD=password
            - MYSQL_DATABASE=database
            - MYSQL_USER=user
            - MYSQL_PASSWORD=password
        ports:
            - "3306:3306"
    php:
        build: ./php
        image: php:fpm
        ports:
            - "9000:9000"
        links:
            - mysql
    nginx:
        build: ./nginx
        image: nginx:latest
        volumes:
            - ./nginx/website.conf:/etc/nginx/conf.d/website.conf
        ports:
            - "80:80"
        links:
            - php
    phpmyadmin:
        image: phpmyadmin/phpmyadmin
        environment:
            - PMA_HOST=mysql
            - MYSQL_USERNAME=user
            - MYSQL_ROOT_PASSWORD=password
        ports:
            - "8181:80"
        links:
            - mysql

Is it possible to use NFS of other file sharing system ?

@BlueBeN82
Copy link

@BlueBeN82 BlueBeN82 commented Nov 5, 2016

Same usecase and behavior for me.

@fmasa
Copy link

@fmasa fmasa commented Nov 22, 2016

@friism One of the solutions would be to allow modification of rsize&wsize parameters of smb mount. PHP projects usually consist of hundreds of small files (which have to be loaded on every request) and large rsize degrades performance for this use case.

@dimasnake Try beta 21 if you have installer, performance is much better.

@dimasnake
Copy link
Author

@dimasnake dimasnake commented Nov 22, 2016

@fmasa I don't have the installer. Where I can download 21 beta?

@saschanaz
Copy link

@saschanaz saschanaz commented Nov 25, 2016

Here is a beta version of Docker for Windows but really beta 21? The latest beta has the version tag 2016-11-10 1.12.3-beta30.

@fmasa
Copy link

@fmasa fmasa commented Nov 25, 2016

@saschanaz Yes. Take a look at the end of the tag (beta30)

@dopee
Copy link

@dopee dopee commented Nov 26, 2016

@dimasnake @saschanaz

I found it!
Build 5971 is beta21
https://download.docker.com/win/beta/1.12.0.5971/InstallDocker.msi

Gonna restart my machine and try it.

@dopee
Copy link

@dopee dopee commented Nov 26, 2016

Can't get it working. Returned to latest beta (30.1).
So frustrating, performance at work on Ubuntu is almost ten times better, while having older hardware.

@fmasa
Copy link

@fmasa fmasa commented Nov 26, 2016

I read in another issue, that installer auto-updates to latest version. :/ So once you updated, you're stuck with that version.

@dimasnake
Copy link
Author

@dimasnake dimasnake commented Nov 28, 2016

@fmasa @dopee No difference in beta 21 and 30.

@dgageot dgageot changed the title Incredibly slow (near unusable) on Docker container Shared Volumes Slow Dec 3, 2016
@wclr
Copy link

@wclr wclr commented Dec 3, 2016

This issue has is more appropriate name then #188

There is also alike issue in docker for mac: docker/for-mac#77

I also experience this for example with git, when I attach as volume some folder and make git operations on it, with quite large code base speed of operation is significantly slower then on dev machine.

To reproduce it you may just take some image with GIT installed and attach some big repo and try do do git status.

@xdrew
Copy link

@xdrew xdrew commented Dec 4, 2016

Facing this issue on symfony app developing.
Symfony

@fmasa
Copy link

@fmasa fmasa commented Dec 5, 2016

@dimasnake Are you sure you have beta 21 installed? I found out that installer auto-updates to latest version no matter what.

@friism Is there any roadmap for this issue?

@dimasnake
Copy link
Author

@dimasnake dimasnake commented Dec 6, 2016

@fmasa I disable auto-update. I have 1.12.0-beta21 (build: 5971). You have in 21 beta performance better? Show output if this command dd if=/dev/zero of=test.dat bs=1024 count=100000 inside docker container.

@friism
Copy link

@friism friism commented Dec 7, 2016

@fmasa we're aware of the problem. Note that volume I/O performance will likely always be slower than pure in-container performance and pure on-host performance because the host-mounted volume filesystem is mounted over a network.

We're interested in making that overhead as low as possible, of course, but we're not keen on adding a lot of toggles that users have to change depending on what software they happen to be running in containers right now.

@fmasa
Copy link

@fmasa fmasa commented Dec 8, 2016

@friism Some performance hit in mounted volumes is expected, but it for this particular use case it's just too much for DfW to be usable on daily basis.
I don't see way to optimize smb shares w/o some sort of toggles. Different stacks have different requirements. Why don' you give an option to tweak some parameters in config files if not in GUI? You're cutting not unsignificant part of community out of DfW, because the only viable workaround right now is Docker Machine or some nasty hacks.
I don't want to flame nor hate, just trying to give you some feedback from PHP world :)

@coderzheng
Copy link

@coderzheng coderzheng commented Apr 18, 2020

So did anyone find a solution to fix the problem in Windows10?I was annoyed with it for a few days.

@er1z
Copy link

@er1z er1z commented Apr 18, 2020

I'm afraid that this problem won't be solved for at least a few years ahead. Only workarounds are available, unfortunately. I've ended with VM with Linux and Unison; tried many ways but this one is the least painful.

@telemmaite
Copy link

@telemmaite telemmaite commented Apr 20, 2020

@er1z how about the upcoming WSL2 which comes with Windows 10 version 2004(year 20 month 04), should be this April.

WSL2 ships an actual Linux kernel and its Linux file system is 5x-10x faster than WSL

@dimasnake mentioned that it works nicely on his end with WSL2.

@azeranex
Copy link

@azeranex azeranex commented Apr 26, 2020

@er1z how about the upcoming WSL2 which comes with Windows 10 version 2004(year 20 month 04), should be this April.

WSL2 ships an actual Linux kernel and its Linux file system is 5x-10x faster than WSL

@dimasnake mentioned that it works nicely on his end with WSL2.

Shared volumes is still slow in wsl2 but it's fast when you put files on wsl itself. It's the best solution for me right now.

@dimasnake
Copy link
Author

@dimasnake dimasnake commented May 9, 2020

it's fast when you put files on wsl itself.

Yes, it's fast, but very slow indexing files in IDE and high cpu usage while files indexing.

@Piepongwong
Copy link

@Piepongwong Piepongwong commented Jul 13, 2020

I was having similar issues and binding all volumes from with wsl itself instead through the windows mount solved it for me. This is recommended best practice: https://docs.docker.com/docker-for-windows/wsl/#best-practices .

I'm not experiencing any significant performance differences between a real linux machine and docker for windows anymore.

@er1z
Copy link

@er1z er1z commented Aug 19, 2020

@Piepongwong — but it assumes that you don't need a performance on host. Good luck on IntelliJ project re-indexing every time you focus main window…

@ForceConstant
Copy link

@ForceConstant ForceConstant commented Sep 25, 2020

I am having a similar issue, not sure what the best way to address. I am not using WSL, but whatever the default docker for windows install is. But writing to mounted volume directory is 10-50x slower than other directories inside the container. How should I address this?

@cortex93
Copy link

@cortex93 cortex93 commented Sep 29, 2020

@Piepongwong — but it assumes that you don't need a performance on host. Good luck on IntelliJ project re-indexing every time you focus main window…

Try vscode remote wsl/container or you may also forward x11 with IntelliJ running inside the container.

@er1z
Copy link

@er1z er1z commented Sep 29, 2020

Tried forwarding, there are some minor issues that makes work a struggle. Apart from GUI lag…

@hsk-kr
Copy link

@hsk-kr hsk-kr commented Nov 2, 2020

Is there anyone else who solved this problem...?
I'm using SFTP instead of docker-volume in windows10....

@mapic100
Copy link

@mapic100 mapic100 commented Nov 19, 2020

it is my solution to increase speed (in my case from 8MB/s even up to 2GB/s on removable SSD disk)

all what we need it is format disk/partition to have on Ext4 (i used free AOMEI Partition Assistant Sotf on W10)

in my case it is (g:) drive
on windows is able to access/manage from path:
"\\wsl$\docker-desktop\mnt\host\g\"

on docker-compose.yml we have to add volume like that:
"
volumes:
- /run/desktop/mnt/host/g/projects:/var/www
"

before:
root@65aad241c5b5:/var/www# dd if=/dev/zero of=test.dat bs=1024 count=100000
100000+0 records in
100000+0 records out
102400000 bytes (102 MB, 98 MiB) copied, 12.0597 s, 8.5 MB/s

after:
root@02c53e37e74f:/var/www# dd if=/dev/zero of=test.dat bs=1024 count=100000
100000+0 records in
100000+0 records out
102400000 bytes (102 MB, 98 MiB) copied, 0.146129 s, 701 MB/s for 100MB

ps i tested also for 5GB and i had result 2.3 GB/s

@er1z
Copy link

@er1z er1z commented Dec 10, 2020

This is still a workaround, @mapic100, and doesn't solve a problem of two-way sync between container and Windows host.

@docker-desktop-robot
Copy link
Collaborator

@docker-desktop-robot docker-desktop-robot commented Mar 10, 2021

Issues go stale after 90 days of inactivity.
Mark the issue as fresh with /remove-lifecycle stale comment.
Stale issues will be closed after an additional 30 days of inactivity.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle stale

@paq85
Copy link

@paq85 paq85 commented Mar 10, 2021

I guess everyone lost their hopes for a solution since this has been over 4 years now =D

Let's get some hope back: /remove-lifecycle stale

@mapic100
Copy link

@mapic100 mapic100 commented Mar 10, 2021

one thing i cannot resolve are lost projects (folders & files) after update Docker Desktop to newest version, maybe someone had the same issue and have some solution?

@dovidweisz
Copy link

@dovidweisz dovidweisz commented Mar 10, 2021

Can someone confirm if this is still an issue when using the new WSL2 based engine?

@Izopi4a
Copy link

@Izopi4a Izopi4a commented Mar 10, 2021

yes it is still very slow. With mounted volumes it probably will be fine, but i am not installing dev previous ( i am on preview circle, but i am not going below that).
The workaround with home dir is decent tho

@peter-gribanov
Copy link

@peter-gribanov peter-gribanov commented Mar 12, 2021

/remove-lifecycle stale

@ken9ross
Copy link

@ken9ross ken9ross commented Apr 20, 2021

/remove-lifecycle stale

@stevekm
Copy link

@stevekm stevekm commented Jun 16, 2021

I was having similar issues and binding all volumes from with wsl itself instead through the windows mount solved it for me. This is recommended best practice: https://docs.docker.com/docker-for-windows/wsl/#best-practices .

I'm not experiencing any significant performance differences between a real linux machine and docker for windows anymore.

from that link;

Performance is much higher when files are bind-mounted from the Linux filesystem, rather than remoted from the Windows host. Therefore avoid docker run -v /mnt/c/users:/users (where /mnt/c is mounted from Windows).

Instead, from a Linux shell use a command like docker run -v ~/my-project:/sources where ~ is expanded by the Linux shell to $HOME.

This is not a feasible solution. By default, the WSL filesystem is located on the Windows OS install drive, and is limited to 256GB in size. It is common for systems with a lot of high throughput disk IO to have dedicated drives such as fast SSD's meant for this purpose; the solution here makes it impossible to use your dedicated fast storage hardware. If you followed this guide, you will likely lock up your entire OS pretty quickly and fill the boot disk.

@docker-desktop-robot
Copy link
Collaborator

@docker-desktop-robot docker-desktop-robot commented Sep 14, 2021

Issues go stale after 90 days of inactivity.
Mark the issue as fresh with /remove-lifecycle stale comment.
Stale issues will be closed after an additional 30 days of inactivity.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle stale

@peter-gribanov
Copy link

@peter-gribanov peter-gribanov commented Sep 14, 2021

/remove-lifecycle stale

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet