New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

File system performance improvements #1592

Open
yallop opened this Issue May 5, 2017 · 74 comments

Comments

Projects
None yet
@yallop
Contributor

yallop commented May 5, 2017

Recent Docker versions (17.04 CE Edge onwards) add additional flags to the -v option for docker run that make it possible to specify the consistency requirements for a bind-mounted directory. The flags are

  • consistent: Full consistency. The container runtime and the host maintain an identical view of the mount at all times. This is the default.
  • cached: The host's view of the mount is authoritative. There may be delays before updates made on the host are visible within a container.
  • delegated: The container runtime's view of the mount is authoritative. There may be delays before updates made in a container are visible on the host.

In 17.04 cached is significantly faster than consistent for many workloads. However, delegated currently behaves identically to cached. The Docker for Mac team plans to release an improved implementation of delegated in the future, to speed up write-heavy workloads. We also plan to further improve the performance of cached and consistent.

We will post updates relating to Docker for Mac file sharing performance in comments to this issue. Users interested in news about performance improvements should subscribe here.

To keep the signal-to-noise ratio high we will actively moderate this issue, removing off-topic advertisements, etc. Questions about the implementation, the behaviour, and the design are very welcome, as are realistic benchmarks and real-world use cases.

@joemewes

This comment has been minimized.

joemewes commented May 6, 2017

A rudimentary real-worked example (Drupal 8):

Docker for Mac : Version 17.05.0-ce-rc1-mac8 (16582)
Mac-mini :: MacOS Sierra : 10.12.2 (16C67)

A simple command line curl test (taken average of 10 calls to URL) Drupal 8 clean install frontend:

old UNISON (custom synced container approach) Volume mount : 0.470s
standard Volume mount: 1.401s
new :cached Volume mount: 0.490

v.easy implementation to add to my compose.yaml files and happy with any delay between host/output using cached on host codebase.

@yallop Is there a rough/expected release date for 17.04 (stable) ?

@yallop

This comment has been minimized.

Contributor

yallop commented May 8, 2017

@yallop Is there a rough/expected release date for 17.04 (stable)

The next stable version will be 17.06, and is likely to be released some time in June. There's been a change to the Docker version numbering scheme recently so that the numbers now indicate the release date, with stable releases every three months (March, June, September, December), and edge releases in the months in between. For example, 17.04 is the Edge release in April 2017 and 17.06 is the stable release in June 2017.

@joemewes

This comment has been minimized.

joemewes commented May 8, 2017

ok, thanks for that. do you expect 17.06 to contain at least the current edge implementation of :cached?

@yallop

This comment has been minimized.

Contributor

yallop commented May 8, 2017

Yes, that's the plan.

@reinout

This comment has been minimized.

reinout commented May 9, 2017

How should this work in a docker-compose.yml file? I tried appending :cached to the existing volume setting:

volumes:
  - .:/srv:cached

... but that got me an error (on OSX):

ERROR: Cannot start service my_service: Mounts denied: 9p: Stack overflow

(Docker version 17.05.0-ce-rc1, build 2878a85)

Note: having a global setting or environment variable to switch my local default to "cached" would also be fine (or rather preferable).

@DanielSchwiperich

This comment has been minimized.

DanielSchwiperich commented May 9, 2017

The syntax looks correct @reinout
here's a stripped but working example docker-compose.yml

version: '2'
services:
  php:
    image: php:7.1-fpm
    ports:
      - 9000
    volumes:
      - .:/var/www/project:cached

tested on Docker version 17.05.0-ce-rc1, build 2878a85

@reinout

This comment has been minimized.

reinout commented May 9, 2017

Your example works. My own one still not (even after really making sure there were no left-over old mounted volumes). So I rebooted. Afterwards, it worked.

So: a reboot might be needed if you've run a docker-compose before upgrading docker to the latest version. Possibly related: I switched from docker stable to edge.


Is there a possibility of a global setting? I don't really want to add this option to the docker-compose.yml that all my linux colleagues are using.

@DanielSchwiperich

This comment has been minimized.

DanielSchwiperich commented May 9, 2017

not as far as I know. When you linux colleagues are running edge the flag should work.

Another workaround would be (that's what we do right now for separating linux and mac volume mounting )
to just put the mount settings for mac in a separate file, like docker-compose-mac.yml and then run docker-compose -f docker-compose.yml -f docker-compose.mac.yml up -d

See https://docs.docker.com/compose/extends/

@carn1x

This comment has been minimized.

carn1x commented May 9, 2017

@reinout You could use an additional compose file to override that of your colleagues? For instance, we have:

docker-compose.yml:

version: '2'

services:

  build_docs:
    image: docs/sphinx
    build: .
    environment:
      - DOCS_NAME='docs'
      - SRC_DIR=src
      - DST_DIR=build
    volumes:
      - "./../../docs/dev:/docs"
    command: /docs/source /docs/build

and volumes-cached.yml:

services:
  build_docs:
    volumes:
      - "./../../docs/dev:/docs:cached"

Which can be run with:

$ docker-compose -f docker-compose.yml -f volumes-cached.yml up
@reinout

This comment has been minimized.

reinout commented May 9, 2017

Yes, I could do that. But.... I'd have to do that for each of the 12 docker-compose projects. And I'd have to keep it in sync with changes to the "master" docker-compose.yml.

As an intermediary measure: fine. Long term: no, as it is not very don't-repeat-yourself :-)

If someone wants to enable the "cached" behaviour, that person probably wants to use it for all/most of the dockers. Would it make sense as a config setting in the Docker app itself? In the preferences' "File sharing" tab? (This should probably be its own ticket, I assume?)

@yallop

This comment has been minimized.

Contributor

yallop commented May 9, 2017

@reinout: :cached is supported across platforms, so there should be no issue in adding it directly to compose files.

@reinout

This comment has been minimized.

reinout commented May 9, 2017

Provided that everybody uses the latest edge version, right? And it seems a bit strange to add an option that only has effect on osx to everybody's docker-compose.yml.

Anyway, it works for now. I won't drag the signal/noise ratio further down :-)

@matthewjosephtaylor

This comment has been minimized.

matthewjosephtaylor commented May 10, 2017

mtaylor(mjt)@mtaylor:~/tmp/docker-disk-perf-test$ time dd if=/dev/zero of=./output bs=8k count=40k; rm ./output
40960+0 records in
40960+0 records out
335544320 bytes transferred in 1.300857 secs (257941007 bytes/sec)

real	0m1.320s
user	0m0.012s
sys	0m0.564s
mtaylor(mjt)@mtaylor:~/tmp/docker-disk-perf-test$ docker run -it --rm -v "$(pwd):/host-disk" ubuntu /bin/bash 
root@4e9c8bc5e5c1:/# cd /host-disk/
root@4e9c8bc5e5c1:/host-disk# time dd if=/dev/zero of=./output bs=8k count=40k; rm ./output
40960+0 records in
40960+0 records out
335544320 bytes (336 MB, 320 MiB) copied, 10.7496 s, 31.2 MB/s

real	0m10.756s
user	0m0.050s
sys	0m1.090s
root@4e9c8bc5e5c1:/host-disk# exit
exit
mtaylor(mjt)@mtaylor:~/tmp/docker-disk-perf-test$ docker run -it --rm -v "$(pwd):/host-disk:cached" ubuntu /bin/bash 
root@597dc640bdeb:/# cd /host-disk/
root@597dc640bdeb:/host-disk# time dd if=/dev/zero of=./output bs=8k count=40k; rm ./output
40960+0 records in
40960+0 records out
335544320 bytes (336 MB, 320 MiB) copied, 11.1683 s, 30.0 MB/s

real	0m11.172s
user	0m0.060s
sys	0m1.080s
root@597dc640bdeb:/host-disk# exit
exit
mtaylor(mjt)@mtaylor:~/tmp/docker-disk-perf-test$ docker run -it --rm -v "$(pwd):/host-disk:delegated" ubuntu /bin/bash 
root@985e4143053b:/# cd /host-disk/
root@985e4143053b:/host-disk# time dd if=/dev/zero of=./output bs=8k count=40k; rm ./output
40960+0 records in
40960+0 records out
335544320 bytes (336 MB, 320 MiB) copied, 12.1589 s, 27.6 MB/s

real	0m12.165s
user	0m0.080s
sys	0m1.000s
root@985e4143053b:/host-disk# exit
exit
mtaylor(mjt)@mtaylor:~/tmp/docker-disk-perf-test$ docker run -it --rm -v "$(pwd):/host-disk:consistent" ubuntu /bin/bash 
root@3377ae356124:/# cd /host-disk/
root@3377ae356124:/host-disk# time dd if=/dev/zero of=./output bs=8k count=40k; rm ./output
40960+0 records in
40960+0 records out
335544320 bytes (336 MB, 320 MiB) copied, 12.5944 s, 26.6 MB/s

real	0m12.601s
user	0m0.060s
sys	0m0.980s
root@3377ae356124:/host-disk# exit
exit
mtaylor(mjt)@mtaylor:~/tmp/docker-disk-perf-test$ docker --version
Docker version 17.05.0-ce, build 89658be
mtaylor(mjt)@mtaylor:~/tmp/docker-disk-perf-test$ 

Perhaps my expectations on how this works are unreasonable, or I'm doing something wrong. Above are some simple disk performance tests and I'm not seeing any differences.

Would be interested in knowing if my expectations, or use of the flag is incorrect.

@barat

This comment has been minimized.

barat commented May 11, 2017

@matthewjosephtaylor ... :cached won't improve dd tests ... for this, you need to check for :delegated to rollout :)

@ToonSpinISAAC

This comment has been minimized.

ToonSpinISAAC commented May 23, 2017

The issue text says:

delegated: The container runtime's view of the mount is authoritative. There may be delays before updates made in a container are visible on the host.

Does this mean that if I were to use delegated (or cached for that matter) that syncing would strictly be a one-way affair?

In other words, to make my question clearer, let's say I have a codebase in a directory and I mount this directory inside a container using delegated. Does this mean that if I update the codebase on the host, the container will overwrite my changes?

What I understood until now was, that using for instance delegated, would keep syncing "two-way", but make it more efficient from the container to the host, but the text leads me to believe that I may have misunderstood, hence the question.

@geerlingguy

This comment has been minimized.

geerlingguy commented May 24, 2017

Just posting another data point:

  • Drupal codebase volume with rw,delegated on Docker Edge (for now, it's only using the cached option though): 37 requests/sec
  • Drupal codebase volume with rw (and no extra options): 2 requests/sec

Drupal 8 is ~18x faster if you're using a Docker volume to share a host codebase into a container, and it's pretty close to native filesystem performance.

With cached, Drupal and Symfony development are no longer insanely painful with Docker. With delegated, that's even more true, as operations like composer update (which results in many writes) will also be orders-of-magnitude faster!

@kostajh

This comment has been minimized.

kostajh commented May 24, 2017

I'm seeing good results for running a set of Behat tests on a Drupal 7 site:

On Mac OS, without :cached:

10:02 $ docker exec clientsite_php bin/behat -c tests/behat.yml --tags=~@failing -f progress
...................................................................... 70
...................................................................... 140
...................................................................... 210
...................................................................... 280
.................................................

69 scenarios (69 passed)
329 steps (329 passed)
11m26.20s (70.82Mb)

With :cached:

09:55 $ docker exec clientsite_php bin/behat -c tests/behat.yml --tags=~@failing -f progress
...................................................................... 70
...................................................................... 140
...................................................................... 210
...................................................................... 280
.................................................

69 scenarios (69 passed)
329 steps (329 passed)
4m33.77s (63.33Mb)

On Travis CI (without :cached):

$ docker exec clientsite_php bin/behat -c tests/behat.yml --tags=~@failing -f progress
...................................................................... 70
...................................................................... 140
...................................................................... 210
...................................................................... 280
.................................................
69 scenarios (69 passed)
329 steps (329 passed)
4m7.07s (55.01Mb)

On Travis CI (with :cached):

247.12s$ docker exec cliensite_php bin/behat -c tests/behat.yml --tags=~@failing -f progress
...................................................................... 70
...................................................................... 140
...................................................................... 210
...................................................................... 280
.................................................
69 scenarios (69 passed)
329 steps (329 passed)
4m6.71s (55.01Mb)

Nice work, Docker team! 👏

@carn1x

This comment has been minimized.

carn1x commented Jun 5, 2017

Is it expected that both :cached and :delegated can be combined or will they be mutually exclusive?

@dsheets

This comment has been minimized.

Contributor

dsheets commented Jun 5, 2017

@ToonSpinISAAC both :cached and (when it lands) :delegated perform two-way "syncing". The text you cite is saying that, with :cached, the container may read stale data if it has changed on the host and the invalidation event hasn't propagated yet. With :cached, the container will write-through and no new write-write conflicts can occur (POSIX still allows multiple writers). Think of :cached as "read caching". With :delegated, if the container writes to a file that write will win even if an intermediate write has occurred on the host. Container writes can be delayed indefinitely but are guaranteed to persist after the container has successfully exited. flush and similar functionality will also guarantee persistence. Think of :delegated as "read-write caching". Even under :delegated, synchronization happens in both directions and updates may occur rapidly (but don't have to). Additionally, you may overlap :cached and :delegated and :cached semantics will override :delegated semantics. See https://docs.docker.com/docker-for-mac/osxfs-caching/#delegated guarantee 5. If you are using :delegated for source code but your container does not write to your code files (this seems unlikely but maybe it auto-formats or something?), there is nothing to worry about. :delegated is currently the same as :cached but will provide write caching in the future.

@carn1x :cached and :delegated (and :default and :consistent) form a partial order (see https://docs.docker.com/docker-for-mac/osxfs-caching/#semantics). They can't be combined but they do degrade to each other. This allows multiple containers with different requirements to share the same bind mount directories safely.

@lmakarov

This comment has been minimized.

lmakarov commented Jun 6, 2017

From within a container is there a way to tell which flag was applied for a volume?

I'm getting the same output from mount regardless of the flag used. Is there another way to check?

$ mount | grep osxfs
osxfs on /var/www/project type fuse.osxfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other,max_read=1048576)
Version 17.06.0-rc1-ce-mac13 (18169)
Channel: edge
2425473dc2
@dsheets

This comment has been minimized.

Contributor

dsheets commented Jun 6, 2017

@lmakarov /Applications/Docker.app/Contents/MacOS/com.docker.osxfs state should show you the host directories that are mounted into containers and their mount options. For instance, after I run docker run --rm -it -v ~:/host:cached alpine ash, I then see:

$ /Applications/Docker.app/Contents/MacOS/com.docker.osxfs state
Exported directories:
 - /Users to /Users (nodes_/Users table size: 62)
 - /Volumes to /Volumes (nodes_/Volumes table size: 0)
 - /tmp to /tmp (nodes_/private/tmp table size: 9)
 - /private to /private (nodes_/private table size: 0)
Container-mounted directories:
 - /Users/dsheets into b8f7765665782501bc1a099f1898911b7eb393b08930be638545a55fd06e420e (state=cached)
@ToonSpinISAAC

This comment has been minimized.

ToonSpinISAAC commented Jun 12, 2017

Thanks for taking the time to clarify and explain @dsheets!

@gondalez

This comment has been minimized.

gondalez commented Jun 13, 2017

Is there any indication from the high sierra dev beta of how this (and docker mac in general) will work under APFS?

Sorry if this is off-topic but it's a question I keep wanting to ask every time I see a new message here :)

@WillSquire

This comment has been minimized.

WillSquire commented Jun 16, 2017

Unable to get @DanielSchwiperich's example working? Retrieving the error:

invalid spec: .:/var/www/project:cached: unknown option: cached

It doesn't like version being set to 2 either. Perhaps I'm missing something? Currently running 17.06.0-rc2-ce-mac14

@piotrekkaminski

This comment has been minimized.

piotrekkaminski commented Oct 13, 2017

Unfortunately the new settings are not enough to make Magento work with similar speed to Vagrant. It's still much slower.

@ajardin

This comment has been minimized.

ajardin commented Oct 13, 2017

@piotrekkaminski, which version of Magento are you using?

With Magento 1EE, my response time (on the homepage) is under 1s without page or blocks cache. And with Magento 2EE (on the homepage also), my response time is about 500ms with the same configuration.

That's so far better with the delegated mode that I left Docker Machine NFS almost immediately!

@piotrekkaminski

This comment has been minimized.

piotrekkaminski commented Oct 14, 2017

i'm using M2. I will check again, maybe the problem is elsewhere.

@aegis123

This comment has been minimized.

aegis123 commented Oct 14, 2017

@piotrekkaminski could you post the dockerfile/docker-compose?

@bpresles

This comment has been minimized.

bpresles commented Oct 23, 2017

Any news on the :delegated implementation ? As far as can see (in the Changelog and in real life benchmarks), there is still no difference between :cachedand :delegated in Docker for Mac 17.09.0-ce-mac34 (edge) or 17.09.ce-mac35 (stable).

@barat

This comment has been minimized.

barat commented Nov 7, 2017

The day when fileshare will be solved on mac/windows will be great for Docker ... docker-sync is somekind of workaround but for example each time lot of files was changed macbook CPU goes crazy :/
So I second bpresles question - any news on :delegated?

@marcell-born

This comment has been minimized.

marcell-born commented Nov 9, 2017

@aegis123 I can confirm what @piotrekkaminski did earlier -- there is no improvement with shared files using option ":delegated" or ":cached" on Mac OS 10.13.1 running Docker v17.09.0-ce-mac35 (19611).

As you can see I am using v2 as shared drives are not working properly on v3:

version: '2'

services:
    db:
        image: mysql
        volumes:
            - "./.data/db:/var/lib/mysql:delegated"
        environment:
            MYSQL_ROOT_PASSWORD: ${MYSQL_ROOT_PASSWORD}
            MYSQL_DATABASE: ${MYSQL_DATABASE}
            MYSQL_USER: ${MYSQL_USER}
            MYSQL_PASSWORD: ${MYSQL_PASSWORD}
    php:
        build:
            context: php7-fpm
            args:
                TIMEZONE: ${TIMEZONE}
        volumes:
            - ${MAGENTO_APP_PATH}:/var/www/magento:delegated
    nginx:
        build: nginx
        ports:
            - 80:80
        volumes_from:
            - php
        volumes:
            - ./logs/nginx/:/var/log/nginx:cached
@pthomas551

This comment has been minimized.

pthomas551 commented Nov 17, 2017

For those using Rails, we have found that disabling the byebug remote debugging server reduces the impact of this issue enough to make development tolerable.

There is a known issue with byebug and remote debugging that substantially reduces performance. I'm not sure if it is file I/O related or not but it absolutely kills our load times on Docker. Meanwhile disabling byebug leads to near native speeds in dev despite using a reasonably large DB.

@ian-axelrod

This comment has been minimized.

ian-axelrod commented Nov 28, 2017

Is there a way to mount a subdirectory of a bind-mounted directory as cached? Let's say I have the following:

volumes:
      - ./static/stylesheets:/app/static/stylesheets:cached
      - .:/app

My understanding is that :cached will have no effect for the above mount because I mount the entire PWD, and overlapping portions of bind mounts obey the most consistent mount's mode. This makes sense in a lot of cases, but it can cause severe configuration bloat in others. To get the above example working, I would have to individually mount each top-level subdirectory / file rather than .:/app so that I can control the overlap. I have projects that have tens of folders and top-level files, with new ones being added at times. You can imagine the volumes config will become hard to maintain, and may get out of sync with what is actually in the pwd. Any ideas?

@aegis123

This comment has been minimized.

aegis123 commented Jan 26, 2018

Is there any update on this with :cached and :delegate? Since this issue has been on the radar for docker for about 1.5 years now and we are coming up on a full version bump to 18.01

@diimdeep

This comment has been minimized.

diimdeep commented Mar 27, 2018

18.03 released, still no improvements?

@barat

This comment has been minimized.

barat commented Mar 28, 2018

Status would be appreciated. I'm using docker-sync as workaround, but it's confusing that blocker like this has no updates about progress (I'm not talking about fix - just information)
Recently docker-sync started to make troubles after mac42 so "native" fix like delegated would be awesome to replace 3rd party tools.

@Saphyel

This comment has been minimized.

Saphyel commented Mar 28, 2018

Maybe we should request to Apple if they can lend a hand?

@markoshust

This comment has been minimized.

markoshust commented Mar 28, 2018

I have noticed much improved performance with the delegated flags on my updated Magento builds - perhaps the addition has already been included in a recent version.

I think everyone needs to remember that this is 100% free software, and being mean to developers giving software away for free or making backhanded comments is a complete noop.

@itaylor

This comment has been minimized.

itaylor commented Mar 28, 2018

@markoshust D4M is not "free software", it's commercial software given away for no cost. It is not open source, nor is the OSXFS file system implementation whose performance is in question here. If they were open source, many of the people involved on this thread (myself included) would have already contributed to improving the perf of these components. Until Docker Inc. decides to open source these components, we have no choice but to whine and complain about the poor performance on Github and hope that Docker team members decide to fix them.

Open source isn't the only model that could work here, either, if this was a standalone piece of commercial software, I'd gladly pay for it, and the purchases could help fund a dev team that was dedicated to actually improving it. Right now D4M is in the worst of all possible places, closed-source commercial software with no income stream to fund improvements to it.

@SpencerMalone

This comment has been minimized.

SpencerMalone commented May 17, 2018

Bringing this back in focus, can we please look at a global setting for the default type of volume? When using a kubernetes setup on edge, you're forced to rely on the slowest volume mounts, and it makes development downright painful.

@docker-for-desktop-robot

This comment has been minimized.

Collaborator

docker-for-desktop-robot commented Aug 15, 2018

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale comment.
Stale issues will be closed after an additional 30d of inactivity.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle stale

@jak

This comment has been minimized.

jak commented Aug 15, 2018

/remove-lifecycle stale

@docker-for-desktop-robot

This comment has been minimized.

Collaborator

docker-for-desktop-robot commented Nov 13, 2018

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale comment.
Stale issues will be closed after an additional 30d of inactivity.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle stale

@cweagans

This comment has been minimized.

cweagans commented Nov 13, 2018

/remove-lifecycle stale

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment