Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Docker Desktop] Improve Mac File system performance #7

Open
nebuk89 opened this issue Mar 6, 2020 · 704 comments
Open

[Docker Desktop] Improve Mac File system performance #7

nebuk89 opened this issue Mar 6, 2020 · 704 comments
Assignees
Labels
docker_desktop

Comments

@nebuk89
Copy link
Contributor

nebuk89 commented Mar 6, 2020

Update: we are now looking at using GRPCFuse rather than mutagen as a simpler path for perf improvement.

Tell us about your request
Integrate the mutagen pluggin within Docker Desktop to provide users with a file caching option to improve performance on modern web frameworks like PHP Symphony

Which service(s) is this request for?
Desktop

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
File system performance is big issue on Mac,our goal improve web page refresh for web languages like PHP from 2.8 seconds to 0.2 seconds

Are you currently working around this issue?
N/A
Additional context
N/A
Attachments
docker/for-mac#77

@nebuk89 nebuk89 changed the title [Docker Desktop] [Docker Desktop] Improve Mac File system performance Mar 6, 2020
@nebuk89 nebuk89 added this to Investigating in docker-roadmap Mar 6, 2020
@nebuk89 nebuk89 moved this from Investigating to Writing Code in docker-roadmap Mar 6, 2020
@pkennedyr pkennedyr added the docker_desktop label Mar 16, 2020
@daveisfera
Copy link

daveisfera commented Mar 26, 2020

These issue are still happening, so can they be added to this or new cards made to include them?
docker/for-mac#1592
docker/for-mac#1899
docker/for-mac#2417
docker/for-mac#3186

@leehambley
Copy link

leehambley commented Apr 6, 2020

I hope leaving a note here is not unwelcome to highlight a particular difficulty in booting Rails applications in Docker on macOS.

Between Rails hot code reloading, which has a semi-hard dependency on working filesystem events (the polling alternative can eat as much as 20% of system CPU) and Bootsnap, which is a caching loader which tries to speed up booting Rails by bypassing the parsing step (and caching compiled btyecode on disk) which means that booting a Rails app with bootsnap loaded writes as many file as it reads (sometimes), but other times (e,g warm start) has a very different read/write ratio.

None of the existing methods account for this very well, even the relatively new mount options to try and performance boost either reads or writes don't particularly help because of the relative symmetry under certain configurations.

Most places I know are still using NFS, with polling, and trying to tune the fsattr cache times to get a reasonable balance of developer experience (speed of code reloading) vs. performance (outright speed), but it's a tightrope.

@nebuk89
Copy link
Contributor Author

nebuk89 commented Apr 6, 2020

hey @daveisfera we haven't got this quite out onto edge yet :) once we do we will move this over to developer/preview and update this ticket. Hopefully then people on those tickets should see some movement!

@mbrioski
Copy link

mbrioski commented Apr 6, 2020

@nebuk89 could u clarify caching? Because at the moment the problem is the slow sync from host machine to docker containers

@Jean85
Copy link

Jean85 commented Apr 6, 2020

@mauri-brioschi With PHP/Symfony apps sometimes the reverse happens: the app itself rewrites a bunch of files in its own var/cache dir, where it dumps a lot of stuff (container definitions, configuration...) and this slows a lot the machine. You could try to ignore that folder from the sync, but you would lose some IDE functionality that relies on that folder.

@mbrioski
Copy link

mbrioski commented Apr 6, 2020

@Jean85 i know the problem and it happens more with "Frameworks" like Magento where there is "generated" code. Usually i exclude these directories so there's not need for sync.
Caching these directories really solve the problem? At the end if u flush the cache and u are synchronizing your volumes, your folder must be sync again.

@markshust
Copy link

markshust commented Apr 6, 2020

Artifacts and caching directories like var/cache, generated, node_modules, vendor, etc. should basically persist on the container, however the issue that @Jean85 pointed out that you miss IDE functionality is definitely valid. As a developer you want assets like these to be generally available, and don't want to have to worry about this folder becoming "out of sync" with the host. I wrote a blog post about all of this at https://markshust.com/2018/12/30/docker-mac-filesystem-volume-mount-approach-performance/ which may be useful to others dealing with filesystem performance issues. Magento 2 seems to be an ideal use-case to test the folder sync issues.

Mutagen offers different modes https://mutagen.io/documentation/synchronization and ideally we should have access to all of these modes. I believe the "one way" is more performant than "two way" since it only needs to check the filesystem changes in one direction. We'd also need to control whether the host or container is alpha or beta. Consistency also probably isn't required for these artifacts directories, it just needs to be "generally there". Being able to control volume mounts on these different modes is crucial to achieving maximum performance.

@mbrioski
Copy link

mbrioski commented Apr 6, 2020

What's about mounting NFS volumes? It is the only working solution i have found, i.e. for Magento2.
No configuration is required and you can still use delegated, cache ...

@markshust
Copy link

markshust commented Apr 6, 2020

@mauri-brioschi osxfs is (much) more performant than nfs in my experience. i haven't had any issues with osxfs as long as i am strategic in the volumes i mount.

@strayer
Copy link

strayer commented Apr 6, 2020

I'd like to add that in our case it is important to not sync folders like node_modules or Elixirs build and deps folders back to the host especially because of IDE support. Elixirs folders contain architecture-specific compiled files that will break IDE functionality because many plugins need to execute these files. While file system performance is one of our most prominent Docker dev issues, the requirement of being able to exclude folders from syncing is very important too.

The obvious workaround would be to simply mount another volume on top of the shared app volume for node_modules and such, but we encountered strange issues with that too. There are filed issues for those problems, but I can't find them right now. I think the issue was the the volumes for node_modules and so on suddenly didn't mount anymore.

@rfay
Copy link

rfay commented Apr 6, 2020

Just a note that all the docker-sync approaches we've tried have failed, and I'd be very suspicious of mutagen. Although the performance was great, the reliability was zero.

@mbrioski
Copy link

mbrioski commented Apr 6, 2020

@markshust i'm following this issue docker/for-mac#1592 since a while, and NFS is definitely not slower than Osxfs... we are using it in all Magento2 projects and performances are not comparable at all. We save minutes with NFS.

@cweagans
Copy link

cweagans commented Apr 6, 2020

@mauri-brioschi it depends on what you're using it for. For PHP projects that all have a zillion tiny files that need to be read every time you send a request, osxfs is definitely not as good. IIRC, for small numbers of files, it does pretty well.

@ifeltsweet
Copy link

ifeltsweet commented Apr 7, 2020

We've tried mutagen on macOS and it worked well but we've experienced syncing issues with Windows. Eventually, we chose not to use mutagen in our team since the difference in setups between Windows and macOS slowed us down more (in terms of maintenance) than a less performant "delegated" mode.

Hopefully, a native integration will make this as simple as adding a flag to your volume.

@ajardin
Copy link

ajardin commented Apr 7, 2020

I'm very excited to have a native implementation of Mutagen within Docker Desktop, as my company and I have been using it for over a year!

We are working on both Magento and Symfony, so we are heavily impacted by the famous file system performance issues... We tried several approaches (Docker Machine NFS, docker-sync, exclude cache/vendors from synchronization, etc.), but the most successful by far is Mutagen, at least for us.

Although having to use a third party solution in addition to Docker is still an issue from my point of view.

@nebuk89
Copy link
Contributor Author

nebuk89 commented Apr 8, 2020

We've tried mutagen on macOS and it worked well but we've experienced syncing issues with Windows. Eventually, we chose not to use mutagen in our team since the difference in setups between Windows and macOS slowed us down more (in terms of maintenance) than a less performant "delegated" mode.

Hopefully, a native integration will make this as simple as adding a flag to your volume.

This is exactly the experience we want! You should be able to check a volume in the Desktop settings and get it cached :) Initially this will just be released on Mac (as our first preview) due to the point some of you have highlighted that behaviour in Windows differs. Our end goal is to ideally hide this difference so it 'just works with a check box'.

@nebuk89
Copy link
Contributor Author

nebuk89 commented May 27, 2020

This is now out on Docker Desktop Edge, we are after feedback on the UX, the functionality and the performance!

@jasonwilliams
Copy link

jasonwilliams commented May 28, 2020

For those of you having permission issues since switching to Edge there's an issue here:
docker/for-mac#4593

@nebuk89
Copy link
Contributor Author

nebuk89 commented May 29, 2020

@jasonwilliams acknowledged, Dave has looked at the issue and we are looking :)

@xaviemirmon
Copy link

xaviemirmon commented May 30, 2022

@Serenacula I ran into the same issue with Gatsby. It's because fsevents aren't supported on VirtioFS or NFS. The workaround for me was twofold.

  1. Set Webpack polling to be time-based

(This looks like what you need) https://github.com/Bikash888/nextjs-docker/blob/main/next.config.js

https://nextjs.org/docs/api-reference/next.config.js/custom-webpack-config

https://webpack.js.org/configuration/watch/

  1. You may need to also enable chokidar polling. This was a requirement for my Gatsby project but I couldn't see anything relevant to Next.js

I wrote an article about my setup here: https://blog.xavie.mirmon.co.uk/running-gatsby-inside-a-docker-container-on-macos-5e98a77787df

@mtibben
Copy link

mtibben commented May 31, 2022

It's because fsevents aren't supported on VirtioFS

That's doesn't seem right. I just tested with inotifywait and inotify seems to be working fine with VirtioFS?

@robsontenorio
Copy link

robsontenorio commented May 31, 2022

@fredericdalleau and docker mates,

Right now, docker 4.8.2 won't start on M1 Macs with macOS 12.4 (21F79) as reported on docker/for-mac#6331 and related issues.

So, it is unusable by now 😢

@robsontenorio
Copy link

robsontenorio commented May 31, 2022

Just fully uninstalled Docker with some App Cleaner, then install 4.8.1, but no lucky!

I am unable to run docker. Any tips?

M1 Mac with macOS 12.4 (21F79).

Update : Ever tried latest macOS 12.5 beta, but no lucky.

@nicholasruunu
Copy link

nicholasruunu commented Jun 1, 2022

@robsontenorio Strange, works for me with same specs. Only issue I'm having is that I have to sigterm my containers started with docker compose up. Other than that it seems to work great.

@robsontenorio
Copy link

robsontenorio commented Jun 1, 2022

@nicholasruunu are you in same macOS 12.4 (==> 21F79 build) ?

There are a lot of people with same issue #6331

The main issue is Docker desktop app won’t start Docker service, it keeps loading forever.

@nicholasruunu
Copy link

nicholasruunu commented Jun 2, 2022

@robsontenorio Yep, same everything only Macbook Pro M1 Max.

@robsontenorio
Copy link

robsontenorio commented Jun 2, 2022

Just tried latest macOS 12.5 beta 2 and newest Docker Desktop 4.9.0 , but no lucky :(

It keeps starting forever.

image

image

UPDATE

Found a workaround docker/for-mac#6331 (comment)

@nightpool
Copy link

nightpool commented Jun 3, 2022

Hey all, we're still really stymied by docker/for-mac#6243 when working with our Postgres databases locally, is there any update on that issue or any workarounds we can use?

@robsontenorio
Copy link

robsontenorio commented Jun 3, 2022

@nightpool the current workaround is to use named volumes instead of binded volumes.

@h1nds1ght
Copy link

h1nds1ght commented Jun 10, 2022

Some time ago I complained about the git performance inside the container. Unfortunately, this is still the case. But, I tracked down the reason a little bit more.

Actually it seems that git is processing much more files with virtio. For example:

virtio

$ strace git status 2>&1 | wc -l #Note: This is the first call after container start
349295
$ strace git status 2>&1 | wc -l #Note: This is the second call after container start
9284

grpc-fuse

$ strace git status 2>&1 | wc -l #Note: This is the first call after container start
7936
$ strace git status 2>&1 | wc -l #Note: This is the second call after container start
7937

Anyone has a clew why this could be the case? (Using MacOS 12.4, DD 4.9.0, git 2.30.2, Debian 11 Container)

@gerrard00
Copy link

gerrard00 commented Jun 10, 2022

I've just begun experimenting with using VirtioFS for a Rails project on my M1 Max. I'm mounting my source code using a bind mount. The performance increase was about 80% which rules. Unfortunately, I'm now dealing with an issue where the synchronization is taking long enough that any modified file is unavailable in the container for up to a minute. That's obviously not workable for a development workflow. I'm trying to avoid adding the complexity of mutagen or docker-sync if possible. Is this something that will be addressed in future releases of the new FS?

@fredericdalleau
Copy link

fredericdalleau commented Jun 10, 2022

@h1nds1ght @gerrard00 can you generate a diagnostic and share id, do you change lots of files on the host, do you share the whole home directory ? You can refer to https://docs.docker.com/desktop/windows/troubleshoot/#diagnose-and-feedback

@dan-cohn-sabre
Copy link

dan-cohn-sabre commented Jun 10, 2022

I see everyone raving about VirtioFS, so I gave it a try on my Intel-based MacBook Pro (16-inch, 2019, 2.6 GHz 6-Core Intel Core i7) with macOS 12.3.1. I am running a large x86_64-based Linux image that I use for development. I have Docker Desktop v4.9.0.

My experience is that VirtioFS slows down the performance so much that I can barely run anything that touches a mounted volume. For example, I have a large source code repo on a shared volume. Running "git status" just now took 6.73 minutes was compared to 44 seconds without VirtioFS (using gRPC FUSE). Compiling a small Java application took over 5 minutes compared to 53 seconds.

Has anyone else had a similar experience?

The shared volume is mounted like this:
-v "/Users/dan/src:/home/username/src:cached"

I tried without the :cached attribute, and the performance is still terrible, so I do not believe that's the issue. Is there something I have to configure differently when using VirtioFS? Should I be unchecking the gRPC FUSE box?

@h1nds1ght
Copy link

h1nds1ght commented Jun 12, 2022

@fredericdalleau actually I tried to run diagnostics. My problem with it is that it collects so much information. Therefore I tried to wipe all my data for Docker Desktop on my Mac (following https://www.mytecbits.com/apple/macos/clean-way-to-uninstall-docker and docker/for-mac#6208 (comment), maybe not the best guide 😕 ).

The result was that diagnostics still contained old logs, which shouldn't be the case if wiping was successfull. Is there any way to wipe all data for Docker Desktop?

Another way would be to send you a scrambled version of my project. Actually I did that already, such that filenames und folders are randomly named and file contents are empty. Interestingly using a fresh Docker Desktop install and scrambled project data, I still observe the same performance drop with virtio compared to grpc-fuse as described in #7 (comment)...

@gerrard00
Copy link

gerrard00 commented Jun 13, 2022

https://docs.docker.com/desktop/windows/troubleshoot/#diagnose-and-feedback

As r

@h1nds1ght @gerrard00 can you generate a diagnostic and share id, do you change lots of files on the host, do you share the whole home directory ? You can refer to https://docs.docker.com/desktop/windows/troubleshoot/#diagnose-and-feedback

As requested: 5961923B-7AFA-46EF-A4A3-47DC9F1DC846/20220613150729

@gerrard00
Copy link

gerrard00 commented Jun 13, 2022

@h1nds1ght @gerrard00 can you generate a diagnostic and share id, do you change lots of files on the host, do you share the whole home directory ? You can refer to https://docs.docker.com/desktop/windows/troubleshoot/#diagnose-and-feedback

Ah, it looks like the issue only happens with Vim because by default it renames your original file and writes the new data to the original filename. Turning that off with set backupcopy=yes is an acceptable workaround for me for now.

@bluz71
Copy link

bluz71 commented Jun 16, 2022

I maintain a personal Ruby-on-Rails application in mixed mode, Ruby language and Ruby-on-Rails run natively on macOS Monterey 12.4 with only the Postgres database running under Docker 4.8.2 (79419).

I am finding on my Apple Macbook Air with M1 chip that VirtioFS performance is inferior to the default (gRPC?) when seeding my database via Rails ActiveRecord.

Performance Results

Performance when seeding Postgres is roughly as follows:

Backend Database Seed Time, avg over multiple runs
Native, no Docker 8mins
Docker, default 20mins
Docker, VirtioFS 29mins

As can be seen, VirtioFS is significantly slower.

I have created this issue which provides steps to reproduce the regression.

@jnfDev
Copy link

jnfDev commented Jul 31, 2022

I was having pretty good results on Docker + WordPress using the new experimental options; however, recently, I started noticing some random slowdowns here and there. I tried to factory reset, but it didn't improve the situation.

Is anyone else noticing the same performance regression with VirtioFS and the new virtualization framework enabled? Because a couple of Docker/macOS updates ago, the performance was way better than now.

@jnfDev
Copy link

jnfDev commented Aug 1, 2022

I was having pretty good results on Docker + WordPress using the new experimental options; however, recently, I started noticing some random slowdowns here and there. I tried to factory reset, but it didn't improve the situation.

Is anyone else noticing the same performance regression with VirtioFS and the new virtualization framework enabled? Because a couple of Docker/macOS updates ago, the performance was way better than now.

Only for comparison, here is the difference between a previous test (my first comment just after trying VirtioFS for the first time) and a current test.

Previous (Feb 25)

before

Nowadays

now

@moreana
Copy link

moreana commented Aug 2, 2022

I am having an issue with permissions with VirtioFS enabled and I wanted to check one of the attached dockers builds which might fix the issue. But when I try to download https://desktop-stage.docker.com/mac/main/amd64/76176/Docker.dmg, it fails with '503 Forbidden'.
Could anybody help with it?

@conradwt
Copy link

conradwt commented Aug 2, 2022

@jnfDev Would it be possible for you to add the following information to your comment?

  • Docker Desktop version
  • OS version

@jnfDev
Copy link

jnfDev commented Aug 3, 2022

@jnfDev Would it be possible for you to add the following information to your comment?

  • Docker Desktop version
  • OS version

Yes, sure, here is:

OS

Screen Shot 2022-08-02 at 11 03 30 PM

Docker

Screen Shot 2022-08-02 at 11 03 57 PM

@djchapm
Copy link

djchapm commented Aug 3, 2022

@jnfDev Yes this is same for me... same OS and Docker Desktop version, experimental features enabled - I'm building an arm64 based image and docker run command will freeze at various points - can't even ctrl-c out of it on my M1 Max. I just docker stop the run from another window and restart it - sometimes it's successful, sometimes not. Never encountered this on my Mac Intel.

@Jiia
Copy link

Jiia commented Aug 9, 2022

In our team we have ran into issues importing a mysql database dump while using VirtioFS. We have a couple hundred compressed csv files totaling 10GB compressed. Those files are uncompressed on the host machine and then imported into the database using mysql cli from the host machine.

Without VirtioFS the import takes about 30 minutes and finishes succesfully. With VirtioFS the import takes about 18 minutes and finishes without errors, but only about 5% of the data gets imported into the database, the rest just vanishes into the void.

We tried removing parallelization of the importing, increasing docker system resources and making sure there's enough disk space. But the issue keeps happening if VirtioFS is enabled. The same issue has been reproduced with all our team members using different Intel MacBook Pro machines and latest macOS and Docker for Desktop. The workaround has been to disable VirtioFS during database imports.

Docker for Desktop: 4.11.1 (84025)
MacOS: 12.5
MacBook Pro (13-inch, 2020)
2,3 GHz Quad-Core Intel Core i7
32 GB 3733 MHz LPDDR4X

The container:

  mysql:
    image: mysql:5.6
    command: >-
      --sql-mode=""
      --innodb-doublewrite=OFF
      --innodb-fast-shutdown=2
      --character-set-server="utf8"
      --collation-server="utf8_general_ci"
      --secure-file-priv="/dumps"
      --innodb-buffer-pool-size=512M
      --innodb-log-buffer-size=64M
      --innodb-log-file-size=64M
      --innodb-write-io-threads=16
      --innodb-flush-log-at-trx-commit=0
      --innodb-lock-wait-timeout=1200
      --innodb-ft-min-token-size=2
      --ft-min-word-len=2
    ports:
      - "127.0.0.1:3306:3306"
    volumes:
      - mysql-data:/var/lib/mysql
      - type: bind
        source: ./dumps
        target: /dumps
    env_file: mysql.env

@PascalBrouwers
Copy link

PascalBrouwers commented Aug 9, 2022

I experience the opposite. Without VirtioFS a mysql import takes 45 minutes for a 10G database. With VirtioFS it takes 15 minutes, but that is because it is loaded in memory as with VirtioFS enabled you have no disk write access (only root can write, bug already mentioned a couple of times about this).

I'll try your settings and see what it does.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
docker_desktop
Projects
docker-roadmap
  
Developer Preview/Experimental
Development

No branches or pull requests