Docker for Mac mutagen caching feature (removed)

[rfay]

Updated 2020-07-31: With Docker Edge 2.3.4.0, this strategy had to be completely changed, so this is an edit.

We need your help testing the new experimental performance feature of Docker Desktop for Mac.

Docker Desktop for Mac has a new experimental performance improvement, mutagen caching. It has some real possibilities for significantly improving DDEV-Local performance on macOS.

The way mutagen caching works is there's a two-way sync happening under the covers (inside the Docker VM and on the host). Changes noted on the mount inside the container get queued for syncing to the host, and changes noted on the host get queued for syncing in the mount inside the container. This means that the mount is not perfectly consistent all the time (the files on the mount inside the container may not be exactly the same as the files on the host until the syncing process catches up). But it allows the system enormous performance advantages: Thousands of files can be created on the mount inside the container without waiting for syncing... but they'll show up eventually on the host (as with a composer install or npm install.) There's a high cost in terms of storage (double), because all the files that you have in your project(s) have to live on both your workstation filesystem and inside Docker's VM.

Here's what we need you to do:

1. Back up any project databases in any projects that are important to you. If you're currently on Docker Desktop stable and switch to edge, you'll lose any ddev databases you have in use. So use ddev export-db or ddev snapshot to save away copies of them. All docker images will be lost, but they'll just be downloaded again. And global caches of other kinds (like composer) are also lost, but will just be recreated when needed.
2. Install Docker Desktop for Mac edge version. You can find the latest at edge release notes. The current edge version as of this writing is 2.3.4.0. Unfortunately, if you're currently on the stable version you'll lose all containers, downloaded images, and volumes (which means databases and composer caches) (see above).
3. Turn off NFS mounting for the project you're going to test: ddev config global --nfs-mount-enabled=false && ddev config --nfs-mount-enabled=false Note that if you had globally been using nfs-mount-enabled you have turned that off.
4. Add to your project a .ddev/docker-compose.mutagen.yaml with these contents:

version: '3.6'
services:
  web:
    volumes:
    - type: bind
      source: ../
      target: /var/www/html
      consistency: delegated

5. ddev start your project. You'll now need to load a database from backup with ddev import-db or ddev restore-snapshot if you were coming to edge from stable.
6. Experiment with your project.
7. Want to verify what kind of mount your project is using? ddev exec df -T /var/www/html. If "Type" is fuse.osxfs, you're using traditional bind-mount. If it's "nfs", it's NFS. If it's "ext4", you're using the mutagen caching.
8. Want to test fsnotify behavior to make sure that filesystem changes in the container are being reflects on that host? Try fswatch, brew install fswatch

Concerns:

• Consistency: We had a feature a few years ago that used a similar technique (bgcache), and people have tried this for years with solutions like docker-sync. There have always been two main problems with docker caching approaches:
  1. If a file changes on both sides (workstation and in container) there's no good way to resolve this. Note that the current Docker mutagen feature just says "If changes are detected on both sides, the host takes precedence and the changes in the containers will be discarded"
  2. At some point the daemon in charge of syncing gets lost or crashes and all syncing stops.
• Disk space usage: This approach doubles the amount of disk storage you're going to need for a project, and all of that goes into Docker's file space, so you'll almost certainly have to increase the amount of disk you have allocated to docker.
• Stops and starts: Every time you change the mutagen caching configuration, docker has to restart, which means all your projects get stopped.
  3. It's likely that ddev import-db won't work for you because of the consistency issues. There are many other ways to import databases.

Advantages:

• Performance, of course. The ddev composer install performance with mutagen is astonishing.
• Filesystem notifications (inotify/fsevents) bubble up to the host from the container, which they do not do with NFS. This is super important for people doing website builds inside the container and waiting for notifications on the host.

Reporting results and problems

You can report results here as they relate to DDEV-Local, but we really want good information to get back to the Docker team. Please report any results or problems on the docker/for-mac issue

Casual performance results

I did a minor study of two kinds of behavior with regular docker bind-mounting, NFS mounting, and mutagen.

The two tests are

• A Drupal 8 ddev composer install with all composer caches already warmed (no downloads). You'll note that mutagen caching absolutely kills this one. The reason is that all updates take place inside the container and then are cached for presentation on the host.
• A Drupal 8 web install. This one shows good results, but not so dramatic. In this case, there is lots of in-container access to thousands of files, and only a few files being changed for update on the host.

[kaystrobach]

@rfay thank you for taking care.

Looks promising, still wonder why the OSX FS Api can’t be directly use with a fuse driver from the docker vm. But that’s something the docker team has to deal with. Afaik they did something similar for windows.

Awesome, first try looked good.

[briantully]

Hmm... doubling the disk space usage — thanks for highlighting that. Sadly my Macbook’s disk space is at a premium, so not sure it's worth it for me, especially since I am quite happy with ddev+NFS performance. Must... resist... the urge... to be on... the bleeding edge... ;)

[shaal]

TIL:
You can run brew cask install homebrew/cask-versions/docker-edge to install docker edge.

[rpkoller]

thanks for the write up of the instructions @rfay ... even though i am short on disk space but i will give it a try anyway on the weekend i suppose. guess with my dated mbp 13" early 2011 setup i am in need of more performance but at the same time it is a good real life test environment aside those latest high powered setups ;)

btw in todays dockercon are a few related sessions if anyone is interested:
Captains on Deck (Q&A hallway track)
https://docker.events.cube365.net/docker/dockercon/content/Videos/jFNDwpTbHotrtbt5x
New Docker Desktop Filesharing Features
https://docker.events.cube365.net/docker/dockercon/content/Videos/92BAM7vob5uQ2spZf
Docker Desktop + WSL 2 Integration Deep Dive
https://docker.events.cube365.net/docker/dockercon/content/Videos/K5D8qKJpX658yY8o9

overview of all available sessions:
https://docker.events.cube365.net/docker/dockercon/agenda

[edysmp]

This is great! It took ~2:45 min without cache and ~45 sec WITH cache to install Drupal 8.

[edysmp]

It seems ddev import-db is broken.
ddev import-db --src dump.sql.gz returns:

pv: /mnt/ddev_config/importdb134619592/*.*sql: No such file or directory
Ensuring write permissions for d7-site
Existing settings.php file includes settings.ddev.php
settings.ddev.php already exists and is managed by the user.
Successfully imported database 'db' for d7-site
Existing database 'db' was dropped before importing

[rfay]

Great catch. ddev import-db expects that file to be immediately available inside the container. The interesting thing is that it's the db container that is using it, and the only container we want caching for is the web container. But come to think of it, they're not giving us that option.

Mentioned your catch in docker/for-mac#1592 (comment)

It also looks to me like there's a bug in ddev, because it says "Successfully imported database 'db' for d7-site" when it obviously didn't.

[rfay]

I took a look at the import-db options with an eye to fixing it. #2295 should fix the fact that no error was reported.

But ways to fix it:

1. Docker provides a way to find out when sync is finished (best way)
2. Pipe from the host into the container. But this loses the wonderful progress bar we get from pv. Possibly we could implement a golang progress bar and use this same technique.
3. Use docker cp to push the files into container before importing them.

[rfay]

From @djs55 on the docker desktop team:

The Mac edge 2.3.2.0 released today https://desktop.docker.com/mac/edge/46268/Docker.dmg has some new file sync (mutagen) behaviour. Since the last edge
docker run -v host:container:delegated will set up the two-way sync so you don’t have to use the UI. This makes it easier to (eg) share optimised docker-compose.yml configurations without asking people to manually change settings
touch file && docker run -v … stat file should work (thanks to a flush on container start) and docker run -v … touch file && stat file should work (thanks to another flush on container stop)
the symlink mode is now posix-raw which should prevent errors where symlinks are dangling
the ~/.mutagen.yml allows further customisation including global excludes (if it’s synchronising too much)
the default file permissions are more permissive, hopefully there will be fewer permission denied errors in the container
Feedback is most welcome!

I'm not sure what this will mean to ddev, whether it helps the problems people hit or not. I'm pretty sure ddev import-db can't work with its current implementation, but I'll give it a try.

[rfay]

There's a new edge release, and there's lots of discussion in docker/for-mac#1592 - anybody is welcome to return to this. I don't think ddev import-db will work with it yet, but the performance is superb, especially if you can handle a little bit of consistency risk.

[rfay]

The latest version of Docker Desktop for Mac Edge is 2.3.2.0, came out today, see https://docs.docker.com/docker-for-mac/edge-release-notes/

With it I am able to use ddev import-db. Lots of people will like Mutagen as opposed to nfs-mount-enabled because of the fsnotify/inotify feature that it does right. see #2361 about gatsby/gulp.

[rfay]

As of 2020-07-31, Docker Edge 2.3.4.0, the strategy outlined here does not work with ddev (you don't get the mutagen syncing at all). Unless Docker changes direction, supporting Mutagen will require explicit code support in DDEV-Local.

[rfay]

You can use a docker-compose.mutagen.yaml to use the mutagen feature.

Place a file named docker-compose.mutagen.yaml in your project's .ddev folder, with these contents:

version: '3.6'
services:
  web:
    volumes:
    - type: bind
      source: ../
      target: /var/www/html
      consistency: delegated

I'll update the OP with that.

[WidgetsBurritos]

@rfay,

Having been using this for over a week now, I'll say that I've had zero problems with ddev import-db. I'm using 2.3.4.0. I've probably done it at least 10 times.

[rfay]

@WidgetsBurritos I'm pretty sure it also relates the the size of the database. The smaller the database, the less likely the problem is to occur. I assume that's why it hasn't happened to me much either, I should test it with a more representative db.

[WidgetsBurritos]

@rfay,

One of the database files I've been working with is a 921MB. The file is not gzip-compressed, mind you, so I'm unsure if that factors in any way, but it seems to import just fine.

[rfay]

Well that's absolutely good news!

[rfay]

It's looking like this Docker Edge feature may not be coming to Docker Stable very soon, or perhaps at all. There are ways to support it without that though.

[rpkoller]

Is the mutagen caching too buggy and unreliable to roll it out in docker stable? or are the devs at docker hoping for the virtualization api coming to macos and being able to provide something similar to wls2 on windows since the apple keynote?

[xenoscopic]

Hey @rfay and @rpkoller! Mutagen developer here (though not involved with the Docker for Mac integration or able to speak for Docker). My impression is that they're still just in something of a testing period, trying to understand if the synchronization heuristics they've chosen will work for a broad enough range of use cases.

It's also possible to use Mutagen independently to achieve the same caching setup with much more granular control over synchronization behavior. Basically you create an in-VM Docker volume and synchronize code into it. This can be done manually or as part of an automated setup (and there are a few options for that, e.g. Mutagen's Compose support). I'd like to work on improving Mutagen's scriptability in v0.13, so I'd be happy to have a chat about what sort of design would be helpful to your integration efforts. This might be a more convenient and performant route for specialized integration cases such as yours, even if Docker for Mac does eventually ship this feature outside the edge channel.

[rfay]

@rpkoller the copy-type integration is so different from any kind of traditional mount that it just causes people confusion. For example, a large file copied into a mounted directory on the host may not appear inside the container for seconds or minutes, and vice versa. It's not actually buggy, it's just a completely different kind of expectation, and that means that if/when they roll it out they are likely to have a lot of support issues on their hands. It's also possible they can rethink how they're doing it and limit that. In my experience, it's far more reliable (as far as not breaking) than our old webcache feature (which was the same idea) from a couple of years ago. But I've seen lots of confusing things happen with it.

@havoc-io thanks for checking in! Yeah, even before the docker integration we added a feature to enable direct use, as described in @cweagans https://cweagans.net/2020/04/23/fast-local-development-on-macos-with-ddev-and-mutagen/ - and we've definitely been talking about your new explicit compose support. I think direct use might be actually more appropriate than using the docker approach, because we'd have more control (and more control of people's expectations, because they'd have to be pretty deliberate about using it. A mutagen-syncing approach is absolutely astonishingly wonderful for applications where everything is happening inside the container (like automated testing), and I think we should plan to do that regardless of what happens with Docker.

[rfay]

The Docker Edge mutagen feature has been removed in Docker Edge 2.3.5.0, so this issue won't be going anywhere any time soon. docker/for-mac#1592 (comment)

[rfay]

Closing this for now, as the feature is not in Docker edge and won't be in stable.

There are at least two good ways to implement mutagen for people who like it.

• @cweagans 's approach in https://cweagans.net/2020/04/fast-local-development-on-macos-with-ddev-and-mutagen/
• Mutagen's own docker-compose support, https://mutagen.io/documentation/orchestration/compose - there are some docker community examples in the comments around File system performance improvements docker/for-mac#1592 (comment) as well.

[cweagans]

For the record, if Mutagen's built-in Docker Compose support works well enough, I plan to deprecate my custom solution in favor of that feature + some automated way to enable it for a ddev project.

[newtondev]

Very sad to see this feature go away, because volume binds are horrible on Mac OSX. Luckily our services now use the mutagen project yaml file and it is lightning fast.