Docker for Mac Mutagen Refugee Discussion

[xenoscopic]

With the removal of Mutagen from Docker for Mac Edge, there's been a lot of interest in continued use of Mutagen for development. While development of gRPC-FUSE continues, I'd like to offer a workaround to developers looking to emulate the previous Docker for Mac Edge functionality. This workaround involves manually creating Docker volumes and Mutagen synchronization sessions. While this isn't as elegant as the Docker for Mac UI, it does provide more granular control over synchronization.

This isn't the only way to use Mutagen with Docker for Mac, it's just an initial workaround proposal. If users have additional setups they'd like to share (e.g. using an SSH sidecar container), please add them below! As @driq already pointed out, Mutagen's Compose integration offers another way to automate this caching.

Please feel free to email me at jacob@mutagen.io if you'd like to discuss your specific setup and needs via email or video chat.

Also, if you'd be interested in a tool that automates this (maybe even a simple GUI), please let me know below!

Install Mutagen

First, you'll need to install Mutagen. If you're on macOS, the best and easiest way to do this is with Homebrew:

brew install mutagen-io/mutagen/mutagen

Create a volume for caching files

Now, for each directory that you want to sync/cache inside the Docker for Mac VM, you'll need to create a named volume:

docker volume create mycache

Create a container to access the volume

In order for Mutagen to access the volume, you'll need to create a container with the volume mounted. You can use the Mutagen sidecar container for this purpose (it's just a no-op entry point):

docker container create --name mycachecontainer -v mycache:/volumes/mycache mutagenio/sidecar
docker container start mycachecontainer

The /volumes directory already exists inside the Mutagen sidecar container, so it makes a good location to create mountpoints.

You can use one sidecar container per volume, or one sidecar container for multiple volumes. The only disadvantage with the latter approach is that you have to mount volumes at container creation time, which makes adding volumes later tricky.

(Optional) Change volume ownership/permissions

Docker volumes default to root ownership with rwxr-xr-x permissions. If you'd like to change that, now is a good time to do that. You can use something like the following command(s):

docker exec mycachecontainer chown 1001:1001 /volumes/mycache
docker exec mycachecontainer chmod go+w /volumes/mycache

The Mutagen sidecar container runs as root, so you can change ownership to any UID/GID you like, I've just chosen 1001 for this example. Your exact needs will vary depending on the containers you're using. If you run your containers as root, this may not matter at all.

Create a Mutagen synchronization session

Finally, you'll want to create a Mutagen synchronization session to sync files into the volume. This will look something like the following:

mutagen sync create [options] <path> docker://mycachecontainer/volumes/mycache

A more concrete example might look something like:

mutagen sync create --name mycache --sync-mode=two-way-resolved --ignore-vcs --ignore 'node_modules/' --default-owner-beta="id:1001" ~/Projects/myapp docker://mycachecontainer/volumes/mycache

Mutagen synchronization sessions have a lot of options that you can use to customize behavior. For a full list, see mutagen sync create --help. I would recommend reading the section on synchronization sessions in the Mutagen documentation, as well as the sections on ignores, version control systems, and permissions.

The most important takeaways are the following:

• Mutagen is a synchronization engine, meaning that it needs to copy files into the target volume. While Mutagen is designed to handle 10,000-100,000 files efficiently, you can't synchronize your entire home directory into the container and expect good performance. You should target caching of individual codebases.
• Mutagen has multiple synchronization modes and defaults to bidirectional synchronization. You probably want to use the two-way-resolved mode (i.e. --sync-mode=two-way-resolved)
• Mutagen allows you to ignore content and you should use this to your advantage. For example, things like node_modules directories make ideal candidates for ignoring. They are big and aren't generally portable between macOS and Linux anyways.
• You shouldn't synchronize version control directories like .git. Use the --ignore-vcs flag.
• Mutagen will execute as root inside the sidecar container. This means that files and directories it creates inside the cache will default to root ownership with restrictive permissions, but it also means that you can use Mutagen's permission and ownership settings to configure this however you'd like to suit the needs of your application.
• For storing common configuration, you can create a file at ~/.mutagen.yml. Mine looks something like:

sync:
  defaults:
    ignore:
      vcs: true
      paths:
        - ".DS_Store"
        - "*.py[cod]"
        - "__pycache__/"
        - "*.egg-info/"
        - "*~"
        - "*.sw[a-p]"

• You can get detailed status information on Mutagen synchronization sessions using the mutagen sync list and mutagen sync monitor commands. For more information on Mutagen's general command structure, check out the getting started guide.

Using the cache

In order to use the cache, you'll most likely want to replace a bind mount in your Docker Compose file with the named volume that you're using as a cache. This transition might look something like the following:

services:
  myapp:
    ...
    volumes:
      - .:/path/in/container

to

volumes:
  mycache:
    external: true

services:
  myapp:
    ...
    volumes:
      - mycache:/path/in/container

You can also use the cache in manually created containers via the -v flag in docker [container] create and docker [container] run.

Tearing it all down

Once you decide that you're done using a particular cache (which could be after an hour, a day, a year, etc.), you can tear it all down with the following:

mutagen sync terminate mycache
docker container stop mycachecontainer
docker container rm mycachecontainer
docker volume rm mycache

[driq]

Much appreciated!

Just to add: For those using docker-compose, if you follow the instructions at https://mutagen.io/documentation/orchestration/compose many of the above-mentioned steps will be done automatically for you.

Docker Compose support has not landed in a stable version of Mutagen yet, so you will have to install it from the beta channel:

brew install mutagen-io/mutagen/mutagen-beta

Once set up, this comes quite close to the setup I used to have with delegated volumes in Docker Edge 2.3.7.0.

[driq]

One gotcha: make sure that you don't use bind mounts for volumes that are cached with Mutagen, as this will result in both Mutagen and the native Docker for Mac implementation getting involved, resulting in truly bad performance.

I had such bind mounts configured when migrating from Docker Sync, which does not use the volumes configured in your docker-compose.yml file.

[ryansch]

I've gone so far as to remove the /Users and /Volumes mounts from Docker Desktop.

Edit: This has the nice side effect of docker itself spitting out a useful error message when I try to use a bind mount.

[ryansch]

@havoc-io I just finished onboarding our dev team last week with our new setup. They've already been using a wrapper I originally adopted from IFTTT so it was straightforward to teach the wrapper some new tricks.

The version we're using currently is here: https://github.com/outstand/dash/blob/master/bin/dev

I found it really helped to remove mutagen's sync volumes entirely when a developer runs the equivalent of mutagen compose down. That behaviour is here: https://github.com/outstand/dash/blob/e9cc77e949608d12b2e6aa57dbd76e5d705bffd8/bin/dev#L140

[dzhgenti]

Hey guys,

I'm trying to set up mutagen sync for local dev environment and getting an error when trying to set the default file/directory permissions. Here is how I try to create a sync:

mutagen sync create \
        source/sites \
        docker://root@drupal/var/www/sites \
        --name=my_sync \
        -i isource/sites/all/themes/custom/slicing/node_modules \
        --ignore-vcs \
        --default-file-mode-beta 0750 \
        --default-directory-mode-beta 0770 \
        --default-group-beta "www-data"

I'm getting this error:

Error: invalid default file mode for beta: executability bits detected in file mode

There is a named volume mounted to var/www/.

Inside the container, nginx is running under www-data and I need it to be able to execute files and write to certain directories.

Any ideas on how to resolve it?

Thanks!

[iperelekhov]

Hey guys,

I'm trying to set up mutagen sync for local dev environment and getting an error when trying to set the default file/directory permissions. Here is how I try to create a sync:

mutagen sync create \
        source/sites \
        docker://root@drupal/var/www/sites \
        --name=my_sync \
        -i isource/sites/all/themes/custom/slicing/node_modules \
        --ignore-vcs \
        --default-file-mode-beta 0750 \
        --default-directory-mode-beta 0770 \
        --default-group-beta "www-data"

I'm getting this error:

Error: invalid default file mode for beta: executability bits detected in file mode

There is a named volume mounted to var/www/.

Inside the container, nginx is running under www-data and I need it to be able to execute files and write to certain directories.

Any ideas on how to resolve it?

Thanks!

Looks like you are trying to build a LAMP stack dev env.
I'm using configuration below for mine dev env (mutagen.yml):

  defaults:
    permissions:
      defaultOwner: "id:501"
      defaultGroup: "id:20"
    symlink:
      mode: "posix-raw"
    mode: "two-way-resolved"
    ignore:
      vcs: true
      paths:
        - "log"
        - "generated"
        - "*~"
        - "*.sw[a-p]"
        - "node_modules/"

501 is my user ID (host system) and 20 - group ID (also at host system).
After sync starts, wait until mutagen hits Waiting for changes status and then execute docker exec mycachecontainer chmod -R +x <absolute path to project>

[dzhgenti]

thanks, @obeygaint! Might be a workaround... But we spin up and destroy containers quite often, so I really hoped there is a way to set permissions in the config. Do you know if it's possible to find out if sync finished from within a container? Perhaps something similar to mounting docker.sock to a container to access Docker API. I would then write a bash script to change permissions after the sync is finished.

Oh, btw, what about new files created on the host? Do I need to re-run chmod on them too?

[xenoscopic]

@dzhgenti The Error: invalid default file mode for beta: executability bits detected in file mode error is because Mutagen wants to control executability bits (i.e. 0111) for files (but not directories) that it synchronizes and thus requires that only read/write bits are specified in file permission configuration. The idea is to implement behavior similar to that of Git and rsync. The error message was designed to avoid confusion; I had hoped it would be better than just implicitly ignoring the 0111 bits of permission settings, but I think maybe it's actually just more confusing. I'll try to improve it. Some more background information can be found here. The solution just to exclude these bits from the specified permissions (i.e. use 0640 instead of 0750, though keep 0770 for directories).

Other than that, the configuration you're using looks good. I might recommend adding --default-owner-beta=www-data as well (if appropriate).

Oh, btw, what about new files created on the host? Do I need to re-run chmod on them too?

New files created on the host will be propagated to the container using the permissions you're specifying when you create the session.

Do you know if it's possible to find out if sync finished from within a container?

There isn't currently. The closest alternative is to use Mutagen's Compose support, which will ensure that the sync is finished before other services start. This would also allow you to codify permissions as part of your Compose configuration.

I'd be happy to follow up by email and/or video chat (email in GitHub profile) if you'd like to have a more direct conversation about what you're trying to accomplish.

[dzhgenti]

@havoc-io, thanks so much for your prompt and in-depth reply!

[fabianx-ai]

Here is an idea (pretty rough though) for avoiding the full-sync without having to ignore everything:

• Docker has support for overlay2
• FUSE file systems are pretty simple (I worked years ago with unionfs and aufs)
• Once a file is accessed that is not actually there, the mutagen daemon is getting a signal to sync it over, then -EAGAIN is returned to try opening the file again (alternatively all calls would need to be delegated via FUSE for this fopen session)

Overall there are always two problems when writing remote network file systems:

• Attributes + stat() + Directory Structure (mutagen efficiently solves that) -> might get away with correctly sized zero byte filled files?

• Actual data reading/writing (mutagen solves that, but it can also sync data that is not needed into the volume) -> There is only a few system calls that actually read / write data: fopen, fread, fwrite, fclose, sendfile, readfile, etc.

The idea is to avoid FUSE calls whenever possible, by using three layers:

• Real data layer: Files with correct sizes and correct data
• FUSE layer: Only reacts to fopen(), fwrite(), while all getdents(), stat() calls are forwarded to the lower level [https://github.com/dimkr/overheadfs/blob/master/overheadfs.c is probably a great starting point]
• Stat layer: Always has all files with correct permissions, etc., but they are zero based and take no space except for metadata.

So a mutagen-on-demand would be really nice as you could just mount your home directory - once a file was synced once it will continue to be monitored indefinitely.

"Cache" would obviously grow over time, but that is to be expected and one could just destroy the mutagen volume and start over. (assuming docker correctly frees space of deleted volumes)

[xenoscopic]

Hey @LionsAd, that's sort of an interesting middle ground between osxfs/gRPC-FUSE and Mutagen. I'm not sure that the Mutagen file index (as it exists now) would be sufficient to return adequate stat() results (and it's not structurally suited to that kind of access), but one could certainly imagine that caching file metadata en masse (with a more purpose-built data structure) would accelerate certain access patterns (e.g. those that stat() but don't open(), such as Git).

@leehambley was looking at building a framework to understand what the access patterns for various tools look like and whether or not a FUSE-based filesystem could be optimized accordingly. In practice, I think part of the problem is that a lot of code (e.g. build tools) out there use interleaved stat()/open()/read()/close() system calls that sustain huge round-trip overheads to the host (regardless of any metadata caching) and it's that latency that builds up to tens or hundreds of seconds of delay for certain operations.

I do think there might be a place for some sort of predictive model that could be used heuristically to pipeline certain options in on-demand virtual filesystems (e.g. if the virtual filesystem sees 2-3 stat()/open() pairs in the same directory after returning a getdents() result, it could start pipelining open() operations for the remaining files in the directory), but one would probably need a lot of data to build such a model, and I don't think there's any guarantee that a generalized heuristic could be formulated.

[leehambley]

I think part of the problem is that a lot of code (e.g. build tools) out there use interleaved stat()/open()/read()/close() system calls that sustain huge round-trip overheads to the host (regardless of any metadata caching) and it's that latency that builds up to tens or hundreds of seconds of delay for certain operations.

Indeed, naïve benchmarks show that most syscalls are under 5µ seconds on Linux Docker (it's native, native native), on Docker4Mac (osxfuse, so not the edge build, which I haven't tested yet) it's more in the order of 130ms. open and read were especially expensive, but even stat/lstat/access were 80 µsec or more, when on native, those "fast check" are more like 1..3µ sec.

In other words, osxfuse (and probably the grpcfuse, but I haven't checked it yet) is something like 25× slower for the fixed costs of accessing the filesystem.