Cannot create a file in the sync volume root folder by non-root user

[sprymiker]

Which version of Mutagen are you using (mutagen version)?

0.12.0-beta1

Which operating system (platform/version/architecture) are you using?

MacOS 10.14.6 (18G6020)

What is the issue that you're experiencing?

A terminating sync session is stuck forever.

What are the steps to reproduce the issue?

1. brew install mutagen-io/mutagen/mutagen-beta
2. cd /tmp
3. git clone https://github.com/mutagen-io/example-voting-app.git ./voting
4. cd ./voting
5. mutagen compose up -d
6. docker exec -it -u 1000:1000 mutagen_result_1 mkdir somedir

What is the expected result of these steps in the absence of the issue?

Folder is created
docker exec -it -u 1000:1000 mutagen_result_1 ls -al somedir

total 8
drwxr-xr-x 2 node node 4096 Aug 27 10:40 .
drwxr-xr-x 8 root root 4096 Aug 27 10:40 ..

What is the actual result?

mkdir: cannot create directory 'somedir': Permission denied

Is there any other information that might be helpful?

Docker set the owner for named volume root folder at the first initialization.
The sidecar image has root owner set for /volumes, therefor volume root folder has root owner and 755 permission that disallow to write into it by a non-root user.

docker exec -it mutagen_result_1 ls -al
total 84
drwxr-xr-x 8 root root  4096 Aug 27 10:40 .
drwxr-xr-x 1 root root  4096 Aug 27 10:15 ..
...

Possible solutions

The sidecar's entry point:

• can set the owner from mutagen configuration, as we do for the contents inside.
• OR can set 777 permission for the root folder. find /volumes -maxdepth 1 -type d -exec chmod 777 "{}" \;

[sprymiker]

This workaround works:

volumes:
    sync-volume:
        external: true

docker volume create --name="sync-volume"
docker run -it --rm -v "sync-volume:/data" busybox chmod 777 /data

$ ls -al 
total 2360
drwxrwxrwx  9 root    root        4096 Aug 27 13:48 .
drwxr-xr-x  1 root    root        4096 Aug 27 13:47 ..
...

However, it is not so comfortable to manage a volume outside docker-compose.
Also, security concerns should be considered.

[Toilal]

I think it's a common docker issue.

You have to create the folder inside the image (with Dockerfile) and chown it (still in Dockerfile) with a user matching your host user id. Then, on volume creation, it will use this gid/uid.

Fixuid may also be an option.

https://github.com/boxboat/fixuid

[xenoscopic]

This is a known Docker (and Compose) issue (see, e.g., docker/compose#3270, moby/moby#3124, and moby/moby#2372).

It would really be more appropriate if there were a way to specify ownership of Compose-created volumes as part of the Compose volumes section rather than something Mutagen-specific. Unfortunately there's been some resistance to that in the past (with good reason), but it's a common enough use case that maybe it's worth having Compose add some sort of exec-based chmod hack to handle this (and it would have to be a hack since the Docker Engine API doesn't support setting volume ownership).

I would recommend opening an issue on the Compose Spec repository and seeing if they're more receptive to it than the previous issues on the Compose repository itself. If that gets shot down, we can reopen this issue and see if maybe there's something that Mutagen can offer as a workaround (such as setting ownership from the sidecar).

[sprymiker]

Hello @havoc-io,

I fully agree, it is a common Docker issue.

However, I do not have the issue as my container runs first and inits the volume with the proper ownership.

Bringing mutagen + compose initiates the "magic" sidecar container that is running at the very first and inits the volume with the root ownership (and bring problems). And I have no control over that. The only way is initing the volume before mutagen compose up, however, that is just overhead and missing of good features of docker-compose to manage volumes.

[xenoscopic]

Hey @sprymiker, thanks for the additional context. I see now how Mutagen would interfere with the existing workaround. I'll reopen and think about what makes the most sense here. There may be other workarounds for other problems that are also broken by the additional sidecar container starting first.

[pjv]

@havoc-io is there a way for the sidecar container to chown the volume root directory to the configured defaultOwner:defaultGroup as @sprymiker suggested?

[xenoscopic]

@pjv There's not at the moment. I'd be hesitant to make such behavior automatic, though perhaps that would be acceptable if Mutagen detected that the volume was empty? This is such a common case that it might be worth doing. If we go this route, we'd still need to make sure the issues I mentioned here don't cause problems, but I don't think they will.

I'm not sure exactly how to implement this... probably just some small executable in the sidecar container that Mutagen Compose will invoke as part of its up sequence. Or, perhaps Mutagen's synchronization sessions could be extended with a setting along the lines of SetSyncRootOwnershipOnStartupIfDirectoryExistsAndIsEmpty. We can brainstorm longer names if this one isn't specific enough :-P

---

Alternatively (or perhaps additionally), Mutagen could allow users to specify a custom sidecar image (based off of the standard image but using the same dynamic image building functionality of Compose) that could pre-create mountpoints and set their ownership/permissions. This would also solve the problem since volumes inherit the permissions of the first mountpoint to which they're attached (and the sidecar is always started first), and it might allow more flexibility in general. The mountpoints would have to be created using a certain scheme, but that could be documented. I just don't want to support whatever other stuff users might inject into this image, so I'd really only consider this as a last resort, or if it provided some significant additional value, though none comes to mind beyond this permissions problem.

[pjv]

@havoc-io I looked at the sidecar repo over the weekend hoping to see if I could help out with a PR but I’m not a go coder and all the mutagen code is pretty greek to me.

WRT making the chowning automatic, I’m thinking that if the user has configured a defaultOwner and/or defaultGroup in the x-mutagen config block then it would be acceptable (and preferred) to chown the volume mount point - maybe always, but certainly if it’s empty. Maybe nothing happens if no defaultOwner:defaultGroup is configured.

Let me know if you need help testing the custom sidecar image idea.

Edit: if this anecdotal data point means anything, I just brought up my little wp dev template project (described in #228) and shelled into the sidecar container and did chown 1000:1000 /volumes/wp-content and it correctly changed the uid:gid in both the sidecar container and also in my running php container.

So at least for the purposes of my particular use-case, executing a simple chown of the volume mount point in the sidecar container during mutagen compose up before other containers are brought up looks like it would be a fix.

[paneq]

WRT making the chowning automatic, I’m thinking that if the user has configured a defaultOwner and/or defaultGroup in the x-mutagen config block then it would be acceptable (and preferred) to chown the volume mount point - maybe always, but certainly if it’s empty.

Yes, please. That sounds great.

When I use docker-compose directly with config of:

    volumes:
      - .:/home/app:cached

I get:

app@0a45308c59dc:~$ ls -alh
drwxr-xr-x   54 app  app  1.7K Sep 17 07:38 .

but when I use mutagen compose with the config of:

version: '3.4'
services:
  frontier-dev-server:
    container_name: frontier-dev-server
    build:
      context: .
      target: frontier-base
    working_dir: /home/app
    volumes:
      - tf-devx-frontier-app:/home/app

x-mutagen:
  sync:
    defaults:
      mode: two-way-resolved
      ignore:
        vcs: true
    code:
      alpha: "./"
      beta: "volume://tf-devx-frontier-app"
      mode: "two-way-resolved"
      configurationBeta:
        permissions:
          defaultOwner: "id:469"
          defaultGroup: "id:469"

I get

app@b49cdf7d893b:~$ ls -alh
drwxr-xr-x   24 root root 4.0K Sep 17 07:50 .

which is unexpected. It would be really nice to get the same behavior, even if it requires a bit of additional configuration, which I need to do anyway.

I like the idea of using defaultOwner and defaultGroup by default, and if we want to make it super-configurable, these could be overwritten by additional configuration options such as mountOwner, mountGroup if present. So the logic would be

mountOwner = config.mountOwner || config.defaultOwner
mountGroup = config.mountGroup || config.defaultGroup

volumes inherit the permissions of the first mountpoint to which they're attached

Is it possible for mutagen to inspect the compose file and the image to evaluate which permissions should be used? Or would that be too complex?

[xenoscopic]

@pjv Thanks for testing the chown behavior, glad to hear that's the case. I'll definitely give you a ping if I decide to implement the custom sidecar idea. In the meantime, I think I'm just going to go with a default heuristic for setting these permissions (see below).

Is it possible for mutagen to inspect the compose file and the image to evaluate which permissions should be used? Or would that be too complex?

@paneq Mutagen's actually already inspecting the Compose file to read configuration information, the issues are just (a) whether or not it should change volume root ownership by default (vs. as a configuration setting) and (b) what mechanism it should use to do that (i.e. should it be done by sync sessions or as something special to the sidecar container).

I've been playing with adding a configuration option to the permissions block that would tell Mutagen synchronization sessions to overwrite existing ownership for empty synchronization roots, but even that behavior becomes a bit unclear in terms of what the configuration option looks like (e.g. whether it should have other more general behaviors), when it's applied, whether it's reapplied, etc. I also don't think I'd even consider adding something so specific if Docker Compose supported volume ownership to start with, so I'm hesitant to add it as a general sync session option.

Instead, I'm leaning towards just enabling this default ownership setting as a heuristic in cases where (a) the synchronization root is empty, (b) it exists under /volumes, and (c) the Mutagen agent is running in the sidecar container. It's such a specific need that I'm not sure I want to expand Mutagen's configuration surface any further to accommodate it or make users dive through the documentation to find some obscure option.

So I'll try to prototype up this default behavior and see how it looks. I expect I'll have a chance to tackle it on Sunday evening and Monday.

[pjv]

@havoc-io have you had any time to look at this?

[xenoscopic]

@pjv Sorry I've been silent on this; I ended up having less time than anticipated. I started prototyping the changes, but there was some edge case behavior that needed further consideration.

For example, at the moment, the Mutagen sessions defined in x-mutagen start up in random order, so if there were two sessions targeting the same volume root, it's not defined which one would take priority. That random startup order can and probably should be fixed, though it's somewhat complex because of the YAML decoding APIs currently being used.

Beyond that, it wasn't immediately clear where in the synchronization code/process the ownership setting should be handled and whether or not it should be re-handled on subsequent synchronization cycles. I'm also wondering if it makes more sense to implement this in the context of a more general permission propagation setting.

I'm still working on it and thinking about it. I'm hoping to have a solution in the next few days. Thanks for following up though!

[pjv]

@havoc-io I’m ignorant of the specifics of what the sidecar is doing altogether, but just imagining creating mountpoints from the linux command line, I’d think that the time to set the owner and group would be right after creating the mountpoint.

mkdir bla-bla-bla
chown preferred_owner:preferred_group bla-bla-bla

I think you could document that the order of the sessions is not defined and then it’s up to the user to make sure that they are specifying the same owner:group for the mount point in all sessions (which I imagine is going to cover 99.5% of use-cases) or know that the owner:group will be undefined due to the undefined ordering.

I guess I think that it should definitely be set per the config on the original creation of the mountpoint and then whether to re-handle on subsequent sync sessions seems to me to be a push; I could see an argument for it either way but I have a slight preference for not re-doing it later on in case someone at some point finds a reason for shelling into the sidecar and manually changing it themselves. Again that seems like an extreme edge case and something that can be documented and ignored (for now).

[ByScripts]

Hi.

Same problem here, and I don't understand how to fix the problem.

In the Dockerfile, I create a dockeruser user and a dockergroup group with the same UID/GID as local macOS user.

Still in the Dockerfile, I create a /app directory and chown it to dockeruser:dockergroup.

In docker-compose.yml I have this (extract):

volumes:
  code:

services:
  my-app:
    volumes:
      - "code:/app"

x-mutagen:
  sync:
    my-app:
      alpha: "/path/to/code"
      beta: "volume://code"

But in the container, if id gives me my user (dockeruser/dockergroup, with correct UID/GID), the /app and its content is still owned by root:root

I tried to add this to x-mutagen.sync.my-app configuration :

configurationAlpha:
        permissions:
          defaultOwner: "id:<my UID>"
          defaultGroup: "id:<my GID>"
configurationBeta:
        permissions:
          defaultOwner: "id:<my UID>"
          defaultGroup: "id:<my GID>"

I also tried only with configurationAlpha OR configurationBeta alone... no success.

And if I replace id:<UID> with dockeruser, I get an error saying the this user is not found.

I'm lost, and have no idea to what to try know...

[pjv]

@havoc-io

i have been able to work around the problem that we've been discussing here (and @ByScripts just repeated) in an impossibly kludgy way by executing something like this once all the containers are up and running:

docker exec -t <name of sidecar container> sh -c "chown <my UID>:<my GID> /volumes/*"

This needs to be repeated every time the container is recreated (i.e. after a docker-compose down or mutagen compose down).

Until you come up with a better way to work this all out, can we get an extension to the x-mutagen configurationBeta permissions, e.g.:

configurationBeta:
        permissions:
          defaultOwner: "id:<my UID>"
          defaultGroup: "id:<my GID>"
          chown: "true"

...that would effectively do the same thing as my docker exec one-liner once the sidecar container is up rather than our having to remember to do this manually?

[ByScripts]

@pjv I tried your solution. It worked until I edited a file, its owner/group was set back to root.

I surrender.

Another thing I don't understand, is that Mutagen was then included in Docker Edge, then removed in favor of gRPC FUSE. This option is activated is my Docker settings, but I don't see any change about the performances.

Without Mutagen, npm install takes 168s, with Mutagen it goes down to 37s.

[TomasLudvik]

Hello, I have just run into this issue, is there any progress? Thank you

[TomasLudvik]

I am setting right owner of root in my Dockerfile, but it gets changed after mutagen compose up -d (worked fine with docker-compose and docker-sync). Currently only way for me, is to run docker-compose exec -u root php-fpm chown -R www-data:www-data /var/www/html after containers are created. @xenoscopic would it be possible to at least add some option to x-mutagen configuration to define commands, that would be run after mutagen-compose up?

[newtondev]

@TomasLudvik this is currently how I am getting around the issue by using the afterCreate lifecycle hook in the mutagen.yml project file which works well for my local environment.

[TomasLudvik]

@newtondev thanks for suggestion. I have tried it, but mutagen.yml config seems to be not working with mutagen compose.

[Jarzebowsky]

Any update regarding this? I'm stuck also with this permissions problem as mutagen compose up returns an error that he can't find user named www-data

[rfay]

Docker is inconsistent about the initial ownership of a volume. You'll likely have to add a way to chown to the user that you're running the container as. For example, ddev has to chown the volume like this:

func SetMutagenVolumeOwnership(app *DdevApp) error {
	// Make sure that if we have a volume mount it's got proper ownership
	uidStr, gidStr, _ := util.GetContainerUIDGid()
	util.Debug("chowning mutagen docker volume for user %s", uidStr)
	_, _, err := app.Exec(
		&ExecOpts{
			Dir: "/tmp",
			Cmd: fmt.Sprintf("sudo chown -R %s:%s /var/www", uidStr, gidStr),
		})
	util.Debug("done chowning mutagen docker volume, result=%v", err)
	return err
}

[xenoscopic]

Indeed, there's just a lot of inconsistency with regard to volume ownership, so setting it explicitly through some mechanism is the best option for now. Unfortunately neither the Docker CLI nor Compose provides a mechanism for this at the moment. This is something that will need to be considered as Mutagen moves to supporting Compose V2, probably as some sort of Mutagen-specific hack/heuristic.

However, the issue with not being able to find the user www-data is likely something different, probably related to no user with that uid existing inside the relevant container image. If you want to post the error message on Slack, I can have a look a bit later.

[TomasLudvik]

@xenoscopic sorry for late response. We have custom user named www-data created in Dockerfile, so we can correctly set user permissions between local machine and containers in order to avoid using root.

On Linux, Windows with WSL and MacOS with Docker-sync rights are working as intended, but when mounting volume using mutagen compose, the files in project root directory are owned by root instead of www-data.

Parameter afterCreate in x-mutagen in docker-compose.yml like it is in mutagen.yml would be fully sufficient for me as I could add docker-compose exec -u root php-fpm chown -R www-data:www-data /var/www/html there and everything would be set as expected.

[pjv]

@xenoscopic I can't say for sure this would work for all use-cases, but I don't see why it wouldn't. Try it and let's see. Change the sidecar behavior so that after that container comes up, and before the other containers defined in the compose file come up, the sidecar chowns everything under /volumes/ to the user and group set in the x-mutagen section of the compose file.

[xenoscopic]

@pjv I also think that's the right approach (but probably conditioned on the volume being empty). I think this will probably just get punted to the Mutagen Compose V2 implementation though, as the V1 implementation is super crufty at the moment. I'll take a look though and see if it's easy enough to do this robustly. It's mostly an agent change anyway.

I've been deep-diving into the compose V2 and compose-cli implementations today and I think I'm ready to take a stab at it this weekend. Will have a better ETA after trying.

[xenoscopic]

Two updates to close out this issue:

First, Mutagen now implements a heuristic to support this in Mutagen v0.13.0-beta2.

Specifically, if a Mutagen Compose synchronization session targets an empty volume root, and a non-default owner, group, or directory mode setting has been specified for that session, then Mutagen will set the permissions of the volume root to match. I think this should offer users enough control to set permissions as desired in most cases. If you need different permissions for different directories, the solution is just to use multiple volumes and synchronization sessions (both of which are cheap).

Second, this new behavior was really implemented to support the Compose-V2-based Mutagen Compose, which will very soon be replacing Mutagen's Compose V1 integration. This is now available to try if you'd like and more information can be found here.

If this new heuristic isn't sufficient, isn't working properly, or needs a bit of tweaking, feel free to comment on this issue or drop by the Mutagen Community Slack Workspace and we can definitely continue the discussion.