chown operation not permitted with mongo:3.4 when mounting NFS volume via Kubernetes

[christianhuening]

Hi there,
i am trying to run MongoDB 3.4 on Kubernetes with an NFS backed volume. The volume gets mounted quite nicely but when the container tries to start here's what it outputs:

chown: changing ownership of '/data/db': Operation not permitted

My settings:

• My NFS share has 'all_squash' set to cope with different uid/gid settings across the network
• Image version: mongo:3.4
• Kubernetes version : 1.5.1
• Docker version 1.12.1
• On the host that mounts the NFS share and then mounts it into the docker container, the directory ownership is set to 999:root (uid 999 equals the mongodb user inside of container) by Kubernetes

Since I am using Kubernetes it seems there is no option to use the --user flag which I thought would be helpful here. I tried setting the security context fsGroup attribute for the Pod / container to 999 but that didn't change a thing.

Any help would be greatly appreciated.

[christianhuening]

Was able to resolve that by providing
"command":["mongod"]

in my Deployment Kubernetes Manifest. That overrides the entrypoint.sh script and thus does not execute the chown. Whether that's ok or not I cannot tell.

[yosifkit]

Yeah, since your volume already has the correct permissions, you don't really gain anything from the entrypoint script. Though we could make it a little smarter to only chown when it detects that it needs to.

[christianhuening]

created a new ticket for making the script smarter : #128

/closed

[jgato]

I am having a similar issue but using Rancher instead of Kubernetes. I have found few issues/help about that. @christianhuening is there any progress about that? any workaround? I have tried overriding entrypoing with mongod --auth, using --user root, different docker version.. nothing seems to work :(

[christianhuening]

@jgato What's your storage solution? What do you mean by "Rancher instead of Kubernetes"? Are you running Cattle or still Kubernetes but through Rancher?

If storage is NFS: Did you set 'all_squash' for the share?

[jgato]

@christianhuening I mean I am using Rancher with Cattle, and NFS Sever and the Rancher-NFS service.

My configuration for the exports is:

/PATH RANCHER-HOST(rw,fsid=0,all_squash,no_subtree_check,sync)

well, 'all_squash' added after your comment, but still the same error:

Expected state running but got error: Error response from daemon: chown /var/lib/rancher/volumes/rancher-nfs/MONGO-Storage: operation not permitted

[christianhuening]

@jgato From your error message it seems your mongo container still tries to run 'chown' during entrypoint. Try to get rid of that by overriding it. Shouldn't be necessary anymore.

Apart from that: I never used Ranacher-NFS, so there might be subtleties to that I cannot help you with.

[jgato]

ummm extrange, ensuring that I am overrinding the entrypoint:

 {
        "Id": "6d6a16396612e0322c8a96d6ee2a4668cbe8ffdfedc0194056658daf6f62fa42",
        "Created": "2017-07-18T10:11:48.472371956Z",
        "Path": "/.r/r",
        "Args": [
            "mongod",
            "--auth"
        ],
...
....
            "WorkingDir": "",
            "Entrypoint": [
                "/.r/r",
                "mongod",
                "--auth"
            ],

But still the same error :(

[jgato]

@christianhuening I have progressed on that, thanks to this issue.

So cool... now I the rancher-nfs is mounting the volume and there is no error by the side of rancher-nfs. But internally, inside the container I am having chown issue. By the way I am using 3.2.14, so I guess the lines that raise this issue are these:

if [ "$originalArgOne" = 'mongod' ]; then
    chown -R mongodb /data/configdb /data/db
fi

So the docker mongo container says:

chown: changing ownership of '/data/db': Operation not permitted

I am creating my own customized docker image, so it would be easy to eliminate these lines, or maybe in my Dockerifle I could chown root:root /data/db, or other options still not tested. But, what should be the best option? what consequences would have if I put these lines away?

I have tried to understand the code, but I am not sure about the intention of these lines. Somethinb about allowing to execute mongo with other different users, but... not clear how is been done.

[michaelfortunato]

Was able to resolve that by providing
"command":["mongod"]

in my Deployment Kubernetes Manifest. That overrides the entrypoint.sh script and thus does not execute the chown. Whether that's ok or not I cannot tell.

This fixed it for me. I was running the aws eks efs storage class and csi driver with all defaults and was getting this error.
https://docs.aws.amazon.com/eks/latest/userguide/efs-csi.html