chown: changing ownership of ‘/var/lib/postgresql/data’: Operation not permitted, when running in kubernetes with mounted "/var/lib/postgres/data" volume

[arnold0617]

I am running docker.io/postgres:9.6.5 in kubernetes, pod crash loop back off all the time.

I found this message in log, "chown: changing ownership of ‘/var/lib/postgresql/data’: Operation not permitted"

[yosifkit]

If the error is coming from the following line, it is just a red herring and thus not really the problem. (I think it was added for when the directory was group writable to fix the ownership since postgres can be particular about that).

postgres/docker-entrypoint.sh

Line 52 in 57213ab

chown -R "$(id -u)" "$PGDATA" 2>/dev/null || :

Since this seems like a general debugging question and not really a problem with the image, it'd be better to post questions like this in the Docker Community Forums, the Docker Community Slack, or Stack Overflow.

[mikygit]

Same problem here. It happens on a docker mounted dir. I suspect a file system weirdy. Still investigating.
If anybody can help ...

[RomanGalochkin]

The same situation. What the problem? Maybe NFS?
I have tried everything.

[mikygit]

Same in a sense that I do have this "chown: changing ownership of ‘/var/lib/postgresql/data’: Operation not permitted" error.
It's probably NFS (the same docker image was perfectly running on another host) but I'm not an expert in that field and I'm struggling to explain our IT team what to do and why ...

Any ideas?

[RomanGalochkin]

To solve this problem do it:

1. Set NFS export dir:
   "nfs/main *(rw,sync,no_subtree_check,no_root_squash)"
   And then restart NFS server:
   sudo /etc/init.d/nfs-kernel-server restart
   or
   sudo exportfs -arv
2. Set NFS volume like this:

  nfs:
    server: ***
    path: "nfs/main/data"

3. Update your entity configuration:

         volumeMounts:
            - name: postgres-storage
              mountPath: /var/lib/postgresql/data
              readOnly: false

Very important to export nfs/main and create volume inside of the folder nfs/main/data.

[mikygit]

Thx!
This looks like a special case for postgre. Would I have to do the same for every mounted volumes?

[RomanGalochkin]

No, you don't need it. Cuz not every container tries to change owner and permissions of files.

[mikygit]

You're right.
Is your third point still a 'NFS" configuration? Because in that case, shouldn't the mountPath point to the host folder that I want to share with the docker container (which in my case is different)?

[RomanGalochkin]

Yes, it is. NFS exported path is a just entry point. And nobody cannot change owner of the path. But, inside of the folder you can do everything you want. It is the main idea of the solution.

[mikygit]

... not sure in the end if you are saying that mountPath should be /xxx/xxx/xxx/xxx if I plan to mount that specific folder ;-) Can you clarify?
nb: could stick to /var/lib/postgresql/data sure, but just to understand properly

[RomanGalochkin]

1. You create sharing folder: "/nfs/shares".
2. You point volume to "/nfs/shares/data".
3. You set volume of a container "/var/lib/postgresql/data".
4. You give to clients users of NFS server rights "no_root_squash" on the shared folder "/nfs/shares".

We have created volume inside of shared folder.

[mikygit]

I deserved that one. My question did sound stupid ;-)
Thanx a lot for your help!

[mikygit]

Oops, another question sorry:
What about 'nfs/main/data' you specified in 2)? Should it be related to the shared folder "/nfs/shares"?

[RomanGalochkin]

Yes, should. I showed another steps with different names to understand better.

[mikygit]

cheers!