Not able to start services with stack scoped volumes in rancher-nfs

[sangeethah]

Rancher Versions: Build from master

Environment - cattle with rancher-nfs - rancher/storage-nfs:v0.6.0
Create a stack using the following yml:

version: '2'
services:
  nginx3:
    image: nginx
    volumes:
    - foo3:/tmp

volumes:
  foo3:
    driver: rancher-nfs
    external: false

Sangeethas-MacBook-Pro:rancher-compose-v0.12.1 sangeethahariharan1$ ./rancher-compose -p new1 -f test1.yml up
INFO[0000] Creating volume template foo3
INFO[0000] [0/1] [nginx3]: Creating
INFO[0000] Creating service nginx3
INFO[0000] [0/1] [nginx3]: Created
INFO[0000] Existing volume template found for foo3
INFO[0000] [0/1] [nginx3]: Starting
INFO[0005] [1/1] [nginx3]: Started
ERRO[0006] Failed to get logs for new1-nginx3-1: Failed to find action: logs

Stack scoped Volume gets created - new1_foo3_82c63 but it in "Dactched" state.
Containers fail to start with the following exception:
Activating (Expected state running but got error: Error response from daemon: chown /var/lib/rancher/volumes/rancher-nfs/new1_foo_3e06f: operation not permitted

[sangeethah]

This issue is seen with rancher-server version - v1.3.0

[atoy40]

I've the exact same issue when creating a service using a "rancher-nfs" volume into a stack directly from the UI. I can see the volume in the storage UI so it has been created (environment scoped) but the service won't start because of the same "chown" issue.

[wilsontayar]

I was having the same issue while trying to use a rancher-nfs volume in a elasticsearch container.
The problem was my nfs server permissions, I had to enable "no_root_squash" so the container could access as root. Otherwise it will try to use "nobody,nogroup".

More info here:

no_root_squash: By default, NFS translates requests from a root user remotely into a non-privileged user on the server. This was intended as security feature to prevent a root account on the client from using the file system of the host as root. no_root_squash disables this behavior for certain shares.

So your exports should look like this:
/nfs [ip](rw,sync,no_root_squash,no_subtree_check)

[pdanysz]

I have got the same problem with "chown" permission on Rancher v.1.3.3 and rancher-nfs from rancher/storage-nfs:v0.6.0 image.

The setup no_root_squash "solve" the issue for now.

[StrongMonkey]

@sangeethah Can you re-test this one with passing the option no_root_squash? I just tried it and can't reproduce it.

[alexleekt]

I haven't been able to get around the problem. I'm using FreeNAS which is BSD based and they don't have a no_root_squash. I've tried maproot as root and mapall as root, but both still run into the chown issue.

[einfallstoll]

I believe setting up NFSv4 in an existing environment or without endless financial resources is impossible. Most people will end up disabling some core features with no_root_squash, which is, in my eyes, not a good solution. And it's not possible in lots of environments (we're using Solaris ZFS on a huge storage cluster - that thing is stable like a tank but - you guess - no option for no_root_squash).

I can't believe how many hours I wasted to setting up NFSv4 in our company just because of rancher-nfs and long story short: it still isn't working and messing with my permissions. The one and only generic solution to NFSv4 that I read over and over again is switching to NFSv3.

[StrongMonkey]

if you can't specify the no_root_squash, don't mount the volume to a existing path inside the container. For all docker named volume, it will copy all the files from container mountpoint to nfs volume, which will raise weird permission error. The solution might be passing nocopy. Right now the workaround is just using a non-existing path inside the container as the mountpoint.

[sangeethah]

Tested with rancher/storage-nfs:v0.8.0 using rancher-server version - v1.6.0-rc1

Able to create volume and attach them to containers successfully when following setting are done for mount points on the nfs server:

/var/nfs *(rw,sync,no_subtree_check,no_root_squash)

[djskinner]

I'm experiencing this issue. Is no_root_squash still the recommended solution?

[djskinner]

Related: #9363

[jmaasing]

I have the same problem, the server is an NFSv3 NAS and does not have a "no_root_squash" option. Are there any other workarounds to make rancher-nfs function?

[corv89]

Also have this problem. "no_root_squash" not available on FreeNAS/FreeBSD.

[ghost]

Also have this problem. "no_root_squash" not available on FreeNAS/FreeBSD.

and also me.... FreeBSD NFS box

[joshuata]

For those with FreeNAS problems, setting a few config options has it working for me:

1. In the NFS service settings set Enable NFSv4
2. Enable NFSv3 ownership model for NFSv4
3. When mounting a volume, pass in the nocopy option, e.g. volume:/mnt/volume:nocopy

This has allowed me to mount nfs volumes from FreeNAS using rancher-nfs

[corv89]

Thanks a lot, I'll try that @joshuata

[joshuata]

I'm realizing that nocopy was only useful because there were permissions issues. The reason it temporarily resolved the problem for me was that it was attempting to write to the nfs share which didn't have write access.

[ideal2545]

yeah, im having the same issue on freenas also, can't pass in no_root_squash

[2fst4u]

@joshuata where exactly do you put these options? In the FreeNAS NFS share or in rancher? I can't find anywhere to put this.

[joshuata]

In rancher. There is an option when setting up an nfs volume to set flags.

Note that this was a year ago with rancher v1, so things may have changed since then. I can't make any promises about more recent versions since I now just use docker-compose because its much lighter weight