Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Not able to start services with stack scoped volumes in rancher-nfs #7334

Closed
sangeethah opened this issue Jan 6, 2017 · 20 comments

Comments

@sangeethah
Copy link
Member

commented Jan 6, 2017

Rancher Versions: Build from master

Environment - cattle with rancher-nfs - rancher/storage-nfs:v0.6.0
Create a stack using the following yml:

version: '2'
services:
  nginx3:
    image: nginx
    volumes:
    - foo3:/tmp

volumes:
  foo3:
    driver: rancher-nfs
    external: false

Sangeethas-MacBook-Pro:rancher-compose-v0.12.1 sangeethahariharan1$ ./rancher-compose -p new1 -f test1.yml up
INFO[0000] Creating volume template foo3
INFO[0000] [0/1] [nginx3]: Creating
INFO[0000] Creating service nginx3
INFO[0000] [0/1] [nginx3]: Created
INFO[0000] Existing volume template found for foo3
INFO[0000] [0/1] [nginx3]: Starting
INFO[0005] [1/1] [nginx3]: Started
ERRO[0006] Failed to get logs for new1-nginx3-1: Failed to find action: logs

Stack scoped Volume gets created - new1_foo3_82c63 but it in "Dactched" state.
Containers fail to start with the following exception:
Activating (Expected state running but got error: Error response from daemon: chown /var/lib/rancher/volumes/rancher-nfs/new1_foo_3e06f: operation not permitted

@sangeethah sangeethah added the kind/bug label Jan 6, 2017

@sangeethah sangeethah added this to the Release 1.4.0 milestone Jan 6, 2017

@sangeethah

This comment has been minimized.

Copy link
Member Author

commented Jan 6, 2017

This issue is seen with rancher-server version - v1.3.0

@atoy40

This comment has been minimized.

Copy link

commented Jan 15, 2017

I've the exact same issue when creating a service using a "rancher-nfs" volume into a stack directly from the UI. I can see the volume in the storage UI so it has been created (environment scoped) but the service won't start because of the same "chown" issue.

@wilsontayar

This comment has been minimized.

Copy link

commented Jan 20, 2017

I was having the same issue while trying to use a rancher-nfs volume in a elasticsearch container.
The problem was my nfs server permissions, I had to enable "no_root_squash" so the container could access as root. Otherwise it will try to use "nobody,nogroup".

More info here:

no_root_squash: By default, NFS translates requests from a root user remotely into a non-privileged user on the server. This was intended as security feature to prevent a root account on the client from using the file system of the host as root. no_root_squash disables this behavior for certain shares.

So your exports should look like this:
/nfs [ip](rw,sync,no_root_squash,no_subtree_check)

@pdanysz

This comment has been minimized.

Copy link

commented Feb 2, 2017

I have got the same problem with "chown" permission on Rancher v.1.3.3 and rancher-nfs from rancher/storage-nfs:v0.6.0 image.

The setup no_root_squash "solve" the issue for now.

@StrongMonkey

This comment has been minimized.

Copy link
Member

commented Apr 17, 2017

@sangeethah Can you re-test this one with passing the option no_root_squash? I just tried it and can't reproduce it.

@alexleekt

This comment has been minimized.

Copy link

commented Apr 17, 2017

I haven't been able to get around the problem. I'm using FreeNAS which is BSD based and they don't have a no_root_squash. I've tried maproot as root and mapall as root, but both still run into the chown issue.

@einfallstoll

This comment has been minimized.

Copy link

commented Apr 18, 2017

I believe setting up NFSv4 in an existing environment or without endless financial resources is impossible. Most people will end up disabling some core features with no_root_squash, which is, in my eyes, not a good solution. And it's not possible in lots of environments (we're using Solaris ZFS on a huge storage cluster - that thing is stable like a tank but - you guess - no option for no_root_squash).

I can't believe how many hours I wasted to setting up NFSv4 in our company just because of rancher-nfs and long story short: it still isn't working and messing with my permissions. The one and only generic solution to NFSv4 that I read over and over again is switching to NFSv3.

@StrongMonkey

This comment has been minimized.

Copy link
Member

commented Apr 19, 2017

if you can't specify the no_root_squash, don't mount the volume to a existing path inside the container. For all docker named volume, it will copy all the files from container mountpoint to nfs volume, which will raise weird permission error. The solution might be passing nocopy. Right now the workaround is just using a non-existing path inside the container as the mountpoint.

@sangeethah

This comment has been minimized.

Copy link
Member Author

commented Apr 28, 2017

Tested with rancher/storage-nfs:v0.8.0 using rancher-server version - v1.6.0-rc1

Able to create volume and attach them to containers successfully when following setting are done for mount points on the nfs server:

/var/nfs *(rw,sync,no_subtree_check,no_root_squash)

@sangeethah sangeethah closed this Apr 28, 2017

@djskinner

This comment has been minimized.

Copy link

commented Jul 18, 2017

I'm experiencing this issue. Is no_root_squash still the recommended solution?

@djskinner

This comment has been minimized.

Copy link

commented Jul 18, 2017

Related: #9363

@jmaasing

This comment has been minimized.

Copy link

commented Aug 30, 2017

I have the same problem, the server is an NFSv3 NAS and does not have a "no_root_squash" option. Are there any other workarounds to make rancher-nfs function?

@corv89

This comment has been minimized.

Copy link

commented Dec 26, 2017

Also have this problem. "no_root_squash" not available on FreeNAS/FreeBSD.

@outbackdingo

This comment has been minimized.

Copy link

commented Jan 23, 2018

Also have this problem. "no_root_squash" not available on FreeNAS/FreeBSD.

and also me.... FreeBSD NFS box

@joshuata

This comment has been minimized.

Copy link

commented Feb 2, 2018

For those with FreeNAS problems, setting a few config options has it working for me:

  1. In the NFS service settings set Enable NFSv4
  2. Enable NFSv3 ownership model for NFSv4
  3. When mounting a volume, pass in the nocopy option, e.g. volume:/mnt/volume:nocopy

This has allowed me to mount nfs volumes from FreeNAS using rancher-nfs

@corv89

This comment has been minimized.

Copy link

commented Feb 3, 2018

Thanks a lot, I'll try that @joshuata

@joshuata

This comment has been minimized.

Copy link

commented Feb 3, 2018

I'm realizing that nocopy was only useful because there were permissions issues. The reason it temporarily resolved the problem for me was that it was attempting to write to the nfs share which didn't have write access.

@ideal2545

This comment has been minimized.

Copy link

commented Apr 24, 2018

yeah, im having the same issue on freenas also, can't pass in no_root_squash

@2fst4u

This comment has been minimized.

Copy link

commented Apr 16, 2019

@joshuata where exactly do you put these options? In the FreeNAS NFS share or in rancher? I can't find anywhere to put this.

@joshuata

This comment has been minimized.

Copy link

commented Apr 18, 2019

In rancher. There is an option when setting up an nfs volume to set flags.

Note that this was a year ago with rancher v1, so things may have changed since then. I can't make any promises about more recent versions since I now just use docker-compose because its much lighter weight

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.