Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Not able to start services with stack scoped volumes in rancher-nfs #7334

Closed
sangeethah opened this issue Jan 6, 2017 · 20 comments
Closed

Not able to start services with stack scoped volumes in rancher-nfs #7334

sangeethah opened this issue Jan 6, 2017 · 20 comments
Assignees
Labels
Milestone

Comments

@sangeethah
Copy link
Contributor

@sangeethah sangeethah commented Jan 6, 2017

Rancher Versions: Build from master

Environment - cattle with rancher-nfs - rancher/storage-nfs:v0.6.0
Create a stack using the following yml:

version: '2'
services:
  nginx3:
    image: nginx
    volumes:
    - foo3:/tmp

volumes:
  foo3:
    driver: rancher-nfs
    external: false

Sangeethas-MacBook-Pro:rancher-compose-v0.12.1 sangeethahariharan1$ ./rancher-compose -p new1 -f test1.yml up
INFO[0000] Creating volume template foo3
INFO[0000] [0/1] [nginx3]: Creating
INFO[0000] Creating service nginx3
INFO[0000] [0/1] [nginx3]: Created
INFO[0000] Existing volume template found for foo3
INFO[0000] [0/1] [nginx3]: Starting
INFO[0005] [1/1] [nginx3]: Started
ERRO[0006] Failed to get logs for new1-nginx3-1: Failed to find action: logs

Stack scoped Volume gets created - new1_foo3_82c63 but it in "Dactched" state.
Containers fail to start with the following exception:
Activating (Expected state running but got error: Error response from daemon: chown /var/lib/rancher/volumes/rancher-nfs/new1_foo_3e06f: operation not permitted

@sangeethah
Copy link
Contributor Author

@sangeethah sangeethah commented Jan 6, 2017

This issue is seen with rancher-server version - v1.3.0

@atoy40
Copy link

@atoy40 atoy40 commented Jan 15, 2017

I've the exact same issue when creating a service using a "rancher-nfs" volume into a stack directly from the UI. I can see the volume in the storage UI so it has been created (environment scoped) but the service won't start because of the same "chown" issue.

@wilsontayar
Copy link

@wilsontayar wilsontayar commented Jan 20, 2017

I was having the same issue while trying to use a rancher-nfs volume in a elasticsearch container.
The problem was my nfs server permissions, I had to enable "no_root_squash" so the container could access as root. Otherwise it will try to use "nobody,nogroup".

More info here:

no_root_squash: By default, NFS translates requests from a root user remotely into a non-privileged user on the server. This was intended as security feature to prevent a root account on the client from using the file system of the host as root. no_root_squash disables this behavior for certain shares.

So your exports should look like this:
/nfs [ip](rw,sync,no_root_squash,no_subtree_check)

@deniseschannon deniseschannon added this to the Release 1.5.0 milestone Jan 26, 2017
@deniseschannon deniseschannon removed this from the Release 1.4.0 milestone Jan 26, 2017
@pdanysz
Copy link

@pdanysz pdanysz commented Feb 2, 2017

I have got the same problem with "chown" permission on Rancher v.1.3.3 and rancher-nfs from rancher/storage-nfs:v0.6.0 image.

The setup no_root_squash "solve" the issue for now.

@will-chan will-chan added this to the Release 1.6.0 milestone Feb 22, 2017
@will-chan will-chan removed this from the Release 1.5.0 milestone Feb 22, 2017
@StrongMonkey
Copy link
Contributor

@StrongMonkey StrongMonkey commented Apr 17, 2017

@sangeethah Can you re-test this one with passing the option no_root_squash? I just tried it and can't reproduce it.

@alexleekt
Copy link

@alexleekt alexleekt commented Apr 17, 2017

I haven't been able to get around the problem. I'm using FreeNAS which is BSD based and they don't have a no_root_squash. I've tried maproot as root and mapall as root, but both still run into the chown issue.

@einfallstoll
Copy link

@einfallstoll einfallstoll commented Apr 18, 2017

I believe setting up NFSv4 in an existing environment or without endless financial resources is impossible. Most people will end up disabling some core features with no_root_squash, which is, in my eyes, not a good solution. And it's not possible in lots of environments (we're using Solaris ZFS on a huge storage cluster - that thing is stable like a tank but - you guess - no option for no_root_squash).

I can't believe how many hours I wasted to setting up NFSv4 in our company just because of rancher-nfs and long story short: it still isn't working and messing with my permissions. The one and only generic solution to NFSv4 that I read over and over again is switching to NFSv3.

@StrongMonkey
Copy link
Contributor

@StrongMonkey StrongMonkey commented Apr 19, 2017

if you can't specify the no_root_squash, don't mount the volume to a existing path inside the container. For all docker named volume, it will copy all the files from container mountpoint to nfs volume, which will raise weird permission error. The solution might be passing nocopy. Right now the workaround is just using a non-existing path inside the container as the mountpoint.

@sangeethah
Copy link
Contributor Author

@sangeethah sangeethah commented Apr 28, 2017

Tested with rancher/storage-nfs:v0.8.0 using rancher-server version - v1.6.0-rc1

Able to create volume and attach them to containers successfully when following setting are done for mount points on the nfs server:

/var/nfs *(rw,sync,no_subtree_check,no_root_squash)

@sangeethah sangeethah closed this Apr 28, 2017
@djskinner
Copy link

@djskinner djskinner commented Jul 18, 2017

I'm experiencing this issue. Is no_root_squash still the recommended solution?

@djskinner
Copy link

@djskinner djskinner commented Jul 18, 2017

Related: #9363

@jmaasing
Copy link

@jmaasing jmaasing commented Aug 30, 2017

I have the same problem, the server is an NFSv3 NAS and does not have a "no_root_squash" option. Are there any other workarounds to make rancher-nfs function?

@corv89
Copy link

@corv89 corv89 commented Dec 26, 2017

Also have this problem. "no_root_squash" not available on FreeNAS/FreeBSD.

@ghost
Copy link

@ghost ghost commented Jan 23, 2018

Also have this problem. "no_root_squash" not available on FreeNAS/FreeBSD.

and also me.... FreeBSD NFS box

@joshuata
Copy link

@joshuata joshuata commented Feb 2, 2018

For those with FreeNAS problems, setting a few config options has it working for me:

  1. In the NFS service settings set Enable NFSv4
  2. Enable NFSv3 ownership model for NFSv4
  3. When mounting a volume, pass in the nocopy option, e.g. volume:/mnt/volume:nocopy

This has allowed me to mount nfs volumes from FreeNAS using rancher-nfs

@corv89
Copy link

@corv89 corv89 commented Feb 3, 2018

Thanks a lot, I'll try that @joshuata

@joshuata
Copy link

@joshuata joshuata commented Feb 3, 2018

I'm realizing that nocopy was only useful because there were permissions issues. The reason it temporarily resolved the problem for me was that it was attempting to write to the nfs share which didn't have write access.

@ideal2545
Copy link

@ideal2545 ideal2545 commented Apr 24, 2018

yeah, im having the same issue on freenas also, can't pass in no_root_squash

@2fst4u
Copy link

@2fst4u 2fst4u commented Apr 16, 2019

@joshuata where exactly do you put these options? In the FreeNAS NFS share or in rancher? I can't find anywhere to put this.

@joshuata
Copy link

@joshuata joshuata commented Apr 18, 2019

In rancher. There is an option when setting up an nfs volume to set flags.

Note that this was a year ago with rancher v1, so things may have changed since then. I can't make any promises about more recent versions since I now just use docker-compose because its much lighter weight

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet