use delegated zfs permissions without sudo access #151
Comments
|
Can you send over the full values files (cleansed of secrets of course)? Generally you can run as non-root for ssh connection/operations but the api is root only. The apiKey setup you’re currently using is ideal for the api atm. The values file does turn the config data into a k8s secret so it is stored as a secret in k8s. If that’s troublesome you can either create the secret manually without using the chart or you can use a project like |
|
Also what version of TrueNAS are you currently running? |
|
Thanks - I am using version: And if I ssh to the TrueNas server - try to create a dataset, i.e. zfs create fast/k8s/nfs/vols/testtruenas ask me to authenticate via its "sudo" functionality and when I type in the password - the dataset is created. In regards to storing the values file inside k8s - I did not realise it was stored as a secret - I just saw the values stored and it looked like it was stored in plain text to me - but that might just have been my "ui" that decrypted it. csiDriver:
# should be globally unique for a given cluster
name: "org.democratic-csi.freenas-nfs"
storageClasses:
- name: freenas-nfs-csi
defaultClass: true
reclaimPolicy: Retain
volumeBindingMode: Immediate
allowVolumeExpansion: true
parameters:
fsType: nfs
mountOptions:
- noatime
secrets:
provisioner-secret:
controller-publish-secret:
node-stage-secret:
node-publish-secret:
controller-expand-secret:
driver:
config:
driver: freenas-nfs
instance_id:
httpConnection:
protocol: http
host: 192.168.0.201
port: 80
allowInsecure: true
apiKey: "2-qEjchKvqI7QimbvONfM4vbNopomKuHpNC0ffnNY8QOPdXlyr0gxPDhSv1BgzXC05"
apiVersion: 2
sshConnection:
host: 192.168.0.201
port: 22
username: k8s
password: "<masked>"
zfs:
cli:
sudoEnabled: true
datasetParentName: fast/k8s/nfs/vols
detachedSnapshotsDatasetParentName: fast/k8s/nfs/snaps
datasetEnableQuotas: true
datasetEnableReservation: false
datasetPermissionsMode: "0777"
datasetPermissionsUser: k8s
datasetPermissionsGroup: wheel
nfs:
shareHost: 192.168.0.201
shareAlldirs: false
shareAllowedHosts: []
shareAllowedNetworks: []
shareMaprootUser: ""
shareMaprootGroup: ""
shareMapallUser: k8s
shareMapallGroup: wheel |
|
I guess helm itself stores the full values in the cluster as secrets as well yes (unless you've configured helm to use configmaps which I wouldn't recommend). My guess is you haven't enabled passwordless sudo for the |
True - I did not notice that - it would be nice to have as a comment in the "values.yaml" example - so people know its a hard requirement that passwordless sudo is set up. But it seems like it does not change a thing - at least not for me: From my "cli" From my sudoers But still my k8s cluster logs:
But to make it fair - even though it seems like its set up correctly on the truenas server - I cannot even do it passwordless when ssh'ing to the server manually and logging on. So it seems like the passwordless sudo on truenas is not working as expected. I will dig into this further - but at least I got a bit further. Thanks |
|
The issue is the group...sudo has strange behavior in this regard. If you're going to enable sudo on the group as well do the following:
It should then work as desired. EDIT: alternatively, just remove sudo access from the group and you should also get the behavior you are after |
|
Its very strange - now I can ssh in manually and do: sudo zfs create fast/k8s/nfs/vols/testWithout getting asked for password - but still I get the same error from k8s. I will try your suggestion to only give it on the group or client. |
|
That is strange yeah :( let me know how it goes. |
It works now - when I removed the sudo from the group. Now I just need to figure out if I can limit my k8s user from destroying my entire pool - just in case there is a bug in the csi framework. But that is more a question for the truenas/zfs forum. Thanks a lot for your support. |
|
That's a very fair concern. The short answer is you can't really (not with how the driver works currently). When using sudo the commands are invoked with root privs anyway so limiting the At some point I may look into leveraging the zfs delegation but nothing is in place yet :( |
|
I have asked the question on the TrueNAS forum - whether or not its possible to limit access to datasets - so a configured user gets full access to a certain dataset and nothing else. And looking at that link you sent it seems like ZFS at least supports it - so perhaps its possible if I do not use sudo, and simply use the access control things in TrueNAS so its only on the explicit dataset that it can get access. Fingers crossed - and if it turns out its possible already - if I stop using sudo, then I will report back here - and it should probably be used as a "best practice" - just like not using the root user :-) |
|
Yeah. There are a couple scenarios where sudo is still required but please do test and see how far you get. Without sudo enabled it will fail to change the ownership and permissions on the share dir (for nfs) and will fail some volume expansion operations (for iscsi). For testing nfs you’ll want to disable the datasetPermission config options to avoid those failures. It’s mostly due to these reasons that I haven’t fully explored the delegated deployment yet…it’s certainly feasible to not use sudo for zfs but use it for the other scenarios but it simply hasn’t been developed yet. |
|
I just tried with delegated permissions - and now I can fully control what the k8s user can do - but it does not have permissions to mount the dataset, which is required I assume to expose it via nfs. So I am kind of back to square one. Perhaps one solution - although it requires changes in the codebase is to "document" that any dataset that is set up for this should use delegated permissions (with documentation of which csi require) - and then only do the ZFS commands via ssh, and the remaining via the api, i.e. If csi needs to create a dataset - it happens via ssh, but the mounting, sharing via nfs etc happens via the API - then it might work with a regular user without sudo access and with delegated permissions to its own dataset? Does it make sense what I propose? Also if bugs sneak in the API integration code, then it would only affect the more "innocent" things, like sharing, mounting etc. Of course this "split" responsibility will only work for truenas installations - but if you can make one implementation more "secure" from potential bugs, I think it could be worth it - if its possible.. Alternatively - on the TrueNAS forums, they also suggested to use a jail for the dataset - I am not sure if that would make it easier on permissions, or if its even possible to interact with the jail from a ssh session in any decent way - but perhaps a jail, with a dataset inside that - it could be managed with the "generic " zfs-generic-nfs implementation, since everything could be done with ssh towards the jail, with full "su" access, and even if bugs come up, it will only affect the jail and the dataset that got mounted to that jail. So basically run a NFS server in a jail, with the dataset mounted to that jail, with full delegation - then whatever runs in the jail can do whatever to the dataset, can mount datasets, share them via ssh - but obviously it would all happen via ssh, since there are probably no api access to jails in truenas. |
|
The mount should happen automatically inheriting the mount point logic from the parent no? I’m with you on making it secure as possible. It just hasn’t been done and requires some thought to make it as robust as possible. |
Unfortunately it does not happen - at least not on truenas. The unpriveleged user is allowed to create a sub dataset in the delegated dataset, but the mount does not happen as far as I see - it even emits a message about this. |
|
Can you get all the properties on the dataset and send it here? |
|
I have solved it: zfs allow -u k8s clone,create,destroy,mount,promote,receive,rename,rollback,send,share,snapshot,sharenfs,mountpoint fast/k8sAdd sysctl setting: So now when I ssh in with my k8s user - I can do whatever i want with the dataset fast/k8s and below - including mounting it - which happens automatically when I do: zfs create fast/k8s/testThen it gets mounted to /mnt/fast/k8s/test So I think it might be possible to do with just using "generic zfs csi" - and by using this way. I can probably remove many of the delegations - I just delegated all of them that I thought the csi might use. So I definately think this might be the way forward - so a pure SSH solution is possible and just by using delegation and allowing non-root mounts. I will definately try it out :-) |
|
Pure ssh is not possible as http is required to manage the shares (ie: insert stuff into the TrueNAS db, etc). Also note that sudo will still be required for some non-zfs operations (such as reloading some services, etc). However all zfs operations could be done over ssh in an underprivileged (non-sudo) fashion however so that is great news! Regarding the specific permissions there are actually quite a few required to cover the full breadth of what the driver does so I think that above list is probably a great start. If anything fails we can add from there. |
|
I see - but I think that is a decent compromise - since then the driver only needs root access to the API - and not directly to the Zfs commands. Of course that still allow a bug in the api code to somehow destroy something, but at least it will not impact the zfs pools - which should be the most precious part of a truenas installation, since thats where the data is stored. Everything else can be recreated - data is harder to recreate - unless you have a very good backup strategy and take snapshots very often. P.S. I just tested - with sudo=false in my values.yaml - and it does not work: Which is strange, since I can just ssh as the k8s user and do: zfs create fast/k8s/vols/pvc-e3061070-b0b1-4af2-9bd3-f85e8f813495Is that because the driver assumes something, like using sudo is required? Even though my values contains: zfs:
cli:
sudoEnabled: falseWhat is even more strange is that it did manage to create the dataset: Perhaps its quotas or something else it fails on - let me try to delegate more :-) |
|
I agree 100% it's a decent compromise. While not ideal to still require sudo for some of the other purposes there's not really any ways around that, and having protection/insurance at the zfs level is indeed highly desirable. |
|
Any where I can see what zfs properties that gets set when the dataset gets created? I have tried with: zfs allow -u k8s clone,create,destroy,mount,promote,receive,rename,rollback,send,share,snapshot,sharenfs,mountpoint,quota,volsize,snapdir,reservation,readonly,exec,copies,compression fast/k8sAnd still it fails. I even removed the "vols" dataset beneath fast/k8s - and that got created again - so I am thinking its some property or setting the drives tries to set, that it gets the permission denied from. |
|
Progress - I added more delegations and now the pvc dataset gets created, but then it fails when it tries to set some properties: And
|
|
output from So properties are getting set by the driver - I wonder where it fails to do something. I have looked into the code - but javascript is not my strong side - I am more of a c++/c/c# kind of guy. |
|
These are my current delegations: zfs allow -u k8s clone,create,destroy,mount,promote,receive,rename,rollback,send,share,snapshot,sharenfs,mountpoint,quota,volsize,snapdir,reservation,readonly,exec,copies,compression,userquota,aclmode,exec,readonly,groupquota,groupused,userprop,userquota,userused,atime,canmount,checksum,compression,devices,nbmand,normalization,readonly,recordsize,refreservation,reservation,setuid,utf8only,version,volsize,volblocksize,vscan,xattr fast/k8s |
|
Any way I can turn on debug logging of some kind - so it logs more what commands it tries to run? |
Regarding the error perhaps what happened is the driver tried to set the |
Can I do this in my values.yaml that I give to helm? I have tried editing via kubectl - but it does not seem to give more logging output.
I already did that - I have removed the dataset, reapplied delegations, uninstalled the helm chart - reinstalled it - but still I get this permission error after it creates the dataset. Edit: So my full delegation command is: zfs allow -u k8s clone,create,destroy,mount,promote,receive,rename,rollback,send,share,snapshot,sharenfs,mountpoint,quota,volsize,snapdir,reservation,readonly,exec,copies,compression,userquota,aclmode,exec,readonly,groupquota,groupused,userprop,userquota,userused,atime,canmount,checksum,compression,devices,nbmand,normalization,readonly,recordsize,refreservation,reservation,setuid,utf8only,version,volsize,volblocksize,vscan,xattr,refquota fast/k8sAnd the most important bit I think that it requires the sysctl setting: Which can be added via the TrueNAS gui - this allows non-root users to mount filesystems. So success - sudo is no longer needed for the ssh connction :-) Thank you for the help. For future reference it would be awesome if it could be documented, what delegations is required precisely, so users only need to delegate the required bits and not do trial and error like I have done. |
|
The debugging must be added to your helm values yes. I would appreciate any contributions to the documentation to get the delegated setup properly documented and explained for easy use. I think in the meantime, I'll go rework some bits of the code to unconditionally use |
Sure - I don't mind helping with the documentation part - if you can just tell me what delegations I should include :-)
I am not sure what you mean by this - will this not be counter-intuitive to what I have been trying to achieve? With my recipy the SSH connection does not require sudo - if you have set up proper delegation and the correct sysctl setting. So perhaps add a setting instead of sudo? i.e. driver:
config:
driver: freenas-nfs
sshConnection:
host: 192.168.0.201
port: 22
delegation: true
sudo: falseAnd then in the driver you check whether or not delegation or sudo is in use. If you have not set up correct delegation, then obviously it requires passwordless sudo to be set up correctly - unless you run as root. But even running as root, in theory it does not guarantee that root has the required permissions - root is just a username - although root usually is the "super user" with all access. Unless I misunderstand you? Right now, my user is not allowed to do any sudo, so if you change the code to always require sudo if the username is not root, then my work would be in vain - and I am back to being a possible "victim" of bugs that can destroy my pool. |
|
I honestly don't have a comprehensive list of which delegations are required to cover the feature set. I suspect you've probably given enough at this point. The project has the potential to change as the driver evolves over time as well. I think we'll just need to run it for a bit and see if we run into any situations where something fails due to lack of permissions. Regarding
I'm updating the code to decouple the scenarios in point 1 from point 2. The situations in point 1 require the use of |
|
I reviewed the I completely removed the Test claim went fine: Tried to edit the claim: I noticed the condition to restart the node, so I tried this: But the condition remains... I don't see any obvious errors.... suggestions? |
|
You can't remove the Regarding the user/group those settings are only applicable to nfs so not an issue with iscsi. I'm referring to these config options: |
|
Went back over my notes, while I removed the SSH section from the yaml, I did not reinstall the helm chart.. so that edit didn't do anything. After some more testing, created a new claim, created a nginx pod to use the claim, and then resized the claim with the nginix pod running and just waited a few seconds and I think everything worked as expected. The 2nd claim bumped to 2Gi Inside the pod: Anything else to check? |
|
Great! Not really other than to make sure the api call was used to reload ctld vs a command over ssh. If the user does not have If you wish to confirm with certainty you would need to review the logs of the controller pod ( |
|
I see this on the TrueNAS side, I assume that isn't important? Found in the Details: |
|
I don't know why you would see that warning...I'm assuming there is indeed a lun defined for the target otherwise you wouldn't be able to attach to it and use the volume. In any case, the logs look exactly like what we're after so that part is good. |
|
It shows lun I just did: TrueNAS Console then shows: Is there an annotation (or something else) which can be used in the claim to populate the extent's "Description" field? |
|
Well, was that error from when the volume was provisioned? You should only see that during initial creation of the volume as the target gets created and then later the lun assigned (never to be removed unless/until the volume is deleted). I guess I never allowed for setting the comment on the extent field.. |
Should I open an issue to request this? |
|
Sure. |
|
Has someone been able to get rootless NFS shares working on Linux? |
|
Can you elaborate more on what you want to achieve? |
|
I want to avoid having to give root access to the CSI driver on the storage box. The problems seems to be that new NFS exports cannot be created by non-root users on Linux. |
|
Which driver are you using? Can you send over the logs? |
|
This is before even using the driver, just manually settings the sharenfs property as a non-root user. I can reproduce this on different hosts. I just wanted to confirm that this is expected and that no workaround exists. |
|
I see so you're using delegated permissions and trying to set the |
|
Correct. Environment is Ubuntu 22.04. |
|
Do you have all the deps installed? openzfs/zfs#4534 |
|
Yes of course. I'm perfectly able to create the export as root. Are you able to create it as non-root on Linux? |
|
I've never actually tried to run a fully delegated setup no. Just wanted to make sure it was specific to delegation and not a general issue. Honestly that behavior seems odd to me but I don't exactly how the internals of it are handled by zfsd or whatever it is that handles those. Do the exports generally show up in |
|
Zfs doesn't use |
|
Well you certainly know more than me about the matter! I've just asked in the irc channel to see if anyone has some input they can provide. |
|
Can you possibly run the zfs command with strace to figure out what folder etc it's trying to create etc? |
|
Thanks for the hint. With strace I found that it was trying to write to /etc/exports.d/zfs.exports. Giving permissions on this file (and directory) setting the sharenfs property did not err anymore, however the share was still not created. I additionally had to give permissions to /var/lib/nfs/etab (and containing directory), then it works.! :) |
|
We definitely need to get that documented! |
|
It would be great to test the same for sharesmb as well and document what is needed there. The upcoming release will debut support for windows nodes/clusters and so smb has far greater test coverage etc than previously. |
|
I managed to get non-root sharesmb going, too: |
Hi,
I really like this project - but would it be possible that the driver uses its required credentials from a secret that is stored inside the kubernetes cluster?
Right now credentials are stored in plain text inside the values, and that seems like a really bad idea.
Edit: Also I seem to not be able to get a non-root user to work at all.
System info:
If I use username/password for httpConnection like this:
Then I get a 401 from the API - my guess is that non-root users are not allowed to use the API.
With the apiKey present instead
I get errors from the ssh connection
if I add:
The error becomes different:
Thanks for any hints - or even a message stating that its not supported properly without root.
The text was updated successfully, but these errors were encountered: