Unable to mount NFS export of NFS Ganesha with a Ceph filesystem

[linkvt]

Hi guys,

we set up a CephFileSystem with rook and want to use NFS Ganesha to expose this cluster, but just can't make it work.
We saw the PRs that fixed the docs just a few days ago to allow creating an export and setup everything as described there.

I created an export via the Ceph Dashboard:

[root@rook-ceph-tools-75d599f794-l556d /]# rados -p fs-data0 -N nfs-ns get export-1 -
EXPORT {
    export_id = 1;
    path = "/";
    pseudo = "/ourpath";
    access_type = "RW";
    squash = "no_root_squash";
    protocols = 3, 4;
    transports = "UDP", "TCP";
    FSAL {
        name = "CEPH";
        user_id = "admin";
        filesystem = "ourfs";
        secret_access_key = "AQDMouradminkey8SKA==";
    }

}

Trying to mount this export via NFS v3 is not working and says, nfs v3 is not supported (but should be according to the export above):

[root@rook-ceph-tools-75d599f794-l556d /]# mount -t nfs -o nolock,nfsvers=3 rook-ceph-nfs-fs-a:/ /mnt --verbose
mount.nfs: timeout set for Thu Apr  1 13:05:48 2021
mount.nfs: trying text-based options 'nolock,nfsvers=3,addr=10.233.18.152'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: portmap query retrying: RPC: Program not registered
mount.nfs: prog 100003, trying vers=3, prot=17
mount.nfs: portmap query failed: RPC: Program not registered
mount.nfs: trying text-based options 'nolock,nfsvers=3,addr=10.233.18.152'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: portmap query retrying: RPC: Program not registered
mount.nfs: prog 100003, trying vers=3, prot=17
mount.nfs: portmap query failed: RPC: Program not registered
mount.nfs: trying text-based options 'nolock,nfsvers=3,addr=10.233.18.152'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: portmap query retrying: RPC: Program not registered
mount.nfs: prog 100003, trying vers=3, prot=17
mount.nfs: portmap query failed: RPC: Program not registered
mount.nfs: requested NFS version or transport protocol is not supported

NFS v4 also doesn't work but I just can't find any docs on how I could authenticate myself. Adding a client for our k8s ips (10.233.0.0/16) also didn't help:

[root@rook-ceph-tools-75d599f794-l556d /]# mount -t nfs -o nolock rook-ceph-nfs-fs-a:/ourpath /mnt --verbose
mount.nfs: timeout set for Thu Apr  1 13:06:00 2021
mount.nfs: trying text-based options 'nolock,vers=4.2,addr=10.233.18.152,clientaddr=10.233.66.88'
mount.nfs: mount(2): Permission denied
mount.nfs: trying text-based options 'nolock,vers=4,minorversion=1,addr=10.233.18.152,clientaddr=10.233.66.88'
mount.nfs: mount(2): Permission denied
mount.nfs: trying text-based options 'nolock,vers=4,addr=10.233.18.152,clientaddr=10.233.66.88'
mount.nfs: mount(2): Permission denied
mount.nfs: trying text-based options 'nolock,addr=10.233.18.152'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: portmap query retrying: RPC: Program not registered
mount.nfs: prog 100003, trying vers=3, prot=17
mount.nfs: portmap query failed: RPC: Program not registered
mount.nfs: access denied by server while mounting rook-ceph-nfs-fs-a:/ourpath

Could somebody give us a tip what we are doing wrong, we tried it a few times but no docs help.
We also had a look at the Ceph docs which sugested that we should set a backend via ceph orch set backend rook but as the rook module was disabled in ceph mgr it seems that this also not the correct way (and enabling it also didn't work).

Thanks in advance and happy holidays!

[travisn]

@varshar16 @parth-gr Any ideas on this issue to get nfs working?

[varshar16]

@linkvt share the mgr and nfs logs. You need to expose the nfs service pod's port before mounting. See the note https://docs.ceph.com/en/latest/cephfs/fs-nfs-exports/#show-nfs-ganesha-cluster-information

If you are using pacific version of ceph then try with 'volumes/nfs' interface https://docs.ceph.com/en/latest/cephfs/fs-nfs-exports/#cephfs-exports-over-nfs

[linkvt]

Hi @varshar16 , thanks this indeed worked, but I don't understand why it didn't work from inside a pod, do you have any idea?
Especially the access denied confused me...

I also didn't any docs about using Kerberos with Rook Ceph NFS Ganesha, is this possible in general or do we need to setup our own NFS Ganesha instance to configure it as needed?
Thanks!