NFSv3 mount hangs in 3510.2.1 NFSv4.1 works fine

[kthommandra]

Description

sudo mount -t nfs -overs=3 NFSSERVER:/SHARE /tmp/mountdir

The above command hangs. Upon dumping the /proc/PID/stack I see the following

[<0>] lockd_up+0x1f/0x2b0 [lockd]
[<0>] nlmclnt_init+0x28/0xc0 [lockd]
[<0>] nfs_start_lockd+0xdd/0x120 [nfs]
[<0>] nfs_init_server.isra.0+0x1f3/0x350 [nfs]
[<0>] nfs_create_server+0x7f/0x220 [nfs]
[<0>] nfs3_create_server+0xc/0x60 [nfsv3]
[<0>] nfs_try_get_tree+0x12c/0x210 [nfs]
[<0>] vfs_get_tree+0x22/0xc0
[<0>] path_mount+0x469/0xa40
[<0>] __x64_sys_mount+0x107/0x140
[<0>] do_syscall_64+0x38/0x90
[<0>] entry_SYSCALL_64_after_hwframe+0x61/0xcb

rpc-statd.service is running on the node

If I re-run the above command with -o nolock then the mount succeeds.

If I use NFSv4 (different NFS server and share) then the mount succeeds.

I tested the same NFSv3 mount on one of the older nodes in our cluster 2983.2.1 and v3 mount works fine.

Impact

This bug is preventing mount of NFSv3 shares. For one of our NFS share we have requirement to use V3 only.

Environment and steps to reproduce

1. sudo mkdir /tmp/nfsv3
2. sudo mount -t nfs -overs=3 nfs-server:/nfs-share /tmp/nfsv3

The command stalls and is Ctl-Cable

Expected behavior

mount using NFSv3 should succeed

Additional information

[kthommandra]

Could someone look into this?

[kthommandra]

bump

[tormath1]

@kthommandra hello, I can't reproduce the issue via a Kubernetes deployment. I tried with two different kernel versions:

$ uname -r
5.15.154-flatcar
$ kubectl exec -ti test-pod-1 -- mount | grep nfs
10.109.64.37:/export/pvc-61c8ad64-c700-4bdb-b996-586ef8bf59f0 on /test type nfs (rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.109.64.37,mountvers=3,mountport=20048,mountproto=udp,local_lock=none,addr=10.109.64.37)
$ uname -r
6.6.30-flatcar
$ kubectl exec -ti pods/test-pod-1 -- mount | grep nfs
10.101.210.186:/export/pvc-10854062-5dc8-4dfe-99c3-62dedcf956ee on /test type nfs (rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.101.210.186,mountvers=3,mountport=20048,mountproto=udp,local_lock=none,addr=10.101.210.186)

Did you try with newer Flatcar versions?

[ader1990]

Hello @kthommandra,

In addition to @tormath1, I have also tried to reproduce the behaviour using:

• an ubuntu 22.04 as NFS host, installed NFS as per the steps documented in this nice doc: https://www.digitalocean.com/community/tutorials/how-to-set-up-an-nfs-mount-on-ubuntu-22-04
• a Flatcar box as client

There were no hangs in my testing.

@kthommandra, Can you please elaborate on how did you install the NFS server or what NFS hardware box are you using?
Maybe there are some logs on the NFS box that might help us with the debug?

Thanks.

[kthommandra]

@kthommandra hello, I can't reproduce the issue via a Kubernetes deployment. I tried with two different kernel versions:

$ uname -r
5.15.154-flatcar
$ kubectl exec -ti test-pod-1 -- mount | grep nfs
10.109.64.37:/export/pvc-61c8ad64-c700-4bdb-b996-586ef8bf59f0 on /test type nfs (rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.109.64.37,mountvers=3,mountport=20048,mountproto=udp,local_lock=none,addr=10.109.64.37)
$ uname -r
6.6.30-flatcar
$ kubectl exec -ti pods/test-pod-1 -- mount | grep nfs
10.101.210.186:/export/pvc-10854062-5dc8-4dfe-99c3-62dedcf956ee on /test type nfs (rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.101.210.186,mountvers=3,mountport=20048,mountproto=udp,local_lock=none,addr=10.101.210.186)

Did you try with newer Flatcar versions?

I have not tried newer flatcar versions yet. Will do.
The kernel version in 3510.2.1 is 5.15.106-flatcar. Can you try with this version ?

[kthommandra]

Hello @kthommandra,

In addition to @tormath1, I have also tried to reproduce the behaviour using:

• an ubuntu 22.04 as NFS host, installed NFS as per the steps documented in this nice doc: https://www.digitalocean.com/community/tutorials/how-to-set-up-an-nfs-mount-on-ubuntu-22-04
• a Flatcar box as client

There were no hangs in my testing.

@kthommandra, Can you please elaborate on how did you install the NFS server or what NFS hardware box are you using? Maybe there are some logs on the NFS box that might help us with the debug?

Thanks.

We are using an NFS appliance and not Linux NFS server.
We are working with the vendor to collect logs.