restarting NFSRODS without re-mounting at the client produces incorrect behavior

[anderbubble]

If I start NFSRODS and mount, I can see the VFS correctly with ls.

[root@irods1 ~]# docker start nfsrods
nfsrods
[root@irods1 ~]# docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS                    NAMES
879249206d41        nfsrods             "./start.sh"        3 days ago          Up 20 seconds       0.0.0.0:2049->2049/tcp   nfsrods
[root@irods1 ~]# mount -o sec=sys,port=2049 localhost:/ /mnt/nfsrods
[root@irods1 ~]# ls /mnt/nfsrods/home/joan5896
check_x509.py  gpfs-expels.csv  quota-test  x-get-selection-owner.c  zeroes

But if I restart NFSRODS without re-mounting, I often get incorrect and inconsistent behavior.

[root@irods1 ~]# docker restart nfsrods
nfsrods
[root@irods1 ~]# ls /mnt/nfsrods/home/joan5896/
ls: cannot access /mnt/nfsrods/home/joan5896/: No such file or directory
[root@irods1 ~]# ls /mnt/nfsrods/
ls: cannot access /mnt/nfsrods/home: No such file or directory
ls: cannot access /mnt/nfsrods/pl: No such file or directory
ls: cannot access /mnt/nfsrods/trash: No such file or directory
home  pl  trash

Note in the last example that the directories were returned and reported "No such file or directory."

This behavior seems to resolve itself after a short time.

[trel]

We've seen similar behavior, but have not been able to pin it down. We're currently thinking it might be on the mount (with caching?) side of things. Any insight is welcome.

[trel]

need to retest - this could have been fixed by f410957?

[korydraughn]

I believe using the DATA_ID as the file handle will resolve this issue.

Will investigate.

[davebiffuk]

I see similar symptoms occasionally here:

dh3@farm5-humgen-nfsrods:~$ grep humgen /proc/mounts 
irods-hum-nfsrods01.internal.sanger.ac.uk:/ /mnt/humgen nfs4 ro,noatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=172.27.71.77,local_lock=none,addr=172.27.71.191 0 0
dh3@farm5-humgen-nfsrods:~$ cd /mnt/humgen/
dh3@farm5-humgen-nfsrods:/mnt/humgen$ ls
ls: cannot access 'archive': No such file or directory
ls: cannot access 'asclepius_testing': No such file or directory
ls: cannot access 'home': No such file or directory
ls: cannot access 'projects': No such file or directory
ls: cannot access 'scratch115': No such file or directory
ls: cannot access 'scratch119-backup': No such file or directory
ls: cannot access 'shepherd_testing': No such file or directory
ls: cannot access 'teams': No such file or directory
ls: cannot access 'test': No such file or directory
ls: cannot access 'trash': No such file or directory
ls: cannot access 'can-you-read-me.txt': Permission denied
archive              home        scratch119-backup  test
asclepius_testing    projects    shepherd_testing   trash
can-you-read-me.txt  scratch115  teams
dh3@farm5-humgen-nfsrods:/mnt/humgen$ ls
ls: cannot access 'can-you-read-me.txt': Permission denied
archive              home        scratch119-backup  test
asclepius_testing    projects    shepherd_testing   trash
can-you-read-me.txt  scratch115  teams

This is nfsrods 2.1.0 on Ubuntu 18.04 for both the NFS client and nfsrods server. I have nfsrods debug logs for this if that's useful.

[korydraughn]

We've been focused on getting 4.2.12 released, so this issue has taken a backseat to that.

We will take care of this as soon as we can.

[kript]

FYI @davebiffuk is one of my fine Sanger colleagues.

[trel]

Excellent work @ganning127! Closing!