nfs: fix race condition of LAYOUTRETURN and LAYOUTGET

Motivation:

in a situation, when two processes on the same client do:

time:------------------------------------------>
P1: OPEN+LAYOUTGET, READ, LAYOUTETURN+CLOSE
P2:                   OPEN,      LAYOUTGET, READ

the LAYOUTGET of the second process might get mover redirect
before while mover kill in progress. As a result, the READ
request will e sent to a pool when mover is already stopped.

Modification:
Return error DELAY as long as shutdown in progress. The next LAYOUTGET
for read will start a new mover, for a write will get EPERM (immutable
files).

Result:
The race condition should be fixed.

Acked-by: Alert Rossi
Target: master, 9.1, 9.0, 8.2
Require-book: no
Require-notes: yes
(cherry picked from commit 421e1dc)
Signed-off-by: Tigran Mkrtchyan <tigran.mkrtchyan@desy.de>

modules/dcache-nfs/src/main/java/org/dcache/chimera/nfsv41/door/NFSv41Door.java

@@ -1497,6 +1497,12 @@ deviceid4[] getPoolDataServers(long timeout) throws
                 throw error;
             }
 
+            if (shutdownInProgress) {
+                // race with competing open. Let wait for pending mover shutdown to complete.
+                _log.info("Waiting for pending mover shutdown to complete: {}@{} ", this.getMoverId(), this.getPool());
+                throw new DelayException("Waiting for pending mover shutdown to complete");
+            }
+
             /*
              * If already have triggered selection process, then there are no
              * reasons to block. The async reply will update _redirectFuture when