Interoperability issue with cookie handling for READDIR operation

[sraillard]

We have noticed an interoperability issue when a Linux NFS client is sending a READDIR command with a cookie value over 0.

For example, when a cookie value of 102 is sent in the READDIR command, the nfs4j server will respond with a list of files, and the first file will have the cookie value of 102. In case of a Linux NFS server, the first file will have a cookie value of 103.

The RFC 7530 (page 270) isn't really clear if the cookie value in the request should be included (like nfs4js) or excluded (like Linux NFS server):
The arguments contain a cookie value that represents where the READDIR should start within the directory. A value of 0 (zero) or the cookie is used to start reading at the beginning of the directory. For subsequent READDIR requests, the client specifies a cookie value that is provided by the server in a previous READDIR request.

I'm joining a NFS Wireshark capture between a Linux client and a Linux server to show the difference (check the second READDIR).
nfs-readdir-linux-server.zip
Another difference noted is the Linux NFS server is always setting the cookie of the last entry to the value 2147483647.

To exclude the cookie provided in the request, this line can be changed:

nfs4j/core/src/main/java/org/dcache/nfs/v4/OperationREADDIR.java

Line 105 in b4af4c3

startValue -= COOKIE_OFFSET;

To:
startValue -= (COOKIE_OFFSET-1);

[kofemann]

Hi,

though the spec doesn't clearly says it, server should return subsequent entries. IOW, if cookie is 102 then server should return directory listing starting from entry 103, as Linux server correctly does. If nfs4j re-sends 102, then this is a clear bug. However, the READDIR operation expects that VFS layer does the right thing

The returned listing will contain only entries with cookies greater than specified value.

nfs4j/core/src/main/java/org/dcache/nfs/vfs/VirtualFileSystem.java

Line 117 in b4af4c3

* is used. The returned listing will contain only entries with cookies

Does your filesystem backend preserves this behavior?

[sraillard]

Thank for pointing that! We have found a bug in th VFS and we'll fix it.

Do you have any opinion about the last directory entry cookie set to 2147483647 by the Linux NFS server?

[kofemann]

Great! The 2147483647 is 0x7fffffff and looks like a magic number used by the backend file system to indicate some special value, last block, for example. Do you use xfs?

[sraillard]

The filesystem used was btrfs with Fedora Linux OS.

It's clearly looking like a magic number, but I didn't find any reference of this value in the RFC.

[kofemann]

This is linux fs specific and have nothing to do with NFS spec.

linux-kernel $ git grep 0x7fffffff fs
fs/affs/file.c: BUG_ON(block > (sector_t)0x7fffffffUL);
fs/afs/internal.h:#define __AFS_PAGE_PRIV_MASK  0x7fffffffUL
fs/ext2/inode.c:                if (inode->i_size > 0x7fffffffULL) {
fs/ext4/hash.c:                 hash -= 0x7fffffff;
fs/ext4/hash.c:                 hash -= 0x7fffffff;
fs/ext4/inode-test.c:#define UPPER_MSB_0 0x7fffffffL
fs/ext4/inode-test.c:                   .expected = {.tv_sec = 0x7fffffffLL, .tv_nsec = 0L},
fs/ext4/inode.c:        if (pos + len <= 0x7fffffffULL)
fs/ext4/inode.c:        if (ei->i_disksize > 0x7fffffffULL) {
fs/kernfs/dir.c:        hash &= 0x7fffffffU;
fs/nfs/proc.c:#define NFS_LOCK32_OFFSET_MAX ((__s32)0x7fffffffUL)
fs/nfsd/vfs.c:          v_mtime = verifier[0]&0x7fffffff;
fs/nfsd/vfs.c:          v_atime = verifier[1]&0x7fffffff;
fs/ntfs/super.c:        if (i_size <= 0 || i_size > 0x7fffffff)
fs/ocfs2/cluster/tcp.h:#define O2NET_TCP_USER_TIMEOUT                   0x7fffffff
fs/ocfs2/quota_local.c: info->dqi_max_spc_limit = 0x7fffffffffffffffLL;
fs/ocfs2/quota_local.c: info->dqi_max_ino_limit = 0x7fffffffffffffffLL;
fs/proc/array.c:        seq_put_decimal_ull(m, " ", task->pending.signal.sig[0] & 0x7fffffffUL);
fs/proc/array.c:        seq_put_decimal_ull(m, " ", task->blocked.sig[0] & 0x7fffffffUL);
fs/proc/array.c:        seq_put_decimal_ull(m, " ", sigign.sig[0] & 0x7fffffffUL);
fs/proc/array.c:        seq_put_decimal_ull(m, " ", sigcatch.sig[0] & 0x7fffffffUL);
fs/quota/quota_v2.c:            info->dqi_max_spc_limit = 0x7fffffffffffffffLL; /* 2^63-1 */
fs/quota/quota_v2.c:            info->dqi_max_ino_limit = 0x7fffffffffffffffLL;
fs/vboxsf/shfl_hostintf.h:      SHFLFSOBJATTRADD_32BIT_SIZE_HACK = 0x7fffffff
fs/xfs/libxfs/xfs_types.h:#define       MAXEXTNUM       ((xfs_extnum_t)0x7fffffff)      /* signed int */
fs/xfs/xfs_dir2_readdir.c:              ctx->pos = dot_offset & 0x7fffffff;
fs/xfs/xfs_dir2_readdir.c:              ctx->pos = dotdot_offset & 0x7fffffff;
fs/xfs/xfs_dir2_readdir.c:              ctx->pos = off & 0x7fffffff;
fs/xfs/xfs_dir2_readdir.c:                                                              0x7fffffff;
fs/xfs/xfs_dir2_readdir.c:              ctx->pos = cook & 0x7fffffff;
fs/xfs/xfs_dir2_readdir.c:                                                              0x7fffffff;
fs/xfs/xfs_dir2_readdir.c:              ctx->pos = xfs_dir2_byte_to_dataptr(curoff) & 0x7fffffff;
fs/xfs/xfs_dir2_readdir.c:              ctx->pos = XFS_DIR2_MAX_DATAPTR & 0x7fffffff;
fs/xfs/xfs_dir2_readdir.c:              ctx->pos = xfs_dir2_byte_to_dataptr(curoff) & 0x7fffffff;
linux-kernel $

[sraillard]

Clearly a magic number widley used...

Thank you for all you help!