Freespace >2TB not supported in 32-bit userspace

[mrjoel]

System information

Distribution Name | Debian
Distribution Version | stretch/9.3 (plus some stretch-backports)
Linux Kernel | 4.14.0-0.bpo.3-amd64
Architecture | i386 (32-bit userspace, 64-bit kernel)
ZFS Version | 0.7.5-1~bpo9+1
SPL Version | 0.7.5-1~bpo9+1

Describe the problem you're observing

An apparent size/type mismatch is present when using 32-bit userspace, a 64-bit kernel, and a filesystem is present with greater than 2TB of free space. It prevents the NFS server from properly exporting the filesystem, and df generates the error below.

df: /srv: Value too large for defined data type

Describe how to reproduce the problem

Import and mount a ZFS filesystem with greater than 2TB available space. From a 64-bit userspace, execute the df command. From the same running system, in a 32-bit userspace, execute the df command and observe the error. The particular system I'm seeing this on is 32-bit root, and I'm testing direct 64-bit binary for confirmation. I'm unable to test to confirm right now, but I expect that with a 64-bit root and 32-bit df userspace the same issue would be exhibited.

Just to confirm isolation of the issue to ZFS, I created a sparse file backed 3TB loopback mounted filesystem to present greater than 2TB free space, and it worked fine with ext4.

As a workaround for now, I've created a 900GB dummy space-hogging zvol which forces my available free space under 2TB. Once that is done, things all seem to start working again. The total size and used values seem to be mapped correctly, only the avail threshold causes the error.

[mrjoel]

I confirmed the reverse on a dev box - running 64-bit root with 64-bit kernel, simply running a 32-bit df command demonstrates the issue.

[adilger]

In the Lustre client, we dynamically scale the ->statfs() return values to fit within 32-bit fields when needed. This can be done by increasing the reported blocksize and correspondingly reducing the total/free/available block counts:

    /*
     * Downshift statfs return fields for 32-bit system calls.
     * Stop before overflowing f_bsize - in which case it is better
     * to just risk EOVERFLOW if caller is using old sys_statfs().
     */
    if (ll_need_32bit_api(ll_s2sbi(sb))) {
            while (osfs.os_blocks > 0xffffffff && sfs->f_bsize < 0x40000000) {
                    sfs->f_bsize <<= 1;

                    osfs.os_blocks >>= 1;
                    osfs.os_bfree >>= 1;
                    osfs.os_bavail >>= 1;
            }
    }

The ll_need_32bit_api() check is determining whether this is a 32-bit syscall or not, for both 32-bit and 64-bit kernels:

static inline bool ll_need_32bit_api(struct ll_sb_info *sbi)
{
#if BITS_PER_LONG == 32
    return true;
#elif defined(CONFIG_COMPAT)
# ifdef CONFIG_X86_X32
    /* in_compat_syscall() returns true when called from a kthread
     * and CONFIG_X86_X32 is enabled, which is wrong. So check
     * whether the caller comes from a syscall (ie. not a kthread)
     * before calling in_compat_syscall(). */
    if (current->flags & PF_KTHREAD)
            return false;
# endif
    return unlikely(in_compat_syscall());
#else
    return false;
#endif
}

[behlendorf]

@adilger thanks for the pointer, would you mind reviewing the proposed fix in #7937. It doesn't incorporate the kernel thread detection since that wasn't an issue in this case, it was primarily being caused by overflowing the f_files and f_ffree. Although block overflow is less likely I did add code to downshift the block counts. You might want to consider adding f_files and f_ffree checks to Lustre too.

[ryao]

@mrjoel Assuming that coreutils andthe nfsd userland utilities are built with -D_FILE_OFFSET_BITS=64, you are likely affected by this kernel bug:

https://patchwork.kernel.org/patch/9987759/

[behlendorf]

@ryao as I understand it this issue is specifically for the 32-bit user space case (all of user space).

An apparent size/type mismatch is present when using 32-bit userspace, a 64-bit kernel