New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to enumerate directories on RHEL 7.5 #7460

Closed
seidelma opened this Issue Apr 18, 2018 · 11 comments

Comments

Projects
  
0.7.9
5 participants
@seidelma

seidelma commented Apr 18, 2018

System information

Type Version/Name
Distribution Name Red Hat Enterprise Linux
Distribution Version 7.5
Linux Kernel 3.10.0-862.el7
Architecture x86_64
ZFS Version 0.7.8-1
SPL Version 0.7.8-1

Describe the problem you're observing

ZFS is unable to enumerate directories on RHEL 7.5, due to changes Red Hat has made to the kernel with 3.10.0-862. Kernel 3.10.0-693.21.1 from RHEL 7.4 does not experience this issue.

Describe how to reproduce the problem

Install RHEL 7.5
Install ZFS

# zpool create test /dev/sdb
# ls -alhtr /test
ls: reading directory /test: Not a directory
total 0

Include any warning/errors/backtraces from the system logs

This issue is caused by backported changes Red Hat made to their 3.10.0-862.el7 kernel. The getdents() system call now calls iterate_dir() (in fs/readdir.c) which also requires that the file be opened with f_mode & FMODE_KABI_ITERATE. In relevant part:

int iterate_dir(struct file *file, struct dir_context *ctx)
{
        struct inode *inode = file_inode(file);
        int res = -ENOTDIR;
        if (!file->f_op ||
            (!file->f_op->readdir && !(file->f_mode & FMODE_KABI_ITERATE)))
                goto out;
...

ZFS, for its part, #defines out the readdir() file op in module/zfs/zpl_file.c since it detects that the kernel supports iterate():

const struct file_operations zpl_dir_file_operations = {
        .llseek         = generic_file_llseek,
        .read           = generic_read_dir,
#ifdef HAVE_VFS_ITERATE_SHARED
        .iterate_shared = zpl_iterate,
#elif defined(HAVE_VFS_ITERATE)
        .iterate        = zpl_iterate,
#else
        .readdir        = zpl_readdir,
#endif
        .fsync          = zpl_fsync,
        .unlocked_ioctl = zpl_ioctl,
#ifdef CONFIG_COMPAT
        .compat_ioctl   = zpl_compat_ioctl,
#endif
};

Downgrading to RHEL's kernel-3.10.0-693.21.1.el7 package fixes the issue.

@behlendorf

This comment has been minimized.

Member

behlendorf commented Apr 18, 2018

@seidelma can you verify that falling back to .readdir works by undefining HAVE_VFS_ITERATE and HAVE_VFS_ITERATE_SHARED in zfs_config.h.

@seidelma

This comment has been minimized.

seidelma commented Apr 18, 2018

@behlendorf Setting both #undef HAVE_VFS_ITERATE and #undef HAVE_VFS_ITERATE SHARED causes a bunch of redefinition errors in include/sys/zpl.h:

In file included from /usr/src/zfs-0.7.8/module/zfs/arc.c:281:0:
/usr/src/zfs-0.7.8/include/sys/zpl.h:135:16: error: redefinition of ‘struct dir_context’
 typedef struct dir_context {
                ^
In file included from include/linux/huge_mm.h:5:0,
                 from include/linux/mm.h:394,
                 from /usr/src/spl-0.7.8//include/sys/thread.h:29,
                 from /usr/src/spl-0.7.8//include/sys/taskq.h:35,
                 from /usr/src/spl-0.7.8//include/sys/kmem_cache.h:28,
                 from /usr/src/zfs-0.7.8/include/sys/zfs_context.h:41,
                 from /usr/src/zfs-0.7.8/include/sys/spa.h:36,
                 from /usr/src/zfs-0.7.8/module/zfs/arc.c:263:
include/linux/fs.h:1775:8: note: originally defined here

and so on.

Unsetting only one causes the preprocessor to nuke the definition of zpl_readdir() in module/zfs/zpl_file.c.

@h1z1

This comment has been minimized.

h1z1 commented Apr 19, 2018

Is this RHEL only or CentOS as well? The rpm build is still a bit broken but as you indicated I can confirm at least CentOS 7.4 doesn't have this. zfs 0.7.8-1

@seidelma

This comment has been minimized.

seidelma commented Apr 19, 2018

@h1z1 This is currently RHEL-only because kernel 3.10.0-862 hasn't been released on CentOS yet (though according to git.centos.org it is in the works, and the offending code in the CentOS version of fs/readdir.c is identical so it will be an issue with the CentOS-branded kernel).

I've submitted this patch, which fixes the issue, to Red Hat.
check_iterate_fileop.txt

behlendorf added a commit to behlendorf/zfs that referenced this issue Apr 20, 2018

RHEL 7.5 compat: FMODE_KABI_ITERATE
As of RHEL 7.5 the mainline fops.iterate() method was added to
the file_operations structure.

This wouldn't have been a problem however in order to maintain
KABI compatibility it was added to the very end of the structure.
Callers intending to use this extended interface were additionally
required to set the FMODE_KABI_ITERATE flag on the file struct
when opening the directory.

Update the ZPL to set this RHEL specific flag when required.
Using this interface will mean that kmods built for RHEL 7.5
will not work on 7.4 without being rebuilt.

Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Issue zfsonlinux#7460

@behlendorf behlendorf referenced this issue Apr 20, 2018

Merged

RHEL 7.5 compat: FMODE_KABI_ITERATE #7463

5 of 13 tasks complete
@behlendorf

This comment has been minimized.

Member

behlendorf commented Apr 20, 2018

@seidelma thanks for looking in to this. I've opened #7463 with an proposed fix which will allow ZFS to use the provided RH (and CentOS) kernel. Please review and test if you have the time.

@seidelma

This comment has been minimized.

seidelma commented Apr 20, 2018

@behlendorf Looks good with some rudimentary tests I've run. Thanks!

ofaaland added a commit to LLNL/zfs that referenced this issue Apr 21, 2018

RHEL 7.5 compat: FMODE_KABI_ITERATE
As of RHEL 7.5 the mainline fops.iterate() method was added to
the file_operations structure.

This wouldn't have been a problem however in order to maintain
KABI compatibility it was added to the very end of the structure.
Callers intending to use this extended interface were additionally
required to set the FMODE_KABI_ITERATE flag on the file struct
when opening the directory.

Update the ZPL to set this RHEL specific flag when required.
Using this interface will mean that kmods built for RHEL 7.5
will not work on 7.4 without being rebuilt.

Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Issue zfsonlinux#7460
@truatpasteurdotfr

This comment has been minimized.

truatpasteurdotfr commented May 1, 2018

3.10.0-862 is released under the CR repository, and I can confirm that both 0.6.5.11 and 0.7.8 have the same issue when rebuilt under that kernel version.

@ElCoyote27

This comment has been minimized.

ElCoyote27 commented May 1, 2018

Hi,
Thanks for fixing this. Any idea when you'll be able to release updated el7.5 packages?
Thanks,
Vincent

@ElCoyote27

This comment has been minimized.

ElCoyote27 commented May 1, 2018

I've opened https://bugzilla.redhat.com/show_bug.cgi?id=1573633 to raise awareness.

behlendorf added a commit to behlendorf/zfs that referenced this issue May 1, 2018

RHEL 7.5 compat: FMODE_KABI_ITERATE
As of RHEL 7.5 the mainline fops.iterate() method was added to
the file_operations structure and is correctly detected by the
configure script.

Normally this is what we want, but in order to maintain KABI
compatibility the RHEL change additionally does the following:

* Requires that callers intending to use this extended interface
  set the FMODE_KABI_ITERATE flag on the file structure when
  opening the directory.
* Adds the fops.iterate() method to the end of the structure,
  without removing fops.readdir().

This change updates the configure check to ignore the RHEL 7.5+
variant of fops.iterate() when detected.  Instead fallback to
the fops.readdir() interface which will be available.

Finally, add the 'zpl_' prefix to the directory context wrappers
to avoid colliding with the kernel provided symbols when both
the fops.iterate() and fops.readdir() are provided by the kernel.

Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Issue zfsonlinux#7460
@behlendorf

This comment has been minimized.

Member

behlendorf commented May 1, 2018

@ElCoyote27 we expect to have an updated point release this week which resolves the issue.

@ElCoyote27

This comment has been minimized.

ElCoyote27 commented May 1, 2018

Thanks Brian.
I'm currently rebuilding rpms of 0.7.8 with your latest patch to solve the issue for my environment.
Refs: https://github.com/zfsonlinux/zfs/commit/d177418748dc81279b080edd50cc6b4d3ed8f450.patch
Thank you very much for the quick fix.
Regards,
Vincent

behlendorf added a commit to behlendorf/zfs that referenced this issue May 2, 2018

RHEL 7.5 compat: FMODE_KABI_ITERATE
As of RHEL 7.5 the mainline fops.iterate() method was added to
the file_operations structure and is correctly detected by the
configure script.

Normally this is what we want, but in order to maintain KABI
compatibility the RHEL change additionally does the following:

* Requires that callers intending to use this extended interface
  set the FMODE_KABI_ITERATE flag on the file structure when
  opening the directory.
* Adds the fops.iterate() method to the end of the structure,
  without removing fops.readdir().

This change updates the configure check to ignore the RHEL 7.5+
variant of fops.iterate() when detected.  Instead fallback to
the fops.readdir() interface which will be available.

Finally, add the 'zpl_' prefix to the directory context wrappers
to avoid colliding with the kernel provided symbols when both
the fops.iterate() and fops.readdir() are provided by the kernel.

Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Issue zfsonlinux#7460

@behlendorf behlendorf closed this in 9464b95 May 2, 2018

tonyhutter added a commit to tonyhutter/zfs that referenced this issue May 3, 2018

RHEL 7.5 compat: FMODE_KABI_ITERATE
As of RHEL 7.5 the mainline fops.iterate() method was added to
the file_operations structure and is correctly detected by the
configure script.

Normally this is what we want, but in order to maintain KABI
compatibility the RHEL change additionally does the following:

* Requires that callers intending to use this extended interface
  set the FMODE_KABI_ITERATE flag on the file structure when
  opening the directory.
* Adds the fops.iterate() method to the end of the structure,
  without removing fops.readdir().

This change updates the configure check to ignore the RHEL 7.5+
variant of fops.iterate() when detected.  Instead fallback to
the fops.readdir() interface which will be available.

Finally, add the 'zpl_' prefix to the directory context wrappers
to avoid colliding with the kernel provided symbols when both
the fops.iterate() and fops.readdir() are provided by the kernel.

Reviewed-by: Olaf Faaland <faaland1@llnl.gov>
Reviewed-by: Tony Hutter <hutter2@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes zfsonlinux#7460 
Closes zfsonlinux#7463

tonyhutter added a commit to tonyhutter/zfs that referenced this issue May 3, 2018

RHEL 7.5 compat: FMODE_KABI_ITERATE
As of RHEL 7.5 the mainline fops.iterate() method was added to
the file_operations structure and is correctly detected by the
configure script.

Normally this is what we want, but in order to maintain KABI
compatibility the RHEL change additionally does the following:

* Requires that callers intending to use this extended interface
  set the FMODE_KABI_ITERATE flag on the file structure when
  opening the directory.
* Adds the fops.iterate() method to the end of the structure,
  without removing fops.readdir().

This change updates the configure check to ignore the RHEL 7.5+
variant of fops.iterate() when detected.  Instead fallback to
the fops.readdir() interface which will be available.

Finally, add the 'zpl_' prefix to the directory context wrappers
to avoid colliding with the kernel provided symbols when both
the fops.iterate() and fops.readdir() are provided by the kernel.

Reviewed-by: Olaf Faaland <faaland1@llnl.gov>
Reviewed-by: Tony Hutter <hutter2@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes zfsonlinux#7460 
Closes zfsonlinux#7463

tonyhutter added a commit to tonyhutter/zfs that referenced this issue May 4, 2018

RHEL 7.5 compat: FMODE_KABI_ITERATE
As of RHEL 7.5 the mainline fops.iterate() method was added to
the file_operations structure and is correctly detected by the
configure script.

Normally this is what we want, but in order to maintain KABI
compatibility the RHEL change additionally does the following:

* Requires that callers intending to use this extended interface
  set the FMODE_KABI_ITERATE flag on the file structure when
  opening the directory.
* Adds the fops.iterate() method to the end of the structure,
  without removing fops.readdir().

This change updates the configure check to ignore the RHEL 7.5+
variant of fops.iterate() when detected.  Instead fallback to
the fops.readdir() interface which will be available.

Finally, add the 'zpl_' prefix to the directory context wrappers
to avoid colliding with the kernel provided symbols when both
the fops.iterate() and fops.readdir() are provided by the kernel.

Reviewed-by: Olaf Faaland <faaland1@llnl.gov>
Reviewed-by: Tony Hutter <hutter2@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes zfsonlinux#7460 
Closes zfsonlinux#7463

tonyhutter added a commit to LLNL/zfs that referenced this issue May 4, 2018

RHEL 7.5 compat: FMODE_KABI_ITERATE
As of RHEL 7.5 the mainline fops.iterate() method was added to
the file_operations structure and is correctly detected by the
configure script.

Normally this is what we want, but in order to maintain KABI
compatibility the RHEL change additionally does the following:

* Requires that callers intending to use this extended interface
  set the FMODE_KABI_ITERATE flag on the file structure when
  opening the directory.
* Adds the fops.iterate() method to the end of the structure,
  without removing fops.readdir().

This change updates the configure check to ignore the RHEL 7.5+
variant of fops.iterate() when detected.  Instead fallback to
the fops.readdir() interface which will be available.

Finally, add the 'zpl_' prefix to the directory context wrappers
to avoid colliding with the kernel provided symbols when both
the fops.iterate() and fops.readdir() are provided by the kernel.

Reviewed-by: Olaf Faaland <faaland1@llnl.gov>
Reviewed-by: Tony Hutter <hutter2@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes zfsonlinux#7460
Closes zfsonlinux#7463

tonyhutter added a commit that referenced this issue May 10, 2018

RHEL 7.5 compat: FMODE_KABI_ITERATE
As of RHEL 7.5 the mainline fops.iterate() method was added to
the file_operations structure and is correctly detected by the
configure script.

Normally this is what we want, but in order to maintain KABI
compatibility the RHEL change additionally does the following:

* Requires that callers intending to use this extended interface
  set the FMODE_KABI_ITERATE flag on the file structure when
  opening the directory.
* Adds the fops.iterate() method to the end of the structure,
  without removing fops.readdir().

This change updates the configure check to ignore the RHEL 7.5+
variant of fops.iterate() when detected.  Instead fallback to
the fops.readdir() interface which will be available.

Finally, add the 'zpl_' prefix to the directory context wrappers
to avoid colliding with the kernel provided symbols when both
the fops.iterate() and fops.readdir() are provided by the kernel.

Reviewed-by: Olaf Faaland <faaland1@llnl.gov>
Reviewed-by: Tony Hutter <hutter2@llnl.gov>
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Closes #7460
Closes #7463
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment