Fedora 31 - Kernel 5.5.5 ZFS crash on startup #10043

greg-hydrogen · 2020-02-23T23:43:41Z

Describe the problem you're observing

ZFS RIP on startup. When attempting to boot into the lastest Kernel I get the following stacktrace

Feb 23 10:15:11 lnxworkstn kernel: BUG: unable to handle page fault for address: 0000000000881bb0
Feb 23 10:15:11 lnxworkstn kernel: #PF: supervisor write access in kernel mode
Feb 23 10:15:11 lnxworkstn kernel: #PF: error_code(0x0002) - not-present page
Feb 23 10:15:11 lnxworkstn kernel: PGD 0 P4D 0
Feb 23 10:15:11 lnxworkstn kernel: Oops: 0002 [#1] SMP NOPTI
Feb 23 10:15:11 lnxworkstn kernel: CPU: 4 PID: 3267 Comm: auditd Tainted: P OE 5.5.5-200.fc31.x86_64 #1
Feb 23 10:15:11 lnxworkstn kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./X470 Taichi Ultimate, BIOS P2.00 12/17/2018
Feb 23 10:15:11 lnxworkstn kernel: RIP: 0010:mutex_lock+0x19/0x30
Feb 23 10:15:11 lnxworkstn kernel: Code: 00 0f 1f 44 00 00 be 02 00 00 00 e9 21 fb ff ff 90 0f 1f 44 00 00 55 48 89 fd e8 c2 dc ff ff 31 c0 65 48 8b 14 25 c0 8b 01 00 48 0f b1 55 00 74 06 48 89 ef 5d e>
Feb 23 10:15:11 lnxworkstn kernel: RSP: 0018:ffffa89c79183910 EFLAGS: 00010246
Feb 23 10:15:11 lnxworkstn kernel: RAX: 0000000000000000 RBX: 0000000000881ba8 RCX: 0000000000000002
Feb 23 10:15:11 lnxworkstn kernel: RDX: ffff9c4ed22e4d00 RSI: ffff9c4f0ad25280 RDI: 0000000000881bb0
Feb 23 10:15:11 lnxworkstn kernel: RBP: 0000000000881bb0 R08: 0000000000000000 R09: 0000000000000000
Feb 23 10:15:11 lnxworkstn kernel: R10: ffffa89c79183988 R11: ffff9c4eedba1a00 R12: ffff9c4ed268f780
Feb 23 10:15:11 lnxworkstn kernel: R13: 0000000000881bb0 R14: 0000000000000002 R15: ffff9c4f0ad25000
Feb 23 10:15:11 lnxworkstn kernel: FS: 00007f23ea736880(0000) GS:ffff9c4f1ed00000(0000) knlGS:0000000000000000
Feb 23 10:15:11 lnxworkstn kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Feb 23 10:15:11 lnxworkstn kernel: CR2: 0000000000881bb0 CR3: 0000000f930ec000 CR4: 00000000003406e0
Feb 23 10:15:11 lnxworkstn kernel: Call Trace:
Feb 23 10:15:11 lnxworkstn kernel: zfs_dirent_unlock+0x20/0x140 [zfs]
Feb 23 10:15:11 lnxworkstn kernel: zfs_setattr_dir+0x405/0x470 [zfs]
Feb 23 10:15:11 lnxworkstn kernel: ? _cond_resched+0x15/0x30
Feb 23 10:15:11 lnxworkstn kernel: ? mutex_lock+0xe/0x30
Feb 23 10:15:11 lnxworkstn kernel: ? txg_list_add+0x73/0x90 [zfs]
Feb 23 10:15:11 lnxworkstn kernel: ? dsl_dataset_dirty+0x4f/0xc0 [zfs]
Feb 23 10:15:11 lnxworkstn kernel: ? dbuf_dirty+0x525/0xa00 [zfs]
Feb 23 10:15:11 lnxworkstn kernel: ? _cond_resched+0x15/0x30
Feb 23 10:15:11 lnxworkstn kernel: ? mutex_lock+0xe/0x30
Feb 23 10:15:11 lnxworkstn kernel: ? sa_attr_op+0x29c/0x3c0 [zfs]
Feb 23 10:15:11 lnxworkstn kernel: ? _cond_resched+0x15/0x30
Feb 23 10:15:11 lnxworkstn kernel: ? dnode_rele_and_unlock+0x53/0xb0 [zfs]
Feb 23 10:15:11 lnxworkstn kernel: zfs_setattr+0x1cc8/0x2240 [zfs]
Feb 23 10:15:11 lnxworkstn kernel: ? lookup_fast+0x107/0x280
Feb 23 10:15:11 lnxworkstn kernel: ? _cond_resched+0x15/0x30
Feb 23 10:15:11 lnxworkstn kernel: ? __kmalloc_node+0x1ff/0x310
Feb 23 10:15:11 lnxworkstn kernel: zpl_setattr+0xff/0x170 [zfs]
Feb 23 10:15:11 lnxworkstn kernel: notify_change+0x333/0x4b0
Feb 23 10:15:11 lnxworkstn kernel: chown_common.isra.0+0xec/0x1a0
Feb 23 10:15:11 lnxworkstn kernel: do_fchownat+0x8f/0xf0
Feb 23 10:15:11 lnxworkstn kernel: __x64_sys_chown+0x1e/0x30
Feb 23 10:15:11 lnxworkstn kernel: do_syscall_64+0x5b/0x1c0
Feb 23 10:15:11 lnxworkstn kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9
Feb 23 10:15:11 lnxworkstn kernel: RIP: 0033:0x7f23eac4c79b
Feb 23 10:15:11 lnxworkstn kernel: Code: 00 00 00 75 a5 48 89 ef e8 e2 c7 f9 ff eb a4 e8 eb dd 01 00 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa b8 5c 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d b>
Feb 23 10:15:11 lnxworkstn kernel: RSP: 002b:00007ffef926ff38 EFLAGS: 00000246 ORIG_RAX: 000000000000005c
Feb 23 10:15:11 lnxworkstn kernel: RAX: ffffffffffffffda RBX: 0000564a6490e100 RCX: 00007f23eac4c79b
Feb 23 10:15:11 lnxworkstn kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000564a656c4320
Feb 23 10:15:11 lnxworkstn kernel: RBP: 0000564a656c4320 R08: 0000564a656c4320 R09: 00007ffef926fe10
Feb 23 10:15:11 lnxworkstn kernel: R10: 0000000000000030 R11: 0000000000000246 R12: 0000564a656c4320
Feb 23 10:15:11 lnxworkstn kernel: R13: 0000000000000028 R14: 0000564a656c2900 R15: 0000000000000000
b 23 10:15:11 lnxworkstn kernel: Modules linked in: vfat fat edac_mce_amd snd_hda_codec_realtek btusb kvm_amd btrtl snd_hda_codec_generic btbcm iwlmvm ledtrig_audio snd_hda_codec_hdmi btintel snd_hda_inte>
Feb 23 10:15:11 lnxworkstn kernel: CR2: 0000000000881bb0
Feb 23 10:15:11 lnxworkstn kernel: ---[ end trace d9880de7770176e7 ]---
Feb 23 10:15:11 lnxworkstn kernel: RIP: 0010:mutex_lock+0x19/0x30
Feb 23 10:15:11 lnxworkstn kernel: Code: 00 0f 1f 44 00 00 be 02 00 00 00 e9 21 fb ff ff 90 0f 1f 44 00 00 55 48 89 fd e8 c2 dc ff ff 31 c0 65 48 8b 14 25 c0 8b 01 00 48 0f b1 55 00 74 06 48 89 ef 5d e>
Feb 23 10:15:11 lnxworkstn kernel: RSP: 0018:ffffa89c79183910 EFLAGS: 00010246
Feb 23 10:15:11 lnxworkstn kernel: RAX: 0000000000000000 RBX: 0000000000881ba8 RCX: 0000000000000002
Feb 23 10:15:11 lnxworkstn kernel: RDX: ffff9c4ed22e4d00 RSI: ffff9c4f0ad25280 RDI: 0000000000881bb0
Feb 23 10:15:11 lnxworkstn kernel: RBP: 0000000000881bb0 R08: 0000000000000000 R09: 0000000000000000
Feb 23 10:15:11 lnxworkstn kernel: R10: ffffa89c79183988 R11: ffff9c4eedba1a00 R12: ffff9c4ed268f780
Feb 23 10:15:11 lnxworkstn kernel: R13: 0000000000881bb0 R14: 0000000000000002 R15: ffff9c4f0ad25000
Feb 23 10:15:11 lnxworkstn kernel: FS: 00007f23ea736880(0000) GS:ffff9c4f1ed00000(0000) knlGS:0000000000000000
Feb 23 10:15:11 lnxworkstn kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Feb 23 10:15:11 lnxworkstn kernel: CR2: 0000000000881bb0 CR3: 0000000f930ec000 CR4: 00000000003406e0

Describe how to reproduce the problem

This occurs on every reboot with kernel 5.5.5 (fedora 31)

Include any warning/errors/backtraces from the system logs

mtb-xt · 2020-02-25T22:47:29Z

@greg-hydrogen kernels 5.5 are not supported yet, I think:
Latest release states: zfs-0.8.3:
Compatible with 2.6.32 - 5.4 Linux kernels

behlendorf · 2020-02-26T00:18:53Z

We'll need to determine if this is related to the updated kernel.

AllKind · 2020-02-28T19:16:37Z

FYI: I'm running zfs 0.8.3 on kernel 5.5.6 (vanilla) on top of a linux mint 19.3 without any crashes.

svde · 2020-02-28T22:36:06Z

running 5.5.5-200.fc31.x86_64, works for me

greg-hydrogen · 2020-02-28T22:56:24Z

interesting... I get the panic every time I boot so is there any additional information I can provide?

rgessner · 2020-03-01T12:40:44Z

kernel-5.5.5-200.fc31.x86_64 with zfs-0.8.3 works without problems here.
Just updated to newly relesaed kernel-5.5.6-201.fc31.x86_64, again no problems with zfs-0.8.3.

# uname -a; zfs --version
Linux janus 5.5.6-201.fc31.x86_64 #1 SMP Mon Feb 24 20:24:13 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
zfs-0.8.3-1
zfs-kmod-0.8.3-1

sparksh · 2020-03-07T07:23:18Z

zfs-0.8.3 - I just updated to kernel-5.5.7, EFI, boot+root on ZFS. It all seems to be working.
With /boot on ZFS, a grub script patch is required for Fedora.

AllKind · 2020-03-11T12:45:01Z

Just FYI, zfs 0.8.3 compiled well with kernel 5.6-rc5, arch amd64. Did not test it more, because the nvidia drivers did not compile.

greg-hydrogen · 2020-04-12T16:52:44Z

Hello Everyone,

I decided to upgrade to Fedora 32 and test a new kernel (5.6) I am getting the same crash as before but this time I narrowed it down to auditd. If auditd doesn't start then I am able to boot and there is no crash, the moment I start auditd I get the following

Apr 11 20:17:54 workstn kernel: kernel BUG at fs/inode.c:1588!
Apr 11 20:17:54 workstn kernel: invalid opcode: 0000 [#1] SMP NOPTI
Apr 11 20:17:54 workstn kernel: CPU: 14 PID: 3364 Comm: auditd Tainted: P OE 5.6.3-300.fc32.x86_64 #1
Apr 11 20:17:54 workstn kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./X470 Taichi Ultimate, BIOS P2.00 12/17/2018
Apr 11 20:17:54 workstn kernel: RIP: 0010:iput+0x19/0x20
Apr 11 20:17:54 workstn kernel: Code: ff ff 0f 0b eb d5 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 85 ff 74 0e f6 87 98 00 00 00 40 75 06 e9 38 fe ff ff c3 <0f> 0b 0f 1f 44 00 00 0f 1f 44 00 00 4>
Apr 11 20:17:54 workstn kernel: RSP: 0018:ffffae0ff896f970 EFLAGS: 00010202
Apr 11 20:17:54 workstn kernel: RAX: 0000000000000002 RBX: ffff93e9decee780 RCX: 0000000000000002
Apr 11 20:17:54 workstn kernel: RDX: 000000000000064b RSI: ffffffffc104ce80 RDI: ffffffffc0f5689a
Apr 11 20:17:55 workstn kernel: RBP: ffffae0ff896fdb0 R08: 0000000000000000 R09: 0000000000000000
Apr 11 20:17:55 workstn kernel: R10: ffffae0ff896f9b8 R11: 0000000000000001 R12: ffff93ea05b70480
Apr 11 20:17:55 workstn kernel: R13: ffff93ea09657000 R14: ffffffffc0f566a2 R15: ffff93e9fc7e1ec0
Apr 11 20:17:55 workstn kernel: FS: 00007f7e3556d880(0000) GS:ffff93ea1ef80000(0000) knlGS:0000000000000000
Apr 11 20:17:55 workstn kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Apr 11 20:17:55 workstn kernel: CR2: 00007f53bb1af014 CR3: 0000000f8b8c2000 CR4: 00000000003406e0
Apr 11 20:17:55 workstn kernel: Call Trace:
Apr 11 20:17:55 workstn kernel: zfs_setattr_dir.isra.0+0x405/0x460 [zfs]
Apr 11 20:17:55 workstn kernel: ? txg_list_add+0x72/0x90 [zfs]
Apr 11 20:17:55 workstn kernel: ? _cond_resched+0x16/0x40
Apr 11 20:17:55 workstn kernel: ? _cond_resched+0x16/0x40
Apr 11 20:17:55 workstn kernel: ? mutex_lock+0xe/0x30
Apr 11 20:17:55 workstn kernel: ? txg_list_add+0x72/0x90 [zfs]
Apr 11 20:17:55 workstn kernel: ? dsl_dataset_dirty+0x4f/0xc0 [zfs]
Apr 11 20:17:55 workstn kernel: ? dbuf_dirty+0x48d/0x9a0 [zfs]
Apr 11 20:17:55 workstn kernel: ? _cond_resched+0x16/0x40
Apr 11 20:17:55 workstn kernel: ? mutex_lock+0xe/0x30
Apr 11 20:17:55 workstn kernel: ? dmu_buf_will_dirty_impl+0x123/0x160 [zfs]
Apr 11 20:17:55 workstn kernel: ? dsl_dir_tempreserve_clear+0x105/0x160 [zfs]
Apr 11 20:17:55 workstn kernel: ? kfree+0xa3/0x200
Apr 11 20:17:55 workstn kernel: ? dmu_tx_destroy+0x7b/0x130 [zfs]
Apr 11 20:17:55 workstn kernel: ? dnode_rele_and_unlock+0x5c/0xc0 [zfs]
Apr 11 20:17:55 workstn kernel: ? zfs_setattr+0x1c80/0x2200 [zfs]
Apr 11 20:17:55 workstn kernel: zfs_setattr+0x1ca6/0x2200 [zfs]
Apr 11 20:17:55 workstn kernel: ? avc_has_perm+0x3b/0x160
Apr 11 20:17:55 workstn kernel: ? _cond_resched+0x16/0x40
Apr 11 20:17:55 workstn kernel: ? __kmalloc_node+0x1fb/0x300
Apr 11 20:17:55 workstn kernel: zpl_setattr+0xff/0x170 [zfs]
Apr 11 20:17:55 workstn kernel: notify_change+0x333/0x4b0
Apr 11 20:17:55 workstn kernel: chown_common+0xde/0x180
Apr 11 20:17:55 workstn kernel: ? _cond_resched+0x16/0x40
Apr 11 20:17:55 workstn kernel: do_fchownat+0x7d/0xc0
Apr 11 20:17:55 workstn kernel: __x64_sys_chown+0x1e/0x30
Apr 11 20:17:55 workstn kernel: do_syscall_64+0x5b/0xf0
Apr 11 20:17:55 workstn kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9
Apr 11 20:17:55 workstn kernel: RIP: 0033:0x7f7e35a968ab
Apr 11 20:17:55 workstn kernel: Code: 00 00 00 75 a5 48 89 ef e8 a2 ca f9 ff eb a4 e8 0b df 01 00 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa b8 5c 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d b>
Apr 11 20:17:55 workstn kernel: RSP: 002b:00007ffd58c3a818 EFLAGS: 00000246 ORIG_RAX: 000000000000005c
Apr 11 20:17:55 workstn kernel: RAX: ffffffffffffffda RBX: 0000000000000018 RCX: 00007f7e35a968ab
Apr 11 20:17:55 workstn kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 000056179aeb0300
Apr 11 20:17:55 workstn kernel: RBP: 000056179aeb0300 R08: 0000000001ffffff R09: 00007f7e35b66a40
Apr 11 20:17:55 workstn kernel: R10: 00000000000001a4 R11: 0000000000000246 R12: 0000000000000028
Apr 11 20:17:55 workstn kernel: R13: 000056179aeb0300 R14: 000056179972a120 R15: 0000000000000000
Apr 11 20:17:55 workstn kernel: Modules linked in: vfat fat edac_mce_amd kvm_amd iwlmvm kvm mac80211 joydev irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel libarc4 snd_hda_codec_realtek iwlwi>
Apr 11 20:17:55 workstn kernel: ---[ end trace 936636b291ab9763 ]---

this issue if very reproducible so if there is anything I can provide to help please let me know

Thank you!

snajpa · 2020-07-20T16:35:49Z

In our environment this only happens with xattr=on, with xattr=sa the problem goes away. It happens for the extended attributes zp in zfs_setattr_dir, on the err2 path after out label in zfs_setattr.

Fixes openzfs#10043 The value of zp is used without having been initialized under some conditions. Initialize the pointer to NULL. Add a regression test case using chown in acl/posix. However, this is not enough because the setup sets xattr=sa, which means zfs_setattr_dir will not be called. Create a second group of acl tests in acl/posix-sa duplicating the acl/posix tests with symlinks, and remove xattr=sa from the original acl/posix tests. This provides more coverage for the default xattr=on code. Signed-off-by: Ryan Moeller <ryan@iXsystems.com>

yocalebo · 2020-10-08T00:22:33Z

This is trivially reproducible in openZFS 2.0.0-RC3 and latest master. This pr fixes it for me: #11025

Reproduction is easy:

zfs create pool/ds
zfs set acltype=posixacl pool/ds
setfacl -d -m u:caleb:rwx /pool/ds
setfacl -b /pool/ds
chown 1777 /pool/ds
panic here

Fixes openzfs#10043 The value of zp is used without having been initialized under some conditions. Initialize the pointer to NULL. Add a regression test case using chown in acl/posix. However, this is not enough because the setup sets xattr=sa, which means zfs_setattr_dir will not be called. Create a second group of acl tests in acl/posix-sa duplicating the acl/posix tests with symlinks, and remove xattr=sa from the original acl/posix tests. This provides more coverage for the default xattr=on code. Signed-off-by: Ryan Moeller <ryan@iXsystems.com>

The value of zp is used without having been initialized under some conditions. Initialize the pointer to NULL. Add a regression test case using chown in acl/posix. However, this is not enough because the setup sets xattr=sa, which means zfs_setattr_dir will not be called. Create a second group of acl tests in acl/posix-sa duplicating the acl/posix tests with symlinks, and remove xattr=sa from the original acl/posix tests. This provides more coverage for the default xattr=on code. Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Ryan Moeller <ryan@iXsystems.com> Closes #10043 Closes #11025

The value of zp is used without having been initialized under some conditions. Initialize the pointer to NULL. Add a regression test case using chown in acl/posix. However, this is not enough because the setup sets xattr=sa, which means zfs_setattr_dir will not be called. Create a second group of acl tests in acl/posix-sa duplicating the acl/posix tests with symlinks, and remove xattr=sa from the original acl/posix tests. This provides more coverage for the default xattr=on code. Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Ryan Moeller <ryan@iXsystems.com> Closes openzfs#10043 Closes openzfs#11025

The value of zp is used without having been initialized under some conditions. Initialize the pointer to NULL. Add a regression test case using chown in acl/posix. However, this is not enough because the setup sets xattr=sa, which means zfs_setattr_dir will not be called. Create a second group of acl tests in acl/posix-sa duplicating the acl/posix tests with symlinks, and remove xattr=sa from the original acl/posix tests. This provides more coverage for the default xattr=on code. Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Ryan Moeller <ryan@iXsystems.com> Closes #10043 Closes #11025

The value of zp is used without having been initialized under some conditions. Initialize the pointer to NULL. Add a regression test case using chown in acl/posix. However, this is not enough because the setup sets xattr=sa, which means zfs_setattr_dir will not be called. Create a second group of acl tests in acl/posix-sa duplicating the acl/posix tests with symlinks, and remove xattr=sa from the original acl/posix tests. This provides more coverage for the default xattr=on code. Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Ryan Moeller <ryan@iXsystems.com> Closes openzfs#10043 Closes openzfs#11025

behlendorf added the Type: Defect Incorrect behavior (e.g. crash, hang) label Feb 26, 2020

snajpa mentioned this issue Jul 20, 2020

kernel NULL pointer dereference, 5.7.0-rc6 + today's zfs tip #10345

Closed

ghost mentioned this issue Oct 7, 2020

Linux: Initialize zp in zfs_setattr_dir #11025

Merged

12 tasks

behlendorf closed this as completed in #11025 Oct 9, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fedora 31 - Kernel 5.5.5 ZFS crash on startup #10043

Fedora 31 - Kernel 5.5.5 ZFS crash on startup #10043

greg-hydrogen commented Feb 23, 2020

mtb-xt commented Feb 25, 2020

behlendorf commented Feb 26, 2020

AllKind commented Feb 28, 2020

svde commented Feb 28, 2020

greg-hydrogen commented Feb 28, 2020

rgessner commented Mar 1, 2020

sparksh commented Mar 7, 2020

AllKind commented Mar 11, 2020

greg-hydrogen commented Apr 12, 2020 •

edited

Loading

snajpa commented Jul 20, 2020

yocalebo commented Oct 8, 2020

Fedora 31 - Kernel 5.5.5 ZFS crash on startup #10043

Fedora 31 - Kernel 5.5.5 ZFS crash on startup #10043

Comments

greg-hydrogen commented Feb 23, 2020

Describe the problem you're observing

Describe how to reproduce the problem

Include any warning/errors/backtraces from the system logs

mtb-xt commented Feb 25, 2020

behlendorf commented Feb 26, 2020

AllKind commented Feb 28, 2020

svde commented Feb 28, 2020

greg-hydrogen commented Feb 28, 2020

rgessner commented Mar 1, 2020

sparksh commented Mar 7, 2020

AllKind commented Mar 11, 2020

greg-hydrogen commented Apr 12, 2020 • edited Loading

snajpa commented Jul 20, 2020

yocalebo commented Oct 8, 2020

greg-hydrogen commented Apr 12, 2020 •

edited

Loading