Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

livelock in dnode_hold_impl() #8426

Closed
bzzz77 opened this issue Feb 17, 2019 · 6 comments · Fixed by #8433
Closed

livelock in dnode_hold_impl() #8426

bzzz77 opened this issue Feb 17, 2019 · 6 comments · Fixed by #8433
Labels
Type: Defect Incorrect behavior (e.g. crash, hang)
Projects
0.7.14

Comments

@bzzz77
Copy link
Contributor

bzzz77 commented Feb 17, 2019

System information

Type Version/Name
Distribution Name
Distribution Version
Linux Kernel
Architecture
ZFS Version 0.7.9
SPL Version 0.7.9

Describe the problem you're observing

dnode_hold_impl() seem to be spinning forever while many threads are trying to initialize same dnode

crash> p *(dnode_handle_t )(0xffff885e2abb20c8+1526)
zr_refcount = 13,
dnh_dnode = 0x2

while dnode_hold_alloc_lock_retry = 124742875

at least one thread between dnode_slots_hold() and dnode_slots_rele() is enough to block dnode_slots_tryenter() ?

Describe how to reproduce the problem

Include any warning/errors/backtraces from the system logs

[  946.702862] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [mdt00_023:36936]
[  946.703859] NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [mdt00_021:32775]
[  946.703923] CPU: 0 PID: 32775 Comm: mdt00_021 Tainted: P           OE  ------------   3.10.0-693.21.1.el7.x86_64 #1
[  946.703924] Hardware name: HP ProLiant DL360 Gen9/ProLiant DL360 Gen9, BIOS P89 04/25/2017
[  946.703925] task: ffff885eef3c0fd0 ti: ffff885ec13a0000 task.ti: ffff885ec13a0000
[  946.703983] RIP: 0010:[<ffffffffc0e314b9>]  [<ffffffffc0e314b9>] zrl_is_locked+0x9/0x20 [zfs]
[  946.703984] RSP: 0018:ffff885ec13a3818  EFLAGS: 00000297
[  946.703985] RAX: 000000000000000a RBX: ffff885ec13a37a0 RCX: 0000000000000011
[  946.703985] RDX: 0000000000000009 RSI: 0000000000000001 RDI: ffff88bc4fbd8160
[  946.703986] RBP: ffff885ec13a3848 R08: ffff885ec13a0000 R09: ffff885eff417a90
[  946.703987] R10: 7fffffffffffffff R11: 0000000000000000 R12: ffff885eabd69d40
[  946.703987] R13: ffff88befcbff1c0 R14: ffff885eabd69d98 R15: ffff885eabd69de8
[  946.703989] FS:  0000000000000000(0000) GS:ffff885eff400000(0000) knlGS:0000000000000000
[  946.703989] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  946.703990] CR2: 0000562c1ac93098 CR3: 000000bec728c000 CR4: 00000000003607f0
[  946.703991] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  946.703992] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[  946.703992] Call Trace:
[  946.704013]  [<ffffffffc0d852e8>] ? dnode_slots_rele+0x48/0x70 [zfs]
[  946.704032]  [<ffffffffc0d85d75>] dnode_hold_impl+0x195/0xc30 [zfs]
[  946.704046]  [<ffffffffc0d69dbe>] ? dmu_buf_rele+0xe/0x10 [zfs]
[  946.704071]  [<ffffffffc0deb1ff>] ? zap_unlockdir+0x3f/0x60 [zfs]
[  946.704097]  [<ffffffffc0decac7>] ? zap_lookup_norm_by_dnode+0x97/0xc0 [zfs]
[  946.704115]  [<ffffffffc0d8682b>] dnode_hold+0x1b/0x20 [zfs]
[  946.704130]  [<ffffffffc0d6f642>] dmu_bonus_hold+0x32/0x1d0 [zfs]
@bunder2015
Copy link
Contributor

does this still happen on 0.8.0-rc3? there were a few recent commits in this area, including #8172 and #8249

@bzzz77
Copy link
Contributor Author

bzzz77 commented Feb 17, 2019

@bunder2015 unfortunately I don't have a reproducer on my side (the issue was hit few times on a different site), so I can't verify that with 0.8*, but looking at the patches I don't think they are relevant.

@bunder2015 bunder2015 added the Type: Defect Incorrect behavior (e.g. crash, hang) label Feb 17, 2019
@benmenadue
Copy link

@bunder2015 The relevant code on master is the same as 0.7.9, what we're running:
https://github.com/zfsonlinux/zfs/blob/f545b6ae00ef77df710ddace328d13e8c5c265bf/module/zfs/dnode.c#L1409-L1446
There haven't been any changes here since Sept 2017 by the look of it.

No thread can progress past the dnode_slots_tryenter on line 1429 if any other thread is between the dnode_slots_hold on line 1410 and the dnode_slots_rele on line 1428. Hence, none ever make it to the dnode_create on line 1443, which would break the other threads out of the loop via the break on line 1415. Nowhere in this loop is there a chance for a threads to wait - the only mutex calls occur after the ZRL is locked, i.e. a successful call to dnode_slots_tryenter.

It's alright with only a couple of threads here, since eventually they'll be in the right place to allow progress. But as you add more threads the chance of this gets much smaller; in the crash dump I looked there were ~20 threads spinning in this loop until it triggered a soft lockup and panic. dnode_hold_alloc_lock_retry was ~125 million when it kdumped.

lidongyang pushed a commit to lidongyang/zfs that referenced this issue Feb 19, 2019
Fix dnode_hold_impl() soft lockup
ec7f4bc 
Soft lockups could happen when multiple threads trying
to get zrl on the same dnode handle in order to allocate
and initialize the dnode marked as DN_SLOT_ALLOCATED.

Don't loop from beginning when we can't get zrl, otherwise
we would increase the zrl refcount and nobody can actually
lock it.

Signed-off-by: Li Dongyang <dongyangli@ddn.com>
Closes openzfs#8426
@lidongyang lidongyang mentioned this issue Feb 19, 2019
Merged
12 tasks
lidongyang pushed a commit to lidongyang/zfs that referenced this issue Feb 20, 2019
Fix dnode_hold_impl() soft lockup
38104e0 
Soft lockups could happen when multiple threads trying
to get zrl on the same dnode handle in order to allocate
and initialize the dnode marked as DN_SLOT_ALLOCATED.

Don't loop from beginning when we can't get zrl, otherwise
we would increase the zrl refcount and nobody can actually
lock it.

Signed-off-by: Li Dongyang <dongyangli@ddn.com>
Closes openzfs#8426
lidongyang pushed a commit to lidongyang/zfs that referenced this issue Feb 20, 2019
Fix dnode_hold_impl() soft lockup
02a0da1 
Soft lockups could happen when multiple threads trying
to get zrl on the same dnode handle in order to allocate
and initialize the dnode marked as DN_SLOT_ALLOCATED.

Don't loop from beginning when we can't get zrl, otherwise
we would increase the zrl refcount and nobody can actually
lock it.

Signed-off-by: Li Dongyang <dongyangli@ddn.com>
Closes openzfs#8426
@cperl82
Copy link

cperl82 commented Apr 10, 2019

I'm running into this issue on some of my systems.

I see the fix has been merged, but is not yet included in any tagged releases (as far as I can tell). Is it only going to be in the next 0.8 tagged release candidate (or release, if that is what is coming next), or will it be in the next 0.7 release as well? If its only going to be included in 0.8, until 0.8 is officially released, is it safe to cherry pick the patch and apply it on top of 0.7.13? It seems to apply cleanly, just with offsets.

@behlendorf
Copy link
Contributor

@cperl82 thanks for the reminder. This change is safe to apply to 0.7.13 and we will be applying it to the next point release. I've added it to the 0.7.14 to track it. And it'll be in 0.8.0-rc4 when tagged.

@gdevenyi
Copy link
Contributor

This issue has not yet been added to the 0.7.14 project tracking.

@behlendorf behlendorf added this to To do in 0.7.14 May 25, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Type: Defect Incorrect behavior (e.g. crash, hang)
Projects
No open projects
0.7.14
  
To do
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix dnode_hold_impl() soft lockup lidongyang/zfs
6 participants
@cperl82 @behlendorf @gdevenyi @bunder2015 @bzzz77 @benmenadue