DAOS-19386 test: Add stall watchdog to NLT FI tests - #18779
Conversation
|
Ticket title is 'NLT stage timing out (~63 min) on master and PR builds — daos container create fault injection appears to hang' |
|
FWIW, I added this to some other PRs and it seems to have worked! https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-18403/7/artifact/nlt_logs/nlt/dnt_fi_cont-create_logs/dnt_stall_dump.log/*view*/ |
|
Test stage Unit Test completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-18779/2/testReport/ |
|
Test stage Functional Hardware Medium MD on SSD completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-18779/2/testReport/ |
Don't allow a stuck child process to tank the entire NLT
run. When a stall is detected, attempt to capture stack
traces and fail early.
The /proc dump shows a real stall directly. From the hang that
motivated this patch: the stalled child holds the parent-dir lock
while waiting on its LOOKUP reply, and every dfuse worker is blocked
on that same lock trying to write a dentry invalidation:
TID 34163: comm=daos wchan=request_wait_answer state=S
[<0>] request_wait_answer+0xfa/0x210 [fuse]
[<0>] fuse_simple_request+0x1b8/0x330 [fuse]
[<0>] fuse_lookup_name+0xa4/0x1c0 [fuse]
[<0>] fuse_lookup+0x66/0x190 [fuse]
[<0>] __lookup_hash+0x70/0xa0
[<0>] __filename_create+0x87/0x150
[<0>] do_mkdirat+0x4c/0x160
[<0>] __x64_sys_mkdir+0x47/0x70
TID 29212: comm=dfuse worker wchan=fuse_reverse_inval_entry state=D
[<0>] fuse_reverse_inval_entry+0x40/0x210 [fuse]
[<0>] fuse_notify+0x287/0x500 [fuse]
[<0>] fuse_dev_do_write+0x305/0x4e0 [fuse]
[<0>] fuse_dev_write+0x50/0x80 [fuse]
TID 29213, 29214: identical (the whole worker pool)
Signed-off-by: Michael MacDonald <github@macdonald.cx>
|
Test stage NLT completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18779/3/execution/node/867/log |
2 participants
Don't allow a stuck child process to tank the entire NLT
run. When a stall is detected, attempt to capture stack
traces and fail early.
The /proc dump shows a real stall directly. From the hang that
motivated this patch: the stalled child holds the parent-dir lock
while waiting on its LOOKUP reply, and every dfuse worker is blocked
on that same lock trying to write a dentry invalidation:
TID 34163: comm=daos wchan=request_wait_answer state=S
[<0>] request_wait_answer+0xfa/0x210 [fuse]
[<0>] fuse_simple_request+0x1b8/0x330 [fuse]
[<0>] fuse_lookup_name+0xa4/0x1c0 [fuse]
[<0>] fuse_lookup+0x66/0x190 [fuse]
[<0>] __lookup_hash+0x70/0xa0
[<0>] __filename_create+0x87/0x150
[<0>] do_mkdirat+0x4c/0x160
[<0>] __x64_sys_mkdir+0x47/0x70
TID 29212: comm=dfuse worker wchan=fuse_reverse_inval_entry state=D
[<0>] fuse_reverse_inval_entry+0x40/0x210 [fuse]
[<0>] fuse_notify+0x287/0x500 [fuse]
[<0>] fuse_dev_do_write+0x305/0x4e0 [fuse]
[<0>] fuse_dev_write+0x50/0x80 [fuse]
TID 29213, 29214: identical (the whole worker pool)
Signed-off-by: Michael MacDonald github@macdonald.cx