async I/O on a zfs file system causes system hang #7335
Projects
Comments
|
I suspect this happens because the page is being locked in mappedread and again in filemap_page_mkwrite. Since the page is already locked, the locking of the page in filemap_page_mkwrite causes the task to block indefinitely. |
|
Touching the mmap'd buffer (e.g. *buf = 0;) before the read on the final do_sync_io() call causes the page to be faulted and then we don't get the hang |
|
issue occurs with latest (today's) zfs + spl on 4.15 |
|
This helps, but is clearly sub-optimal. |
|
Clearly it's a double-lock on the page causing the problem, if I unlock it before the uiomove we don't get the lockup hang: Any ideas of a suitable fix? |
behlendorf
added a commit
to behlendorf/zfs
that referenced
this issue
Mar 24, 2018
Calling uiomove() in mappedread() can result in deadlock if the user space page needs to be faulted in. This issue is that uiomove() must be called with the page lock held in order to safely populate the page date. If the page needs to be faulted in by filemap_page_mkwrite() then it will also take the page lock resulting in a double-lock. Normally this isn't an issue since the pages are very likely to be already faulted in. This patch makes sure that is always the case by prefaulting in the user space pages. Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Issue openzfs#7335
|
@ColinIanKing thanks for getting to the heart of the matter for putting together such a nice test case! I've opened #7339 with a proposed fix for your review and verification. What I'm suggesting is that we simply always ensure the page is prefaulted. |
|
Thanks for the speedy turnaround on this fix. I'll test it out on Monday with our own Canonical file system tests in the mix to give it a thorough test. |
behlendorf
added a commit
to behlendorf/zfs
that referenced
this issue
Mar 25, 2018
Calling uiomove() in mappedread() can result in deadlock if the user space page needs to be faulted in. This issue is that uiomove() must be called with the page lock held in order to safely populate the page date. If the page needs to be faulted in by filemap_page_mkwrite() then it will also take the page lock resulting in a double-lock. Normally this isn't an issue since the pages are very likely to be already faulted in. This patch makes sure that is always the case by prefaulting in the user space pages. Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Issue openzfs#7335
behlendorf
added a commit
to behlendorf/zfs
that referenced
this issue
Mar 26, 2018
Calling uiomove() in mappedread() can result in deadlock if the user space page needs to be faulted in. This issue is that uiomove() must be called with the page lock held in order to safely populate the page date. If the page needs to be faulted in by filemap_page_mkwrite() then it will also take the page lock resulting in a double-lock. Normally this isn't an issue since the pages are very likely to be already faulted in. This patch makes sure that is always the case by prefaulting in the user space pages. Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Issue openzfs#7335
behlendorf
added a commit
to behlendorf/zfs
that referenced
this issue
Mar 26, 2018
Calling uiomove() in mappedread() can result in deadlock if the user space page needs to be faulted in. This issue is that uiomove() must be called with the page lock held in order to safely populate the page date. If the page needs to be faulted in by filemap_page_mkwrite() then it will also take the page lock resulting in a double-lock. Normally this isn't an issue since the pages are very likely to be already faulted in. This patch makes sure that is always the case by prefaulting in the user space pages. Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Issue openzfs#7335
behlendorf
added a commit
to behlendorf/zfs
that referenced
this issue
Mar 26, 2018
Calling uiomove() in mappedread() can result in deadlock if the user space page needs to be faulted in. This issue is that uiomove() must be called with the page lock held in order to safely populate the page date. If the page needs to be faulted in by filemap_page_mkwrite() then it will also take the page lock resulting in a double-lock. Normally this isn't an issue since the pages are very likely to be already faulted in. This patch makes sure that is always the case by prefaulting in the user space pages. Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Issue openzfs#7335
behlendorf
added a commit
to behlendorf/zfs
that referenced
this issue
Mar 27, 2018
Calling uiomove() in mappedread() can result in deadlock if the user space page needs to be faulted in. Resolve the issue by only taking a reference on the page when copying it and not the page lock. The inode range lock protects against concurrent updates via zfs_read() and zfs_write(). Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Issue openzfs#7335
behlendorf
added a commit
to behlendorf/zfs
that referenced
this issue
Mar 27, 2018
Calling uiomove() in mappedread() can result in deadlock if the user space page needs to be faulted in. Resolve the issue by only taking a reference on the page when copying it and not the page lock. The inode range lock protects against concurrent updates via zfs_read() and zfs_write(). Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Issue openzfs#7335
behlendorf
added a commit
to behlendorf/zfs
that referenced
this issue
Mar 27, 2018
Calling uiomove() in mappedread() can result in deadlock if the user space page needs to be faulted in. Resolve the issue by only taking a reference on the page when copying it and not the page lock. The inode range lock protects against concurrent updates via zfs_read() and zfs_write(). Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Issue openzfs#7335
behlendorf
added a commit
to behlendorf/zfs
that referenced
this issue
Mar 27, 2018
Calling uiomove() in mappedread() can result in deadlock if the user space page needs to be faulted in. Resolve the issue by only taking a reference on the page when copying it and not the page lock. The inode range lock protects against concurrent updates via zfs_read() and zfs_write(). Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Issue openzfs#7335
tonyhutter
pushed a commit
to tonyhutter/zfs
that referenced
this issue
Apr 16, 2018
Calling uiomove() in mappedread() under the page lock can result in a deadlock if the user space page needs to be faulted in. Resolve the issue by dropping the page lock before the uiomove(). The inode range lock protects against concurrent updates via zfs_read() and zfs_write(). Reviewed-by: Albert Lee <trisk@forkgnu.org> Reviewed-by: Chunwei Chen <david.chen@nutanix.com> Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Closes openzfs#7335 Closes openzfs#7339
tonyhutter
pushed a commit
to tonyhutter/zfs
that referenced
this issue
May 4, 2018
Calling uiomove() in mappedread() under the page lock can result in a deadlock if the user space page needs to be faulted in. Resolve the issue by dropping the page lock before the uiomove(). The inode range lock protects against concurrent updates via zfs_read() and zfs_write(). Reviewed-by: Albert Lee <trisk@forkgnu.org> Reviewed-by: Chunwei Chen <david.chen@nutanix.com> Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Closes openzfs#7335 Closes openzfs#7339
tonyhutter
pushed a commit
that referenced
this issue
May 10, 2018
Calling uiomove() in mappedread() under the page lock can result in a deadlock if the user space page needs to be faulted in. Resolve the issue by dropping the page lock before the uiomove(). The inode range lock protects against concurrent updates via zfs_read() and zfs_write(). Reviewed-by: Albert Lee <trisk@forkgnu.org> Reviewed-by: Chunwei Chen <david.chen@nutanix.com> Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Closes #7335 Closes #7339
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
System information
Running the reproducer program shown below will cause a soft lockup. This occurs after the async I/O read occurs. strace on the program shows it hangs on the read io_submit. If the read/write buffer being passed is NOT mmap'd then the hang does not happen.
io_setup(1024, [0x7f0a46d58000]) = 0
openat(AT_FDCWD, "testfile", O_RDWR|O_CREAT, 04130) = 3
ftruncate(3, 512) = 0
brk(NULL) = 0x55645fa55000
brk(0x55645fa76000) = 0x55645fa76000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_SHARED, 3, 0) = 0x7f0a46d7a000
io_submit(0x7f0a46d58000, 1, [{aio_lio_opcode=IOCB_CMD_PWRITE, aio_fildes=3, aio_buf="\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., aio_nbytes=512, aio_offset=0}]) = 1
io_getevents(0x7f0a46d58000, 0, 1, [{data=0, obj=0x7ffda2080710, res=512, res2=0}], {tv_sec=30, tv_nsec=0}) = 1
io_submit(0x7f0a46d58000, 1, [{aio_lio_opcode=IOCB_CMD_PREAD, aio_fildes=3, aio_buf=0x7f0a46d7a000, aio_nbytes=512, aio_offset=0}]
Describe how to reproduce the problem
Compile the C source listed below and run:
gcc test.c -laio
run on a ZFS file system:
./a.out
..and it will hang and get soft lockup timeouts after 120 seconds on Ubuntu.
The text was updated successfully, but these errors were encountered: