Please sign in to comment.
Detect IO errors during device removal
* Detect IO errors during device removal While device removal cannot verify the checksums of individual blocks during device removal, it can reasonably detect hard IO errors from the leaf vdevs. Failure to perform this error checking can result in device removal completing successfully, but moving no data which will permanently corrupt the pool. Situation 1: faulted/degraded vdevs In the configuration shown below, the removal of mirror-0 will permanently corrupt the pool. Device removal will preferentially copy data from 'vdev1 -> vdev3' and from 'vdev2 -> vdev4'. Which in this case will result in nothing being copied since one vdev in each of those groups in unavailable. However, device removal will complete successfully since all IO errors are ignored. tank DEGRADED 0 0 0 mirror-0 DEGRADED 0 0 0 /var/tmp/vdev1 FAULTED 0 0 0 external fault /var/tmp/vdev2 ONLINE 0 0 0 mirror-1 DEGRADED 0 0 0 /var/tmp/vdev3 ONLINE 0 0 0 /var/tmp/vdev4 FAULTED 0 0 0 external fault This issue is resolved by updating the source child selection logic to exclude unreadable leaf vdevs. Additionally, unwritable destination child vdevs which can never succeed are skipped to prevent generating a large number of write IO errors. Situation 2: individual hard IO errors During removal if an unexpected hard IO error is encountered when either reading or writing the child vdev the entire removal operation is cancelled. While it may be possible to reconstruct the data after removal that cannot be guaranteed. The only strictly safe thing to do is to cancel the removal. As a future improvement we may want to instead suspend the removal process and allow the damaged region to be retried. But that work is left for another time, hard IO errors during the removal process are expected to be exceptionally rare. Reviewed-by: Serapheim Dimitropoulos <email@example.com> Reviewed-by: Tony Hutter <firstname.lastname@example.org> Reviewed-by: Tom Caputi <email@example.com> Signed-off-by: Brian Behlendorf <firstname.lastname@example.org> Issue #6900 Closes #8161
- Loading branch information...
Showing with 350 additions and 19 deletions.
- +16 −0 man/man5/zfs-module-parameters.5
- +4 −3 man/man8/zpool.8
- +89 −11 module/zfs/vdev_removal.c
- +3 −3 tests/runfiles/linux.run
- +21 −1 tests/zfs-tests/include/libtest.shlib
- +2 −1 tests/zfs-tests/tests/functional/removal/Makefile.am
- +2 −0 tests/zfs-tests/tests/functional/removal/removal.kshlib
- +109 −0 tests/zfs-tests/tests/functional/removal/removal_with_errors.ksh
- +104 −0 tests/zfs-tests/tests/functional/removal/removal_with_faulted.ksh
Oops, something went wrong.