New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Autoreplace not working #2449

Closed
cohdavid opened this Issue Jul 1, 2014 · 9 comments

Comments

Projects
None yet
5 participants
@cohdavid
Copy link

cohdavid commented Jul 1, 2014

I'm currently testing ZFS on Linux v0.6.3 on Debian Wheezy, using a LSI2008 based controller with the disks in JBOD mode. And I have configured the system as shown:

zpool status tank

pool: tank
state: ONLINE
scan: none requested
config:

    NAME                                STATE     READ WRITE CKSUM
    tank                                ONLINE       0     0     0
      raidz1-0                          ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:2:0   ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:3:0   ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:4:0   ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:5:0   ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:6:0   ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:7:0   ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:8:0   ONLINE       0     0     0
      raidz1-1                          ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:9:0   ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:10:0  ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:11:0  ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:12:0  ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:13:0  ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:14:0  ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:15:0  ONLINE       0     0     0
      raidz1-2                          ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:16:0  ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:17:0  ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:18:0  ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:19:0  ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:20:0  ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:21:0  ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:22:0  ONLINE       0     0     0
    logs
      mirror-3                          ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:24:0  ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:25:0  ONLINE       0     0     0
    spares
      pci-0000:03:00.0-scsi-0:0:23:0    AVAIL   

errors: No known data errors

And then I pull out a disk to see what happens in an error condition.

zpool status tank

pool: tank
state: DEGRADED
status: One or more devices could not be used because the label is missing
or
invalid. Sufficient replicas exist for the pool to continue
functioning in a degraded state.
action: Replace the device using 'zpool replace'.
see: http://zfsonlinux.org/msg/ZFS-8000-4J
scan: none requested
config:

    NAME                                STATE     READ WRITE CKSUM
    tank                                DEGRADED     0     0     0
      raidz1-0                          DEGRADED     0     0     0
        pci-0000:03:00.0-scsi-0:0:2:0   ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:3:0   ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:4:0   ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:5:0   ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:6:0   UNAVAIL      0     0     0
        pci-0000:03:00.0-scsi-0:0:7:0   ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:8:0   ONLINE       0     0     0
      raidz1-1                          ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:9:0   ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:10:0  ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:11:0  ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:12:0  ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:13:0  ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:14:0  ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:15:0  ONLINE       0     0     0
      raidz1-2                          ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:16:0  ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:17:0  ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:18:0  ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:19:0  ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:20:0  ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:21:0  ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:22:0  ONLINE       0     0     0
    logs
      mirror-3                          ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:24:0  ONLINE       0     0     0
        pci-0000:03:00.0-scsi-0:0:25:0  ONLINE       0     0     0
    spares
      pci-0000:03:00.0-scsi-0:0:23:0    AVAIL   

errors: No known data errors

zpool get autoreplace tank

NAME PROPERTY VALUE SOURCE
tank autoreplace on local

And now I wonder why the system has not started resilvering using the hot spare.......

@dweeezil

This comment has been minimized.

Copy link
Member

dweeezil commented Jul 1, 2014

See https://github.com/zfsonlinux/zfs/blob/master/cmd/zed/zed.d/io-spare.sh#L19-22 and the rest of the script for details (and also the checksum-spare.sh script). At the moment, the replace operation will only be performed in response to I/O or checksum errors and then, only if you've set the appropriate flags in the zed.rc file.

The zed and its scripts are new for 0.6.3 and are a work-in-progress at the moment.

@ryao

This comment has been minimized.

Copy link
Member

ryao commented Jul 2, 2014

@cohdavid You need to have the zed running as @dweeezil insinuated. Not all distributions include the rc scripts (better known as "init" scripts") to start zed at this time.

@cohdavid

This comment has been minimized.

Copy link

cohdavid commented Jul 3, 2014

I understand what you are writing, so please consider this as a feature request instead, to get the situation where a disk becoming unavailable will be considered an error condition, triggering the use of a spare disk.
PS. I have zed running, and manually triggering the resilvering using a 'zpool replace' command works.

@behlendorf behlendorf added zed labels Jul 14, 2014

@behlendorf behlendorf added this to the 0.6.5 milestone Jul 14, 2014

@behlendorf behlendorf modified the milestone: 0.6.5 Nov 8, 2014

@mailinglists35

This comment has been minimized.

Copy link

mailinglists35 commented May 20, 2015

oh, I can't wait to see this in 0.6.5!
If I modify the io-spare.sh in 0.6.4, then in order to perform a correct upgrade from 0.6.4 to 0.6.5, will I have to restore it to original state before the upgrade, right?

update: I have replaced my io-spare.sh with the version above referenced by @dweeezil and restarted the zed daemon, but it still does not initiate a resilver :(
using 0.6.4 on debian jessie kernel 3.16

@mailinglists35

This comment has been minimized.

Copy link

mailinglists35 commented Jun 7, 2015

this appears to work on a testcase using latest released version - 0.6.4-1.1-2 on debian jessie - see comment #3478 (comment)

however it fails with io-spare.sh exit code 4 under cciss storage driver.

how can I make zed/io-spare detect errors when my disks are not named /dev/sdX but /dev/cciss/cXdY ?

@behlendorf

This comment has been minimized.

Copy link
Member

behlendorf commented Jun 9, 2015

@mailinglists35 to make the io-spare.sh script more robust it should use the ZEVENT_VDEV_SPARE_GUIDS rather than the ZEVENT_VDEV_SPARE_PATHS to do the replacement. This way the path name itself becomes irrelevant and there is no ambiguity because the guids are for all intents and purposes unique.

The io-spare.sh script has all of this information today but can't use it because the zpool replace command needs to be extended to accept a guid for the spare device. For example, the following syntax should be supported.

zpool replace <failed vdev guid> <spare vdev guid>
@mailinglists35

This comment has been minimized.

Copy link

mailinglists35 commented Jun 16, 2015

the zpool replace command needs to be extended to accept a guid for the spare device

@behlendorf thanks for the light; is this the right issue to follow to see when it is woking? #2012

@behlendorf

This comment has been minimized.

Copy link
Member

behlendorf commented Jun 16, 2015

@mailinglists35 that's related but unnecessary to fix this. The script already has the guid, we just need to extend the utilities and I don't think there an issue open for that.

@mailinglists35

This comment has been minimized.

Copy link

mailinglists35 commented Jun 16, 2015

@behlendorf I'm confused, then where/what should I watch for to know when the utilities can do that?

The problem that actually I need to solve is to have zfs automatically use a hot spare when i/o or checksum happen on an old hp P400 controller having each physical drive behind a corresponding logicaldrive under /dev/cciss/c0dX (the new "hpsa" driver replacing "cciss" which has normal scsi naming /dev/sdX does not support the p400 controller).
I did some workaround in issue #3478 by creating with mknod equivalent entries in /dev/sdXY for each cciss/c0dX and cciss/c0dXpY (because otherwise zpool would not partition the disks when creating the pool) but I'm not sure autoreplace will happen when a drive will fail and the system will not have remote hands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment