zpool remove on mirrored logs fails silently

[arthurzenika]

On the setup described in : #1420 we have a mirrored log ZIL ::

[...]
logs
mirror-3 ONLINE 0 0 0
wwn-0x5001517bb2a7cca9 ONLINE 0 0 0
wwn-0x5001517bb2a7acae ONLINE 0 0 0
[...]

doing "zpool remove data mirror-3" doesn't complain, returns a 0 error code, but doesn't seem to work.

Doing zpool status still shows the same thing.

We tried to put the disks offline (zpool offline data wwn-0x5001517bb2a7acae) before the remove but this does not help.

Is it possible that zpool status is showing wrong information (we don't have any disk activity on the ZIL, dstat -d -D sdw,sdx shows zeros) ? Does anyone have an idea on how to debug this ?

[behlendorf]

To remove a ZIL device you need to remove each individual disk vdev not the mirror vdev. In your case try the followings:

zpool remove data wwn-0x5001517bb2a7cca9
zpool remove data wwn-0x5001517bb2a7acae

However, there may still be a bug here since the tools should have given you a more useful error message rather than silently returning.

As for the ZIL device not being used, keep in mind that the ZIL is only used for synchronous I/O. If your application never performs synchronous I/O you won't see any benefit from adding the ZIL. See the following link for a nice description of how the ZIL works:

http://pthree.org/2013/04/19/zfs-administration-appendix-a-visualizing-the-zfs-intent-log

[arthurzenika]

Thanks for the reply.

We'd unsuccessfully tried to remove those disks directly :

[root@centaurus ~]$ zpool remove data wwn-0x5001517bb2a7cca9
cannot remove wwn-0x5001517bb2a7cca9: operation not supported on this type of pool
[root@centaurus ~]$ zpool remove data wwn-0x5001517bb2a7acae
cannot remove wwn-0x5001517bb2a7acae: operation not supported on this type of pool

Is this a bug ?

For the ZIL, I've tried testing with dbench -s 5 (-s is for syncronous IO) and the iozone command in the article you mention but this did not spur any activity on the ZIL disks.

[douardda]

@behlendorf the man page explicitly states that removing a mirrored slog must be done at top-level mirror. As stated by @arthurlogilab we tried to remove devices independently, but zpool refuses to do so with an error message (which is consistent with man pages). When we try to remove the mirror, it does not complain, but does nothing either.

We also tried to offline both devices involved in the mirrored slog, does not work either.

Concerning your remind about ZIL "As for the ZIL device not being used, keep in mind that the ZIL is only used for synchronous I/O.", we know this, but even when I run things like:

dd if=/dev/zero of=/data/test bs=1k count=1000 oflag=sync

I see no activity on the slog devices...

David

[mumblepins]

I am also seeing a similar problem, though in my case I have an unmirrored ZIL device. zpool remove pool zil-device is failing silently (log device still present with zpool status), and trying to offline the device prior to removing it does not work either. Any ideas on what to try? Thanks

Daniel

[GregorKopka]

Had the same situation on a pool with two mirrored zil vdevs, needed to remove one zil mirror.
Offlining/detaching one side of the worked as expected.
Offlining the (now unmirrored) zil works, removing it silently fails.
i got rid of the zil by exporting the pool and disconnecting the device before reimporting, then removing the unavail.

What is also happening: adding a mirror to a zil triggers a complete pool resilver, which i find a bit excessive since the zil content is basically dropped on every txg.

[GregorKopka]

Update: Tried to remove a defect zil with a ZoL 6.1 pool - didn't work, had to replace it (which triggered complete pool scrub/resilver)

[byteharmony]

zpool remove tank log mirror-2
does deliver an error, but zpool status shows the log device was dropped. I've had this on 2x raid 10 systems.

RC14
FYI
BK

[treydock]

I just tried this with 0.6.1 and succeeded in removing a mirrored ZIL.

Before:

# zpool status
  pool: tank
 state: ONLINE
  scan: none requested
config:

        NAME            STATE     READ WRITE CKSUM
        tank            ONLINE       0     0     0
          raidz2-0      ONLINE       0     0     0
            d01         ONLINE       0     0     0
            d02         ONLINE       0     0     0
            d03         ONLINE       0     0     0
            d04         ONLINE       0     0     0
            d05         ONLINE       0     0     0
            d06         ONLINE       0     0     0
            d07         ONLINE       0     0     0
            d08         ONLINE       0     0     0
            d09         ONLINE       0     0     0
            d10         ONLINE       0     0     0
            d11         ONLINE       0     0     0
          raidz2-1      ONLINE       0     0     0
            d13         ONLINE       0     0     0
            d14         ONLINE       0     0     0
            d15         ONLINE       0     0     0
            d16         ONLINE       0     0     0
            d17         ONLINE       0     0     0
            d18         ONLINE       0     0     0
            d19         ONLINE       0     0     0
            d20         ONLINE       0     0     0
            d21         ONLINE       0     0     0
            d22         ONLINE       0     0     0
            d23         ONLINE       0     0     0
        logs
          mirror-2      ONLINE       0     0     0
            ssd0-part2  ONLINE       0     0     0
            ssd1-part2  ONLINE       0     0     0
        cache
          ssd0-part4    ONLINE       0     0     0
          ssd1-part4    ONLINE       0     0     0
        spares
          d12           AVAIL
          d24           AVAIL

errors: No known data errors

Removing ZIL:

# zpool remove tank mirror-2
# echo $?
0
# zpool status
  pool: tank
 state: ONLINE
  scan: none requested
config:

        NAME          STATE     READ WRITE CKSUM
        tank          ONLINE       0     0     0
          raidz2-0    ONLINE       0     0     0
            d01       ONLINE       0     0     0
            d02       ONLINE       0     0     0
            d03       ONLINE       0     0     0
            d04       ONLINE       0     0     0
            d05       ONLINE       0     0     0
            d06       ONLINE       0     0     0
            d07       ONLINE       0     0     0
            d08       ONLINE       0     0     0
            d09       ONLINE       0     0     0
            d10       ONLINE       0     0     0
            d11       ONLINE       0     0     0
          raidz2-1    ONLINE       0     0     0
            d13       ONLINE       0     0     0
            d14       ONLINE       0     0     0
            d15       ONLINE       0     0     0
            d16       ONLINE       0     0     0
            d17       ONLINE       0     0     0
            d18       ONLINE       0     0     0
            d19       ONLINE       0     0     0
            d20       ONLINE       0     0     0
            d21       ONLINE       0     0     0
            d22       ONLINE       0     0     0
            d23       ONLINE       0     0     0
        cache
          ssd0-part4  ONLINE       0     0     0
          ssd1-part4  ONLINE       0     0     0
        spares
          d12         AVAIL
          d24         AVAIL

errors: No known data errors

Re-adding the 2 ZIL devices worked just fine too.

[douardda]

@treydock yes I also have been able to remove mirrored logs from some pools, my problem here is with my production pool, for which I cannot remove the log. I still cannot remove it with zfs 0.6.2.

Thus my pool must be in some weird state, and I suspect this is related to the fact I never see any activity on my log devices, and to the fact I have very poor performances with this pool. Is there a way to investigate this weird state?

The only solution I can imagine now is to create some other pool nearby and copy the whole data in this temporary pool, drop the problematic pool and recreate it from scratch. But how can I be sure not to see this problem appear again then? If I must recreate my pool from scratch every few months, it won't be an acceptable situation.

David

[GregorKopka]

As far as i remember it was possible to offline the LOG, remove the drive, reimport the pool and then remove the unavail LOG vdev.

Gregor

[douardda]

On 08/09/2013 08:41, GregorKopka wrote:

As far as i remember it was possible to offline the LOG, remove the
drive, reimport the pool and then remove the unavail LOG vdev.

I've tried this, in vain.

I had 2 mirrored SSD as log device. The only thing I've able to achieve
is to detach one of the SSD from the mirror and add it as another log
device. So I now have one log device that is actually used, and the
performances of the pool are now more acceptable (I still find my pool
too slow, and I still haven't found why, but this is another story).

This last week-end I've tried hard to remove this "broken" log device
from my pool, but I failed.

When I tried to do what you describe here (offline, remove the drive,
reimport), the pool stated the log device as unavailable, but I still
wasn't able to remove it. I've been obliged to do a "zpool replace -f"
to bring my pool back to the ONLINE state.

One thing to note is that if the device is in fact not used at all (no
I/O activity on the "broken" log device as reported by zpool iostat or
dstat), zpool iostat always reports 128K of allocated space (whereas the
other log device, the one that is relly used now, has no allocated space
when there is no activity on the pool).

David

Gregor

David DOUARD LOGILAB
Directeur du département Outils & Systèmes

+33 1 45 32 03 12 david.douard@logilab.fr
+33 1 83 64 25 26 http://www.logilab.fr/id/david.douard

Formations - http://www.logilab.fr/formations
Développements - http://www.logilab.fr/services
Gestion de connaissances - http://www.cubicweb.org/

[GregorKopka]

Update for zpool remove pool slog-device

I tried with 0.6.0-rc14 (freshly recompiled, installed, initrd-updated and rebooted) before:
'operation not supported on this type of pool'

Upgraded to 0.6.2 using (cheat-sheet for gentoo):

emerge zfs
genkernel all --no-clean --no-mountboot --zfs --bootloader=grub2 --callback="emerge @module-rebuild"
grub2-install <boot drives>
reboot

And finally with 0.6.2: IT WORKED, zil slog successfully removed!
Still the problem with attaching slog mirrors or replacing slogs is that a full pool resilver is started

$ zpool status
  pool: data
 state: ONLINE
  scan: resilvered 0 in 2h4m with 0 errors on Wed Sep 11 01:06:25 2013
config:

        NAME            STATE     READ WRITE CKSUM
        data            ONLINE       0     0     0
          mirror-0      ONLINE       0     0     0
            data-0-1    ONLINE       0     0     0
            data-0-2    ONLINE       0     0     0
          mirror-1      ONLINE       0     0     0
            data-1-1    ONLINE       0     0     0
            data-1-2    ONLINE       0     0     0
        logs
          data-zil-1    ONLINE       0     0     0
          data-zil-2    ONLINE       0     0     0
        cache
          data-cache-1  ONLINE       0     0     0

$ zpool remove data data-zil-2
$ zpool status
  pool: data
[...]
config:
[...]
        NAME            STATE     READ WRITE CKSUM
        logs
          data-zil-1    ONLINE       0     0     0
[...]

$ zpool attach data data-zil-1 /dev/mapper/data-zil-2
$ zpool status
  pool: data
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Wed Sep 11 02:43:44 2013
    8.50G scanned out of 1.18T at 161M/s, 2h7m to go
    0 resilvered, 0.70% done
config:

        NAME            STATE     READ WRITE CKSUM
[...]
        logs
          mirror-2      ONLINE       0     0     0
            data-zil-1  ONLINE       0     0     0
            data-zil-2  ONLINE       0     0     0
[...]

So i can only suggest that everyone who has problems with removing slogs updates to 0.6.2, reboots and tries again.

[douardda]

I've got news too. The machine has been rebooted yesterday thanks (!) to a hard freeze (see below), and since then, both the log devices now are ONLINE and used for true (there are IOs on both of them). Note that I did reboot the machine several times before, when I was trying to remove this "dead" log device from my pool...

BUT I still cannot remove it from the pool... I'll try within the next few days several magic incantations (offline, remove the disk, etc.) just ni case.

And since I tried to remove the device, it's back to a no-IOs state... It's there in the zpool, marked as ONLINE, but zpool iostat never shows any activity on it any more (as before the reboot). And refuses (silenttly) to remove the device from the pool.

About this latest hard freeze, it seems to be related to a situation where I have a ZFS partition with a huge amount of small XML files (I know I know), that is exported by NFS4. This exported directory is used by a fellow colleague at Logilab as an input for a database importation script. And it looks like this situation is a killer for the system, in which NFS and ZFS consume all the memory (I've got 16G for now, should be upgraded to 32G soon).

David

[b333z]

I recently experienced this issue on a FreeBSD 8.4 and64, was unable to remove a mirrored log from pool, tried all permutations to remove, would get no error, from memory, zdb showed log vdev with attribute: Removing: 1, after attempts to remove, attempted to return it to previous state by adding back the other mirror vdev then rebooted but left log broken, from then on no further io to log....

On similar hardware, same os and version, did not show same problem, making me think it was a pool specific issue.

Was production server and needed fast sync writes, under pressure was forced to rebuild pool, so collected little info of use to log issue, but thought I would mention that I had also sighted this on FreeBSD...

[tuxis-ie]

I tried to replace my Zil's last night and succeeded partially because of this issue.

I started out with spl 0.6.0 and the following zpool:
root@zstore01:~# zpool status -v
pool: export
state: ONLINE
scan: resilvered 687G in 5h13m with 0 errors on Fri Mar 22 08:59:51 2013
config:

     NAME                        STATE     READ WRITE CKSUM
     export                      ONLINE       0     0     0
       raidz1-0                  ONLINE       0     0     0
         scsi-350014ee207cf0919  ONLINE       0     0     0
         scsi-350014ee0ae12a103  ONLINE       0     0     0
         scsi-350014ee0ae12a202  ONLINE       0     0     0
         scsi-350014ee2074a070c  ONLINE       0     0     0
     logs
       mirror-1                  ONLINE       0     0     0
         scsi-350026b7226012d28    ONLINE       0     0     0
         scsi-350026b722c0602af    ONLINE       0     0     0

I tried removing the log-mirror with:
zpool remove export mirror-1
as it's stated in the manpage. This did cause the log-mirror to be out of use, but did not release the devices. I then found this issue on github and upgraded to 0.6.2. Unfortunately, that did not help.

zdb did report the log-mirror as 'removing: 1', and that's just about it.

I rebooted the box several times, but could not get ZFS to release the devices. I did manage to get one disk by running:
zpool detach export scsi-350026b722c0602af

I then added my new disks and re-added scsi-350026b722c0602af as cache-device. But now I'm stuck with an unused log-device which I want to use as a cache-device...

    root@zstore01:~# zpool status -v
      pool: export
     state: ONLINE
      scan: resilvered 687G in 5h13m with 0 errors on Fri Mar 22 08:59:51 2013
    config:

         NAME                        STATE     READ WRITE CKSUM
         export                      ONLINE       0     0     0
           raidz1-0                  ONLINE       0     0     0
             scsi-350014ee207cf0919  ONLINE       0     0     0
             scsi-350014ee0ae12a103  ONLINE       0     0     0
             scsi-350014ee0ae12a202  ONLINE       0     0     0
             scsi-350014ee2074a070c  ONLINE       0     0     0
         logs
           scsi-350026b7226012d28    ONLINE       0     0     0
           mirror-2                  ONLINE       0     0     0
             scsi-355cd2e404b406b83  ONLINE       0     0     0
             scsi-355cd2e404b406ab1  ONLINE       0     0     0
         cache
           scsi-350026b722c0602af    ONLINE       0     0     0

If anyone has a safe idea what to do next, I'd be glad to hear about it. IIRC, I cannot just remove the scsi-350026b7226012d28 from the server, since log-devices may not fail. Although this disk is marked as removed, I'm kind of afraid that removing it breaks my whole pool.

Any hint, tip or tric is welcome. :)