Incorrect ashift value may cause faulted pools

[gunnarbeutner]

While investigating another issue I've come across this:

root@zt:~# zpool create -f -o ashift=16 tank sdb
root@zt:~# dd if=/dev/zero of=/tank/test bs=1M &
[1] 3405
root@zt:~# sleep 5; echo b > /proc/sysrq-trigger

after the reboot:

root@zt:~# zpool status
  pool: tank
 state: FAULTED
status: One or more devices could not be used because the label is missing
        or invalid.  There are insufficient replicas for the pool to continue
        functioning.
action: Destroy and re-create the pool from
        a backup source.
   see: http://www.sun.com/msg/ZFS-8000-5E
 scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        tank        FAULTED      0     0     0  corrupted data
          sdb       UNAVAIL      0     0     0  corrupted data
root@zt:~#

I'm able to reproduce this same behavior with ashift values as low as 14. The disk in this test is an iSCSI LUN (with 64kB sectors) that's passed-through to the VM by VMware as a disk with 512-byte sectors.

There are two things I'm currently wondering about:

a) Can this possibly happen with lower ashift values (e.g. ashift=12 with 512-bytes-per-sector disks) and is this because ZFS assumes that single-sector-writes are atomic (I'm really just guessing here).

b) Depending on how (a) turns out, is this suggestion (from zpool(8)) such a great idea:

       For optimal performance, the pool sector size should be greater than or equal to the
       sector size of the underlying disks. Since the property cannot be changed after pool
       creation,  if in a given pool, you ever want to use drives that report 4KiB sectors,
       you must set ashift=12 at pool creation time.

[gunnarbeutner]

Actually, even without dd/reboot the pool faults when it's exported/re-imported or when I try to scrub the pool. Hmm.

[behlendorf]

That's not good. Why larger ashift values are damaging isn't immediately clear to me but it's clearly causing damage to the label. The larger ashift will impact the total number of uberblocks which can be stored in the fixed size labels and perhaps we have an overrun. It would probably be wise to restrict the maximum ashift size to 12 which has been well tested.