On-disk block allocation and 512e/4kn drives

[ossimoi]

I've been doing some testing with a mixture of 512/4096 and 4096/4096 disks and there's one thing which I can't seem to find a good read about. Are there any gains from formatting drives to 4096 logical block size? I noticed that 512/4096 drives can't be replaced with 4096/4096 drives, but other way around works just fine. Ashift is always 12.

I guess this would require in-depth knowledge about metaslabs and space maps. I'm interested how ZFS maps these into actual disk sectors. When 10TB drive is in 512/4096 mode (logical/physical), it reports 10000831348736 bytes space in 19532873728 sectors to the OS. For 4096/4096 drive 10000831348736 bytes, 2441609216 sectors. One would imagine, that less sectors means smaller block map, which would potentially save some memory, but is this the case? Does ashift tackle this issue by calculating the offsets so that each 4k IO hits a physical block even in 512e mode? Does the disk firmware still process IO in logical addresses? What does sg_format --size=4096 actually do for the disk?

edit: There's a thing called DVA, apparently. But even DVAs calculate the physical location using offset of 512b blocks. If this is the case for 4kn drives as well, does ZFS actually care about the sectors at all?

[rincebrain]

Replacing 512e drives with 4kn drives should be fine assuming they're the same capacity, how have you seen that fail?

ashift on ZFS is a per-top-level-vdev property for the minimum allocation size it will make on the disk - 12 in this case being 2 << 12, or 4096. So replacing 512e drives with 4kn drives should work, and if it doesn't, that's probably some kind of shortcoming that should be addressed, though might not be addressable retroactively depending on why (my immediate guess would be if the amount of space after a partition table is less on 4kn than 512e, because the partition data locations are on logical size, so if, say, you needed to pad the first MB out to avoid being unable to replace a disk with a larger ashift, that could, conceivably, be biting you...)

[ossimoi]

Disk /dev/sda: 9.1 TiB, 10000831348736 bytes, 2441609216 sectors
Sector size (logical/physical): 4096 bytes / 4096 bytes

Disk /dev/sds: 9.1 TiB, 10000831348736 bytes, 19532873728 sectors
Sector size (logical/physical): 512 bytes / 4096 bytes

Disk /dev/sdt: 9.1 TiB, 10000831348736 bytes, 19532873728 sectors
Sector size (logical/physical): 512 bytes / 4096 bytes

~# zpool create test mirror /dev/sds /dev/sdt
~# zpool replace test /dev/sds /dev/sda
cannot replace /dev/sds with /dev/sda: device is too small

You're probably right. It fails because Solaris reserved partition is always 16384 sectors long, which eats a bit more space with 4k logical sector size:

/dev/sds1         2048 19532855295 19532853248  9.1T Solaris /usr & Apple ZFS
/dev/sds9  19532855296 19532871679       16384    8M Solaris reserved 1

/dev/sda1        2048 2441590783 2441588736  9.1T Solaris /usr & Apple ZFS
/dev/sda9  2441590784 2441607167      16384   64M Solaris reserved 1

[rincebrain]

Yeah, I proposed just deleting that recently, because I couldn't get a really good answer on why have it at all these days, and it seems like unnecessary complexity and confusion if nobody can justify why it's there.

[IvanVolosyuk]

Looks like a workaround before that fix would be to use a manually created disk partition, which will be larger than whole drive for ZFS. :)