Test case: zpool_expand_*

[FransUrbo]

Way to reproduce:

#!/bin/bash

rm -f /var/tmp/zfs_test-1
truncate -s 25000m /var/tmp/zfs_test-1

zpool create test /var/tmp/zfs_test-1
for i in {1..3}; do
        zfs create -V 1g test/vol$i
done
sleep 1

zpool create -o autoexpand=on test2 /dev/zvol/test/vol1  /dev/zvol/test/vol2 /dev/zvol/test/vol3

autoexp=$(zpool get autoexpand test2 2> /dev/null | tail -1 | awk '{print $3}')
echo "Autoexpand on test2: '$autoexp'"

prev_size=$(zpool get size test2 2> /dev/null | tail -1 | awk '{print $3}')
zfs_prev_size=$(zfs get -pH -o value avail test2)

for i in {1..3}; do
        zfs set volsize=2g test/vol$i
done

sync
sleep 10
sync

expand_size=$(zpool get size test2 2> /dev/null | tail -1 | awk '{print $3}')
zfs_expand_size=$(zfs get -pH -o value avail test2)

echo "test has previous size: '$prev_size' ($zfs_prev_size) and expanded size: '$expand_size' ($zfs_expand_size)"

Results in:

# /tmp/testme.sh
Autoexpand on test2: 'on'
test has previous size: '2.98G' (3145056256) and expanded size: '2.98G' (3145031680)

[behlendorf]

This is expected, although it is a bug / missing feature. It's my understanding that the way this works on Illumos is that an autoexpand event is handled by their FMA infrastructure. In turn FMA calls zpool online -e pool device to expand the device. We'll want to do something equivalent with the ZED on Linux.

[FransUrbo]

@behlendorf why is this tagged 'Test Suite'? It was FOUND by the test suite, but it's not a problem with it, but with ZFS (ZED).

[andyeff]

Apologies if this isn't quite on topic, but I'm having a problem with manual autoexpand after a LUN expansion, not an automatic one.

Environment: RHEL 6.5, ZFS 0.6.5.3-1, VMWare Workstation 9.0.4.

I have added a 1GB disk, /dev/sdb, and created a pool: zpool create tank sdb
I poweroff the VM and expand the disk to 2GB in vmware.
I power on, and then I attempt to grow the zpool to available space.

I have tried: enabling autoexpand on tank, rebooting, zpool online -e tank sdb, exporting/importing pool, but it will not grow.

Partition view:

[root@rheltest ~]# parted /dev/sdb unit s print
Model: VMware, VMware Virtual S (scsi)
Disk /dev/sdb: 4194304s
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start     End       Size      File system  Name  Flags
 1      2048s     2078719s  2076672s               zfs
 9      2078720s  2095103s  16384s

I am unsure if the two partitions created are standard? Is the small second partition 9 blocking the first from being able to expand? The extra disk in vmware was turned into a zpool without doing anything else on it first - I did not create any existing partitions, just gave it the whole disk.

[andyeff]

I did some further testing, in this case I created an LVM physical volume, volume group and logical volume. If I use that logical volume for ZFS, I can both extend the LVM logical volume and the ZFS pool / filesystem quite happily.

I suspect I'm running into an unsupported case by trying to extend ZFS when a raw disk device is used during pool creation.

[odoucet]

We experience the same issue as @andyeff (with Xen instead of VMware).
zpool online -e does not do the trick, nor reboot.
We previously did get this working, but cannot remember which version we used at that time (0.6.2 or 0.6.3). It is not working on 0.6.5.4-1

[ilovezfs]

@odoucet did you try offlining and then re-onlining -e?

[odoucet]

Cannot be done on this specific zpool because it is made of only one device, and zfs refused to offline the device in this case ("no valid replica").

[ThomasRasmussen]

I've hit the same issue as @andyeff but on Ubuntu 16.04 LTS with its standard ZFS implementation.
I've created a blank disk in vmware (4GiB), run zpool create -o autoexpand=on -f tank /dev/sdb and now parted lists two partitions.
I've then extended the vmware disk to 5GiB and made sure that kernel actually recognizes this change (fdisk et.al shows disk is 5GiB).
I suspect that the small partition (8MiB in size) is infact blocking autoexpand from working. I can delete this partition and adjust partition 1 to the new size, and then zpool online -e tank /dev/sdb actually adjust the pool to a new size.

But deleting this 8MiB partition doesn't sound like a good solution, there must be a reason why it is there? I haven't experienced any problems yet, but still...

Any thoughts, comments or recommendations as to the correct procedure to expand a disk?

[FransUrbo]

In ZoL, this isn't done automatically (yet). You need to online this new disk (actually, the new space) for the additional space to be seen.

[ThomasRasmussen]

If I understand everything correct, then this should be a proper guide:

• /dev/sdb attached, 5GB
• zpool create -o autoexpand=on -f tank /dev/sdb
• zpool list confirms that it is 5GB
• expand disk in vmware to 10GB
• * reboot server, to ensure kernel sees 10GB
• * confirmed using parted -l /dev/sdb
• zpool online -e tank /dev/sdb

But this is not true... My tests shows that I need to do this:

• /dev/sdb attached, 5GB
• zpool create -o autoexpand=on -f tank /dev/sdb
• zpool list confirms that it is 5GB
• expand disk in vmware to 10GB
• * reboot server (or rescan scsi bus), to ensure kernel sees 10GB
• * confirmed using parted -l /dev/sdb
• parted: delete partition 9 on /dev/sdb
• parted: resize partition 1 on /dev/sdb to 100%
• zpool online -e tank /dev/sdb
• confirm correct 10GB on tank

[ilovezfs]

zpool online -e is supposed to do the re-partitioning – not you – if you're using whole_disk=1.

[ThomasRasmussen]

@ilovezfs in which version of ZoL did whole_disk appear? On my host I have

ii zfsutils-linux 0.6.5.6-0ubuntu9 amd64 Native OpenZFS management utilities for Linux

[andyeff]

@ilovezfs Hmm this might have some relation to #2296 whereby /dev/mapper drives / multipath drives are being left as whole_disk=0 (which I believe is an issue I have)

At a guess, I can't switch whole_disk from 0 to 1 on an existing pool?

[ilovezfs]

whole_disk=0/1 is an upstream thing that's older than the hills.

Yeah, anything that's a partition OR not "real" is supposed to be whole_disk=0, so that is correct behavior for dev mapper.

[ThomasRasmussen]

@ilovezfs hmmm... How is whole_disk enabled?

sudo zpool create -o whole_disk=1 pool02 /dev/sdc
property 'whole_disk' is not a valid pool property
sudo zpool create -O whole_disk=1 pool02 /dev/sdc
cannot create 'pool02': invalid property 'whole_disk'

[ilovezfs]

It's handled under the hood automatically. If you have a physical disk, and you use "sdc" not "sdc#{number}", then it will be using whole_disk=1 (i.e., it will autopartition, putting the pool on partition 1 and reserving some space in a partition 9). zpool online -e handles the task of rewriting the partition table in this case.

[andyeff]

@ilovezfs Is there a standard method for doing this manually, e.g. if using SAN-provided disks as /dev/mapper devices where whole disk won't be 1? What's involved in the partition rewriting?

[ilovezfs]

For whole_disk=0 devices, you resize the faux-device or partition yourself and then run zpool online -e #{pool} #{faux device or partition}. autoexpand=on seems to have no impact in that case, even if you export and import.

[FransUrbo]

@andyeff I'm not sure what you mean by SAN-provided disks, but I can access my iSCSI targets (when logged in to them/it) as a /dev/sd? device..

[andyeff]

@FransUrbo ah, I don't believe I can get them to be represented that way in my environment. We're using a pair of fibre-channel paths to access the disks, so multipathd presents the disks as /dev/mapper/mpath? items

[FransUrbo]

Ah, ok. Fair enough. I haven't used in FC in 10+ years, so I have no idea how it works now :).

[ThomasRasmussen]

@ilovezfs If I understand correctly, this should have worked? I would have expected that partition 9 was moved and partition 1 was extended... but what am I doing wrong?

# parted /dev/sdc print
Error: /dev/sdc: unrecognised disk label
Model: VMware Virtual disk (scsi)
Disk /dev/sdc: 8590MB
Sector size (logical/physical): 512B/512B
Partition Table: unknown
Disk Flags:

# zpool create -f -o autoexpand=on pool02 /dev/sdc

# zpool list
NAME     SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
pool02  7.94G  62.5K  7.94G         -     0%     0%  1.00x  ONLINE  -

### Expand vmdisk from 8GiB to 9GiB

# for device in `ls /sys/class/scsi_device/`; do echo 1 > /sys/class/scsi_device/$device/device/rescan; done

dmesg | tail -4
[66593.367036]  sdc: sdc1 sdc9
[66593.375151]  sdc: sdc1 sdc9
[66694.262278] sd 2:0:2:0: [sdc] 18874368 512-byte logical blocks: (9.66 GB/9.00 GiB)
[66694.262321] sdc: detected capacity change from 8589934592 to 9663676416

# parted /dev/sdc print
Warning: Not all of the space available to /dev/sdc appears to be used, you can fix the GPT to use all of the space (an extra 2097152
blocks) or continue with the current setting?
Fix/Ignore? f
Model: VMware Virtual disk (scsi)
Disk /dev/sdc: 9664MB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name  Flags
 1      1049kB  8580MB  8579MB               zfs
 9      8580MB  8589MB  8389kB

# zpool online -e pool02 /dev/sdc

# zpool list
NAME     SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
pool02  7.94G   192K  7.94G         -     0%     0%  1.00x  ONLINE  -

# parted /dev/sdc print
Model: VMware Virtual disk (scsi)
Disk /dev/sdc: 9664MB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name  Flags
 1      1049kB  8580MB  8579MB               zfs
 9      8580MB  8589MB  8389kB

[ThomasRasmussen]

I have then tried to remove partition 9 before running zpool online, and then it is actually expanded:

# parted /dev/sdc rm 9

# zpool online -e pool02 /dev/sdc

Then the partition is infact resized, but what is partition 9 used for? is it critical for ZFS to function or is it something that is "just" added?

[FransUrbo]

what is partition 9 used for? is it critical for ZFS to function or is it something that is "just" added?

It's not actually used for anything, it's a "buffer" partition. Meaning, you don't have to add a new disk with exactly the same number of blocks etc.

[ilovezfs]

If I understand correctly, this should have worked?

@ThomasRasmussen No that procedure is not expected to work because you're changing the virtual disk size, but the original partition table would still be there. zpool online -e is supposed to work as a follow up to zpool replace.

[ThomasRasmussen]

@ilovezfs so it is not possible to perform resize of a pool when the virtual-disk is expanded?

On Solaris (and I also believe it is true for FreeBSD) this is infact working without problems... so I would have thought that it was true for Linux as well :-(

[gaurkuma]

@behlendorf @ilovezfs @ThomasRasmussen

We had a similar problem of expanding current vdevs. We are running as a VM solution so the approach we are thinking of getting around all the expansion hassle is as follows:

1. We ask for a really large virtual disk.
2. Instead of saying vdev->asize = vdev->max_asize we assign vdev_max_asize to the device size obtained via bdev_capacity. But vdev->asize is assigned a smaller value (x % of max_asize)
3. As we start filling the vdevs, at some point (e.g. when we start allocation from the second last metaslab) we raise a zed event that will invoke zpool online -e which will increase the vdev size by some finite size not to exceed the max_asize.

Initial tests results look promising. Looking for some comments from the experts as I am not sure why first of all max_asize and asize exists if there were assigned the same value and secondly, what is the purpose of the expandsize zpool property. i dont think it is being used.