Root Filesystem Expansion

[arithx]

Issue

Now that we have LUKS volumes & the ability to reprovision the root filesystem coming soon we need to be able to handle expanding the root partition in those cases. Currently we are doing the expansion via coreos-growpart but that only handles the most basic case without any support for complex device types and is only inside of the initramfs. It would also be nice to provide a way for users to expand root after the first boot (especially for platforms like VMware which can easily resize the underlying disks).

Potential Solutions

1. Expand the existing coreos-growpart code to support the new cases
   • Pros:
     • Some code already exists
   • Cons:
     • Bash
     • Requires discovery of the filesystems / partitions / disks
2. Add a new Ignition stage which can expand the root filesystem (and underlying partitions)
   • Pros:
     • Not bash
   • Cons:
     • Not driven by the Ignition config (doesn't really fit the existing Ignition model)
     • Ignition currently only runs on the first boot inside the initramfs
     • Requires discovery of the filesystems / partitions / disks on second boot (first boot can look at the cache config but we can't guarantee that the config will be fetchable on secondary boots)
3. Write a new tool in not-bash
   • Pros:
     • Not bash
   • Cons:
     • More tooling sprawl

[arithx]

cc @cgwalters @jlebon @lucab @bgilbert @miabbott

[bgilbert]

I think using Ignition is a tough sell here. Ignition is config-driven, but the config is no longer available and wouldn't help us if it was, since the system may have changed since initial provisioning. And the task is neither first-boot nor provisioning. If we want to avoid bash, writing a new tool makes sense to me.

[cgwalters]

I think we do want two tools...but they really need to share as much code as is feasibly possible.

One simple implementation of that would be to have the logic in the ignition binary and we have a wrapper which calls out to it. Another is to make it a Go shared library.

But the more I think about it...it'd be great to use systemd-repart. If it comes to it...we could try fighting to get that backported to RHEL8 systemd too.

[cgwalters]

But...let me lay out an argument here that the ignition-disks phase should implicitly grow the root device partition unless provided a config that says otherwise, and should also grow the filesystem. We might actually be able to just use systemd-growfs for the latter today.

For "day 2 resize"...I have come around to the idea that this should require a manual invocation (as I think jlebon mentioned), because the user might want to use the new space for a separate partition.
This also partially gets us out of having to support arbitrary block/root device setups because with e.g. LVM/stratis/btrfs instead of trying to grow the blockdev you can make a new partition on the new space and just add it to the pool easily enough.

It would be great to handle "day 2" resizes for the case of LUKS + [filesystem] though for sure out of the box.

[bgilbert]

But...let me lay out an argument here that the ignition-disks phase should implicitly grow the root device partition unless provided a config that says otherwise, and should also grow the filesystem.

Hmm, please do; that's not obvious to me. Ignition doesn't currently have any functionality for resizing partitions or filesystems, and doesn't know what the root device is. (Also, a change in default behavior technically requires a spec bump to version 4.0.0, though I think it's arguable because the OS-level behavior would be unchanged.) At the moment the separate resize script makes more sense to me.

[miabbott]

I think we do want two tools...but they really need to share as much code as is feasibly possible.

One simple implementation of that would be to have the logic in the ignition binary and we have a wrapper which calls out to it. Another is to make it a Go shared library.

I like the idea of a wrapper that calls out to the Ignition binary. It allows us to bake in the functionality in Ignition itself, doesn't require us to package/ship an entirely new binary, and allows us to easily share the feature across FCOS + RHCOS (once RHCOS is using spec v3).

But the more I think about it...it'd be great to use systemd-repart. If it comes to it...we could try fighting to get that backported to RHEL8 systemd too.

In the manpage, systemd-repart is described as:

systemd-repart is intended to be used when deploying OS images, to automatically adjust them to the system they are running on, during first boot

So maybe it would be a good fit for initial install/configuration, but doesn't seem like the right tool for day 2 resize operations.

[bgilbert]

It allows us to bake in the functionality in Ignition itself

I guess I'm still missing something here. Why is that good? What Ignition functionality is useful for implementing this feature?

[cgwalters]

Hmm, please do; that's not obvious to me. Ignition doesn't currently have any functionality for resizing partitions or filesystems, and doesn't know what the root device is.

It knows how to create partitions though, resizing is pretty straightforward relatively. I am not arguing Ignition should know about growing filesystems (see above) - if we're reprovisioning the rootfs then we're implicitly sizing to the target blockdev (which, if we're doing it "in place", then Ignition growing the root partition would work).

[cgwalters]

Or to say this all another way: For the case of "reprovision rootfs in place", it's bonkers to separately growpart + resize, because we just created that filesystem; we might as well create it in the desired layout to start. (I believe it's also more efficient for many Linux filesystems to start out at a reasonable approximation of their target size)

[miabbott]

It allows us to bake in the functionality in Ignition itself

I guess I'm still missing something here. Why is that good? What Ignition functionality is useful for implementing this feature?

Nope, the fault is mine. I attached myself to the idea of landing functionality in Ignition without thinking through the use case. Imagining a system being ignited and doing a grow partition operation during that phase is...well, wrong. As Colin points out above, Ignition will have just created the partition table and filesystem...why would it need to grow a partition immediately after creation?

I guess that means we need Something Else ™️ to handle a day 2 resize operation. Perhaps another tool that can leverage a shared library is the better path.

[bgilbert]

Or to say this all another way: For the case of "reprovision rootfs in place", it's bonkers to separately growpart + resize, because we just created that filesystem; we might as well create it in the desired layout to start.

With coreos/ignition#1087 we can document that the user should request partition resizing, and then the created filesystem will have the correct size immediately.

As to day 2 resizing: if we don't want to continue implementing in shell, we could put it in rdcore, which already includes a lot of block device support code. Presumably it's easier to do resizing from the initramfs before the root is mounted. However, if resize is a manual operation, we'd need some way to cause it to be invoked. If instead we want resize to run from userspace, we could consider adding yet another binary to the coreos-installer repo.

[jlebon]

Related discussions about this in #586.

[jlebon]

Circling back to this, so the situation now is that ignition-ostree-growfs (the script formerly known as coreos-growpart) handles all the cases it already needs to handle, so there is no missing functionality there (in the case of not-in-place root reprovisioning, it's up to the user, with the help of FCC sugar, to size everything correctly in their Ignition config; no automatic growing will occur).

So this issue now is strictly about supporting root filesystem growing as a day-2 manual operation. This tool would of course need to be smarter than just what's in ignition-ostree-growfs because it needs to handle complex (e.g. arbitrary RAID layering, multipath, etc...) setups as well.

I'd be in favour of a new binary (as mentioned above, possibly in coreos-installer; or maybe we need to figure out how to share the block device-related bits in there) for this. And then we could also ship it in the initrd and make ignition-ostree-growfs call out to it instead of re-implementing a subset in shell.

[zanoniluiz]

Do you have any forecast when we will have this feature in CoreOS?

[dustymabe]

Do you have any forecast when we will have this feature in CoreOS?

No concrete plans right now.

@jlebon from your summary above do I read right that what's missing here is a tool (that handles complex root devices) a user can run manually to resize the rootfs after the initial boot.

[jlebon]

Do you have any forecast when we will have this feature in CoreOS?

No concrete plans right now.

@jlebon from your summary above do I read right that what's missing here is a tool (that handles complex root devices) a user can run manually to resize the rootfs after the initial boot.

Yup, exactly.

This work is tracked internally as well, but is not yet scheduled for a specific timeline. Assigning to @sohankunkerkar to reflect the internal state.

[cgwalters]

FTR I briefly looked into this; as of RHEL9 systemd-growfs handles more cases and there's now systemd-repart which looks like it may be useful here, but I had trouble making things work on top of our LUKS setup - and we need to handle that.

[jlebon]

Definitely would be nice to centralize on systemd-growfs. We should ask them if they'd accept patches to support the more complex cases we care about (e.g. LUKS, RAID1, and multipath). Re. LUKS, I do see LUKS-related bits in the source for systemd-growfs at least.