strict ordering of resize service #2522
Conversation
The pid look ok, without a real test filesystem to enlarge the issue might be not blocking the next remount step while resizing. Wonder how cloud-guest-utils does the blocking. |
|
@tim-moody comments? |
|
This looks like a bug fix that if tested for #2517 should be merged shortly? Thanks @jvonau & @tim-moody |
holta
changed the title
|
I'm not sure what this aims to fix. Don't see how it can fix an upstream bug that prevents resizing from even taking place on a W. I have not seen a timeout when resizing actually runs, as on a 4. The amount of time it takes can be considerable and depends on the size of the card compared to the image, but the service seems just to launch resizing and then terminate. For example if you ssh in at the first opportunity on first boot you can use df to watch the partition size grow. If this were to block, users would be locked out I think for up to a couple of minutes. Resize does not require the fs to be mounted read only. I can see the benefit of fsck running before resize as the resized portion of the fs should not need checking, and there is a possible conflict. I'm not sure if there is any conflict with fake-hwclock. To be clear I use images that require resizing frequently. Except on W I never see it fail, though I sometimes need to wait for it to finish. Even W used to succeed, though it required even more patience. |
holta
changed the title
RaspOS has a too old of a version to work correctly on the W, understood.
Yes I understand you can watch the progress, trouble is the whole machine is starting to write to the disk, with the boot paused while resizing is preformed the resizing should complete faster with no contention for CPU cycles or disk IO.
fake-hwclock was used as a reference for the starting order, based on the pids of the services started.
Resizing is something that should be done at the equivalent to runlevel 1 same as where fsck is being run prior to letting the services access the filesystem for the resource contention issue to provide a nice clean slate to work with once the filesystem is remounted read write. See the potential speed benefits mentioned above. Note RaspOS does a reboot after resizing the initial filesystem for this reason, and Ubuntu blocks during resizing using the cloud-init package to get a clean filesystem into place. |
|
Thanks everyone for the many above clarifications. What OS / layouts are most needed to test this PR? |
|
Testing this is a bit tricky, one would need a shrunken image with iiab-rpi-root-resize.service in place. |
getting an image is trivial as there are several on archive.org including https://archive.org/details/iiab-PRE-7.2-200914-medical-content-ready-g01d418e.img which I just put there. Write it to an sd card and boot in rpi 4. |
|
the biggest point of contention is whether the file system needs to be locked during resizing. I believe it does not. The speed varies greatly. A little while ago I tried the image above on a 32G card on rpi 4. It had resized before I could ssh in. |
|
If it can be demonstrated that this PR greatly increases the speed of resizing then I think it is worthwhile. But if not, users will be locked out and may power cycle the machine, which may prevent resizing from continuing. Please use a 128G card to test as 32G may not take long enough. |
|
Let's seek out methodical testers to try this out (often one or two show up, some weeks more than others!) Thanks @jvonau & @tim-moody for the hard work making this possible. |
Ubuntu's first boot would be an indication of how growpart is implemented, and what delay to expect. Do user's power cycle during first boot of Ubuntu? No, the delay is hidden behind cloud-init pretty display if you have a monitor hooked up otherwise you are waiting on ssh to become active.
The one that I have is busy atm... I'll try to pick another up, but if you want, write the image to a sdcard of any size and make a machine available on the vpn with the sdcard available in the usb adapter and I'll do the mods just to see if I'm barking up a wrong tree or not. You can watch it with your own eyes, would be a live test demo and that would be a data point to see if this works as I envision it or not. Let just leave the W's out of the equation, different issue. What do you think? |
I can wait |
|
How can setting up the base filesystem be considered a "good 1st issue" for somebody new to IIAB or linux in general? This skill set involved is way beyond that of a general user. |
True, but somebody new to IIAB (who is already browsing GitHub, reviewing this ticket, hi if that's you!) is rarely new to Linux. e.g. if it happens to be a new IIAB volunteer with RPi/filesystem expertise, this could possibly be a "good 1st issue" for just such a person. |
me too. is there a target that is before sshd is turned on, if the objective is to prevent user login before resizing?
How could I be? There is code to lock the disk in growpart, though I have no idea if it works, but there is absolutely nothing in growpart that would prevent running it on a fully booted system. I have no idea what was in the mind of its author, but I would guess they might have tried to sense the proper conditions in which to run if they thought it necessary. While I acknowledge all your points above, I have serious misgivings about the practicality of avoiding first boot when building anything but the most trivial images. I have built many images; most are 'content ready' but some include content. They are all tweaked in some way because of user requirements. So I need to be able to shrink an image so that it can then resize on next boot. I wanted to use the RaspOS stock resizing, but after numerous attempts to shrink the rootfs, even when offline, in such a way that it would work I gave up and cobbled this together with help from others. For me the bottom line remains that when testing on rpi I write my images that need to resize to an sdcard multiple times per day, mostly to rpi 4, and I never see them fail. Before I upload an image to archive.org, I always write it to an sdcard and boot it up to test it. Until the incident I reported that started this whole chain, I did not experience failures. If someone else has experienced a failure that this PR fixes, I'm all ears. |
|
Required reading:
In a nutshell it appears cloud-init authors envision growpart being run by dracut in the initramfs while the disk is unmounted long before systemd gets a hold of the disk(4). On Ubuntu the automatic resizing can be re-enabled see the first-boot-determination(1). On RaspOS an initramfs is not used at all and mounts the disk directly, this is why RaspOS's uses systemd to launch raspi-config to expand the partition using fdisk on first boot via do_expand_rootfs, then places /etc/init.d/resize2fs_once to expand the filesystem on the next boot. Looks like the distros take great care to insure the root partition is expanded while not much else is running during the resizing operation. |
If that's @jvonau's recommendation, I can merge it today. With so many more OS versions now appearing on RPi (64-bit RaspiOS & Ubuntu) it's entirely possible this will need to evolve again in future? Regardless: everyone should acknowledge a ton of thought has gone into this over the past 10 days. |
tim-moody
changed the title
|
Why merge this rather untested process when there are other PR's pending that have had wider testing is beyond my ability to understand what makes the cut for release. |
|
Just as another data point, I wrote the latest raspios to sdcard, booted, ssh in as pi, and noted the following: |
|
Just a public service announcement that there are 2 distinct operations, partition resizing and filesystem resizing.
To clarify, that is after the quick reboot that does the partition resizing, then you can watch the filesystem resizing. |
|
As part of that public service could you mention if it would be possible to write to the resized portion of a partition prior to resizing the file system. |
|
All this info is a tremendous public service to the roughly 40 Million Raspberry Pi's out there. If folks want this current PR merged into IIAB 7.2 RC2 (expected within about 24h) let me know. |
|
This PR (or something very similar?) could possibly be just what's needed — to solve: |
See also @jvonau's 2022-04-02 summary:
[*] In fact growpart is installed as part of all IIAB's since 2022-03-15: |
|
Several people read over this ticket quite carefully in recent days. And are impressed by the progress that was made over ~12 days, by @jvonau and @tim-moody trying to work this out. Hopefully much of this (or most, or possibly all) still applies almost 2 years later. |
|
The opening post points straight at the race condition, went 747, I didn't think I needed to point out how wrong starting the resize before fsck can be but that was a bad assumption on my part. |
|
This issue is 2 years old, and who are these "Several people read over this ticket quite carefully in recent days." that you speak of and for? I'm sick of the compartmentalization of the reports through a single channel, what is this, a branch of the CIA or an open source project with full transparency? |
Straw Man 1-prep/templates/iiab-expand-rootfs.service based on 2020's PR #2522 + /usr/sbin/iiab-expand-rootfs "bash -xe" exit-on-error (to defer deleting /.expand-rootfs)
Strict ordering to preform the resizing while mounted read only.
124
136
137
139
Could allow these two to run and resize before remounting read write, that will be the second commit
143