Filesystem failed to expand [new RasPiOS opportunity to show visual progress during boot, as rootfs expands]

[jvonau]

From #3375 (comment)
Filesystem failed to expand
Filesystem Size Used Avail Use% Mounted on
/dev/root 6.7G 5.7G 877M 87% / ZIMs: 3 OER2Go: 1 Apps2B: 0

=IIAB==========================================================================
COMMAND: /usr/bin/df -ah # Disk usage detail
Filesystem Size Used Avail Use% Mounted on
/dev/root 6.7G 5.7G 877M 87% /
=IIAB==========================================================================
COMMAND: /usr/bin/lsblk # Partition mount points
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
mmcblk0 179:0 0 58G 0 disk
├─mmcblk0p1 179:1 0 256M 0 part /boot
└─mmcblk0p2 179:2 0 57.7G 0 part /

Originally posted by @jvonau in #3375 (comment)

[holta]

Perhaps this is to be expected — if @drszxn had not yet rebooted?

CLARIF: After PR #3337 is merged, a manual reboot will no longer be required to expand the root filesystem (with Raspberry Pi OS).

Related:

• rootfs can fail to fully expand: race condition betw iiab-expand-rootfs & systemd-fsck ? (journalctl confusingly/regularly portrays 2 simultaneous boots due to lack of RTC?) #3325
• PR Revert "1-prep/templates/iiab-expand-rootfs: Avoid reboot" #3336
• PR Straw Man 1-prep/templates/iiab-expand-rootfs.service based on 2020's PR #2522 + /usr/sbin/iiab-expand-rootfs "bash -xe" exit-on-error (to defer deleting /.expand-rootfs) #3337

[jvonau]

Post writing http://download.iiab.io/8.0/iiab-8.0preview3-220820-SMALL-raspios64-lite-g2543831e.img.zip in host machine:

lsblk
sdb           8:16   1  59.5G  0 disk 
├─sdb1        8:17   1   256M  0 part /media/jvonau/boot
└─sdb2        8:18   1   6.8G  0 part /media/jvonau/rootfs

df
/dev/sdb2         6969504   5054960    1755796  75% /media/jvonau/rootfs
/dev/sdb1          258095     30943     227153  12% /media/jvonau/boot

Firstboot in device:

iiab-admin@box:~ $ journalctl -u iiab-expand-rootfs.service --no-pager
-- Journal begins at Mon 2022-04-04 10:41:41 EDT, ends at Sat 2022-09-24 01:09:09 EDT. --
Aug 20 10:20:45 box systemd[1]: Starting Root Filesystem Auto-Expander...
Aug 20 10:20:45 box iiab-expand-rootfs[403]: + '[' -f /.expand-rootfs ']'
Aug 20 10:20:45 box iiab-expand-rootfs[403]: + '[' -f /.resize-rootfs ']'
Aug 20 10:20:45 box systemd[1]: iiab-expand-rootfs.service: Succeeded.
Aug 20 10:20:45 box systemd[1]: Finished Root Filesystem Auto-Expander.
-- Boot 80e8ca626f6e485681009957f0bb7830 --
Aug 20 10:24:24 box systemd[1]: Starting Root Filesystem Auto-Expander...
Aug 20 10:24:24 box iiab-expand-rootfs[447]: + '[' -f /.expand-rootfs ']'
Aug 20 10:24:24 box iiab-expand-rootfs[447]: + echo '/usr/sbin/iiab-expand-rootfs: Expanding rootfs partition'
Aug 20 10:24:24 box iiab-expand-rootfs[447]: /usr/sbin/iiab-expand-rootfs: Expanding rootfs partition
Aug 20 10:24:24 box iiab-expand-rootfs[447]: + '[' -x /usr/bin/raspi-config ']'
Aug 20 10:24:24 box iiab-expand-rootfs[447]: + raspi-config --expand-rootfs
Aug 20 10:24:26 box iiab-expand-rootfs[613]: Welcome to fdisk (util-linux 2.36.1).
Aug 20 10:24:26 box iiab-expand-rootfs[613]: Changes will remain in memory only, until you decide to write them.
Aug 20 10:24:26 box iiab-expand-rootfs[613]: Be careful before using the write command.
Aug 20 10:24:26 box iiab-expand-rootfs[613]: Command (m for help): Disk /dev/mmcblk0: 59.52 GiB, 63910707200 bytes, 124825600 sectors
Aug 20 10:24:26 box iiab-expand-rootfs[613]: Units: sectors of 1 * 512 = 512 bytes
Aug 20 10:24:26 box iiab-expand-rootfs[613]: Sector size (logical/physical): 512 bytes / 512 bytes
Aug 20 10:24:26 box iiab-expand-rootfs[613]: I/O size (minimum/optimal): 512 bytes / 512 bytes
Aug 20 10:24:26 box iiab-expand-rootfs[613]: Disklabel type: dos
Aug 20 10:24:26 box iiab-expand-rootfs[613]: Disk identifier: 0x7790a859
Aug 20 10:24:26 box iiab-expand-rootfs[613]: Device         Boot  Start      End  Sectors  Size Id Type
Aug 20 10:24:26 box iiab-expand-rootfs[613]: /dev/mmcblk0p1        8192   532479   524288  256M  c W95 FAT32 (LBA)
Aug 20 10:24:26 box iiab-expand-rootfs[613]: /dev/mmcblk0p2      532480 14768911 14236432  6.8G 83 Linux
Aug 20 10:24:26 box iiab-expand-rootfs[613]: Command (m for help): Partition number (1,2, default 2):
Aug 20 10:24:26 box iiab-expand-rootfs[613]: Partition 2 has been deleted.
Aug 20 10:24:26 box iiab-expand-rootfs[613]: Command (m for help): Partition type
Aug 20 10:24:26 box iiab-expand-rootfs[613]:    p   primary (1 primary, 0 extended, 3 free)
Aug 20 10:24:26 box iiab-expand-rootfs[613]:    e   extended (container for logical partitions)
Aug 20 10:24:26 box iiab-expand-rootfs[613]: Select (default p): Partition number (2-4, default 2): First sector (2048-124825599, default 2048): Last sector, +/-sectors or +/-size{K,M,G,T,P} (532480-124825599, default 124825599):
Aug 20 10:24:26 box iiab-expand-rootfs[613]: Created a new partition 2 of type 'Linux' and of size 59.3 GiB.
Aug 20 10:24:26 box iiab-expand-rootfs[613]: Partition #2 contains a ext4 signature.
Aug 20 10:24:26 box iiab-expand-rootfs[613]: Command (m for help):
Aug 20 10:24:26 box iiab-expand-rootfs[613]: Disk /dev/mmcblk0: 59.52 GiB, 63910707200 bytes, 124825600 sectors
Aug 20 10:24:26 box iiab-expand-rootfs[613]: Units: sectors of 1 * 512 = 512 bytes
Aug 20 10:24:26 box iiab-expand-rootfs[613]: Sector size (logical/physical): 512 bytes / 512 bytes
Aug 20 10:24:26 box iiab-expand-rootfs[613]: I/O size (minimum/optimal): 512 bytes / 512 bytes
Aug 20 10:24:26 box iiab-expand-rootfs[613]: Disklabel type: dos
Aug 20 10:24:26 box iiab-expand-rootfs[613]: Disk identifier: 0x7790a859
Aug 20 10:24:26 box iiab-expand-rootfs[613]: Device         Boot  Start       End   Sectors  Size Id Type
Aug 20 10:24:26 box iiab-expand-rootfs[613]: /dev/mmcblk0p1        8192    532479    524288  256M  c W95 FAT32 (LBA)
Aug 20 10:24:26 box iiab-expand-rootfs[613]: /dev/mmcblk0p2      532480 124825599 124293120 59.3G 83 Linux
Aug 20 10:24:26 box iiab-expand-rootfs[613]: Command (m for help): The partition table has been altered.
Aug 20 10:24:26 box iiab-expand-rootfs[613]: Syncing disks.
Aug 20 10:24:28 box iiab-expand-rootfs[475]: Please reboot
Aug 20 10:24:28 box iiab-expand-rootfs[447]: + rm -f /.expand-rootfs /.resize-rootfs
Aug 20 10:24:28 box systemd[1]: iiab-expand-rootfs.service: Succeeded.
Aug 20 10:24:28 box systemd[1]: Finished Root Filesystem Auto-Expander.

lsblk
NAME        MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
mmcblk0     179:0    0 59.5G  0 disk 
├─mmcblk0p1 179:1    0  256M  0 part /boot
└─mmcblk0p2 179:2    0 59.3G  0 part /

df
Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/root        6969504 5097992   1712764  75% /

cat /etc/init.d/resize2fs_once 
#!/bin/sh
### BEGIN INIT INFO
# Provides:          resize2fs_once
# Required-Start:
# Required-Stop:
# Default-Start: 3
# Default-Stop:
# Short-Description: Resize the root filesystem to fill partition
# Description:
### END INIT INFO

. /lib/lsb/init-functions

case "$1" in
  start)
    log_daemon_msg "Starting resize2fs_once" &&
    resize2fs "/dev/mmcblk0p2" &&
    update-rc.d resize2fs_once remove &&
    rm /etc/init.d/resize2fs_once &&
    log_end_msg $?
    ;;
  *)
    echo "Usage: $0 start" >&2
    exit 3
    ;;
esac

The file should not be present if it ran but I didn't reboot

sudo /etc/init.d/resize2fs_once start
Starting resize2fs_once (via systemctl): resize2fs_once.service
.
iiab-admin@box:~ $ 
iiab-admin@box:~ $ journalctl -u resize2fs_once.service --no-pager
-- Journal begins at Mon 2022-04-04 10:41:41 EDT, ends at Sat 2022-09-24 01:27:54 EDT. --
Apr 04 10:41:59 raspberrypi systemd[1]: Starting LSB: Resize the root filesystem to fill partition...
Apr 04 10:41:59 raspberrypi resize2fs_once[417]: Starting resize2fs_once:
Apr 04 10:41:59 raspberrypi resize2fs_once[501]: resize2fs 1.46.2 (28-Feb-2021)
Aug 20 09:33:00 raspberrypi resize2fs_once[501]: Filesystem at /dev/mmcblk0p2 is mounted on /; on-line resizing required
Aug 20 09:33:00 raspberrypi resize2fs_once[501]: old_desc_blocks = 1, new_desc_blocks = 4
Aug 20 09:33:00 raspberrypi resize2fs_once[501]: The filesystem on /dev/mmcblk0p2 is now 15563776 (4k) blocks long.
Aug 20 09:33:00 raspberrypi resize2fs_once[417]: .
Aug 20 09:33:00 raspberrypi systemd[1]: Started LSB: Resize the root filesystem to fill partition.
-- Boot 80e8ca626f6e485681009957f0bb7830 --
Sep 24 01:26:54 box systemd[1]: Starting LSB: Resize the root filesystem to fill partition...
Sep 24 01:26:54 box resize2fs_once[2174]: Starting resize2fs_once:
Sep 24 01:26:54 box resize2fs_once[2177]: resize2fs 1.46.2 (28-Feb-2021)
Sep 24 01:27:51 box resize2fs_once[2177]: Filesystem at /dev/mmcblk0p2 is mounted on /; on-line resizing required
Sep 24 01:27:51 box resize2fs_once[2177]: old_desc_blocks = 1, new_desc_blocks = 4
Sep 24 01:27:51 box resize2fs_once[2177]: The filesystem on /dev/mmcblk0p2 is now 15536640 (4k) blocks long.
Sep 24 01:27:52 box resize2fs_once[2174]: .
Sep 24 01:27:52 box systemd[1]: Started LSB: Resize the root filesystem to fill partition.

iiab-admin@box:~ $ df
Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/root       61139384 5098000  55333352   9% /
devtmpfs         1777020       0   1777020   0% /dev
tmpfs            1942748       0   1942748   0% /dev/shm
tmpfs             777100     944    776156   1% /run
tmpfs               5120       4      5116   1% /run/lock
/dev/mmcblk0p1    258095   30943    227153  12% /boot
tmpfs             388548       0    388548   0% /run/user/1000

[jvonau]

Think the reboot could be avoided just calling

resize2fs "replace with variable"
update-rc.d resize2fs_once remove
rm /etc/init.d/resize2fs_once

or borrow parts (fdisk) of the do_expand_rootfs() function from raspi-config

[holta]

Think the reboot could be avoided just calling

resize2fs "replace with variable" update-rc.d resize2fs_once remove rm /etc/init.d/resize2fs_once

or borrow parts (fdisk) of the do_expand_rootfs() function from raspi-config

Apologies I'm confused by the above:

Isn't this ticket mainly discussing RasPiOS (where PR #3337 as it stands should handle rapid auto-reboot very cleanly) ?

Or other OS's with resize2fs (where reboot is apparently not required) ?

[jvonau]

Neither, I spotted that the created image in the report didn't have a resized file system but jumped to just running the script in an effort to suppress the reboot.. I'll circle back with just reboot after the initial login on firstboot cause I'm wondering if using any of the pre-seed values in Rpi-imager changes the sequence of events given Rpi-imager creates 'systemd.run=/boot/firstrun.sh systemd.run_success_action=reboot systemd.unit=kernel-command-line.target' in the /boot/cmdline.txt file and /boot/firstrun.sh is created when you do on the newly written iiab image.

[holta]

To reduce complexity, PR #3337 (or something close to it) needs be merged and broadly tested shortly, after much progress over 7 weeks now:

Certainly #3375 (and this ticket) are about RasPiOS. As such, is there solid evidence of a bug in raspi-config --expand-rootfs ?

How do we best identify conditions to reproduce that bug if so, as a public service for others?

(Or conversely, if there's solid but complicated evidence — let's report that upstream as clearly as we can in coming days — so Raspberry Pi Foundation people can hopefully investigate!)

[jvonau]

No a simple reboot works on the produced images but doesn't give any user feedback that it is running. I'm looking towards the future with what was released on the current upstream images where there is feedback that the resize is in progress.