Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FS#2231 - fstools: PREINIT calling of block extroot doesn't acknowledge non-MTD rootfs overlays #7352

Open
openwrt-bot opened this issue Apr 9, 2019 · 63 comments
Labels

Comments

@openwrt-bot
Copy link

openwrt-bot commented Apr 9, 2019

knodel:

Extroot overlay mount fails on my ZyXEL NBG6817.

I can confirm this bug on both the latest stable 18.06.2, and the snapshot from the 4th of April.

When PREINIT calls 'block extroot', block fails to load the custom fstab from the eMMC ext4 overlay mounted at /tmp/overlay:

[...]
Thu Apr  4 12:01:51 2019 user.info kernel: [    3.508150] init: - preinit -
Thu Apr  4 12:01:51 2019 kern.info kernel: [    8.592387] EXT4-fs (loop0): recovery complete
Thu Apr  4 12:01:51 2019 kern.info kernel: [    8.595484] EXT4-fs (loop0): mounted filesystem with ordered data mode. Opts: (null)
Thu Apr  4 12:01:51 2019 user.info kernel: [    8.598875] mount_root: loading kmods from internal overlay
Thu Apr  4 12:01:51 2019 user.info kernel: [    8.649260] kmodloader: loading kernel modules from /tmp/overlay/upper/etc/modules-boot.d/*
Thu Apr  4 12:01:51 2019 user.info kernel: [    8.663223] kmodloader: done loading kernel modules from /tmp/overlay/upper/etc/modules-boot.d/*
Thu Apr  4 12:01:51 2019 user.info kernel: [   10.119972] block: attempting to load /etc/config/fstab
Thu Apr  4 12:01:51 2019 user.err kernel: [   10.120211] block: unable to load configuration (fstab: Entry not found)
Thu Apr  4 12:01:51 2019 user.err kernel: [   10.124238] block: no usable configuration
Thu Apr  4 12:01:51 2019 user.info kernel: [   10.132103] mount_root: switching to ext4 overlay
Thu Apr  4 12:01:51 2019 user.info kernel: [   10.292110] procd: - early -
Thu Apr  4 12:01:51 2019 user.info kernel: [   10.292264] procd: - watchdog -
Thu Apr  4 12:01:51 2019 user.info kernel: [   10.988093] procd: - watchdog -
Thu Apr  4 12:01:51 2019 user.info kernel: [   10.988475] procd: - ubus -
Thu Apr  4 12:01:51 2019 user.info kernel: [   11.046160] procd: - init -
[...]

Unrelated dmesg entries omitted, full log here:
https://gist.github.com/knuddelknoedel/2985ce7777a0263fbc22a02f8ef5307c

Custom modules are loaded with the correct overlay /tmp/overlay/upper prefix by libfstools, however the forked 'block extroot' process behaves differently when searching for /etc/config/fstab configuration.

Steps to reproduce:

  • Flash any current openwrt sysupgrade image on a device where the rootfs+overlay don't reside on MTD storage
  • Configure an appropriate /overlay uci fstab extroot entry as specified in the respective wiki documentation
  • Reboot

Further notes: adding /etc/config/fstab with the desired /overlay entry to the sysupgrade squashfs image before flashing allows block to successfully find the uci fstab config, however the mounting of the therein configured /overlay mount still fails.

@openwrt-bot
Copy link
Author

openwrt-bot commented Apr 9, 2019

knodel:

I wrote a small patch that allows proper rootfs and fstab discovery on devices where the rootfs+overlay reside on a block device partition.

Here the dmesg of a current openwrt HEAD build with the patch:

Mon Apr  8 22:25:35 2019 user.info kernel: [    3.335289] init: - preinit -
Mon Apr  8 22:25:35 2019 user.info kernel: [    8.035073] mount_root: loading kmods from internal overlay
Mon Apr  8 22:25:35 2019 user.info kernel: [    8.046998] kmodloader: loading kernel modules from //etc/modules-boot.d/*
Mon Apr  8 22:25:35 2019 user.info kernel: [    8.048462] kmodloader: done loading kernel modules from //etc/modules-boot.d/*
Mon Apr  8 22:25:35 2019 user.info kernel: [    8.828623] block: attempting to load /tmp/overlay/upper/etc/config/fstab
Mon Apr  8 22:25:35 2019 user.err kernel: [    8.828798] block: unable to load configuration (fstab: Entry not found)
Mon Apr  8 22:25:35 2019 user.info kernel: [    8.834600] block: attempting to load /tmp/overlay/etc/config/fstab
Mon Apr  8 22:25:35 2019 user.err kernel: [    8.841281] block: unable to load configuration (fstab: Entry not found)
Mon Apr  8 22:25:35 2019 user.info kernel: [    8.847486] block: attempting to load /etc/config/fstab
Mon Apr  8 22:25:35 2019 user.err kernel: [    8.854319] block: unable to load configuration (fstab: Entry not found)
Mon Apr  8 22:25:35 2019 user.err kernel: [    8.859098] block: no usable configuration
Mon Apr  8 22:25:35 2019 kern.info kernel: [    8.985003] EXT4-fs (loop0): recovery complete
Mon Apr  8 22:25:35 2019 kern.info kernel: [    8.986207] EXT4-fs (loop0): mounted filesystem with ordered data mode. Opts: (null)
Mon Apr  8 22:25:35 2019 user.info kernel: [    8.988754] mount_root: loading kmods from internal overlay
Mon Apr  8 22:25:35 2019 user.info kernel: [    9.013356] kmodloader: loading kernel modules from /tmp/overlay/upper/etc/modules-boot.d/*
Mon Apr  8 22:25:35 2019 user.info kernel: [    9.013492] kmodloader: done loading kernel modules from /tmp/overlay/upper/etc/modules-boot.d/*
Mon Apr  8 22:25:35 2019 user.info kernel: [   10.011824] block: attempting to load /tmp/overlay/upper/etc/config/fstab
Mon Apr  8 22:25:35 2019 kern.info kernel: [   10.017872] EXT4-fs (mmcblk0p10): mounted filesystem with ordered data mode. Opts:
Mon Apr  8 22:25:35 2019 user.info kernel: [   10.162997] mount_root: switched to extroot
Mon Apr  8 22:25:35 2019 user.info kernel: [   10.331230] procd: - early -
Mon Apr  8 22:25:35 2019 user.info kernel: [   10.331412] procd: - watchdog -
Mon Apr  8 22:25:35 2019 user.info kernel: [   11.009306] procd: - watchdog -
Mon Apr  8 22:25:35 2019 user.info kernel: [   11.009775] procd: - ubus -
Mon Apr  8 22:25:35 2019 user.info kernel: [   11.064866] procd: - init -

Unrelated dmesg entries omitted, full log here: https://gist.github.com/knuddelknoedel/46db572959056d2e69fd861d59ad0daf

@openwrt-bot
Copy link
Author

val-kulkov:

The proposed patch works beautifully to enable extroot on my x86_64 device. With it, I can finally make use of the available space on my /dev/sda without worrying about losing data on it upon a sysupgrade.

The sysupgrade process on a x86_64 system that I use to avoid losing data is not yet a straightforward one. First, I burn openwrt-x86-64-combined-squashfs.img on a USB drive. Then, I copy first two partitions from the USB drive to the corresponding partitions on my /dev/sda. Finally, I copy the MBR bootstrap only, without the partition table: "dd if=/dev/sdc of=/dev/sda bs=446 count=1". This procedure retains sda3, my root overlay partition, and sda4, a swap partition.

@openwrt-bot
Copy link
Author

utrumo:

Now on Zyxel Armor Z2 (NBG6817) extroot doesn't work without this patch.
Is there a developer of openwrt? Please add this patch to master! :)

To make work extroot on this router we need compile own firmware and apply this patch:

  1. clone src and change dir to it: git clone https://github.com/openwrt/openwrt.git ~/source && cd ~/source
  2. make directory for patch: mkdir ./package/system/fstools/patches
  3. write patch content from https://patchwork.ozlabs.org/patch/1082599/ to ./package/system/fstools/patches/001-add_propper_rootfs_and_fstab_discovery_on_a_block_device_partitions.patch
  4. build own image:
    ./scripts/feeds update -a
    ./scripts/feeds install -a
    export MAKEFLAGS=-j5
    make menuconfig
    ===================================================
    system => Qualcomm Atheros IPQ806X y
    Target Profile => Zyxel NBG6817 y
    Base system =>
    => block-mount y (for extroot on mmcblk0p10)
    => blockd y (for extroot on mmcblk0p10)
    Kernel modules => USB Support => kmod-usb-storage y (for usb-hdd support)
    LuCI =>
    => Collections => luci y
    => Applications => luci-app-advanced-reboot y
    ===================================================
    make defconfig
    make download
    ionice -c 3 nice -n19 make

results you can find in:
./bin/targets/ipq806x/generic/

  1. flash this firmware to router
    scp ./bin/targets/ipq806x/generic/openwrt-ipq806x-zyxel_nbg6817-squashfs-sysupgrade.bin root@openwrt:/tmp
    ssh root@openwrt
    sysupgrade -v /tmp/openwrt-ipq806x-zyxel_nbg6817-squashfs-sysupgrade.bin

  2. set extroot:
    ssh root@openwrt
    DEVICE="$(awk -e '/\s/overlay\s/{print $1}' /etc/mtab)"
    uci -q delete fstab.rwm
    uci set fstab.rwm="mount"
    uci set fstab.rwm.device="${DEVICE}"
    uci set fstab.rwm.target="/rwm"
    uci set fstab.rwm.enabled="1"
    uci commit fstab

DEVICE="/dev/mmcblk0p10" // or if you want to use usb-hdd: DEVICE="/dev/sda1"
mkfs.ext4 "${DEVICE}"
eval $(block info "${DEVICE}" | grep -o -e "UUID=\S*")
uci -q delete fstab.overlay
uci set fstab.overlay="mount"
uci set fstab.overlay.uuid="${UUID}"
uci set fstab.overlay.target="/overlay"
uci set fstab.overlay.enabled="1"
uci commit fstab

mount "${DEVICE}" /mnt
cp -a -f /overlay/. /mnt
umount /mnt

reboot

@openwrt-bot
Copy link
Author

rmilecki:

That fstools patch cannot be applied as is (I reviewed & commented on it).

I pushed three related fstools changes:

  • [[https://git.openwrt.org/?p=project/fstools.git;a=commitdiff;h=d70774de24413f91087e9c3c873cf7a4ec3df813|d70774de2441 ("block: add some basic extroot documentation")]]
  • [[https://git.openwrt.org/?p=project/fstools.git;a=commitdiff;h=37c9148a3ae17665e75f99b459b6ff2b461d6739|37c9148a3ae1 ("block: simplify check_extroot() a bit")]]
  • [[https://git.openwrt.org/?p=project/fstools.git;a=commitdiff;h=8b9e601160712f74ff46313579d0225a7f90ca3c|8b9e60116071 ("block: always use st_dev (device ID) of / when looking for root")]]

That moves us closer to fixing this issue.

Someone now needs to improve main_extroot() and make it:

  • Find standard overlay device (e.g. /dev/sda3)
  • Mount it using f2fs
  • Call mount_extroot() for it

It's basically the same logic as what we already have for JFFS2 and UBIFS in that main_extroot() function.

@openwrt-bot
Copy link
Author

leifliddy:

If we're going to use f2fs as the filesystem for the extroot overlay device (which is a great idea), then we should probably include these package as part of the base build for the zyxel_nbg6817:
f2fsck
f2fs-tools
kmod-fs-f2fs
mkf2fs

Actually, this brings up another question, why aren't we using the f2fs filesystem for the default overlay device?

/dev/loop0 on /overlay type ext4 (rw,noatime,data=ordered)

@openwrt-bot
Copy link
Author

rmilecki:

I just noticed that orignal report was about ext4. Recently I was chatting with m4t who was using f2fs.

So this task seems more generic: we need to check /dev/loop0 for /etc/config/fstab with overlay setup. That /dev/loop0 may be ext4 or f2fs (or other?).

@openwrt-bot
Copy link
Author

leifliddy:

I'm not sure if this is the correct forum for this or not, but here goes.
If you look at partition schedule for the nbg6817

https://openwrt.org/toh/zyxel/nbg6817

You'll see that mmcblk0p5 is divided up into two parts

mmcblk0p5 rootfs 64 MiB

/rom (squashfs) 4MB
/overlay (ext4) 60MB (/dev/loop0)

So, if you've configured mmcblk0p10 to act as the /overlay,
then the 4MB squashfs filesystem from mmcblk0p5 will continue to act as the 4Mb read-only filesystem?

Is there a reason we can't just re-partition this entire device, I mean there's no reason for openwrt to need 10 partitions.
Or is there some sort of unspoken rule that you need to operate within the confines of the OEM flash layout?

If we need to maintain the OEM partitioning scheme, then we should make mmcblk0p1 the 4MB /rom partition
...and make mmcblk0p10 the default /overlay partition.

That's makes the most sense...but I'll save this discussion for another day. Need to do a bit more research on how openwrt is designed....

@openwrt-bot
Copy link
Author

leifliddy:

@ken would you be able to create a new patch based on the changes Rafał made to fstools?

@openwrt-bot
Copy link
Author

openwrt-bot commented May 17, 2020

tofurky:

leifliddy, this works for me with f2fs on 19.07.3. note that the change to check_filesystem() isn't strictly necessary, but i found that newer f2fs-tools gave some issues without it.

diff --git a/block.c b/block.c
index 569bf56..b88e208 100644
--- a/block.c
+++ b/block.c
@@ -747,7 +747,7 @@ static void check_filesystem(struct probe_info *pr)
        pid = fork();
        if (!pid) {
                if(!strncmp(pr->type, "f2fs", 4)) {
-                       execl(ckfs, ckfs, "-f", pr->dev, NULL);
+                       execl(ckfs, ckfs, "-p", "2", "-f", pr->dev, NULL);
                        exit(EXIT_FAILURE);
                } else if(!strncmp(pr->type, "btrfs", 5)) {
                        execl(ckfs, ckfs, "--repair", pr->dev, NULL);
@@ -1591,7 +1591,7 @@ static int main_extroot(int argc, char **argv)
 #endif
 
        /* As a last resort look for /etc/config/fstab on "rootfs" partition */
-       return mount_extroot(NULL);
+       return mount_extroot("/tmp/overlay");
 }
 
 static int main_mount(int argc, char **argv)

i spent a bit of time trying to figure out the proper way to do it per rafal's suggestion, but was running into segfaults with find_block(NULL, NULL, "loop0", NULL) so i just took the easy way out ;) not sure if i was even close to taking the right approach with that, though.

@openwrt-bot
Copy link
Author

katrat:

Hi,

I guess this fix wasn't included in the 19.07.3 release yet, since I am struggling to set up overlayfs on my new NBG6817 with usb hard disk and ext4.

Are there any patched images to download/test? Any chance this will be fixed in one of the next releases?

I am not that experienced to build/figure out this myself, so any links/instructions are much appreciated. On the other hand, my zyxel isn't productive yet, so I am willing to test...

Thanks,
kat

@openwrt-bot
Copy link
Author

rmilecki:

Noone cared to develop a proper patch, people keep posting some hacks that are unacceptable due to breaking other setups.

@openwrt-bot
Copy link
Author

katrat:

Thanks for the quick, if also a bit unsatisfying, answer. I got the zyxel because it seemed to be properly supported by openwrt (and having a bit more oomph than my ageing tp-link). I don't really understand where the problem is: ext4 of f2fs, hard disk or flash drive, or the drives not being ready at boot time, or just some configuration hiccup.

Can you suggest what I could do? I don't mind playing around a bit.

@openwrt-bot
Copy link
Author

rmilecki:

Someone now needs to improve main_extroot() and make it:

1. Find standard overlay device (e.g. /dev/sda3)
2. Mount it using f2fs
3. Call mount_extroot() for it

It's basically the same logic as what we already have for JFFS2 and UBIFS in that main_extroot() function.

I just noticed that orignal report was about ext4. Recently I was chatting with m4t who was using f2fs.

So this task seems more generic: we need to check /dev/loop0 for /etc/config/fstab with overlay setup. That /dev/loop0 may be ext4 or f2fs (or other?).

@openwrt-bot
Copy link
Author

visualage:

Is it reasonable to use libfstools to search for rootfs_data volume and mount it, just like what mount_root.c does?

@openwrt-bot
Copy link
Author

visualage:

I made a patch to openwrt to support extroot on non-MTD rootfs_data volumes. It uses libfstools in block.c to support other squashfs volumes, such as /dev/sd* or /dev/mmc*.

The patch itself contains a patch to fstools project that can be applied to that codebase directly.

I tested the build locally on a x86-64 virtualbox environment as well as rockchip friendlyarm nanopi r2s. It should work on any other environments as long as mount_root works since they use the same underlying library.

@openwrt-bot
Copy link
Author

visualage:

This is the fstools patch. The openwrt patch is just to make sure block-mount depends on libfstools when packaging.
Rafal, would you please take a look?

@openwrt-bot
Copy link
Author

kofec:

Hi,
Aby update about this issue ?

@openwrt-bot
Copy link
Author

uutrumo:

Qi Liu, for wich version of openwrt your patch?
I try it on v19.07.7 and have got an error:
Package block-mount is missing dependencies for the following libraries:
libfstools.so

@openwrt-bot
Copy link
Author

openwrt-bot commented May 10, 2021

uutrumo:

tofurky, your hack works, thank you.
Some patch for v19.07.7 and v21.02.0-rc1:

--- a/block.c
+++ b/block.c
@@ -747,7 +747,7 @@ static void check_filesystem(struct prob
 	pid = fork();
 	if (!pid) {
 		if(!strncmp(pr->type, "f2fs", 4)) {
-			execl(ckfs, ckfs, "-f", pr->dev, NULL);
+			execl(ckfs, ckfs, "-p", "2", "-f", pr->dev, NULL);
 			exit(EXIT_FAILURE);
 		} else if(!strncmp(pr->type, "btrfs", 5)) {
 			execl(ckfs, ckfs, "--repair", pr->dev, NULL);
@@ -1591,7 +1591,7 @@ static int main_extroot(int argc, char *
 #endif
 
 	/* As a last resort look for /etc/config/fstab on "rootfs" partition */
-	return mount_extroot(NULL);
+	return mount_extroot("/tmp/overlay");
 }
 
 static int main_mount(int argc, char **argv)

@openwrt-bot
Copy link
Author

luce.nera:

How do I apply these patches and which one to use?

@openwrt-bot
Copy link
Author

flowercow:

i met this bug in 21.02.0-rc4 installed in kvm.

Is there any other method to solve this problem?

For example, do something in /etc/preinit?

@openwrt-bot
Copy link
Author

Gandalf:

Still got the issue in 21.02.0 !!!
Is there any work in this patch ?

@openwrt-bot
Copy link
Author

flowercow:

i have tried to recompile firmware applying patch as mentioned by utrumo above.
after upgrade to that firmware and install block-mount package that recompiled, this issue was not fixed.

@openwrt-bot
Copy link
Author

Op3n:

The same bug occurs on the latest stable release 20.02, Is there any way to avoid this bug other than by compiling it myself? I prefer use official pre-compiled version on downloads.openwrt.com

@openwrt-bot
Copy link
Author

katrat:

Well, any solutions apart from patching and compiling openwrt ourselves? Does anyone have a precompiled working solution for 21.02?

@openwrt-bot
Copy link
Author

siicosmos:

Need some solution for 21.02 as well. Either patches or a pre-compiled image is fine...

@openwrt-bot
Copy link
Author

siicosmos:

Updates:
Actually, the patch tofurky provided actually works in 21.02. You need to apply the patch using Quilt or edit directly on block.c file.

@VA1DER
Copy link

VA1DER commented Feb 28, 2022

Is there a timeline for this being officially fixed?

@MaxSKash
Copy link

MaxSKash commented Mar 7, 2022

There is an alternative way for Zyxel NBG6817: change the uboot-env variable so that rootfs is loaded from /dev/mmсblk0p10

Example:
Boot into an alternative configuration (which is 01, not FF)
We copy the sysupgrade.bin to the router in the folder /tmp
Connect to it via ssh
Manual flash firmvare:
(based on a standard script: /lib/upgrade/zyxel.sh)

view at the contents of the firmware:

root@OpenWrt:~# cd /tmp
root@OpenWrt:/tmp# tar tf openwrt-ipq806x-generic-zyxel_nbg6817-squashfs-sysupgrade.bin
tar: sysupgrade-nbg6817/CONTROL: time stamp 2022-03-07 10:14:29 is 42124622.3872312 s in the future
tar: sysupgrade-nbg6817/kernel: time stamp 2022-03-07 10:14:29 is 42124622.35407792 s in the future
tar: sysupgrade-nbg6817/root: time stamp 2022-03-07 10:14:29 is 42124622.32335552 s in the future
tar: sysupgrade-nbg6817: time stamp 2022-03-07 10:14:29 is 42124622.32289632 s in the future

flash: (tar xf <<filename.bin>> <<contentname>> -O > <<path>> )

root@OpenWrt:/tmp# tar xf openwrt-ipq806x-generic-zyxel_nbg6817-squashfs-sysupgrade.bin sysupgrade-nbg6817/kernel -O > /dev/mmcblk0p4
root@OpenWrt:/tmp# tar xf openwrt-ipq806x-generic-zyxel_nbg6817-squashfs-sysupgrade.bin sysupgrade-nbg6817/root -O > /dev/mmcblk0p10

make an overlay:
find out the offset:

root@OpenWrt:/tmp# tar xf openwrt-ipq806x-generic-zyxel_nbg6817-squashfs-sysupgrade.bin sysupgrade-nbg6817/root -O | wc -c
6553600

find free loop:

root@OpenWrt:/tmp# losetup -f
/dev/loop1

Create and format it:

root@OpenWrt:/tmp# losetup -o 6553600 /dev/loop1 /dev/mmcblk0p10
root@OpenWrt:/tmp# mkfs.ext4 -F -L rootfs_data /dev/loop1
mke2fs 1.44.5 (15-Dec-2018)
Creating filesystem with 849847 4k blocks and 212576 inodes
Filesystem UUID: ae0d6b77-d336-4bdc-a899-12436c82e0f9
Superblock backups stored on blocks: 
        32768, 98304, 163840, 229376, 294912, 819200

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done 

Now change uboot-env:
get the current value:

root@OpenWrt:/tmp# fw_printenv flashargs flashargs_1
flashargs=setenv bootargs board=NBG6817 root=/dev/mmcblk0p5 rootwait ${bootmode} ${zld_ver}
flashargs_1=setenv bootargs board=NBG6817 root=/dev/mmcblk0p8 rootwait ${bootmode} ${zld_ver}

The first variable is responsible for booting from the main partition (FF), the second for booting from the alternate partition (01)
change the first variable:

root@OpenWrt:/tmp# fw_setenv flashargs 'setenv bootargs board=NBG6817 root=/dev/mmcblk0p10 rootwait ${bootmode} ${zld_ver}'

checking:

root@OpenWrt:/tmp# fw_printenv flashargs flashargs_1
flashargs=setenv bootargs board=NBG6817 root=/dev/mmcblk0p10 rootwait ${bootmode} ${zld_ver}
flashargs_1=setenv bootargs board=NBG6817 root=/dev/mmcblk0p8 rootwait ${bootmode} ${zld_ver}

Everything. Now you can reboot into the main partition and check the result:

root@OpenWrt:~# df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/root                 6.5M      6.5M         0 100% /rom
tmpfs                   232.6M    264.0K    232.4M   0% /tmp
/dev/loop0                3.1G    104.0K      2.9G   0% /overlay
overlayfs:/overlay        3.1G    104.0K      2.9G   0% /
tmpfs                   512.0K         0    512.0K   0% /dev

And no block-mount, blockd are no longer needed ;)
Sorry for my english, google translate.

@lu-zero
Copy link

lu-zero commented Dec 26, 2022

I do not see a github mirror of fstools, I stumbled upon this issue today while trying to configure extroot

@lu-zero
Copy link

lu-zero commented Dec 26, 2022

I started digging
I applied

diff --git a/block.c b/block.c
index 4b45200..ff2a433 100644
--- a/block.c
+++ b/block.c
@@ -1479,17 +1479,17 @@ static int check_extroot(char *path)
                err = find_block_ubi_RO(libubi, "rootfs", devpath, sizeof(devpath));
                libubi_close(libubi);
        }
 #endif
        if (err) {
                err = find_root_dev(devpath, sizeof(devpath));
        }
        if (err) {
-               ULOG_ERR("extroot: unable to determine root device\n");
+               ULOG_ERR("extroot: unable to determine root device %s\n", devpath);
                return -1;
        }

And I got:

[ 5693.351056] block: attempting to load /etc/config/fstab
[ 5693.360249] F2FS-fs (mmcblk0p7): Found nat_bits in checkpoint
[ 5693.383255] F2FS-fs (mmcblk0p7): Mounted with checkpoint version = 75800596
[ 5693.392275] block: extroot: unable to determine root device /dev/pts ***

my /etc/config/fstab

config global
        option anon_swap '0'
        option anon_mount '0'
        option auto_swap '1'
        option auto_mount '1'
        option delay_root '5'
        option check_fs '0'

config mount
        option target '/mnt/mmcblk0p6'
        option uuid '28057978-e6ed-45cb-a549-6c55345b2bd6'
        option enabled '0'

config mount
        option target '/rom'
        option uuid '8a8ae03f-9a5174cd-6534c080-f8f29497'
        option enabled '0'

config mount
        option target '/overlay'
        option enabled '0'
        option uuid '40a18348-4dd2-4388-bbcb-a340fd0a963a'

config mount 'overlay'
        option uuid '40a18348-4dd2-4388-bbcb-a340fd0a963a'
        option target '/overlay'

my block info:

 block info
/dev/ubiblock0_2p1: UUID="d9be54b5-31f14e0c-915790f6-94e8904c" VERSION="4.0" TYPE="squashfs"
/dev/ubi0_5: UUID="6e9ddd50-3fa6-41a3-8341-c5060df51216" VERSION="w5r0" TYPE="ubifs"
/dev/mmcblk0p6: UUID="28057978-e6ed-45cb-a549-6c55345b2bd6" VERSION="1.15" TYPE="f2fs"
/dev/mmcblk0p65: UUID="8a8ae03f-9a5174cd-6534c080-f8f29497" VERSION="4.0" TYPE="squashfs"
/dev/mmcblk0p66: UUID="28217c12-37bc-47a2-8ac0-5e0744af0405" LABEL="rootfs_data" VERSION="1.15" TYPE="f2fs"
/dev/mmcblk0p7: UUID="40a18348-4dd2-4388-bbcb-a340fd0a963a" LABEL="extroot" VERSION="1.15" TYPE="f2fs"

@dangowrt
Copy link
Member

dangowrt commented Dec 26, 2022

Please try this

diff --git a/block.c b/block.c
index 009023d..d328da5 100644
--- a/block.c
+++ b/block.c
@@ -1471,7 +1471,7 @@ static int check_extroot(char *path)
        FILE *fp;
        int err;
 
-       err = find_block_mtd("\"rootfs\"", devpath, sizeof(devpath));
+       err = find_root_dev(devpath, sizeof(devpath));
 #ifdef UBIFS_EXTROOT
        if (err) {
                libubi_t libubi;
@@ -1482,8 +1482,7 @@ static int check_extroot(char *path)
        }
 #endif
-        if (err) {
+        if (err)
-               err = find_root_dev(devpath, sizeof(devpath));
-       }
+               err = find_block_mtd("\"rootfs\"", devpath, sizeof(devpath));
        if (err) {
                ULOG_ERR("extroot: unable to determine root device\n");
                return -1;

@lu-zero
Copy link

lu-zero commented Dec 26, 2022

the problem remains the same.

@lu-zero
Copy link

lu-zero commented Dec 26, 2022

block extroot should look at the gpt partition with extroot or it should mount its base overlay and then read the fstab present there. but if the /overlay is mounted on / then doing stat would produce the strange result, I'm afraid.

@dangowrt
Copy link
Member

So let's improve the behavior of find_root_dev() then, e.g.:

diff --git a/block.c b/block.c
index 009023d..09755c4 100644
--- a/block.c
+++ b/block.c
@@ -1404,25 +1404,28 @@ static int find_root_dev(char *buf, int len)
 	dev_t root;
 	struct stat s;
 	struct dirent *e;
-
-	if (stat("/", &s))
-		return -1;
+	int overlay = 0;
 
 	if (!(d = opendir("/dev")))
 		return -1;
 
-	root = s.st_dev;
+	while (overlay < 2) {
+		if (stat(overlay++?"/rom":"/", &s))
+			continue;
 
-	while ((e = readdir(d)) != NULL) {
-		snprintf(buf, len, "/dev/%s", e->d_name);
+		root = s.st_dev;
+		while ((e = readdir(d)) != NULL) {
+			snprintf(buf, len, "/dev/%s", e->d_name);
 
-		if (stat(buf, &s) || s.st_rdev != root)
-			continue;
+			if (stat(buf, &s) || s.st_rdev != root)
+				continue;
 
-		closedir(d);
-		return 0;
-	}
+			closedir(d);
+			return 0;
+		}
 
+		rewinddir(d);
+	}
 	closedir(d);
 	return -1;
 }
@@ -1471,7 +1474,7 @@ static int check_extroot(char *path)
 	FILE *fp;
 	int err;
 
-	err = find_block_mtd("\"rootfs\"", devpath, sizeof(devpath));
+	err = find_root_dev(devpath, sizeof(devpath));
 #ifdef UBIFS_EXTROOT
 	if (err) {
 		libubi_t libubi;
@@ -1482,8 +1485,7 @@ static int check_extroot(char *path)
 	}
 #endif
 	if (err) {
-		err = find_root_dev(devpath, sizeof(devpath));
-	}
+		err = find_block_mtd("\"rootfs\"", devpath, sizeof(devpath));
 	if (err) {
 		ULOG_ERR("extroot: unable to determine root device\n");
 		return -1;

@lu-zero
Copy link

lu-zero commented Dec 26, 2022

It fails on boot in a different way, but from a clean reboot issuing PREINIT=1 block extroot completes correctly.

@lu-zero
Copy link

lu-zero commented Dec 26, 2022

After completing the overlay is still wrong.

@lu-zero
Copy link

lu-zero commented Dec 26, 2022

On boot:

[    8.815688] F2FS-fs (mmcblk0p66): Mounted with checkpoint version = 6147d103
[    8.824595] mount_root: loading kmods from internal overlay
[    8.833702] kmodloader: loading kernel modules from /tmp/overlay/upper/etc/modules-boot.d/*
[    8.842969] kmodloader: done loading kernel modules from /tmp/overlay/upper/etc/modules-boot.d/*
[    9.987280] random: crng init done
[   10.313806] UBIFS (ubi0:5): Mounting in unauthenticated mode
[   10.319538] UBIFS (ubi0:5): background thread "ubifs_bgt0_5" started, PID 667
[   10.787701] UBIFS (ubi0:5): UBIFS: mounted UBI device 0, volume 5, name "rootfs_data"
[   10.795515] UBIFS (ubi0:5): LEB size: 126976 bytes (124 KiB), min./max. I/O unit sizes: 2048 bytes/2048 bytes
[   10.805411] UBIFS (ubi0:5): FS size: 88883200 bytes (84 MiB, 700 LEBs), max 711 LEBs, journal size 4444160 bytes (4 MiB, 35 LEBs)
[   10.817036] UBIFS (ubi0:5): reserved for root: 4198173 bytes (4099 KiB)
[   10.823633] UBIFS (ubi0:5): media format: w5/r0 (latest is w5/r0), UUID 6E9DDD50-3FA6-41A3-8341-C5060DF51216, small LPT model
[   10.842466] block: attempting to load /tmp/ubifs_cfg/upper/etc/config/fstab
[   10.867877] block: unable to load configuration (fstab: Entry not found)
[   10.874585] block: attempting to load /tmp/ubifs_cfg/etc/config/fstab
[   10.881052] block: unable to load configuration (fstab: Entry not found)
[   10.887751] block: attempting to load /etc/config/fstab
[   10.893001] block: unable to load configuration (fstab: Entry not found)
[   10.899700] block: no usable configuration
[   10.914512] UBIFS (ubi0:5): un-mount UBI device 0
[   10.919220] UBIFS (ubi0:5): background thread "ubifs_bgt0_5" stops
[   10.931202] mount_root: switching to f2fs overlay
[   10.937393] overlayfs: null uuid detected in lower fs '/', falling back to xino=off,index=off,nfs_export=off.

post boot:

root@OpenWrt:/# PREINIT=1 block extroot
[  196.975863] UBIFS (ubi0:5): Mounting in unauthenticated mode
[  196.981593] UBIFS (ubi0:5): background thread "ubifs_bgt0_5" started, PID 2674
[  197.449569] UBIFS (ubi0:5): UBIFS: mounted UBI device 0, volume 5, name "rootfs_data"
[  197.457392] UBIFS (ubi0:5): LEB size: 126976 bytes (124 KiB), min./max. I/O unit sizes: 2048 bytes/2048 bytes
[  197.467287] UBIFS (ubi0:5): FS size: 88883200 bytes (84 MiB, 700 LEBs), max 711 LEBs, journal size 4444160 bytes (4 MiB, 35 LEBs)
[  197.478914] UBIFS (ubi0:5): reserved for root: 4198173 bytes (4099 KiB)
[  197.485512] UBIFS (ubi0:5): media format: w5/r0 (latest is w5/r0), UUID 6E9DDD50-3FA6-41A3-8341-C5060DF51216, small LPT model
[  197.504316] block: attempting to load /tmp/ubifs_cfg/upper/etc/config/fstab
[  197.529714] block: unable to load configuration (fstab: Entry not found)
[  197.536435] block: attempting to load /tmp/ubifs_cfg/etc/config/fstab
[  197.542891] block: unable to load configuration (fstab: Entry not found)
[  197.549596] block: attempting to load /etc/config/fstab
[  197.566449] UBIFS (ubi0:5): un-mount UBI device 0
[  197.571157] UBIFS (ubi0:5): background thread "ubifs_bgt0_5" stops

This is what we get:

root@OpenWrt:/# mount
/dev/root on /rom type squashfs (ro,relatime,errors=continue)
proc on /proc type proc (rw,nosuid,nodev,noexec,noatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,noatime)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,noatime)
/dev/mmcblk0p66 on /overlay type f2fs (rw,lazytime,noatime,background_gc=on,discard,no_heap,user_xattr,inline_xattr,inline_data,inline_dentry,flush_merge,extent_cache,mode=adaptive,active_logs=6,alloc_mode=reuse,checkpoint_merge,fsync_mode=posix,discard_unit=block)
overlayfs:/overlay on / type overlay (rw,noatime,lowerdir=/,upperdir=/overlay/upper,workdir=/overlay/work,xino=off)
tmpfs on /dev type tmpfs (rw,nosuid,noexec,noatime,size=512k,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,noatime,mode=600,ptmxmode=000)
debugfs on /sys/kernel/debug type debugfs (rw,noatime)
bpffs on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,noatime,mode=700)
pstore on /sys/fs/pstore type pstore (rw,noatime)
/dev/mmcblk0p7 on /mnt type f2fs (rw,lazytime,relatime,background_gc=on,discard,no_heap,user_xattr,inline_xattr,inline_data,inline_dentry,flush_merge,extent_cache,mode=adaptive,active_logs=6,alloc_mode=reuse,checkpoint_merge,fsync_mode=posix,discard_unit=block)
/dev/mmcblk0p7 on /tmp/extroot/overlay type f2fs (rw,lazytime,relatime,background_gc=on,discard,no_heap,user_xattr,inline_xattr,inline_data,inline_dentry,flush_merge,extent_cache,mode=adaptive,active_logs=6,alloc_mode=reuse,checkpoint_merge,fsync_mode=posix,discard_unit=block)

@lu-zero
Copy link

lu-zero commented Dec 27, 2022

This fixes mounting at boot

diff --git a/block.c b/block.c
index f2fab5b..f520419 100644
--- a/block.c
+++ b/block.c
@@ -1647,16 +1647,21 @@ static int main_extroot(int argc, char **argv)
        }

        make_devs();
        cache_load(1);

        /* enable LOG_INFO messages */
        ulog_threshold(LOG_INFO);

+       /* try the currently mounted overlay if exists */
+       err = mount_extroot("/tmp/overlay");
+       if (!err)
+           return err;
+
        /*
         * Look for "rootfs_data". We will want to mount it and check for
         * extroot configuration.
         */

        /* Start with looking for MTD partition */
        find_block_mtd("\"rootfs_data\"", blkdev_path, sizeof(blkdev_path));
        if (blkdev_path[0]) {

@wjiec
Copy link

wjiec commented Dec 28, 2022

For me (OpenWrt 22.03.2)

ubus call system board
{
	"kernel": "5.10.146",
	"hostname": "OpenWrt",
	"system": "Intel(R) Core(TM) i7-4770HQ CPU @ 2.20GHz",
	"model": "innotek GmbH VirtualBox",
	"board_name": "innotek-gmbh-virtualbox",
	"rootfs_type": "squashfs",
	"release": {
		"distribution": "OpenWrt",
		"version": "22.03.2",
		"revision": "r19803-9a599fee93",
		"target": "x86/64",
		"description": "OpenWrt 22.03.2 r19803-9a599fee93"
	}
}

applying the patch below, recompiling and reinstalling block-mount and fstools can fix the bug.

--- a/block.c
+++ b/block.c
@@ -1697,7 +1697,7 @@ static int main_extroot(int argc, char **argv)
 #endif

 	/* As a last resort look for /etc/config/fstab on "rootfs" partition */
-	return mount_extroot(NULL);
+	return mount_extroot("/tmp/overlay");
 }

 static int main_mount(int argc, char **argv)

The complete main_extroot function is shown below(only the last line of return has been modified)

/**
 * Look for extroot config and mount it if present
 *
 * Look for /etc/config/fstab on all supported partitions and use it for
 * mounting extroot if specified.
 */
static int main_extroot(int argc, char **argv)
{
	struct probe_info *pr;
	char blkdev_path[32] = { 0 };
	int err = -1;
#ifdef UBIFS_EXTROOT
	libubi_t libubi;
#endif

	if (!getenv("PREINIT"))
		return -1;

	if (argc != 2) {
		ULOG_ERR("Usage: block extroot\n");
		return -1;
	}

	make_devs();
	cache_load(1);

	/* enable LOG_INFO messages */
	ulog_threshold(LOG_INFO);

	/*
	 * Look for "rootfs_data". We will want to mount it and check for
	 * extroot configuration.
	 */

	/* Start with looking for MTD partition */
	find_block_mtd("\"rootfs_data\"", blkdev_path, sizeof(blkdev_path));
	if (blkdev_path[0]) {
		pr = find_block_info(NULL, NULL, blkdev_path);
		if (pr && !strcmp(pr->type, "jffs2")) {
			char cfg[] = "/tmp/jffs_cfg";

			/*
			 * Mount MTD part and try extroot (using
			 * /etc/config/fstab from that partition)
			 */
			mkdir_p(cfg, 0755);
			if (!mount(blkdev_path, cfg, "jffs2", MS_NOATIME, NULL)) {
				err = mount_extroot(cfg);
				umount2(cfg, MNT_DETACH);
			}
			if (err < 0)
				rmdir("/tmp/overlay");
			rmdir(cfg);
			return err;
		}
	}

#ifdef UBIFS_EXTROOT
	/* ... but it also could be an UBI volume */
	memset(blkdev_path, 0, sizeof(blkdev_path));
	libubi = libubi_open();
	find_block_ubi(libubi, "rootfs_data", blkdev_path, sizeof(blkdev_path));
	libubi_close(libubi);
	if (blkdev_path[0]) {
		char cfg[] = "/tmp/ubifs_cfg";

		/* Mount volume and try extroot (using fstab from that vol) */
		mkdir_p(cfg, 0755);
		if (!mount(blkdev_path, cfg, "ubifs", MS_NOATIME, NULL)) {
			err = mount_extroot(cfg);
			umount2(cfg, MNT_DETACH);
		}
		if (err < 0)
			rmdir("/tmp/overlay");
		rmdir(cfg);
		return err;
       }
#endif

	/* As a last resort look for /etc/config/fstab on "rootfs" partition */
	return mount_extroot("/tmp/overlay");
}

The full steps are as follows

mkdir -p package/system/fstools/patches

cat > package/system/fstools/patches/fix-squashfs-extroot.patch <<EOF
--- a/block.c
+++ b/block.c
@@ -1697,7 +1697,7 @@ static int main_extroot(int argc, char **argv)
 #endif

 	/* As a last resort look for /etc/config/fstab on "rootfs" partition */
-	return mount_extroot(NULL);
+	return mount_extroot("/tmp/overlay");
 }

 static int main_mount(int argc, char **argv)
EOF

make package/system/fstools/compile

final, reinstall the block-mount and fstools in bin/targets/x86/64/packages (additional library file dependencies are located at bin/packages/x86_64/base)

or you can also use my compiled ipk file (install the lib*.ipk first) block-mount.zip

@lu-zero
Copy link

lu-zero commented Dec 28, 2022

You have to try it first otherwise if you have multiple storage systems you exit earlier.

In general would probably nice to improve block extroot so it works even if the /overlay is already mounted but the changes required are more.

@dangowrt is it possible to have a mirror of fstools here on github? I couldn't find one so far.

@rtgiskard
Copy link

rtgiskard commented Dec 29, 2022

I have tried OpenWrt 22.03.2 x86 image with libvirt/kvm, the extroot to mount /overlay does not work for squashfs image, but work for ext4 image.

The ext4 image boot without overlay mount by default, you can add /overlay mount entry in /etc/config/fstab, after reboot you will have overlay mounted root. This might be a better choice for x86 machine(not knowning why, works for the vm, but not for another x86 machine, sdx vs vdx?)

@lu-zero
Copy link

lu-zero commented Jan 3, 2023

I stored the patches discussed here in this branch, https://github.com/domo-iot/fstools/tree/extroot-squashfs I hope it helps.

@lu-zero
Copy link

lu-zero commented Jan 18, 2023

Potentially the situation could be improved further by setting the lower option to merge in the starting overlay instead of copying it.

@taylor-cz
Copy link

taylor-cz commented Mar 13, 2023

Is this gonna be included in the next release, pls?
I'm struggling with this issue on Turris Omnia router (it mounts /overlay from /dev/loop0 too), having release 22.03.2

@taylor-cz
Copy link

Is there any chance to fix it without building custom installation image?

@dangowrt
Copy link
Member

Is there any chance to fix it without building custom installation image?

Of course. All it takes is a pull-request (in compliance with our submission guideline) against openwrt/fstools:

https://github.com/openwrt/fstools/pulls

@lu-zero
Copy link

lu-zero commented Mar 19, 2023

I couldn't find the repository somehow, let me do it now :)

@taylor-cz
Copy link

@lu-zero that is awesome, you deserve a bottle ;)

@taylor-cz
Copy link

I couldn't find the repository somehow, let me do it now :)

Can I help somehow?

@lu-zero
Copy link

lu-zero commented Mar 30, 2023

openwrt/fstools#1 if you want to test it.

@0sma
Copy link

0sma commented Apr 2, 2023

@lu-zero Thanks, it works well with the latest snapshot/master version.

@dsmatilla
Copy link

@dangowrt Do you have any information about when this fix will be released that you can share with us? thanks!

@psychowood
Copy link

I'd say I see very similar behaviour for my Zyxel Armor G5 (NBG7815), I can't mount extroot outside the loop0 defaults:

loop0: detected capacity change from 0 to 122880
loop0: detected capacity change from 122880 to 108928
EXT4-fs (loop0): recovery complete
EXT4-fs (loop0): mounted filesystem with ordered data mode. Opts: (null). Quota mode: disabled.
mount_root: loading kmods from internal overlay
kmodloader: loading kernel modules from /tmp/overlay/upper/etc/modules-boot.d/*
kmodloader: done loading kernel modules from /tmp/overlay/upper/etc/modules-boot.d/*
block: attempting to load /etc/config/fstab
block: unable to load configuration (fstab: Entry not found)
block: no usable configuration
mount_root: switching to ext4 overlay
overlayfs: null uuid detected in lower fs '/', falling back to xino=off,index=off,nfs_export=off.

Looking eagerly for the merge :)

@MaxSKash how does your solution/workaround behave with sysupgrade? Shoud it be reapplied at each upgrade?

@psychowood
Copy link

psychowood commented May 31, 2023

Following my previous comment, I applied @lu-zero patch to my Zyxel Armor G5 (NBG7815) image (SNAPSHOT, r23135-7068a0737a). After a factory reset (I probably messed up something on the first try) everything is working as expected.

@lu-zero
Copy link

lu-zero commented May 31, 2023

If somebody has a good idea on how to remount overlayfs the whole behavior could be improved otherwise enabling extroot always require a reboot

@psychowood
Copy link

AFAIK it is not possible to do it without a reboot, it must be done during the preinit stage

psychowood added a commit to psychowood/openwrt-armor-g5-nbg7815 that referenced this issue Jun 1, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests