FS#2231 - fstools: PREINIT calling of block extroot doesn't acknowledge non-MTD rootfs overlays

[openwrt-bot]

knodel:

Extroot overlay mount fails on my ZyXEL NBG6817.

I can confirm this bug on both the latest stable 18.06.2, and the snapshot from the 4th of April.

When PREINIT calls 'block extroot', block fails to load the custom fstab from the eMMC ext4 overlay mounted at /tmp/overlay:

[...]
Thu Apr  4 12:01:51 2019 user.info kernel: [    3.508150] init: - preinit -
Thu Apr  4 12:01:51 2019 kern.info kernel: [    8.592387] EXT4-fs (loop0): recovery complete
Thu Apr  4 12:01:51 2019 kern.info kernel: [    8.595484] EXT4-fs (loop0): mounted filesystem with ordered data mode. Opts: (null)
Thu Apr  4 12:01:51 2019 user.info kernel: [    8.598875] mount_root: loading kmods from internal overlay
Thu Apr  4 12:01:51 2019 user.info kernel: [    8.649260] kmodloader: loading kernel modules from /tmp/overlay/upper/etc/modules-boot.d/*
Thu Apr  4 12:01:51 2019 user.info kernel: [    8.663223] kmodloader: done loading kernel modules from /tmp/overlay/upper/etc/modules-boot.d/*
Thu Apr  4 12:01:51 2019 user.info kernel: [   10.119972] block: attempting to load /etc/config/fstab
Thu Apr  4 12:01:51 2019 user.err kernel: [   10.120211] block: unable to load configuration (fstab: Entry not found)
Thu Apr  4 12:01:51 2019 user.err kernel: [   10.124238] block: no usable configuration
Thu Apr  4 12:01:51 2019 user.info kernel: [   10.132103] mount_root: switching to ext4 overlay
Thu Apr  4 12:01:51 2019 user.info kernel: [   10.292110] procd: - early -
Thu Apr  4 12:01:51 2019 user.info kernel: [   10.292264] procd: - watchdog -
Thu Apr  4 12:01:51 2019 user.info kernel: [   10.988093] procd: - watchdog -
Thu Apr  4 12:01:51 2019 user.info kernel: [   10.988475] procd: - ubus -
Thu Apr  4 12:01:51 2019 user.info kernel: [   11.046160] procd: - init -
[...]

Unrelated dmesg entries omitted, full log here:
https://gist.github.com/knuddelknoedel/2985ce7777a0263fbc22a02f8ef5307c

Custom modules are loaded with the correct overlay /tmp/overlay/upper prefix by libfstools, however the forked 'block extroot' process behaves differently when searching for /etc/config/fstab configuration.

Steps to reproduce:

• Flash any current openwrt sysupgrade image on a device where the rootfs+overlay don't reside on MTD storage
• Configure an appropriate /overlay uci fstab extroot entry as specified in the respective wiki documentation
• Reboot

Further notes: adding /etc/config/fstab with the desired /overlay entry to the sysupgrade squashfs image before flashing allows block to successfully find the uci fstab config, however the mounting of the therein configured /overlay mount still fails.

[openwrt-bot]

knodel:

I wrote a small patch that allows proper rootfs and fstab discovery on devices where the rootfs+overlay reside on a block device partition.

Here the dmesg of a current openwrt HEAD build with the patch:

Mon Apr  8 22:25:35 2019 user.info kernel: [    3.335289] init: - preinit -
Mon Apr  8 22:25:35 2019 user.info kernel: [    8.035073] mount_root: loading kmods from internal overlay
Mon Apr  8 22:25:35 2019 user.info kernel: [    8.046998] kmodloader: loading kernel modules from //etc/modules-boot.d/*
Mon Apr  8 22:25:35 2019 user.info kernel: [    8.048462] kmodloader: done loading kernel modules from //etc/modules-boot.d/*
Mon Apr  8 22:25:35 2019 user.info kernel: [    8.828623] block: attempting to load /tmp/overlay/upper/etc/config/fstab
Mon Apr  8 22:25:35 2019 user.err kernel: [    8.828798] block: unable to load configuration (fstab: Entry not found)
Mon Apr  8 22:25:35 2019 user.info kernel: [    8.834600] block: attempting to load /tmp/overlay/etc/config/fstab
Mon Apr  8 22:25:35 2019 user.err kernel: [    8.841281] block: unable to load configuration (fstab: Entry not found)
Mon Apr  8 22:25:35 2019 user.info kernel: [    8.847486] block: attempting to load /etc/config/fstab
Mon Apr  8 22:25:35 2019 user.err kernel: [    8.854319] block: unable to load configuration (fstab: Entry not found)
Mon Apr  8 22:25:35 2019 user.err kernel: [    8.859098] block: no usable configuration
Mon Apr  8 22:25:35 2019 kern.info kernel: [    8.985003] EXT4-fs (loop0): recovery complete
Mon Apr  8 22:25:35 2019 kern.info kernel: [    8.986207] EXT4-fs (loop0): mounted filesystem with ordered data mode. Opts: (null)
Mon Apr  8 22:25:35 2019 user.info kernel: [    8.988754] mount_root: loading kmods from internal overlay
Mon Apr  8 22:25:35 2019 user.info kernel: [    9.013356] kmodloader: loading kernel modules from /tmp/overlay/upper/etc/modules-boot.d/*
Mon Apr  8 22:25:35 2019 user.info kernel: [    9.013492] kmodloader: done loading kernel modules from /tmp/overlay/upper/etc/modules-boot.d/*
Mon Apr  8 22:25:35 2019 user.info kernel: [   10.011824] block: attempting to load /tmp/overlay/upper/etc/config/fstab
Mon Apr  8 22:25:35 2019 kern.info kernel: [   10.017872] EXT4-fs (mmcblk0p10): mounted filesystem with ordered data mode. Opts:
Mon Apr  8 22:25:35 2019 user.info kernel: [   10.162997] mount_root: switched to extroot
Mon Apr  8 22:25:35 2019 user.info kernel: [   10.331230] procd: - early -
Mon Apr  8 22:25:35 2019 user.info kernel: [   10.331412] procd: - watchdog -
Mon Apr  8 22:25:35 2019 user.info kernel: [   11.009306] procd: - watchdog -
Mon Apr  8 22:25:35 2019 user.info kernel: [   11.009775] procd: - ubus -
Mon Apr  8 22:25:35 2019 user.info kernel: [   11.064866] procd: - init -

Unrelated dmesg entries omitted, full log here: https://gist.github.com/knuddelknoedel/46db572959056d2e69fd861d59ad0daf

[openwrt-bot]

val-kulkov:

The proposed patch works beautifully to enable extroot on my x86_64 device. With it, I can finally make use of the available space on my /dev/sda without worrying about losing data on it upon a sysupgrade.

The sysupgrade process on a x86_64 system that I use to avoid losing data is not yet a straightforward one. First, I burn openwrt-x86-64-combined-squashfs.img on a USB drive. Then, I copy first two partitions from the USB drive to the corresponding partitions on my /dev/sda. Finally, I copy the MBR bootstrap only, without the partition table: "dd if=/dev/sdc of=/dev/sda bs=446 count=1". This procedure retains sda3, my root overlay partition, and sda4, a swap partition.

[openwrt-bot]

utrumo:

Now on Zyxel Armor Z2 (NBG6817) extroot doesn't work without this patch.
Is there a developer of openwrt? Please add this patch to master! :)

To make work extroot on this router we need compile own firmware and apply this patch:

1. clone src and change dir to it: git clone https://github.com/openwrt/openwrt.git ~/source && cd ~/source
2. make directory for patch: mkdir ./package/system/fstools/patches
3. write patch content from https://patchwork.ozlabs.org/patch/1082599/ to ./package/system/fstools/patches/001-add_propper_rootfs_and_fstab_discovery_on_a_block_device_partitions.patch
4. build own image:
   ./scripts/feeds update -a
   ./scripts/feeds install -a
   export MAKEFLAGS=-j5
   make menuconfig
   ===================================================
   system => Qualcomm Atheros IPQ806X y
   Target Profile => Zyxel NBG6817 y
   Base system =>
   => block-mount y (for extroot on mmcblk0p10)
   => blockd y (for extroot on mmcblk0p10)
   Kernel modules => USB Support => kmod-usb-storage y (for usb-hdd support)
   LuCI =>
   => Collections => luci y
   => Applications => luci-app-advanced-reboot y
   ===================================================
   make defconfig
   make download
   ionice -c 3 nice -n19 make

results you can find in:
./bin/targets/ipq806x/generic/

5. flash this firmware to router
   scp ./bin/targets/ipq806x/generic/openwrt-ipq806x-zyxel_nbg6817-squashfs-sysupgrade.bin root@openwrt:/tmp
   ssh root@openwrt
   sysupgrade -v /tmp/openwrt-ipq806x-zyxel_nbg6817-squashfs-sysupgrade.bin

6. set extroot:
   ssh root@openwrt
   DEVICE="$(awk -e '/\s/overlay\s/{print $1}' /etc/mtab)"
   uci -q delete fstab.rwm
   uci set fstab.rwm="mount"
   uci set fstab.rwm.device="${DEVICE}"
   uci set fstab.rwm.target="/rwm"
   uci set fstab.rwm.enabled="1"
   uci commit fstab

DEVICE="/dev/mmcblk0p10" // or if you want to use usb-hdd: DEVICE="/dev/sda1"
mkfs.ext4 "${DEVICE}"
eval $(block info "${DEVICE}" | grep -o -e "UUID=\S*")
uci -q delete fstab.overlay
uci set fstab.overlay="mount"
uci set fstab.overlay.uuid="${UUID}"
uci set fstab.overlay.target="/overlay"
uci set fstab.overlay.enabled="1"
uci commit fstab

mount "${DEVICE}" /mnt
cp -a -f /overlay/. /mnt
umount /mnt

reboot

[openwrt-bot]

rmilecki:

That fstools patch cannot be applied as is (I reviewed & commented on it).

I pushed three related fstools changes:

• [[https://git.openwrt.org/?p=project/fstools.git;a=commitdiff;h=d70774de24413f91087e9c3c873cf7a4ec3df813|d70774de2441 ("block: add some basic extroot documentation")]]
• [[https://git.openwrt.org/?p=project/fstools.git;a=commitdiff;h=37c9148a3ae17665e75f99b459b6ff2b461d6739|37c9148a3ae1 ("block: simplify check_extroot() a bit")]]
• [[https://git.openwrt.org/?p=project/fstools.git;a=commitdiff;h=8b9e601160712f74ff46313579d0225a7f90ca3c|8b9e60116071 ("block: always use st_dev (device ID) of / when looking for root")]]

That moves us closer to fixing this issue.

Someone now needs to improve main_extroot() and make it:

• Find standard overlay device (e.g. /dev/sda3)
• Mount it using f2fs
• Call mount_extroot() for it

It's basically the same logic as what we already have for JFFS2 and UBIFS in that main_extroot() function.

[openwrt-bot]

leifliddy:

If we're going to use f2fs as the filesystem for the extroot overlay device (which is a great idea), then we should probably include these package as part of the base build for the zyxel_nbg6817:
f2fsck
f2fs-tools
kmod-fs-f2fs
mkf2fs

Actually, this brings up another question, why aren't we using the f2fs filesystem for the default overlay device?

/dev/loop0 on /overlay type ext4 (rw,noatime,data=ordered)

[openwrt-bot]

rmilecki:

I just noticed that orignal report was about ext4. Recently I was chatting with m4t who was using f2fs.

So this task seems more generic: we need to check /dev/loop0 for /etc/config/fstab with overlay setup. That /dev/loop0 may be ext4 or f2fs (or other?).

[openwrt-bot]

leifliddy:

I'm not sure if this is the correct forum for this or not, but here goes.
If you look at partition schedule for the nbg6817

https://openwrt.org/toh/zyxel/nbg6817

You'll see that mmcblk0p5 is divided up into two parts

mmcblk0p5 rootfs 64 MiB

/rom (squashfs) 4MB
/overlay (ext4) 60MB (/dev/loop0)

So, if you've configured mmcblk0p10 to act as the /overlay,
then the 4MB squashfs filesystem from mmcblk0p5 will continue to act as the 4Mb read-only filesystem?

Is there a reason we can't just re-partition this entire device, I mean there's no reason for openwrt to need 10 partitions.
Or is there some sort of unspoken rule that you need to operate within the confines of the OEM flash layout?

If we need to maintain the OEM partitioning scheme, then we should make mmcblk0p1 the 4MB /rom partition
...and make mmcblk0p10 the default /overlay partition.

That's makes the most sense...but I'll save this discussion for another day. Need to do a bit more research on how openwrt is designed....

[openwrt-bot]

leifliddy:

@ken would you be able to create a new patch based on the changes Rafał made to fstools?

[openwrt-bot]

tofurky:

leifliddy, this works for me with f2fs on 19.07.3. note that the change to check_filesystem() isn't strictly necessary, but i found that newer f2fs-tools gave some issues without it.

diff --git a/block.c b/block.c
index 569bf56..b88e208 100644
--- a/block.c
+++ b/block.c
@@ -747,7 +747,7 @@ static void check_filesystem(struct probe_info *pr)
        pid = fork();
        if (!pid) {
                if(!strncmp(pr->type, "f2fs", 4)) {
-                       execl(ckfs, ckfs, "-f", pr->dev, NULL);
+                       execl(ckfs, ckfs, "-p", "2", "-f", pr->dev, NULL);
                        exit(EXIT_FAILURE);
                } else if(!strncmp(pr->type, "btrfs", 5)) {
                        execl(ckfs, ckfs, "--repair", pr->dev, NULL);
@@ -1591,7 +1591,7 @@ static int main_extroot(int argc, char **argv)
 #endif
 
        /* As a last resort look for /etc/config/fstab on "rootfs" partition */
-       return mount_extroot(NULL);
+       return mount_extroot("/tmp/overlay");
 }
 
 static int main_mount(int argc, char **argv)

i spent a bit of time trying to figure out the proper way to do it per rafal's suggestion, but was running into segfaults with find_block(NULL, NULL, "loop0", NULL) so i just took the easy way out ;) not sure if i was even close to taking the right approach with that, though.

[openwrt-bot]

katrat:

Hi,

I guess this fix wasn't included in the 19.07.3 release yet, since I am struggling to set up overlayfs on my new NBG6817 with usb hard disk and ext4.

Are there any patched images to download/test? Any chance this will be fixed in one of the next releases?

I am not that experienced to build/figure out this myself, so any links/instructions are much appreciated. On the other hand, my zyxel isn't productive yet, so I am willing to test...

Thanks,
kat

[openwrt-bot]

rmilecki:

Noone cared to develop a proper patch, people keep posting some hacks that are unacceptable due to breaking other setups.

[openwrt-bot]

katrat:

Thanks for the quick, if also a bit unsatisfying, answer. I got the zyxel because it seemed to be properly supported by openwrt (and having a bit more oomph than my ageing tp-link). I don't really understand where the problem is: ext4 of f2fs, hard disk or flash drive, or the drives not being ready at boot time, or just some configuration hiccup.

Can you suggest what I could do? I don't mind playing around a bit.

[openwrt-bot]

rmilecki:

Someone now needs to improve main_extroot() and make it:

1. Find standard overlay device (e.g. /dev/sda3)
2. Mount it using f2fs
3. Call mount_extroot() for it

It's basically the same logic as what we already have for JFFS2 and UBIFS in that main_extroot() function.

I just noticed that orignal report was about ext4. Recently I was chatting with m4t who was using f2fs.

So this task seems more generic: we need to check /dev/loop0 for /etc/config/fstab with overlay setup. That /dev/loop0 may be ext4 or f2fs (or other?).

[openwrt-bot]

visualage:

Is it reasonable to use libfstools to search for rootfs_data volume and mount it, just like what mount_root.c does?

[openwrt-bot]

visualage:

I made a patch to openwrt to support extroot on non-MTD rootfs_data volumes. It uses libfstools in block.c to support other squashfs volumes, such as /dev/sd* or /dev/mmc*.

The patch itself contains a patch to fstools project that can be applied to that codebase directly.

I tested the build locally on a x86-64 virtualbox environment as well as rockchip friendlyarm nanopi r2s. It should work on any other environments as long as mount_root works since they use the same underlying library.