build: config: image: add redesigned options for squashfs compression

[Spudz76]

This is an attempt at making SquashFS compression fully selectable, and enables all the initramfs compression options (previously selectable, but broken). I wanted to try LZ4 since it might boot a few seconds faster and my device has no shortage of flash space, but nobody wired up the options, and the host-build of mksquashfs4 was missing several compressors for no real reason (space not an issue on the compilation host).

• Add tools/liblz4, tools/liblzo, tools/lzop for the host toolchain, set tools/squashfskit4 to use them all (and already existing zstd)
• Add config menus for overriding the default squashfs compression (unset follows board-default or otherwise behaves as it always did)
• Fix wiring for initramfs compression methods which were already in menuconfig but didn't really work

This is all untested past building and inspecting output on the compile-host ; awaiting serial cable before I can test it on real hardware for actually-boots status

Did not set myself as maintainer for the new tools packages since they are roughly based on the target packages of similar names. Also did not copy maintainer from those sources, but IMHO it should just be the same maintainer.

[mpratt14]

every single tool you add should be a separate commit

also the tools should not be always enabled by default, but rather use the config you are adding to select them

[Spudz76]

I will split commits as suggested, and I should have done so in the first place.

However zstd is always included, therefore what I followed with the other compression types. I will attempt to make them connect with the TARGET_(INITRAMFS|SQUASHFS)_COMPRESSION_* settings (but for consistency, will also map zstd...)

zlib and xz are also enabled always in squashfskit4 and always built, even if it is unused by options... another precident I was generally following. Those should also be set up to follow the selected options.

[mpratt14]

zlib and xz should for sure be always enabled, since other tools that are always needed are dependent on them somehow

I would just focus on the ones you're adding

[Spudz76]

zlib and xz should for sure be always enabled, since other tools that are always needed are dependent on them somehow

If anything is truly dependent, then its deps should be wired up in the tools/Makefile like everything else.

I have already stripped them and things still generally work, but I have not tested with a fully clean/virgin tree (maybe the pre-existing tools are already previously installed and leftover in my staging_dir).

[mpratt14]

I have already stripped them and things still generally work

careful with that concept

things can still "work" but that's not always the point, some tools are meant to be just "our copy" of whatever it is, and on many build systems you can install the same thing on the host machine, but that version of the tool differs between build systems. one of the purposes of the xz tool for example is version control to ensure every downloaded package that is an .xz image is unpacked the same way guaranteed and not subject to the differences between versions of xz provided by different OSes. Some host machines might have every tool installed already, but the build system is designed to be able to run on an almost completely "dry" system with only build-essential packages like git and make and autotools, etc. installed.

[hauke]

Could you provide some sizes for the different compression standards please.

Maybe we should switch the default from XZ to zstd. The compression ratio in zstd can be configured.

[Spudz76]

Could you provide some sizes for the different compression standards please.

Maybe we should switch the default from XZ to zstd. The compression ratio in zstd can be configured.

Yes, I collected stats from various build (all the same rootfs content)

Filesystem size 7725990 bytes (7544.91 Kbytes / 7.37 Mbytes) Compression xz Block size 1048576
Filesystem size 7863310 bytes (7679.01 Kbytes / 7.50 Mbytes) Compression xz Block size 524288
Filesystem size 7996442 bytes (7809.03 Kbytes / 7.63 Mbytes) Compression xz Block size 262144
Filesystem size 8180238 bytes (7988.51 Kbytes / 7.80 Mbytes) Compression xz Block size 131072
Filesystem size 8637166 bytes (8434.73 Kbytes / 8.24 Mbytes) Compression zstd Block size 1048576
Filesystem size 8795350 bytes (8589.21 Kbytes / 8.39 Mbytes) Compression zstd Block size 524288
Filesystem size 8924443 bytes (8715.28 Kbytes / 8.51 Mbytes) Compression zstd Block size 262144
Filesystem size 9088083 bytes (8875.08 Kbytes / 8.67 Mbytes) Compression zstd Block size 131072
Filesystem size 9673987 bytes (9447.25 Kbytes / 9.23 Mbytes) Compression gzip Block size 1048576
Filesystem size 9679301 bytes (9452.44 Kbytes / 9.23 Mbytes) Compression gzip Block size 524288
Filesystem size 9696926 bytes (9469.65 Kbytes / 9.25 Mbytes) Compression gzip Block size 262144
Filesystem size 9735833 bytes (9507.65 Kbytes / 9.28 Mbytes) Compression gzip Block size 131072
Filesystem size 10532724 bytes (10285.86 Kbytes / 10.04 Mbytes) Compression lzo Block size 1048576
Filesystem size 10548119 bytes (10300.90 Kbytes / 10.06 Mbytes) Compression lzo Block size 524288
Filesystem size 10587339 bytes (10339.20 Kbytes / 10.10 Mbytes) Compression lzo Block size 262144
Filesystem size 10670590 bytes (10420.50 Kbytes / 10.18 Mbytes) Compression lzo Block size 131072
Filesystem size 11170138 bytes (10908.34 Kbytes / 10.65 Mbytes) Compression lz4 Block size 1048576
Filesystem size 11203138 bytes (10940.56 Kbytes / 10.68 Mbytes) Compression lz4 Block size 524288
Filesystem size 11276815 bytes (11012.51 Kbytes / 10.75 Mbytes) Compression lz4 Block size 262144
Filesystem size 11435621 bytes (11167.60 Kbytes / 10.91 Mbytes) Compression lz4 Block size 131072
Filesystem size 13879944 bytes (13554.63 Kbytes / 13.24 Mbytes) Compression lz4 Block size 4096

And, I collected live performance stats for some of the same builds (linksys EA7300v1 / ramips-mt7261):

target.lz4.0004:root@test:~# sync; echo 3 > /proc/sys/vm/drop_caches; time /bin/sh -c "find /rom -type f | sort | xargs -n1 cat > /dev/null"
target.lz4.0004:real    0m 12.55s
target.lz4.0004:user    0m 5.26s
target.lz4.0004:sys     0m 3.12s
target.lz4.0128:root@test:~# sync; echo 3 > /proc/sys/vm/drop_caches; time /bin/sh -c "find /rom -type f | sort | xargs -n1 cat > /dev/null"
target.lz4.0128:real    0m 14.02s
target.lz4.0128:user    0m 5.13s
target.lz4.0128:sys     0m 6.69s
target.lz4.0256:root@test:~# sync; echo 3 > /proc/sys/vm/drop_caches; time /bin/sh -c "find /rom -type f | sort | xargs -n1 cat > /dev/null"
target.lz4.0256:real    0m 11.50s
target.lz4.0256:user    0m 5.10s
target.lz4.0256:sys     0m 3.39s
target.lz4.0512:root@test:~# sync; echo 3 > /proc/sys/vm/drop_caches; time /bin/sh -c "find /rom -type f | sort | xargs -n1 cat > /dev/null"
target.lz4.0512:real    0m 11.61s
target.lz4.0512:user    0m 5.16s
target.lz4.0512:sys     0m 3.39s
target.lz4.1024:root@test:~# sync; echo 3 > /proc/sys/vm/drop_caches; time /bin/sh -c "find /rom -type f | sort | xargs -n1 cat > /dev/null"
target.lz4.1024:real    0m 11.09s
target.lz4.1024:user    0m 5.16s
target.lz4.1024:sys     0m 3.23s
target.xz.0128:root@test:~# sync; echo 3 > /proc/sys/vm/drop_caches; time /bin/sh -c "find /rom -type f | sort | xargs -n1 cat > /dev/null"
target.xz.0128:real    0m 13.58s
target.xz.0128:user    0m 4.95s
target.xz.0128:sys     0m 6.38s
target.xz.0256:root@test:~# sync; echo 3 > /proc/sys/vm/drop_caches; time /bin/sh -c "find /rom -type f | sort | xargs -n1 cat > /dev/null"
target.xz.0256:real    0m 13.85s
target.xz.0256:user    0m 4.98s
target.xz.0256:sys     0m 6.43s
target.xz.0512:root@test:~# sync; echo 3 > /proc/sys/vm/drop_caches; time /bin/sh -c "find /rom -type f | sort | xargs -n1 cat > /dev/null"
target.xz.0512:real    0m 13.93s
target.xz.0512:user    0m 5.09s
target.xz.0512:sys     0m 6.50s
target.xz.1024:root@test:~# sync; echo 3 > /proc/sys/vm/drop_caches; time /bin/sh -c "find /rom -type f | sort | xargs -n1 cat > /dev/null"
target.xz.1024:real    0m 13.04s
target.xz.1024:user    0m 5.02s
target.xz.1024:sys     0m 6.11s
target.zstd.0512:root@test:~# sync; echo 3 > /proc/sys/vm/drop_caches; time /bin/sh -c "find /rom -type f | sort | xargs -n1 cat > /dev/null"
target.zstd.0512:real    0m 11.30s
target.zstd.0512:user    0m 5.06s
target.zstd.0512:sys     0m 3.78s
target.zstd.1024:root@test:~# sync; echo 3 > /proc/sys/vm/drop_caches; time /bin/sh -c "find /rom -type f | sort | xargs -n1 cat > /dev/null"
target.zstd.1024:real    0m 10.86s
target.zstd.1024:user    0m 5.09s
target.zstd.1024:sys     0m 3.68s

ZSTD is indeed the winner overall (slightly larger than XZ but way faster and less CPU load), but LZ4 can be really fast and lowest CPU load, if you don't care as much about space.

LZMA1 support should also be fully dropped, and squashfskit4 is abandoned while the real squashfs-tools is much newer (and has dropped LZMA1 and fixed a bunch of other things since squashfskit4 forked from it...)

[Spudz76]

Also, force-pushed my current work, unclassified work in progress lumped in WIP

[hauke]

There is work ongoing in #10967 to switch back to squashfs-tools.
I think you can configure the compression ratio in zstd. did you experiment with that?

[Spudz76]

Rebased to current master and fixed a build error and a missing autoload in the lz4 kmod changes.

Verified it now builds from a clean kernel tree, and the lz4hc compressor kmod exists and gets loaded.

Currently reorganizing the menuconfig stuff since I don't like how it works yet (and it needs a setting for which of the enabled compression methods is actually used in the image, currently it just uses whichever the last-enabled one is).

Also spent some time investigating the squashfskit4 situation, and some work on making a compressor-agnostic loader so that we might choose zstd-loader or others rather than just good old lzma-loader and save like 200ms on booting.

Should I set this to Draft and does that block the constant re-testing cycles (that probably will crash anyway until this is polished)?

[dangowrt]

Please write commit descriptions for each commit, not just titlte and SoB (just in case you were not planning to still do that).
Apart from that and changes to tools/Makefile being in the wrong commit it already looks very nice up to this point.
Thank you for working on this and contributing!

[Spudz76]

improved commit descriptions, picked most of those oneliner cherries into the appropriate atomic commits; thank you for the tips/pointers/direction

the remaining lines have to be added to the matching commits without the BUILD_LZ*_TOOLS since those don't exist until a later commit, but agree they should get built when all-host-tools is active as of the package commit (even though literally nothing is wired up to depend on them until later)

[Spudz76]

I believe this single commit c9d2056 is complete and could be merged, can that occur from here or should I make a totally separate branch and PR for it atomically?

Made a separate PR#11358 for semi-unrelated, completely separable, and unused-here repair that seems to be unlikely to change in this PR.

[tapper82]

This looks interesting.

[Spudz76]

Did a whole bunch of cleanup and commit-splitting, now it actually looks almost good, and much less noise with unrelated things split out to:

• build: fix incomplete initramfs compression options #11381
• kernel: modules/lib-lz4: add lz4hc_compress #11358
• tools: fix Makefile sorting and logic for readability #11359
• build: images: squashfs: add help, fix description #11366
• build: config: cleanup impossible dependency logic #11367
• build: fix incomplete initramfs compression options #11381

[dkadioglu]

Any news regarding this?

[Spudz76]

I'll fast forward it shortly.

[Spudz76]

I forgot it's way worse than a fast-forward considering some other changes to squashfs4 since, which was why I backburnered it. Will take me longer, but at least I'm actually digging into it now.

[aparcar]

@Spudz76 hey could you please give this another go or if you can't, do you mind I look into rebasing (parts) of it? I'm specifically interested in ZSTD for x86 images, so this would come in handy! Thanks for your work!

[Spudz76]

@aparcar Finally got around to this. Still crashes on completely unrelated/untouched things in the tests for whatever mysterious reason. But still "Works For Me" (tm)

[Spudz76]

Still very confused as to why two platforms crash on:
zstd: can't stat /__w/openwrt/openwrt/openwrt/dl/squashfs4-4.6.1.tar.zst : No such file or directory -- ignored
...while everything else finds the source just fine.

Or even how to debug that.

[backslashxx]

works well. tried both lz4 and zstd on ipq60xx and ipq40xx, theres a bit of faster bootup benefit on my ipq40xx. good job.
no issues at all on lz4

EDIT:
when I tested zstd theres this weird error on my dmesg
on ipq40xx (23.05-SNAPSHOT / 5.15)

[    3.449494] UBIFS error (ubi0:1 pid 1): ubifs_read_node: bad magic 0x73717368, expected 0x6101831
[    3.449566] UBIFS error (ubi0:1 pid 1): ubifs_read_node: bad node at LEB 0:0
[    3.457442] Not a node, first 24 bytes:
[    3.457460] 00000000: 68 73 71 73 40 0d 00 00 3c bd 14 66 00 00 04 00 86 00 00 00 06 00 12 00                          hsqs@...<..f............
[    3.468089] CPU: 1 PID: 1 Comm: swapper/0 Not tainted 5.15.153 #0
[    3.480905] Hardware name: Generic DT based system
[    3.487159] [<c031245c>] (unwind_backtrace) from [<c030e230>] (show_stack+0x10/0x14)
[    3.491847] [<c030e230>] (show_stack) from [<c0727f30>] (dump_stack_lvl+0x40/0x4c)

and on ipq60xx (SNAPSHOT / 6.1)

[    0.745139] UBIFS error (ubi0:0 pid 1): ubifs_check_node: bad magic 0x73717368, expected 0x6101831
[    0.745194] UBIFS error (ubi0:0 pid 1): ubifs_check_node: bad node at LEB 0:0
[    0.753048] Not a node, first 24 bytes:
[    0.753058] 00000000: 68 73 71 73 53 0d 00 00 22 dd 15 66 00 00 04 00 d2 00 00 00 06 00 12 00                          hsqsS..."..f............
[    0.763891] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 6.6.25 #0
[    0.763901] Hardware name: Linksys MR7350 (DT)
[    0.763906] Call trace:

but it seems theres no issue so its like 🤷

does not happen at all on xz and lz4

inb4 zstd backdoor

[Spudz76]

Yeah I get those errors (also zstd) and I thought it was just "normal" / unrelated to this patch. Admittedly I haven't run a vanilla build in quite some time to see what errors are just regular complaints and which are new. As you note, everything works fine so I wasn't concerned and decided it was all "informational warnings" or "expected failures" that have workarounds or Plan-B's (or everything works anyway).

Similarly I assumed these are "normal":

[   26.748521] mtdblock: MTD device 'devinfo' is NAND, please consider using UBI block devices instead.
[   27.017884] mtdblock: MTD device 'devinfo' is NAND, please consider using UBI block devices instead.
[   27.708573] mtdblock: MTD device 'devinfo' is NAND, please consider using UBI block devices instead.
[   33.898778] do_page_fault(): sending SIGSEGV to uci for invalid write access to 00000000
[   33.907029] epc = 77dac984 in libc.so[77d7c000+a9000]
[   33.912156] ra  = 77dacd94 in libc.so[77d7c000+a9000]
[   34.190490] do_page_fault(): sending SIGSEGV to uci for invalid write access to 00000000
[   34.198624] epc = 77e5e984 in libc.so[77e2e000+a9000]
[   34.203718] ra  = 77e5ed94 in libc.so[77e2e000+a9000]

But could be from other patches I have in my "hax" tree that I build from. I shall try a vanilla build from main and see what sort of boot messages I get on this platform (ramips/mt7621).

[Spudz76]

@backslashxx Added kernel patches that eliminate the ubifs "bad magic" spew when the magic is squashfs so that it falls through quietly (unless the magic is actually wrong and not squashfs).

[backslashxx]

Works as expected. Thanks.

You sending that upstream?

[Spudz76]

It appears the entire CONFIG_MTD_ROOTFS_ROOT_DEV option which triggers trying ubifs before anything else comes from OpenWRT patches anyway, thus is not even part of upstream (yet?).

The offensive code comes from target/linux/generic/pending-*/492-try-auto-mounting-ubi0-rootfs-in-init-do_mounts.c.patch

I am not sure why that's in pending and not hack though, perhaps it has been sent upstream.

IMO there should be a CONFIG_UBIFS_ROOT similar to the options for CIFS or NFS root that can control the inclusion of the UBIFS code in the above patch, rather than the fairly more global CONFIG_MTD_ROOTFS_ROOT_DEV which should only do what it says and check for a MTD partition name rootfs and set it as the root filesystem, while remaining agnostic about what filesystem, and not do things like assume/try UBIFS before others without explicit indication by another config item, or by setting rootfstypes=ubifs,squashfs or such for the current behavior where warranted. But that might be used for some device where ubifs root is actually being used and the kernel args cannot be changed.

Also considering some investigation toward a similar PR as this one based on UBIFS since it supports most of the same compression methods including zstd, and might be superior to squashfs anyway although that is unclear (when the rootfs is read-only anyway). I do think squashfs is still better since the block size can be set quite large which generally improves compression ratios, and it would seem at initial glance that ubifs only works in LEB sizes (128K on my devices NAND flash) although that may be different in read-only mode (since eraseblocks are irrelevant when updates are not allowed anyway). I have been using 1MB with zstd and according to my previous testing that saves 450917 bytes or 3.44 LEBs, that can then be used by the writable overlay ubifs. There are some patches floating around on squashfs upstream for even larger compression blocksizes that have potential to save even more.

EDIT: okay ubifs compression is even worse than LEB-size... it uses 4KB no matter what. So it is likely not worth pursuing for the read-only rootfs at all

"UBIFS splits all data on 4KiB chunks and compresses each chunk independently. This is not optimal, because larger chunks of data would compress better, but this still provides noticeable flash space economy."

[Spudz76]

Tested UBIFS on my platform, the rootfs was enormous compared to squashfs even with zstd enabled, so that's why almost nothing uses it unless it absolutely must (and most seem to only make a ubifs partition containing the kernel to satisfy the minimums of the bootloader, and then pivot to squashfs from there).

Anyway I implemented the CONFIG_UBIFS_ROOT option so now it doesn't even include the code that was patched to be quiet by the previous patches. And furthermore attempted to find all platforms where there may be an actual rootfs in ubifs form and have set those to re-enable the behavior in case they rely on it. Everything else will just skip that definitely useless code.

Might split this PR into completely separate ones since now some of the commits could probably stand alone, and apparently the squashfs4 tarball situation will never be fixed so this will never pass checks (still sure it's not my fault - maybe doing so many parallel builds hits some connection/spam limit for git clones against github?).

[Spudz76]

I have split the UBIFS related stuff into a different PR here since these work separately. Building from just this PR will reintroduce the boot-spew, apply the other patches on top of (or before) this to achieve the quiet operation again.