image: respect TARGET_PER_DEVICE_ROOTFS for initramfs

[luizluca]

This PR tries to solve the problem of devices that requires a specific package to work (i.e.: switch drivers) that are not required or even conflict with other devices.

I'm not an expert in Makefile and most of this PR was done using try-n-error. I'm not sure, for example, if it is OK to include kernel-defaults.mk in image.mk. It might be better ways to get the same result.

The PR was still not tested with other situation, like without TARGET_PER_DEVICE_ROOTFS, with CONFIG_TARGET_ROOTFS_INITRAMFS_SEPARATE or CONFIG_EXTERNAL_CPIO.

It also failed sometimes with parallel jobs (-j21) because multiple initramfs are built in parallel:

rm -f openwrt/trunk/build_dir/target-mips_24kc_musl/linux-ath79_generic/linux-5.15.118/.config.prev
rm -f openwrt/trunk/build_dir/target-mips_24kc_musl/linux-ath79_generic/linux-5.15.118/.config.prev
rm -f openwrt/trunk/build_dir/target-mips_24kc_musl/linux-ath79_generic/linux-5.15.118/.config.prev
mv openwrt/trunk/build_dir/target-mips_24kc_musl/linux-ath79_generic/linux-5.15.118/.config openwrt/trunk/build_dir/target-mips_24kc_musl/linux-ath79_generic/linux-5.15.118/.config.old
rm -f openwrt/trunk/build_dir/target-mips_24kc_musl/linux-ath79_generic/linux-5.15.118/.config.prev
mv openwrt/trunk/build_dir/target-mips_24kc_musl/linux-ath79_generic/linux-5.15.118/.config openwrt/trunk/build_dir/target-mips_24kc_musl/linux-ath79_generic/linux-5.15.118/.config.old
mv openwrt/trunk/build_dir/target-mips_24kc_musl/linux-ath79_generic/linux-5.15.118/.config openwrt/trunk/build_dir/target-mips_24kc_musl/linux-ath79_generic/linux-5.15.118/.config.old
grep -v -e INITRAMFS -e CONFIG_RD_ -e CONFIG_BLK_DEV_INITRD openwrt/trunk/build_dir/target-mips_24kc_musl/linux-ath79_generic/linux-5.15.118/.config.old > openwrt/trunk/build_dir/target-mips_24kc_musl/linux-ath79_generic/linux-5.15.118/.config
mv openwrt/trunk/build_dir/target-mips_24kc_musl/linux-ath79_generic/linux-5.15.118/.config openwrt/trunk/build_dir/target-mips_24kc_musl/linux-ath79_generic/linux-5.15.118/.config.old
mv: cannot stat 'openwrt/trunk/build_dir/target-mips_24kc_musl/linux-ath79_generic/linux-5.15.118/.config': No such file or directory

How can I avoid it?

Fixes initramfs after 575ec7a b168a07 6e0f0ea when building for multiple devices.

[luizluca]

Pushed a fix for images without TARGET_PER_DEVICE_ROOTFS

There is still an issue with parallel build. I noticed that .config for initramfs might be based on an empty .config.old. Maybe two initramfs images are being built in parallel or .config.old was never generated.

[castillofrancodamian]

Does this PR fix this or has nothing to do with it?

[luizluca]

Does this PR fix this or has nothing to do with it?

Nothing to do. However, if that issue is related to a too big kernel/rootfs, this PR might allow to move some code/packages from the default selection into the devices.

[luizluca]

The parallel build fails because multiple images are built in parallel. How can we avoid parallelization only during kernel rebuild?

[luizluca]

I cleaned the changes to keep as much as possible the original calls.

The issue is how to mutex Kernel/CompileImage/Initramfs. If I let them as is, Makefile will try to rebuild kernel images in parallel for each set of packages and Kernel/CompileImage/Initramfs is not thread-safe. You'll get a broken kernel config.

[luizluca]

As a workaround, I'm simply disabling parallel jobs in image.mk when initramfs and multi rootfs are both enabled.

[Ansuel]

@luizluca Hi, this is a very good feature and I think we should try to make it work... but we need to be careful and it may take some time...

A good approach to this given how crucial this is follow some practice:

• I know it's a good opportunity to do some cleanup but we may cause indirect regression so if we are not sure about some thing I think we should keep them as is and investigate later. (example the MTIME doubt)
• We should split thing that would change from commit that are "preparation". Ideally these preparation commit would be merged first and see if these change cause any regression.

Hope you understand why I'm asking to follow these 2 thing. Aside from them I read the last commit and I have a question... I wonder if it's possible to create something like a 2 stage creation.

• Considering on buildbot we build every package
• And the current implementation is to create a generic rootfs

I wonder if it's possible to create the generic rootfs and then add the required packages for each device. This way in theory the generic rootfs implementation can be used and just modified for the task.

One thing we should do is check if appending stuff results in saving some build time or not.

[luizluca]

@luizluca Hi, this is a very good feature and I think we should try to make it work... but we need to be careful and it may take some time...

What really matters is to get the feature merged. ;-)

A good approach to this given how crucial this is follow some practice:

• I know it's a good opportunity to do some cleanup but we may cause indirect regression so if we are not sure about some thing I think we should keep them as is and investigate later. (example the MTIME doubt)

Revisiting that, I believe we do need to touch that file. As an alternative approach is to copy the file into a temporary directory and append that directory. The goal of that commit is to keep the rootfs untouched as it will be shared between initramfs and squashfs images. Also, /generic/other-files/ might not be the best name for a single file used only by initramfs. Maybe /generic/initramfs-files/ and copy them as a directory?

• We should split thing that would change from commit that are "preparation". Ideally these preparation commit would be merged first and see if these change cause any regression.

Sure. I'll isolate the first commit and submit it. The second one does nothing if you don't give an argument to Kernel/CompileImage/Initramfs.

Hope you understand why I'm asking to follow these 2 thing. Aside from them I read the last commit and I have a question... I wonder if it's possible to create something like a 2 stage creation.

• Considering on buildbot we build every package
• And the current implementation is to create a generic rootfs

I wonder if it's possible to create the generic rootfs and then add the required packages for each device. This way in theory the generic rootfs implementation can be used and just modified for the task.

I don't create a new rootfs for initramfs. I just reuse the one already created by squashfs images. And that one is already copied from a shared $(TARGET_DIR_ORIG) (not the $(TARGET_DIR( that is currently used by initramfs).

One thing we should do is check if appending stuff results in saving some build time or not.

It might not save now as copying/deleting a file is not relevant for the build time. However, it will save the time allowing the reuse of an rootfs already generated for squashfs.

[mcprat]

I'd like to start taking a look at this problem, but I don't know much about this part of the makefile yet.

the $eval that you added to me is a red flag and shouldn't be necessary. do you remember why you added it?

[luizluca]

I'd like to start taking a look at this problem, but I don't know much about this part of the makefile yet.

the $eval that you added to me is a red flag and shouldn't be necessary. do you remember why you added it?

You mean this one?
https://github.com/openwrt/openwrt/pull/12959/files#diff-a2b0aa97bfb01148df53ca7ec71c30d039c22fc67e1d52685613b5b2597da4aeL734

In order to expand $$(ROOTFS_ID/$(1)), you need that eval. It is not different from other defines that use ROOTFS_ID

[luizluca]

The difficult here is that when you have a "define" with a sequence of independent commands (new line not escaped), you cannot run that as a single mutex context (i.e.: $(call locked ...). So, we either need to convert Kernel/CompileImage/Initramfs into a single shell command (with lots of "", including defines called from it), some other Makefile black magic to avoid two initramfs to be built at the same time or simply disable parallel image building when initramfs with multi rootfs.

Also, there is the single extra file (/init) copied for initramfs. It would be nice if we could inject that respecting the $SOURCE_DATE_EPOCH. I really don't like the cp....build...rm strategy in use today nor a new temporary directory only for setting the mtime. However, it seems that the rm approach is the easier one. Kernel does use $KBUILD_BUILD_TIMESTAMP for links, dirs and devs. Maybe we could add a new option to set it for all file.

[luizluca]

I pushed a simplified PR. It has just two simple commits. The first is just a preparation, allowing the user to define a different target-dir. I removed the part of including the /init as an appended directory for now. I still believe it would be cleaner than cp/rm but I would need to use a tmp directory to set mtime.

There is a list of FIXMEs in the last commit message:

• Do we still need a generic initramfs image? If so, do we need multiple .debug and .elf initramfs kernel? Isn't the generic one equivalent for debugging the kernel?
• Can the extra /init land in the non-initramfs rootfs image?
• is it OK to include kernel-default.mk in image.mk?
• Solve the mutual exclusive build of initramfs images (call to Kernel/CompileImage/Initramfs) without using .NOTPARALLEL? Maybe convert it into a single shell command (escaping newlines) and using it with $(call locked, Kernel/CompileImage/Initramfs)?

For this last one, I even created a simple PoC:

define xxx
target$(1):
        @echo starting xxx $(1)
        $(call yyy,$(1))
endef

define yyy
        @echo starting yyy $(1)
        @sleep 1
        $(call zzz,v$(1))
        @echo finishing yyy $(1)
endef

define zzz
        @echo doing zzz $(1)
        @sleep 1
endef

all: $(foreach i,1 2 3,target$i)

$(foreach i,1 2 3,$(eval $(call xxx,$(i))))

If I make -j1, I get:

starting xxx 1
starting yyy 1
doing zzz 1
finishing yyy 1
starting xxx 2
starting yyy 2
doing zzz 2
finishing yyy 2
starting xxx 3
starting yyy 3
doing zzz 3
finishing yyy 3

And with make -j3, I get:

starting xxx 1
starting xxx 2
starting xxx 3
starting yyy 1
starting yyy 2
starting yyy 3
doing zzz 1
doing zzz 2
doing zzz 3
finishing yyy 1
finishing yyy 2
finishing yyy 3

And I need xxx to run in parallel but yyy in mutex:

starting xxx 1
starting xxx 2
starting xxx 3
starting yyy 1
doing zzz 1
finishing yyy 1
starting yyy 2
doing zzz 2
finishing yyy 2
starting yyy 3
doing zzz 3
finishing yyy 3

[Ansuel]

@dangowrt btw i analyzed the problem a bit and this is not trivial at all...

One solution might be get out the big guns and manually create alle the files...
Problem is that kernel really base everything around the config for the initramfs files...

And this cause all kind of problems as notice here with parallel build...

But I may have found a funny solution... Considering this is the last step and everything has been already built....
The only things that needs to be regenerated are the vmlinux the Image and the initramfs object and those are in arch, in /usr and in the linux top directory...

We can do a meme thing and create a separate linux directory for each device, symbolic link for everyone and copy only the directory that will be affected...

I just tested this and it seems to work correctly... I still need to test this with the PER_DEVICE_ROOTFS tho... but it seems to be the only solution currently...

[mcprat]

adding a new .NOTPARALLEL: may not be finely tuned enough to make the individual images not parallel while having every other component parallel, in that case I think the best solution is to set dependency rules between each of those items instead, something like this:

A: B C D
A B: C D
A B C: D
A B C D:

which can be done with a pattern substitution
(one of my PR's is a demonstration of this)

the tricky part is debuging enough to figure out which variable represents that list of items that we must build strictly in order. I'm not sure what it is just by looking at the diff here... I wish I had time to dive deeper into this but I'm too busy in real life, let alone all of the smaller problems here that are in my todo list.

However, I have a suspicion that some of the changes currently presented here may not be necessary if we set up structured dependency chains like this between unique images.

the only things that needs to be regenerated are the vmlinux the Image and the initramfs object and those are in arch, in /usr and in the linux top directory...

We can do a meme thing and create a separate linux directory for each device, symbolic link for everyone and copy only the directory that will be affected

I think we ought to have unique file names for each instance of each component of each image using the device name as the beginning of the filename, just like it is for the final image in the bin directory, but only when this configure option is enabled, otherwise the prepended part of the filename could just be "common".

For the rootfs there ought to be a "common" subdirectory for all files of the rootfs which will always be common, which will then become the template for each individual image by copying everything from there into the actual rootfs directory that will be squashed. That way, uniqueness of the name of the rootfs directory before squashing is not necessary. The rootfs directory should also be deleted after each image has finished building since unlike the kernel, one can select many packages and make the rootfs extremely large, so having a rootfs directory for each image could be a massive hog on disk space.

[Ansuel]

Further test scrapped my idea of link and stuff... Locking is mandatory for initramfs generation... Alternative is to copy the entire linux directory but that is really not something doable...

The only remaining thing is the handling of the separate cpio... It seems they might comflict. Aside from that this is pretty much ready.

[mcprat]

yeah we should stay away from symlinking here...

there ought to be a way to get the linux build system to use existing binaries or objects to wrap up a new compresssed image without a full rebuild and install it under a new name... but maybe that would require patching the linux makefiles?