GRUB env block corrupt at first boot on aarch

[marta-lewandowska]

We have this bug for fedora https://bugzilla.redhat.com/show_bug.cgi?id=2457333 which only appears on Live ISOs created by kiwi on aarch64. While I'm certainly not sure that kiwi is causing this, but I would love to have someone take a look.
Already on the iso the grubenv looks like:

jlinton@fedora:/boot/grub2$ hexdump -C grubenv 
00000000  23 20 47 52 55 42 20 45  6e 76 69 72 6f 6e 6d 65  |# GRUB Environme|
00000010  6e 74 20 42 6c 6f 63 6b  0a 23 20 57 41 52 4e 49  |nt Block.# WARNI|
00000020  4e 47 3a 20 44 6f 20 6e  6f 74 20 65 64 69 74 20  |NG: Do not edit |
00000030  74 68 69 73 20 66 69 6c  65 20 62 79 20 74 6f 6f  |this file by too|
00000040  6c 73 20 6f 74 68 65 72  20 74 68 61 6e 20 67 72  |ls other than gr|
00000050  75 62 2d 65 64 69 74 65  6e 76 21 21 21 0a 65 6e  |ub-editenv!!!.en|
00000060  76 5f 62 6c 6f 63 6b 3d  35 31 32 2b 31 0a 73 61  |v_block=512+1.sa|
00000070  76 65 64 5f 65 6e 74 72  79 3d 65 39 61 30 32 34  |ved_entry=e9a024|
00000080  34 33 33 36 38 31 34 64  31 31 39 65 62 62 32 62  |4336814d119ebb2b|
00000090  37 32 39 30 35 37 36 36  37 65 2d 36 2e 31 39 2e  |729057667e-6.19.|
000000a0  31 30 2d 33 30 30 2e 66  63 34 34 2e 61 61 72 63  |10-300.fc44.aarc|
000000b0  68 36 34 0a 6d 65 6e 75  5f 61 75 74 6f 5f 68 69  |h64.menu_auto_hi|
000000c0  64 65 3d 31 0a 62 6f 6f  74 5f 73 75 63 63 65 73  |de=1.boot_succes|
000000d0  73 3d 31 0a 23 23 23 23  23 23 23 23 23 23 23 23  |s=1.############|
000000e0  23 23 23 23 23 23 23 23  23 23 23 23 23 23 23 23  |################|
*
00000400

The offending line being: env_block=512+1 This causes an error to get written out and the GRUB menu doesn't show. Removing the grubenv and recreating it via grub2-editenv does not write out this line.

This is related to a btrfs patch set that was requested: https://bugzilla.redhat.com/show_bug.cgi?id=2372973 and subsequently sent upstream & pulled into fedora. It seems to be doing what's expected in other environments than this Live ISO, though, which is why I'm suspecting kiwi. ;)

thanks!

[schaefi]

@marta-lewandowska Hi, I was looking into envblock and how kiwi could have caused this. I found two places in code where kiwi deals with the grubenv.

1. If the image description provides an environment block like this

<bootloadersettings>
    <environment>
        <env name="menu_auto_hide" value="1"/>
        <env name="boot_success" value="1"/>
     </environment>
</bootloadersettings>

In such a case kiwi calls grub2-editenv. I believe this cannot cause a problem given the tool does the right thing. I'm mentioning it such that you can check your buildlog for this calls

2. There is some _fix_grub_root_device_reference method in kiwi here https://github.com/OSInside/kiwi/blob/main/kiwi/bootloader/config/grub2.py#L1540 This code scared me from the beginning on but as you can see from the comment it is still needed to call this frankenstein operations. Part of the code is a raw editing of the grubenv file under certain circumstances. Maybe at this point there is a mistake or we better change this code to call grub2-editenv if possible. Can you investigate if this is called in your case, if yes try to disable the grubenv change and see if that fixes it.

As it's not so easy for me to reproduce your exact build, I appreciate if we can work together on the topic.

Thanks

[Conan-Kudo]

We aren't using that stanza yet in the fedora descriptions, but we are calling grub2-editenv: https://pagure.io/fedora-kiwi-descriptions/blob/rawhide/f/config.sh#_44-50

This is only called for ARM disk images, though. It shouldn't be happening at all for live ISOs.

[Conan-Kudo]

@schaefi According to the RHBZ, it can be reproduced by simply building one of the live ISOs for aarch64 and booting it.

[marta-lewandowska]

Thank you both for your comments. I'm still trying to track down the iso build logs. @Conan-Kudo you don't happen to know where those are, do you?

[Conan-Kudo]

They are in Koji. For example, here's KDE-Desktop-Live and Workstation-Live.

[marta-lewandowska]

They are in Koji. For example, here's KDE-Desktop-Live and Workstation-Live.

thank you! well, I can't find anything useful in there unfortunately.

[schaefi]

@Conan-Kudo we see this only on ARM correct ?

[Conan-Kudo]

That's what the bug says...

[lsandov1]

@schaefi Also observed in x64, see

https://bugzilla.redhat.com/show_bug.cgi?id=2457333#c23
https://bugzilla.redhat.com/show_bug.cgi?id=2457333#c24

[schaefi]

I can only imagine that this is caused by the one place in the kiwi code that manually modifies the grubenv file without using grub2-editenv. I provided a patch to fix this but I'm not 100% sure if this fixes the problem on your end. I will also double check with our integration tests. Thanks

[schaefi]

The problem why I'm not sure if the proposed fix really fixes it for you is because our integration test from here:

• https://build.opensuse.org/package/show/Virtualization:Appliances:Images:Testing_x86:fedora/test-image-live-disk

Doesn't show the problem initially. The grubenv blob looks like this

[root@fedora grub2]# hexdump -C grubenv
00000000  23 20 47 52 55 42 20 45  6e 76 69 72 6f 6e 6d 65  |# GRUB Environme|
00000010  6e 74 20 42 6c 6f 63 6b  0a 23 20 57 41 52 4e 49  |nt Block.# WARNI|
00000020  4e 47 3a 20 44 6f 20 6e  6f 74 20 65 64 69 74 20  |NG: Do not edit |
00000030  74 68 69 73 20 66 69 6c  65 20 62 79 20 74 6f 6f  |this file by too|
00000040  6c 73 20 6f 74 68 65 72  20 74 68 61 6e 20 67 72  |ls other than gr|
00000050  75 62 2d 65 64 69 74 65  6e 76 21 21 21 0a 23 23  |ub-editenv!!!.##|
00000060  23 23 23 23 23 23 23 23  23 23 23 23 23 23 23 23  |################|
*
00000400

so basically empty...

I'm sure this is because your image builds in a different way than my simple integration test.

Nevertheless I believe the proposed patch is a good change :)

@Conan-Kudo maybe you can help with testing if it also helps with the real issue.

Thanks