F39/F40: Fails to boot on with "vmlinuz has invalid signature" #543

Mershl · 2024-03-28T20:10:45Z

Describe the bug
Trying to rebase an existing SB39 to SB40 fails to boot showing vmlinuz-6.8.1-300.fc40.x86.x64 has invalid signature. you need to load the kernel first on an Thinkpad T495. I was not able to reproduce the issue on my desktop systems.

coming from deployment:

fedora:fedora/39/x86_64/silverblue
                  Version: 39.20240328.0 (2024-03-28T00:39:32Z)
               BaseCommit: a1ac8885d05ca13728d92c2bcf1ded67a7ddb409d657e446a808397366a463b1
             GPGSignature: Valid signature by E8F23996F23218640CB44CBE75CF5AC418B8E74C

going for:

fedora:fedora/40/x86_64/silverblue
                  Version: 40.20240328.n.0 (2024-03-28T08:09:28Z)
               BaseCommit: f726d0be3361a42a8ac175b08851de67a2d97c9c01ee130a3abcd32720120f9c
             GPGSignature: Valid signature by 115DF9AEF857853EE8445D0A0727707EA15B79CC

What I've tried so far

cleanup + rpm-ostree initramfs --enable + rebase
cleanup -r (removing all other deployments) + rebase
rebase to fedora:fedora/40/x86_64/testing/silverblue instead, same error except error message points to vmlinuz-6.8.2

To Reproduce

rpm-ostree rebase fedora:fedora/40/x86_64/silverblue
systemctl reboot

The text was updated successfully, but these errors were encountered:

travier · 2024-03-29T10:45:46Z

I've hit the same issue and I've disabled Secure Boot for now.

It's likely due to the bootloader being too old. bootupd should land in the image after the release and will then let people update their bootloader.

Or maybe we should include bootupd in Fedora 39 now so that people can update their bootloader before upgrading to Fedora 40.

Mershl · 2024-04-07T11:13:29Z

Oh Oh.

SB39 (stable) just updated (no rebase) to kernel 6.8.4 showing the very same issue.

  kernel 6.7.11-200.fc39.x86_64 -> 6.8.4-200.fc39.x86_64

breaks boot, showing

vmlinuz-6.8.4-200.fc39.x86_64 has invalide signature.
you need to load the kernel first.

(Thinkpad T495 using fedora:fedora/39/x86_64/silverblue)

HassoSigbjoernson · 2024-04-08T11:40:29Z

Or maybe we should include bootupd in Fedora 39 now so that people can update their bootloader before upgrading to Fedora 40.

And people have to be told what to do. I got a bad shim signature. you need to load the kernel first error while updating my SB39 system (no rebase to 40) and was able to fix it by copying files from /sysroot/ostree/deploy/fedora/deploy/[...]/usr/lib/ostree-boot/efi/EFI/fedora to /boot/efi/EFI/fedora but I was only able to do this because I had already encountered this problem when it affected arm64 systems a year ago.

travier · 2024-04-08T15:41:34Z

Pinging Universal Blue folks as it's likely going to impact them soon: @castrojo @noelmiller @EyeCantCU @KyleGospo

travier · 2024-04-09T16:38:41Z

Unfortunately, now that the kernel has landed in an update, we can not add bootupd to a future F39 build as you won't be able to boot it to get the fix.

Overlaying bootupd via rpm-ostree install is not enough as it needs to run a command during the commit build process.

So we have coreos/bootupd#635 as an option (that I have not tested yet) to create container images with bootupd based on an older commit with the previous kernel.

Or we document how to manually fix this until bootupd finally lands in Atomic Desktops. Unfortunately it will not fully be in Fedora 40 by default yet. See: https://gitlab.com/fedora/ostree/sig/-/issues/1

hferreiro · 2024-04-09T19:50:46Z

Can't the update to the kernel package be reverted in the same commit that bootupd is added?

travier · 2024-04-09T19:54:58Z

No unfortunately, for Fedora Atomic Desktops, we always take the latest RPMs from the repos.

jorgeml · 2024-04-10T15:59:12Z

@travier I'm watching this one with interest as my Silverblue image is now one week old. If I understand it correctly, the only solution at the moment is to disable Secure Boot? And not even upgrading to Fedora 40 will solve it in the near future?

travier · 2024-04-10T18:31:19Z

The workaround for this issue is going to be commands that do what bootupd does but manually unfortunately.

This is mostly doing a copy from /usr/lib/ostree-boot to /boot/efi on EFI systems, and a /usr/sbin/grub2-install call on BIOS ones, with the correct arguments.

I have not tested any of this so I'm not providing ready made commands and everything is at your own risk. Please make backups and make sure that you are confortable rescuing a broken system before trying thing out.

Help with testing in VMs or on real hardware is welcomed.

I recommend disabling Secure Boot in the meantime until this is fixed.

travier · 2024-04-10T20:06:06Z

Warning: Do at your own risk, only lightly tested

Here is the set of commands I've just used to update my (x86_64) EFI booted system successfully:

# Enter a root shell on the host (i.e. not in a toolbox)
$ sudo -i

# Make a backup of the content of the EFI partition
$ cd /boot/efi/
$ cp -a EFI EFI.bkp

# Copy updated bootloader versions
$ cp /usr/lib/ostree-boot/efi/EFI/BOOT/{BOOTIA32.EFI,BOOTX64.EFI,fbia32.efi,fbx64.efi} /boot/efi/EFI/BOOT/
$ cp /usr/lib/ostree-boot/efi/EFI/fedora/{BOOTIA32.CSV,BOOTX64.CSV,grubia32.efi,grubx64.efi,mmia32.efi,mmx64.efi,shim.efi,shimia32.efi,shimx64.efi} /boot/efi/EFI/fedora/

# Only needed if it exists already on your system
$ cp /usr/lib/ostree-boot/efi/EFI/fedora/shimx64.efi /boot/efi/EFI/fedora/shimx64-fedora.efi

# Sync changes to the disk
$ sync

# Reboot

Once reboot is successful, you can remove the backup copies:

# Enter a root shell on the host (i.e. not in a toolbox)
$ sudo -i

# Make a backup of the content of the EFI partition
$ cd /boot/efi/
$ rm -ri ./EFI.bkp

# Sync changes to the disk
$ sync

Edit: Updated to add 32bits EFI binaries as well.

jorgeml · 2024-04-10T20:23:58Z

Thanks @travier ! Just did this on my machine and it worked. I was also able to reenable Secure Boot. As a positive side effect, it also looks like the UEFI dbx update was applied. That had been blocked for a while.

cgwalters · 2024-04-11T13:14:32Z

OK sorry yes, this is an embarrassing problem.

@travier and I had a chat, a few things here:

We can safely add bootupd to f39
I think we can also document a flow of rpm-ostree upgrade but without restarting, then
Run bootupd from the new root

Alternatively, we can document how to run bootupd from a privileged container. (In fact, running it from the existing quay.io/fedora/fedora-coreos:stable container for example. One minor stumbling block here is that bootupd wants to act in a client/server model with systemd, but I think we can change it to not do that)

Longer term yes, the technical debt here is high and makes coreos/bootupd#454 important to do and just turn on automatic updates, or at least automatic updates when they're needed.

Mershl · 2024-04-11T18:56:04Z

Copy updated bootloader versions
$ cp /usr/lib/ostree-boot/efi/EFI/BOOT/{BOOTIA32.EFI,BOOTX64.EFI,fbia32.efi,fbx64.efi} /boot/efi/EFI/BOOT/
$ cp /usr/lib/ostree-boot/efi/EFI/fedora/{BOOTIA32.CSV,BOOTX64.CSV,grubia32.efi,grubx64.efi,mmia32.efi,mmx64.efi,shim.efi,shimia32.efi,shimx64.efi} /boot/efi/EFI/fedora/

Only needed if it exists already on your system
$ cp /usr/lib/ostree-boot/efi/EFI/fedora/shimx64.efi /boot/efi/EFI/fedora/shimx64-fedora.efi

@travier I've noticed that the affected Thinkpad T495 was missing boot/efi/EFI/BOOT completely. I skipped the line in your test steps and it worked out perfectly. SecureBoot reenabled, no regressions found so far.

Looking into other systems I see the mentioned boot/efi/EFI/BOOT. Does someone know the difference, does it depend on the birth date of the Fedora/Silverblue installation?

UPDATE: upgraded two other desktop systems successfully (SB39->SB40 with enabled SecureBoot) using the test steps.

A6GibKm · 2024-04-13T17:39:34Z

Warning: Do at your own risk, only lightly tested

Here is the set of commands I've just used to update my (x86_64) EFI booted system successfully:

# Enter a root shell on the host (i.e. not in a toolbox)
$ sudo -i

# Make a backup of the content of the EFI partition
$ cd /boot/efi/
$ cp -a EFI EFI.bkp

# Copy updated bootloader versions
$ cp /usr/lib/ostree-boot/efi/EFI/BOOT/{BOOTIA32.EFI,BOOTX64.EFI,fbia32.efi,fbx64.efi} /boot/efi/EFI/BOOT/
$ cp /usr/lib/ostree-boot/efi/EFI/fedora/{BOOTIA32.CSV,BOOTX64.CSV,grubia32.efi,grubx64.efi,mmia32.efi,mmx64.efi,shim.efi,shimia32.efi,shimx64.efi} /boot/efi/EFI/fedora/

# Only needed if it exists already on your system
$ cp /usr/lib/ostree-boot/efi/EFI/fedora/shimx64.efi /boot/efi/EFI/fedora/shimx64-fedora.efi

# Sync changes to the disk
$ sync

# Reboot

Once reboot is successful, you can remove the backup copies:

# Enter a root shell on the host (i.e. not in a toolbox)
$ sudo -i

# Make a backup of the content of the EFI partition
$ cd /boot/efi/
$ rm -ri ./EFI.bkp

# Sync changes to the disk
$ sync

Edit: Updated to add 32bits EFI binaries as well.

I tested these steeps on my workstation and my Thinkpad T480. I managed to get secure boot on both devices.

On the desktop I had to use fwupdmgr to update the UEFI dbx.

rugk · 2024-04-25T09:50:24Z

Has the same issue reported here (also includes a kernel bug report though I am unsure how helpful that is), so, but this question AFAIK remains unanswered:

And not even upgrading to Fedora 40 will solve it in the near future?

So does updating fix it? I mean I can boot into an old working Fedora 39 version.
If I upgrade from there to Fedora 40, does that work? (Edit: tested, does not work)

Alternatively, if I need to disable Secure Boot temporarily, can I just do so – upgrade to Fedora 40 and boot successfully, afterwards? (Edit: tested, does not work, I can only boot with SecureBoot disabled)

Also, would this possibly be a candidate for a common issue if it prevents upgrading to Fedora 40?

travier · 2024-04-25T11:44:22Z

It's a good idea to make it a common issue.

It's basically impossible to fix automatically right now for F39 users updating as the new kernel already landed in Fedora 39.

We'll try to get bootupd support to a sufficiently good shape to have a small set of commands to enter manually but in the meantime, the commands in #543 (comment) are the best that we have.

rugk · 2024-04-25T15:00:45Z

Okay then, I have created a draft for this here:
https://discussion.fedoraproject.org/t/booting-fails-with-vmlinuz-has-invalid-signature/114354

Unfortunately I miss the technical details etc. to properly document it. So feel free to edit it please.

rugk · 2024-04-29T22:16:27Z

FYI I can confirm the workaround posted worked:

# tree EFI
EFI
├── BOOT
│   ├── BOOTIA32.EFI
│   ├── BOOTX64.EFI
│   ├── fbia32.efi
│   └── fbx64.efi
└── fedora
    ├── BOOTIA32.CSV
    ├── BOOTX64.CSV
    ├── fonts
    ├── grub.cfg
    ├── grub.cfg.old
    ├── grubenv
    ├── grubenvLa7zjw
    ├── grubia32.efi
    ├── grubx64.efi
    ├── mmia32.efi
    ├── mmx64.efi
    ├── shim.efi
    ├── shimia32.efi
    ├── shimx64.efi
    └── shimx64-fedora.efi

4 directories, 18 files
# tree EFI.bkp/
EFI.bkp/
├── BOOT
│   ├── BOOTX64.EFI
│   └── fbx64.efi
└── fedora
    ├── BOOTX64.CSV
    ├── fonts
    ├── grub.cfg
    ├── grub.cfg.old
    ├── grubenv
    ├── grubenvLa7zjw
    ├── grubx64.efi
    ├── mmx64.efi
    ├── shim.efi
    ├── shimx64.efi
    └── shimx64-fedora.efi

fbruetting · 2024-05-02T12:28:28Z

Warning: Do at your own risk, only lightly tested

Here is the set of commands I've just used to update my (x86_64) EFI booted system successfully:

# Enter a root shell on the host (i.e. not in a toolbox)
$ sudo -i

# Make a backup of the content of the EFI partition
$ cd /boot/efi/
$ cp -a EFI EFI.bkp

# Copy updated bootloader versions
$ cp /usr/lib/ostree-boot/efi/EFI/BOOT/{BOOTIA32.EFI,BOOTX64.EFI,fbia32.efi,fbx64.efi} /boot/efi/EFI/BOOT/
$ cp /usr/lib/ostree-boot/efi/EFI/fedora/{BOOTIA32.CSV,BOOTX64.CSV,grubia32.efi,grubx64.efi,mmia32.efi,mmx64.efi,shim.efi,shimia32.efi,shimx64.efi} /boot/efi/EFI/fedora/

# Only needed if it exists already on your system
$ cp /usr/lib/ostree-boot/efi/EFI/fedora/shimx64.efi /boot/efi/EFI/fedora/shimx64-fedora.efi

# Sync changes to the disk
$ sync

# Reboot

Once reboot is successful, you can remove the backup copies:

# Enter a root shell on the host (i.e. not in a toolbox)
$ sudo -i

# Make a backup of the content of the EFI partition
$ cd /boot/efi/
$ rm -ri ./EFI.bkp

# Sync changes to the disk
$ sync

Edit: Updated to add 32bits EFI binaries as well.

Thereafter, I get the following:

After rpm-ostree update:

❯ rpm-ostree update
…
…
rpm-md repo 'updates-archive' (cached); generated: 2024-05-02T02:13:06Z solvables: 45817
Resolving dependencies... done
Applying 2 overrides and 201 overlays
Processing packages... done
Running pre scripts... done
Running post scripts... done
Running posttrans scripts... done
Writing rpmdb... done
Writing OSTree commit... done
Staging deployment... done
error: Staging deployment: Cleaning deployments: Removing ostree/deploy/fedora/deploy/3fd29d12b510aabcc32bd5ab5a6262b4d1a4da43efc7a70788c11837f1afa140.0: unlinkat(classic-topbar-network-cellular-acquiring.svg): Das Dateisystem ist nur lesbar (engl.: „File system is read-only“)

At second try:

❯ LANG=en_US.utf8; rpm-ostree update
error: Remounting /sysroot read-write: Das Argument ist ungültig (engl.: „Argument not valid“)

rugk · 2024-05-02T13:56:32Z

@fbruetting seems like a different issue, OSTree updates should not have anything to do with the bootloader update. Maybe ask at the Fedora discussions forum for help?

Mershl added the bug Something isn't working label Mar 28, 2024

travier added the f40 Related to Fedora 40 label Mar 29, 2024

travier mentioned this issue Mar 29, 2024

Is it safe to run /usr/bin/bootupctl backend generate-update-metadata in a Containerfile? coreos/bootupd#635

Open

travier added the f39 Related to Fedora 39 label Apr 8, 2024

castrojo mentioned this issue Apr 8, 2024

bootupd upgrade issue tracking (F40) ublue-os/main#551

Closed

travier changed the title ~~rebase to SB40 fails to boot on notebook ("vmlinuz has invalid signature")~~ F39/F40: Fails to boot on with "vmlinuz has invalid signature" Apr 9, 2024

travier mentioned this issue Apr 11, 2024

Can't update dbx via fwupdmgr #355

Closed

travier pinned this issue Apr 11, 2024

travier mentioned this issue Apr 11, 2024

Use bootupd and remove ostree-grub2 (was: All deployments are shown twice in grub) #120

Open

This comment was marked as off-topic.

Sign in to view

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

F39/F40: Fails to boot on with "vmlinuz has invalid signature" #543

F39/F40: Fails to boot on with "vmlinuz has invalid signature" #543

Mershl commented Mar 28, 2024

travier commented Mar 29, 2024

Mershl commented Apr 7, 2024 •

edited

HassoSigbjoernson commented Apr 8, 2024 •

edited

travier commented Apr 8, 2024

travier commented Apr 9, 2024

hferreiro commented Apr 9, 2024

travier commented Apr 9, 2024

jorgeml commented Apr 10, 2024

travier commented Apr 10, 2024 •

edited

travier commented Apr 10, 2024 •

edited

jorgeml commented Apr 10, 2024

cgwalters commented Apr 11, 2024

Mershl commented Apr 11, 2024 •

edited

A6GibKm commented Apr 13, 2024

rugk commented Apr 25, 2024 •

edited

travier commented Apr 25, 2024

rugk commented Apr 25, 2024

rugk commented Apr 29, 2024

fbruetting commented May 2, 2024 •

edited

rugk commented May 2, 2024

This comment was marked as off-topic.

F39/F40: Fails to boot on with "vmlinuz has invalid signature" #543

F39/F40: Fails to boot on with "vmlinuz has invalid signature" #543

Comments

Mershl commented Mar 28, 2024

travier commented Mar 29, 2024

Mershl commented Apr 7, 2024 • edited

HassoSigbjoernson commented Apr 8, 2024 • edited

travier commented Apr 8, 2024

travier commented Apr 9, 2024

hferreiro commented Apr 9, 2024

travier commented Apr 9, 2024

jorgeml commented Apr 10, 2024

travier commented Apr 10, 2024 • edited

travier commented Apr 10, 2024 • edited

jorgeml commented Apr 10, 2024

cgwalters commented Apr 11, 2024

Mershl commented Apr 11, 2024 • edited

A6GibKm commented Apr 13, 2024

rugk commented Apr 25, 2024 • edited

travier commented Apr 25, 2024

rugk commented Apr 25, 2024

rugk commented Apr 29, 2024

fbruetting commented May 2, 2024 • edited

rugk commented May 2, 2024

This comment was marked as off-topic.

Mershl commented Apr 7, 2024 •

edited

HassoSigbjoernson commented Apr 8, 2024 •

edited

travier commented Apr 10, 2024 •

edited

travier commented Apr 10, 2024 •

edited

Mershl commented Apr 11, 2024 •

edited

rugk commented Apr 25, 2024 •

edited

fbruetting commented May 2, 2024 •

edited