Secure Boot support for OUTPUT=USB

[jsmeix]

• Type: Enhancement

• Impact: Normal

• Reference to related issue (URL):

#3025 (comment)

• How was this pull request tested?

I tested it same (on same VMs) as I did in
#3025 (comment)
but now with the changes here
Secure Boot with OUTPUT=USB works for me.

• Brief description of the changes in this pull request:

In output/USB/Linux-i386/100_create_efiboot.sh
add Secure Boot support for OUTPUT=USB by using
SECURE_BOOT_BOOTLOADER as first stage Secure Boot bootloader BOOTX64.efi
and using grub*.efi as second stage Secure Boot bootloader files
in the same way as already done for OUTPUT=ISO
in output/ISO/Linux-i386/250_populate_efibootimg.sh
see #3025 (comment)

My current implementation here is just a first step.
The whole code looks somewhat convoluted
and needs at least some more generic cleanup
to make it easier to further develop things in this area.

[jsmeix]

First things first:
I will not clean up the whole UEFI and Secure Boot code
via this pull request.
Via this pull request only what is needed
to get Secure Boot support with OUTPUT=USB
in the current code environment
should be implemented.

Later - as needed and as time permits - I would like
to clean up step by step the whole UEFI and Secure Boot code.

In particular currently I do not like that
via SECURE_BOOT_BOOTLOADER
UEFI_BOOTLOADER is overwritten
because that makes it needlessly hard (at least for me)
to understand the UEFI and Secure Boot code
because it is not clear if in a particular piece of code
UEFI_BOOTLOADER means a non-Secure-Boot bootloader
OR
if UEFI_BOOTLOADER means a Secure Boot first stage bootloader
AND
in the latter case the user cannot configure
the Secure Boot second stage bootloader.
I would like to Keep Separated Items Separated - "KSIS" ;-)

[jsmeix]

@pcahyna @rear/contributors
could you please -as time permits - have a look here?

I would very much appreciate it if you could test it
on non-SUSE Linux distributions, in particular RHEL
and perhaps also Ubuntu and Debian.

[jsmeix]

@pcahyna @rear/contributors
I have a question regarding the
Secure Boot second stage bootloader files:

Currently those are (hardcoded) all files
that match the bash globbing grub*.efi
in the directory where SECURE_BOOT_BOOTLOADER is.

I wonder if it is more fail-safe to use all *.efi files
in the directory where SECURE_BOOT_BOOTLOADER is,
probably even all *.efi files with ignore case matching?

[jsmeix]

@pcahyna @rear/contributors
I would like to merge it tomorrow (Thursday) afternoon
unless there are objections.

[pcahyna]

@jsmeix thank you, this is a very important change, sorry for being late with review. I will have a look at it tomorrow.

[jsmeix]

@pcahyna
thank you in advance for your review.
No rush - take your time.
I will wait until you did your review.

[pcahyna]

@jsmeix I am a bit late, but looking now.
Can you see
https://bugzilla.redhat.com/show_bug.cgi?id=2196445#c0
where @rmetrich has provided a quite detailed analysis
of what is happening in the current code?

[jsmeix]

@pcahyna
no worries take your time - I will wait until you did your review.

I can access
https://bugzilla.redhat.com/show_bug.cgi?id=2196445#c0

Tomorrow I will try to understand it.
I need time because I am not at all an expert in this area.

[jsmeix]

Wow - I am so impressed how fast I am :-))

I think I do understand the initial decription of @rmetrich
https://bugzilla.redhat.com/show_bug.cgi?id=2196445#c0
and - as far as I understand it - this is what I implemented
with this pull request here - at least basically.

@rmetrich
could you please also have a look here (as time permits).
I would much appreciate it to get a review also from you.

[jsmeix]

@pcahyna
as far as I understand it (which I described in my comments)

# Shim is a signed EFI binary that is a first stage bootloader
# that loads and executes another (signed) EFI binary
# which normally is a second stage bootloader
# which normally is a GRUB EFI binary
# which normally is available as a file named grub*.efi
# so when SECURE_BOOT_BOOTLOADER is used as UEFI_BOOTLOADER
# (cf. rescue/default/850_save_sysfs_uefi_vars.sh)
# then Shim (usually shim.efi) must be copied as EFI/BOOT/BOOTX64.efi
# and Shim's second stage bootloader must be also copied where Shim already is.

the crucial point for Secure Boot is that
the Shim first stage bootloader 'shim.efi' is a signed EFI binary
and Shim's second stage bootloader (usually named 'grub*.efi')
is also a signed EFI binary
which means
both signed EFI binaries from the Linux distribution
must be used unchanged for Secure Boot
instead of using an unsigned EFI binary
that gets created by build_bootx86_efi()

[jsmeix]

Regarding the second part of @rmetrich in
https://bugzilla.redhat.com/show_bug.cgi?id=2196445#c0

UEFI_BOOTLOADER ... is *overwritten* by SECURE_BOOT_BOOTLOADER

see my second part about "currently I do not like" in
#3031 (comment)

[jsmeix]

@pcahyna
please, first things first:
Does it work for you on RHEL (and/or Fedora) or not?
If yes, on which RHEL (and/or Fedora) versions does it work?
If no, where and how does it fail?

@rear/contributors
it would be nice if someone could test it on Ubuntu or Debian.
But as we do not have an active Ubuntu or Debian maintainer
those distributions are of rather low interest for me.

[pcahyna]

@pcahyna please, first things first: Does it work for you on RHEL (and/or Fedora) or not? If yes, on which RHEL (and/or Fedora) versions does it work? If no, where and how does it fail?

I have not tried it yet - I merely looked at the code. I can schedule a test on non-secureboot hardware to make sure that the change does not break the case that was working before. I am interested to test with Secure Boot as well, but this will have to wait a bit as I don't have a test environment set up (I will probably be able to do it over the weekend).

Regarding versions: Fedora 37 and later probably won't work because of #2971 (same problem with RHEL 9), although I could probably work around this by configuring ReaR to use Secure Boot shim even if the firmware is not actually set up for Secure Boot.

[jsmeix]

On non-secureboot hardware
SECURE_BOOT_BOOTLOADER must not be specified and
then there should be no change in what gets done
(because of the if ... $SECURE_BOOT_BOOTLOADER).

[pcahyna]

On non-secureboot hardware SECURE_BOOT_BOOTLOADER must not be specified

Why is that the case?
The shim bootloader is installed even on non-secureboot hardware,
shouldn't it just work even without secureboot enabled?

[jsmeix]

Because SECURE_BOOT_BOOTLOADER is nowhere set to a non-empty value.
Currently there is no Secure Boot autodetection code in ReaR
as far as I understand things in this code area
where I am not yet familiar with.

The UEFI_BOOTLOADER autodetection code in
rescue/default/850_save_sysfs_uefi_vars.sh
happens only for non-secureboot cases
as far as I understand things
because 'shim' (ignore case) does not appear in
rescue/default/850_save_sysfs_uefi_vars.sh

Currently 'shim' (ignore case) does only appear in
finalize/SUSE_LINUX/i386/675_install_shim.sh
output/ISO/Linux-i386/250_populate_efibootimg.sh

[pcahyna]

Right, one has to set SECURE_BOOT_BOOTLOADER manually in order to test it.
But, I would expect it to work even if the firmware is not configured for Secure Boot.
Wouldn't you?

[jsmeix]

Of course Secure Boot autodetection in ReaR
would be an "expected" feature.

But this is nothing that belongs to this pull request
and it is even nothing that belongs to the subsequent
cleanup of the whole UEFI booting setup code in ReaR
(because 'cleanup' != 'new features').

So I think Secure Boot autodetection in ReaR
should not be impemented as long as we still
have some mess of UEFI booting setup code
but after we properly cleaned up our current code
to make further development more fail-safe.

General improvements of UEFI related things in ReaR
are on my longer term TODO list.

What I am currently doing here is trying to move
step by step and carefully into this direction.

By the way:
Didn't I somewhere already mention "first things first"?
;-))

[jsmeix]

Now it's weekend time!
I wish you all a relaxed and recovering weekend!

[pcahyna]

@jsmeix maybe you misunderstood what I meant. It is fine at this point that Secure Boot is not autodetected and has to be configured manually (which has been always the case with ISO boot, as far as I understand). What I meant is that if the hardware is not configured for secure booting, configuring Secure Boot in ReaR (by setting SECURE_BOOT_BOOTLOADER) should result in a medium that boots fine. While Secure Boot requires a signed UEFI shim, non-secureboot UEFI firmware does not mind using this shim as well, it just does not enforce it.

I tested this on RHEL 8: my UEFI hardware is not configured for secure booting and ReaR has been booting fine on it from disk (OUTPUT=USB). With your change, I can now set
SECURE_BOOT_BOOTLOADER=/boot/efi/EFI/redhat/shimx64.efi
(RHEL is using this shim always, even if secure boot is not used) and ReaR rescue disk boots again correctly. At the end of recovery, the boot entry is restored to something like HD(1,GPT,ca381585-f4e2-418e-b583-13d02fba640b,0x800,0x12c000)/File(\EFI\redhat\shimx64.efi) which is correct.

[pcahyna]

I see only one minor problem: during boot I get

error: ../../grub-core/fs/fshelp.c:258:file `/EFI/redhat/x86_64-efi/xzio.mod'
 not found.
 Loading kernel /EFI/BOOT/kernel ...
 Loading initial ramdisk /EFI/BOOT/initrd.cgz ...
 
 Press any key to continue...

because the menu contains

    insmod xzio

and xzio is apparently not present in our default GRUB image. This does not seem to cause any harm and if I don't press any key, the boot continues in about 10 seconds. I wonder whether the insmod xzio part can be safely removed (it won't work in any case when Secure Boot is turned on, AIUI).

[pcahyna]

Regarding future plans: instead of Secure Boot autodetection, I would prefer to use the Secure Boot code path always (even if Secure Boot is not used) and get entirely rid of our own GRUB image generation for the UEFI case (build_bootx86_efi). It would simplify the code and more importantly, the GRUB image is broken anyway (#2971).

[pcahyna]

@jsmeix

I have a question regarding the Secure Boot second stage bootloader files:

Currently those are (hardcoded) all files that match the bash globbing grub*.efi in the directory where SECURE_BOOT_BOOTLOADER is.

I wonder if it is more fail-safe to use all *.efi files in the directory where SECURE_BOOT_BOOTLOADER is, probably even all *.efi files with ignore case matching?

No. I would do what OUTPUT=ISO is doing, unless there are known problems with it. If you copy all *.efi files, you may also copy fbx64.efi (https://www.rodsbooks.com/efi-bootloaders/fallback.html), and that's probably not what one wants.

[jsmeix]

@pcahyna
many thanks for your review and your explanatory comments!
It is much appeciated!

I think I also experienced the same as you wrote in your
#3031 (comment)
As it "somehow works" we should leave it "as is" for now
and focus more on the general cleanup of UEFI booting.

Regarding your
#3031 (comment)

I think also SUSE always boots via Shim on UEFI systems
(i.e. Shim = fist stage, GRUB2 = second stage) and
I was also wondering if ReaR should for UEFI booting
always "just copy" the (signed) EFI binaries of the
Linux distribution (i.e. shim*.efi and grub*.efi)
instead of making its own ReaR specific EFI binaries.

My current offhanded reasoning:

When the EFI binaries of the Linux distribution
make the original system boot,
then exactly those EFI binaries
should also make the ReaR recovery system boot
on replacement hardware
because replacement hardware must be sufficiently
compatible with the original system hardware.

Another (hopefully reasonable) assumption is that
the (signed) EFI binaries of the Linux distribution
should be sufficiently feature complete
to boot the original system on various hardware
from various kind of boot media (disk, CDROM, ISO).
I assume Linux distribution use exactly the same
(signed) EFI binaries on their installation media
to boot their installation systems and
also later on the disk of the installed system
to boot the installed system.
So I think exactly those (signed) EFI binaries
should also make the ReaR recovery system boot
on sufficiently compatible replacement hardware
from various kind of boot media.
Again there is the requirement that the replacement hardware
is sufficiently compatible with the original system hardware
which means in particular that the ReaR recovery system
must be booted from a medium that is a supported medium
to install the original system.

[jsmeix]

@pcahyna
regarding your
#3031 (comment)

My "secret final plan" is to have
some reasonable UEFI boot autodetection
so UEFI_BOOTLOADER and SECURE_BOOT_BOOTLOADER
do not need to be specified by the user
but the user can use them if needed to specify
the first stage Secure Boot bootloader EFI binary
via SECURE_BOOT_BOOTLOADER and
the second stage Secure Boot bootloader EFI binary
via UEFI_BOOTLOADER
or - when only UEFI_BOOTLOADER is specified
the single stage UEFI bootloader EFI binary.

[jsmeix]

@rear/contributors
I will merge it today afternoon unless objections appear.

[pcahyna]

@jsmeix thanks a lot for this improvement, in the meantime I tried it on a Secure Boot enabled hardware with RHEL 8 (it works - both the rescue image an the recovered system) and on Secure Boot disabled (but with configured Secure Boot in ReaR via SECURE_BOOT_BOOTLOADER=/boot/efi/EFI/redhat/shimx64.efi) with RHEL 9 (I don't test Secure Boot this way, but it confirms that using the shim + the default GRUB avoids #2971).

[pcahyna]

Another (hopefully reasonable) assumption is that
the (signed) EFI binaries of the Linux distribution
should be sufficiently feature complete
to boot the original system on various hardware
from various kind of boot media (disk, CDROM, ISO).

I believe that with Secure Boot, you are not allowed to insmod any GRUB modules, so this should be a reasonable assumption - the GRUB image should be indeed feature complete, avoiding the need for grub2-mkstandalone (it also does not need to have the path to the configuration file hardcoded, unlike on PC BIOS, because EFI passes to GRUB the directory it was loaded from and GRUB locates and loads grub.cfg in the same directory).

[pcahyna]

When the EFI binaries of the Linux distribution
make the original system boot,
then exactly those EFI binaries
should also make the ReaR recovery system boot
on replacement hardware
because replacement hardware must be sufficiently
compatible with the original system hardware.

It is not only the recovery (rescue) system (where some deviations may be necessary), it is also the recovered system. In my experience, when recovering without SECURE_BOOT_BOOTLOADER set, the recovered system has a UEFI entry of this type

Boot0016* RedHatEnterpriseServer 8      HD(1,GPT,f18ee8ed-2fdd-4984-8e71-95a82f99318d,0x800,0x12c000)/File(\EFI\redhat\grubx64.efi)

so, GRUB is directly booted, in contrast with the original system, which has

Boot000E* Red Hat Enterprise Linux      HD(1,GPT,a1d3cffa-2eb7-4a8a-b559-7f2241cba776,0x800,0x12c000)/File(\EFI\redhat\shimx64.efi)

(as it uses the shim always) - so the recovered system is different in this respect from the original system. This will be a problem if the user decides to turn Secure Boot on after the system was backed up and recovered.

My "secret final plan" is to have
some reasonable UEFI boot autodetection
so UEFI_BOOTLOADER and SECURE_BOOT_BOOTLOADER
do not need to be specified by the user
but the user can use them if needed to specify
the first stage Secure Boot bootloader EFI binary
via SECURE_BOOT_BOOTLOADER and
the second stage Secure Boot bootloader EFI binary
via UEFI_BOOTLOADER
or - when only UEFI_BOOTLOADER is specified
the single stage UEFI bootloader EFI binary.

regarding autodetection, I believe it is needed to examine the current boot entry (BootCurrent in efibootmgr output) during mkrescue and use it to set the variables - something that is AFAIK not being done now.

[pcahyna]

FYI that's what's inside the system ESP on RHEL:

# ls -lR /boot/efi/EFI/
/boot/efi/EFI/:
total 8
drwx------. 2 root root 4096 Aug  8 00:03 BOOT
drwx------. 3 root root 4096 Aug  8 05:21 redhat

/boot/efi/EFI/BOOT:
total 1016
-rwx------. 1 root root 943520 Jun  7  2022 BOOTX64.EFI
-rwx------. 1 root root  90568 Jun  7  2022 fbx64.efi

/boot/efi/EFI/redhat:
total 4872
-rwx------. 1 root root     182 Jun  7  2022 BOOTX64.CSV
drwx------. 2 root root    4096 Feb  6 16:27 fonts
-rwx------. 1 root root    6201 Aug  8 05:21 grub.cfg
-rwx------. 1 root root    6201 Aug  8 05:21 grub.cfg.rearbak
-rwx------. 1 root root    1024 Aug  8 00:07 grubenv
-rwx------. 1 root root 2217416 Feb  6 16:27 grubx64.efi
-rwx------. 1 root root  852552 Jun  7  2022 mmx64.efi
-rwx------. 1 root root  943520 Jun  7  2022 shimx64.efi
-rwx------. 1 root root  936520 Jun  7  2022 shimx64-redhat.efi

/boot/efi/EFI/redhat/fonts:
total 0

and what's on the ESP of the ReaR bootable disk after your change, when SECURE_BOOT_BOOTLOADER=/boot/efi/EFI/redhat/shimx64.efi

# ls -lR /mnt/rear/EFI/
/mnt/rear/EFI/:
total 4
drwxr-xr-x. 2 root root 4096 Aug  8 05:05 BOOT

/mnt/rear/EFI/BOOT:
total 540420
-rwxr-xr-x. 1 root root    943520 Aug  8 05:02 BOOTX64.efi
-rwxr-xr-x. 1 root root      1411 Aug  8 05:05 grub.cfg
-rwxr-xr-x. 1 root root   2217416 Aug  8 05:02 grubx64.efi
-rwxr-xr-x. 1 root root 539374222 Aug  8 05:02 initrd.cgz
-rwxr-xr-x. 1 root root  10838352 Aug  8 05:02 kernel