rear should automatically use ebiso if UEFI bootloader is found on SLES/OpenSUSE/…

[thomas-merz]

• ReaR version ("/usr/sbin/rear -V"): Relax-and-Recover 2.7 / 2022-07-13

• OS version ("cat /etc/os-release" or "lsb_release -a" or "cat /etc/rear/os.conf"): SUSE Linux Enterprise Server 15 SP4

• ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat /etc/rear/local.conf"):

OUTPUT=ISO
BACKUP=CDM
OUTPUT_URL=file:///rear/iso
OUTPUT_PREFIX="sap-testdt8-02"
export TMPDIR="/tmp"
TIMESYNC=NTP
NETFS_KEEP_OLD_BACKUP_COPY=N
EXCLUDE_VG=()
ONLY_INCLUDE_VG=(systemvg binvg hanasharedvg)
WAIT_SECS=120
SKIP_CFG2HTML=Y
USE_CFG2HTML=N

No site.conf at all.

• Hardware vendor/product (PC or PowerNV BareMetal or ARM) or VM (KVM guest or PowerVM LPAR): VMware

• System architecture (x86 compatible or PPC64/PPC64LE or what exact ARM device): x86

• Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or ELILO or Petitboot): UEFI

• Storage (local disk or SSD) and/or SAN (FC or iSCSI or FCoE) and/or multipath (DM or NVMe): local disk(s)

• Storage layout ("lsblk -ipo NAME,KNAME,PKNAME,TRAN,TYPE,FSTYPE,LABEL,SIZE,MOUNTPOINT"):

NAME                                       KNAME      PKNAME    TRAN TYPE FSTYPE      LABEL  SIZE MOUNTPOINT
/dev/sda                                   /dev/sda                  disk                    120G
|-/dev/sda1                                /dev/sda1  /dev/sda       part vfat               512M /boot/efi
|-/dev/sda2                                /dev/sda2  /dev/sda       part ext3               512M /boot
`-/dev/sda3                                /dev/sda3  /dev/sda       part LVM2_member        119G
  |-/dev/mapper/systemvg-usr_lv            /dev/dm-0  /dev/sda3      lvm  xfs                 15G /usr
  |-/dev/mapper/systemvg-swap_lv           /dev/dm-1  /dev/sda3      lvm  swap                16G [SWAP]
  |-/dev/mapper/systemvg-root_lv           /dev/dm-2  /dev/sda3      lvm  xfs                 15G /
  |-/dev/mapper/systemvg-opt_lv            /dev/dm-5  /dev/sda3      lvm  xfs                  5G /opt
  |-/dev/mapper/systemvg-var_lv            /dev/dm-7  /dev/sda3      lvm  xfs                 10G /var
  |-/dev/mapper/systemvg-varlogaudit_lv    /dev/dm-8  /dev/sda3      lvm  xfs                 10G /var/log/audit
  |-/dev/mapper/systemvg-varlog_lv         /dev/dm-9  /dev/sda3      lvm  xfs                 10G /var/log
  |-/dev/mapper/systemvg-vartmp_lv         /dev/dm-11 /dev/sda3      lvm  xfs                 10G /var/tmp
  |-/dev/mapper/systemvg-tmp_lv            /dev/dm-13 /dev/sda3      lvm  xfs                 10G /tmp
  `-/dev/mapper/systemvg-home_lv           /dev/dm-15 /dev/sda3      lvm  xfs                 15G /home
/dev/sdb                                   /dev/sdb                  disk                    200G
`-/dev/sdb1                                /dev/sdb1  /dev/sdb       part LVM2_member        200G
  |-/dev/mapper/binvg-usrsap_lv            /dev/dm-10 /dev/sdb1      lvm  xfs                 50G /usr/sap
  |-/dev/mapper/binvg-saphome_lv           /dev/dm-12 /dev/sdb1      lvm  xfs                  2G /home/sap
  |-/dev/mapper/binvg-uc4home_lv           /dev/dm-14 /dev/sdb1      lvm  xfs                  2G /home/uc4
  `-/dev/mapper/binvg-sapinst_lv           /dev/dm-16 /dev/sdb1      lvm  xfs                115G /sapinst
/dev/sdc                                   /dev/sdc                  disk                    100G
`-/dev/sdc1                                /dev/sdc1  /dev/sdc       part LVM2_member        100G
  `-/dev/mapper/hanadatavg-hanadata_lv     /dev/dm-3  /dev/sdc1      lvm  xfs                100G /hana/data
/dev/sdd                                   /dev/sdd                  disk                    100G
`-/dev/sdd1                                /dev/sdd1  /dev/sdd       part LVM2_member        100G
  `-/dev/mapper/hanasharedvg-hanashared_lv /dev/dm-4  /dev/sdd1      lvm  xfs                100G /hana/shared
/dev/sde                                   /dev/sde                  disk                     50G
`-/dev/sde1                                /dev/sde1  /dev/sde       part LVM2_member         50G
  `-/dev/mapper/hanalogvg-hanalog_lv       /dev/dm-6  /dev/sde1      lvm  xfs                 50G /hana/log
/dev/sr0                                   /dev/sr0             ata  rom                    1024M

* Description of the issue (ideally so that others can reproduce it):

ERROR: Could not create ISO image (with /usr/bin/mkisofs)

• Workaround, if any:
  Setting ISO_MKISOFS_BIN=/usr/bin/ebiso in site.conf via https://www.suse.com/support/kb/doc/?id=000019856

• Attachments, as applicable ("rear -D mkrescue/mkbackup/recover" debug log files):
  rear.log

==> Please let rear automatically use ebiso if UEFI bootloader is found, so that no one has to explicit set it.

Resolution
If your system boots with a UEFI boot loader, install the package ebiso and add the following line into /etc/rear/local.conf:
ISO_MKISOFS_BIN=/usr/bin/ebiso
Cause
To allow disaster recovery on UEFI systems, you need at least Rear version 1.18.a and the package ebiso.
Only this version supports the new helper tool /usr/bin/ebiso.
This helper tool is used to create a UEFI-bootable Rear system ISO image.

[thomas-merz]

I asked Ralph from SUSE some weeks ago to contact @jsmeix from SUSE directly.

[jsmeix]

@thomas-merz
I cannot reply to your request in a helpful way because
I am not at all an expert in bootloader area
and even less an expert in UEFI area.
I am also not at all an expert in making bootable ISO images
and even less an expert in making UEFI bootable ISO images.

So I can neither comment whether or not it is possible
to be implemented so that it works reasonably well
nor do I have an idea how much effort it could be
to implement it in a reasonable way.

Could you explain your reason why configuring ReaR
(i.e. with ISO_MKISOFS_BIN=/usr/bin/ebiso)
is not feasible for your particular use case?

[thomas-merz]

Hello @jsmeix , I just wanted that someone Rear-related from SUSE is aware of this and got no reaction since opening this issue…
I also found an older (closed, not fixed) issue opened by you with the exact same issue/idea: #805 😄

Regarding your question:

We see that Rear is already aware of "this is system has been booted with EFI" and it should know that mkisofs (without extra parameters?) can't make an ISO for UEFI boot.

But why should the user (or admin of many systems) be responsible to tell Rear what it already knows? Please think of your customers that have many systems - some EFI and some not - and need to configure this! So please let sysadmins "relax" by letting Rear do what needs to be done 😃

Maybe @schlomo oder @gdha can explain because I found both very interested in this discussion: https://relax-and-recover.org/rear-user-guide/issues/2015-09-18.657.pr.merged.html

[jsmeix]

@thomas-merz
to avoid misunderstandings:

It is not that I don't understand why you request
that ReaR should do things automatically.
I do fully understand that desire and I have the same.
Almost each day I wished computers won't behave so "utterly stupid"
when "obvious stuff" does not work and needs awkward interaction
(most annoyance with Windows and Android, far less with Linux).

All I say here is that currently I cannot implement it
because I do not have the needed overall understanding
how all the related code in ReaR works or is meant to work
and I am not alone, cf.
#657 (comment)

code around UEFI_BOOTLOADER I find it very complex (meaning that
by reading it I don't understand exactly what goes on there)

see also my
#2980 (comment)

Simply put:
Currently we at ReaR upstream do not have an active contributor
who is an expert in the booting area.

Something (hopefully) more constructive:

Regarding
"many systems - some EFI and some not - and need to configure this":

Do I understand it right that your actual issue is
that you like to use one same /etc/rear/local.conf file
for all your systems so that you don't need to configure
each system manually and separately?

If my assumtion is right, here a general note
that could perhaps help in your specific case:

ReaR config files are read via 'source'.
Because 'source' executes the content as bash script
you can run commands within your configuration files,
in particular commands to set different configuration values
depending on certain conditions as you need like

CONDITION_COMMAND && VARIABLE="special_value" || VARIABLE="usual_value"

cf.
https://github.com/rear/rear/blob/master/etc/rear/local.conf

If you can distinguish your systems that boot with UEFI
from those that boot with BIOS via some CONDITION_COMMAND
you could set the right ISO_MKISOFS_BIN automatically.

[thomas-merz]

Currently we at ReaR upstream do not have an active contributor who is an expert in the booting area.

"Challenge accepted" - let's see if I find some time to have a look and try a PR…

Something (hopefully) more constructive:

Regarding "many systems - some EFI and some not - and need to configure this":

Do I understand it right that your actual issue is that you like to use one same /etc/rear/local.conf file for all your systems so that you don't need to configure each system manually and separately?

We use a config management (sorry, not SUSE Manager, but Puppet) so we just have to set an entry in a YAML file for the hosts with EFI to inform Puppet that this host needs another line in site.conf. That's really not impossible to do. But… I thought that if Rear already knows about EFI it could or it should already use the right tool - automatically.

CONDITION_COMMAND && VARIABLE="special_value" || VARIABLE="usual_value"

I understand, but to make it more complicated:
rear mkrescue is executed remotely by our backup system via a backup agent everytime a system is backupped… We could put this into a script and let the script be executed… and so on… this feels very "hacky" and not very "relaxing" 😉

Please give me some time to have a look into it and if a PR might be possible to make and if you can adopt/adapt it…

[jsmeix]

@thomas-merz
I do very much appreciate it that you yourself
will have a look and try to contribute a PR!

Your PR does not need to be some kind of "general solution".
Just what works for you in your particular environment
so we have a starting point wherefrom further development
can be done as needed.

The basically only mandatory condition for a PR is
that things behave reasonably backward compatible, cf.
https://github.com/rear/rear/wiki/Coding-Style#maintain-backward-compatibility

Of course I will help you as good as I can.

Many thanks in advance for your contribution!

[jsmeix]

@thomas-merz
before you invest time right now to find out
how to let ReaR use 'ebiso' automatically
please wait a bit
because I am currently testing something:

Based on my testing of UEFI booting
in #3025
and #3031
on a KVM/QEMU VM with OVMF "TianoCore" UEFI firmware
with SLES15-SP4
it seems UEFI booting had worked (at least for me)
with the nowadays default /usr/bin/xorrisofs

# xorrisofs is now used as the preferred method for generating the iso image
# with mkisofs and genisoimage as second and third option
ISO_MKISOFS_BIN="$( type -p xorrisofs || type -p mkisofs || type -p genisoimage )"

see default.conf online for ReaR 2.7 at
https://github.com/rear/rear/blob/rear-2.7/usr/share/rear/conf/default.conf#L893

So perhaps 'ebiso' has meanwhile become obsoleted by 'xorrisofs'?

@thomas-merz
did you try out - and if not could you try out - whether or not
UEFI booting works in your case when you use xorrisofs?

On my SLES15-SP4 system I have the RPM package
xorriso-1.4.6-1.29.x86_64 installed
that is normally available for SLE15:

# rpm -qi xorriso
...
Packager    : https://www.suse.com/
Vendor      : SUSE LLC <https://www.suse.com/>
...
Distribution: SUSE Linux Enterprise 15

# zypper info xorriso
...
Repository     : sle-module-basesystem
Name           : xorriso
Version        : 1.4.6-1.29
Arch           : x86_64
Vendor         : SUSE LLC <https://www.suse.com/>
Support Level  : Level 3

[pcahyna]

If SUSE ships xorriso and it works, it would be the best option. Otherwise it possibly could be enough to add ebiso to the line

rear/usr/share/rear/conf/default.conf

Line 1023 in 7c5c9bc

ISO_MKISOFS_BIN="$( type -p xorrisofs || type -p mkisofs || type -p genisoimage )"

ahead of mkisofs and genisoimage? Although "ebiso silently corrupts files greater or equal 2GiB" (cf #2525) does not sound very encouraging for preferring ebiso, but perhaps mkisofs is no better in this respect.

[jsmeix]

@pcahyna
as far as I know 'ebiso' is only for EFI boot
but not for legacy BIOS boot.

What I found at its upstream location
https://gitlab.com/gozora/ebiso
all documentation tells only about [U]EFI and
its sources contain neither 'bios' nor 'legacy' (with ignore case).

This is my primary problem:
When 'ebiso' cannot be used for BIOS boot
a reliably working method is needed
to distinguish [U]EFI boot from BIOS boot.

But there is the general problem that
it is impossible to determine in a reliable way
how a running system was actually booted, cf. the
section "Disaster recovery does not just work" in
https://en.opensuse.org/SDB:Disaster_Recovery

[thomas-merz]

I will test with xorriso on friday and give feedback. This might be a much easier solution - for maintainers (just an update in docu) and for users/admins/customers…!

[jsmeix]

Yesterday and today I tested UEFI booting with OUTPUT=ISO
on the same KVM/QEMU VM with OVMF "TianoCore" UEFI firmware
with SLES15-SP4 that I had used with OUTPUT=USB
in #3025
and #3031

I did not use the ReaR 2.7 release
but a bit older 'git clone' of our current GitHub master code.
I used what there was already on that KVM/QEMU VM which is exactly

# git log | head -n7
commit 283efdaea10ff62dc94e968f74e1136b8384a954
Merge: 41c2d9b1 70a39382
Author: Johannes Meixner <jsmeix@suse.com>
Date:   Fri Jul 21 14:56:34 2023 +0200

    Merge pull request #3025 from rear/jsmeix-create_grub2_cfg

Yesterday I tested without Secure Boot
which worked for me.

Today I tested with Secure Boot
which also worked for me.

For Secure Boot I specify in etc/rear/local.conf

SECURE_BOOT_BOOTLOADER="/boot/efi/EFI/sles/shim.efi"

cf.
#3025 (comment)

My whole etc/rear/local.conf

FIRMWARE_FILES=( 'no' )
MODULES=( 'loaded_modules' )
PROGRESS_MODE="plain"
PROGRESS_WAIT_SECONDS="5"
SSH_ROOT_PASSWORD="rear"
USE_DHCLIENT="yes"
USE_SERIAL_CONSOLE="no"
SECURE_BOOT_BOOTLOADER="/boot/efi/EFI/sles/shim.efi"
OUTPUT=ISO
BACKUP=NETFS
BACKUP_OPTIONS="nfsvers=3,nolock"
BACKUP_URL=nfs://192.168.122.1/nfs

Excerpts from my today's "rear -D mkbackup":

Found EFI system partition /dev/vda1 on /boot/efi type vfat
Using UEFI Boot Loader for Linux (USING_UEFI_BOOTLOADER=1)
Using '/usr/bin/xorrisofs' to create ISO filesystem images
...
Using '/boot/efi/EFI/sles/shim.efi' as UEFI Secure Boot bootloader file
...
Using Shim '/boot/efi/EFI/sles/shim.efi' as first stage UEFI bootloader BOOTX64.efi
Using second stage UEFI bootloader files for Shim: /boot/efi/EFI/sles/grub.efi /boot/efi/EFI/sles/grubx64.efi
Let GRUB2 load kernel /isolinux/kernel
Let GRUB2 load initrd /isolinux/initrd.cgz
Set GRUB2 default root device via 'set root=cd0'
Let GRUB2 search root device via 'search --no-floppy --set=root --file /boot/efiboot.img'
...
Making ISO image
Wrote ISO image: /root/rear/var/lib/rear/output/rear-localhost.iso (180M)

Excerpts from my today's "rear -D recover":

Start system layout restoration.
Disk '/dev/vda': creating 'gpt' partition table
Disk '/dev/vda': creating partition number 1 with name ''vda1''
Disk '/dev/vda': creating partition number 2 with name ''vda2''
Disk '/dev/vda': creating partition number 3 with name ''vda3''
Creating filesystem of type ext4 with mount point / on /dev/vda2.
Mounting filesystem /
Creating filesystem of type vfat with mount point /boot/efi on /dev/vda1.
Mounting filesystem /boot/efi
Creating swap on /dev/vda3
Disk layout created.
...
Creating EFI Boot Manager entries...
Creating  EFI Boot Manager entry 'SUSE_LINUX 15.4' for 'EFI\sles\shim.efi' (UEFI_BOOTLOADER='/boot/efi/EFI/sles/shim.efi') 
Installing secure boot loader (shim)...

After reeboot of the recreated system
I had the minor issue hat I described at
#3025 (comment)

Reeboot of the recreated system works with Secure Boot enabled
(at least on my KVM/QEMU VM with OVMF "TianoCore" UEFI firmware).
After GRUB loaded kernel and initrd
the message

EFI stub: UEFI Secure Boot is enabled.

is visible for a short time (about one second or so).

[gdha]

@jsmeix SECURE_BOOT_BOOTLOADER="/boot/efi/EFI/sles/shim.efi" - was it not possible that ReaR guessed this automatically?

[jsmeix]

@gdha
please just implement what you ask for
(SECURE_BOOT_BOOTLOADER is nowhere set to a non-empty value)

# find usr/sbin/rear usr/share/rear -type f | xargs grep 'SECURE_BOOT_BOOTLOADER='

usr/share/rear/conf/default.conf:# 1. UEFI boot without secure boot (SECURE_BOOT_BOOTLOADER="")
usr/share/rear/conf/default.conf:# For example: SECURE_BOOT_BOOTLOADER="/boot/efi/EFI/BOOT/shim.efi"
usr/share/rear/conf/default.conf:# so when for example SECURE_BOOT_BOOTLOADER="/boot/efi/EFI/BOOT/shim.efi" is specified
usr/share/rear/conf/default.conf:SECURE_BOOT_BOOTLOADER=""

See also
#3031 (comment)

[jsmeix]

@rear/contributors
could you - please - do you and me a favour
and contribute when you like to improve ReaR
instead of asking me that I should do for you
what you think is needed to make you feel better?

To avoid misunderstandings:
When someone pays to get ReaR improved
then the one can of course determine
what others should do for the money
provided both parties agree on a contract.

[thomas-merz]

With xorriso installed I don't need to set ISO_MKISOFS_BIN=/usr/bin/ebiso anymore. So with the help/tipp/hint of @jsmeix I solved my "problem" 👍🏼

I also tested booting the ISO that xorriso created with success. So I will install it on all SLES15 servers and let Rear use it instead of mkisofs.

[jsmeix]

@thomas-merz
thank you very much for your testing and your feedback!
It helps so much to have feedback how things behave
for users "in real world out there" - and in particular
positive feedback when things work (reasonably) well.

I will adapt the ReaR upstream documentation soon.
I will also have a look at the SUSE documentation.

[thomas-merz]

So shall we close this issue or do you need it as a reminder for updating docu(s)?

[jsmeix]

I like to keep it open until I adapted
the ReaR upstream documentation.

[schlomo]

Can it be that the "problem" is that SUSE LINUX installs by default ebiso and not xorriso?

So we could solve the problem by adding a dependency on xorriso in the SUSE RPMs (or maybe even all RPMs, if xorriso is our preferred tool)?

If we don't want to add such a dependency then I'd like us to add auto-detection for ebiso as suggested in #3084 (comment)

In any case, I would expect from ReaR to work well out-of-the-box even for SUSE LINUX and UEFI is the standard boot method for most systems nowadays.

[jsmeix]

When xorriso works sufficiently well
to make a UEFI bootable ISO image
then I would prefer to phase out ebiso
because I think xorriso is standard software
on nowadays Linux distributions.

In contrast ebiso is no such kind of standard software.
I was developed by @gozora and (if I remember correctly)
it was developed at least to some extent as band-aid
for an isse that SUSE created with its early SLES12.
At that time (i.e. when SLES12 came out)
SUSE did not provide some standard software tool
to make a UEFI bootable ISO image regardless that SUSE
"supported" UEFI boot but it seems that had only meant
booting via UEFI but not also making a UEFI bootable ISO.
So with early SLES12 (before SLES12-SP2 as far as I see)
one could not make a UEFI bootable ISO on a UEFI system :-(
I will always be grateful to @gozora that he solved
this SUSE specific issue!

[jsmeix]

Regarding adding a RPM dependency in
https://github.com/rear/rear/blob/master/packaging/rpm/rear.spec
to get xorriso:

Certainly doable but as far as I see at first glance
this could become rather ugly legwork because it seems
there is no common standard what RPM package names
and/or RPM capablity names are used in Linux distributions
to specify "THE common standard tool to make ISO images".

[pcahyna]

@jsmeix the same could be said of any other dependencies in the spec file, but in practice the situation is not that bad, we have had

Requires:   xorriso

in the RHEL spec since RHEL 7, so this change would be ok (even preferred) on all RHEL versions since then, also on Fedora.

[jsmeix]

Is a hard RPM requirement i.e. Requires: something really needed?

I assume Requires: ISO-making-tool is only needed for OUTPUT=ISO
so RPM Recommends: ISO-making-tool would be better
because a hard RPM requirement cannot be skipped by the user
without having having unresolved RPM dependencies.

Do RHEL and Fedora support RPM Recommends: ... ?

Cf.
https://bugzilla.opensuse.org/show_bug.cgi?id=776080#c39

[pcahyna]

@gdha please just implement what you ask for (SECURE_BOOT_BOOTLOADER is nowhere set to a non-empty value)

# find usr/sbin/rear usr/share/rear -type f | xargs grep 'SECURE_BOOT_BOOTLOADER='

usr/share/rear/conf/default.conf:# 1. UEFI boot without secure boot (SECURE_BOOT_BOOTLOADER="")
usr/share/rear/conf/default.conf:# For example: SECURE_BOOT_BOOTLOADER="/boot/efi/EFI/BOOT/shim.efi"
usr/share/rear/conf/default.conf:# so when for example SECURE_BOOT_BOOTLOADER="/boot/efi/EFI/BOOT/shim.efi" is specified
usr/share/rear/conf/default.conf:SECURE_BOOT_BOOTLOADER=""

See also #3031 (comment)

I am working on it. Mostly done, actually. Can anyone test the ISO part, please?