Ubuntu 20.04 initrd "decoding failed" - needs special command to make initrd

[will-code-for-pizza]

Relax-and-Recover (ReaR) Issue Template

Fill in the following items before submitting a new issue
(quick response is not guaranteed with free support):

• ReaR version ("/usr/sbin/rear -V"):
  Relax-and-Recover 2.5 / Git (Ubuntu 20.04 distro package)

• OS version ("cat /etc/os-release" or "lsb_release -a" or "cat /etc/rear/os.conf"):

NAME="Ubuntu"
VERSION="20.04.3 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.3 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

• ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat /etc/rear/local.conf"):

/etc/rear/site.conf

BACKUP=NETFS
OUTPUT=ISO
USE_CFG2HTML=y
BACKUP_PROG=tar
BACKUP_PROG_COMPRESS_OPTIONS="--gzip"
LOGFILE="/var/log/rear/$HOSTNAME.log"
BACKUP_PROG_EXCLUDE=( '/media/*' '/mnt/*' '/BACKUP/*' '/cache/*' '/tmp/*' '/dev/shm/*' $VAR_DIR/output/\* )

/etc/rear/local.conf

NETFS_KEEP_OLD_BACKUP_COPY=yes
KEEP_BUILD_DIR="no"
BACKUP_URL=nfs://139.1.xxx.xxx/srv/IMAGES/rear/myserver
EXCLUDE_COMPONENTS=( 'swap:/dev/mapper/rootvg-swaplv' 'fs:/tmp' 'fs:/var/cache' )
REQUIRED_PROGS+=( 'ifenslave' )
GRUB2_MODULES_LOAD+=( 'bonding' )

• Hardware vendor/product (PC or PowerNV BareMetal or ARM) or VM (KVM guest or PowerVM LPAR):

HPe Proliant DL580 Gen8

• System architecture (x86 compatible or PPC64/PPC64LE or what exact ARM device):

x86

• Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or ELILO or Petitboot):

Legacy BIOS activated, GRUB

• Storage (local disk or SSD) and/or SAN (FC or iSCSI or FCoE) and/or multipath (DM or NVMe):

local disks (HDD and SSD)

• Storage layout ("lsblk -ipo NAME,KNAME,PKNAME,TRAN,TYPE,FSTYPE,SIZE,MOUNTPOINT"):

# lsblk -ipo NAME,KNAME,PKNAME,TRAN,TYPE,FSTYPE,SIZE,MOUNTPOINT
NAME                            KNAME      PKNAME    TRAN TYPE FSTYPE        SIZE MOUNTPOINT
/dev/loop0                      /dev/loop0                loop squashfs     61.8M /snap/core20/1081
/dev/loop1                      /dev/loop1                loop squashfs     55.5M /snap/core18/2074
/dev/loop2                      /dev/loop2                loop squashfs     67.6M /snap/lxd/20326
/dev/loop3                      /dev/loop3                loop squashfs     67.3M /snap/lxd/21545
/dev/loop4                      /dev/loop4                loop squashfs     32.3M /snap/snapd/12883
/dev/loop6                      /dev/loop6                loop squashfs     55.4M /snap/core18/2128
/dev/loop7                      /dev/loop7                loop squashfs     32.3M /snap/snapd/13170
/dev/sda                        /dev/sda             sas  disk             558.7G
|-/dev/sda1                     /dev/sda1  /dev/sda       part                 1M
|-/dev/sda2                     /dev/sda2  /dev/sda       part ext4            2G /boot
`-/dev/sda3                     /dev/sda3  /dev/sda       part LVM2_member 556.7G
  |-/dev/mapper/rootvg-rootlv   /dev/dm-2  /dev/sda3      lvm  ext4            4G /
  |-/dev/mapper/rootvg-swaplv   /dev/dm-3  /dev/sda3      lvm  swap            4G [SWAP]
  |-/dev/mapper/rootvg-tmplv    /dev/dm-4  /dev/sda3      lvm  ext4            4G /tmp
  |-/dev/mapper/rootvg-varlv    /dev/dm-5  /dev/sda3      lvm  ext4            6G /var
  |-/dev/mapper/rootvg-varloglv /dev/dm-6  /dev/sda3      lvm  ext4            2G /var/log
  |-/dev/mapper/rootvg-homelv   /dev/dm-7  /dev/sda3      lvm  ext4            8G /home
  |-/dev/mapper/rootvg-usrlv    /dev/dm-8  /dev/sda3      lvm  ext4            4G /usr
  `-/dev/mapper/rootvg-optlv    /dev/dm-9  /dev/sda3      lvm  ext4            2G /opt
/dev/sdb                        /dev/sdb             sas  disk LVM2_member   4.6T
|-/dev/mapper/datavg-varliblv   /dev/dm-0  /dev/sdb       lvm  ext4            2G /var/lib
`-/dev/mapper/datavg-srvlv      /dev/dm-1  /dev/sdb       lvm  ext4            2G /srv

• Description of the issue (ideally so that others can reproduce it):

After 'mkrescue' and 'mkbackuponly' without any errors I tried to restore the machine,
bootin´ with rescue-iso-image.

System loads ISO:

Rescue image kernel 5.4.0-84-generic Thu, 30 Sept 2021 15:35:23 +0200
BACKUP_NETFS OUTPU=ISO BACKUP_URL=nfs://139.1.xxx.xxx/srv/IMAGES/rear/myserver
Loading kernel... ok
Loading initrd.cgz...ok

Decoding failed

--System halted

• Workaround, if any:

No workaround

• Attachments, as applicable ("rear -D mkrescue/mkbackup/recover" debug log files):

No attachements

Thanks in advance !
Cheers!

[jsmeix]

By googling for initrd "decoding failed" I found e.g.
https://forums.linuxmint.com/viewtopic.php?t=323152
and
https://askubuntu.com/questions/1269855/usb-installer-initramfs-unpacking-failed-decoding-failed
that both have links to (rather old from 2019)
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1835660
so it seems the root cause is in Ubuntu and not in ReaR
but I am not a Ubuntu user so I cannot check Ubuntu issues myself.

Specifying an appropriate REAR_INITRD_COMPRESSION (see default.conf)
should help for systems that cannot decompress the ReaR default initrd.

[will-code-for-pizza]

Hi,
unfortunatelly none of the compression options worked for me.

REAR_INITRD_COMPRESSION="fast"
REAR_INITRD_COMPRESSION="best"
REAR_INITRD_COMPRESSION="lzma"
REAR_INITRD_COMPRESSION="lz4"

All options result in "decoding failed" --- System halted.

Regards
Will Code

[jsmeix]

@will-code-for-pizza
I think you need to find out what the right command is
that creates an initrd for the specific system you have
so that this initrd can then be decompressed/decoded
by the specific kernel that runs on your particular system.

When you know that command you can modify your
usr/share/rear/pack/GNU/Linux/900_create_initramfs.sh
to let it run that command instead of what it currently does.
If it then works to boot the ReaR recovery system in your case
please tell us that exact command that created the initrd for you
so we could have a look what the differences are compared to
what 900_create_initramfs.sh currently does so that we
may finally enhance 900_create_initramfs.sh accordingly.
accordingly.

[will-code-for-pizza]

Ok, I will do that.
Many thanks for the hint. Give me a few hours, please.
Cheers.

[will-code-for-pizza]

After some days of searching the internet, I found serveral depending issues.
The command for creating initrd in Ubuntu 20.04 is done in 'mkinitramfs'.

Regards.

[will-code-for-pizza]

I think, I found it....

/usr/share/rear/pack/GNU/Linux/900_create_initramfs.sh (ReaR)

(lz4)
cpio -H newc --create --quiet | lz4 -l > "$TMP_DIR/$REAR_INITRD_FILENAME"

(lzma)
cpio -H newc --create --quiet | xz --format=lzma --compress --stdout > "$TMP_DIR/$REAR_INITRD_FILENAME"

(fast)
cpio -H newc --create --quiet | gzip --fast > "$TMP_DIR/$REAR_INITRD_FILENAME"

(best)
cpio -H newc --create --quiet | gzip --best > "$TMP_DIR/$REAR_INITRD_FILENAME"

(*)
cpio -H newc --create --quiet | gzip > "$TMP_DIR/$REAR_INITRD_FILENAME"

/usr/sbin/mkinitramfs (Ubuntu 20.04)

cpio --quiet $cpio_owner_root --reproducible -o -H newc | gzip >>"${outfile}"

cpio --quiet $cpio_owner_root --reproducible -o -H newc 4>&-; echo "ec2=$?;" >&4

where

cpio_owner_root="-R 0:0"

[will-code-for-pizza]

So I will create a new
(custom) entry in /usr/share/rear/pack/GNU/Linux/900_create_initramfs.sh

(custom)
cpio -R 0:0 --reproducible -o -H newc | gzip > "$TMP_DIR/$REAR_INITRD_FILENAME"

and try out...

[will-code-for-pizza]

SOLVED !

/etc/rear/local.conf

REAR_INITRD_COMPRESSION="ubuntu2004"

/usr/share/rear/pack/GNU/Linux/900_create_initramfs.sh

    (ubuntu2004)
        # Create initrd.cgz with gzip compression (the UBUNTU way)
        REAR_INITRD_FILENAME="initrd.cgz"
        LogPrint "Creating recovery/rescue system initramfs/initrd $REAR_INITRD_FILENAME with gzip the UBUNTU way"
        if find . ! -name "*~" | cpio -R 0:0 --reproducible -o -H newc | gzip > "$TMP_DIR/$REAR_INITRD_FILENAME" ; then
            needed_seconds=$(( $( date +%s ) - start_seconds ))
            LogPrint "Created $REAR_INITRD_FILENAME with gzip the UBUNTU way ($( stat -c%s $TMP_DIR/$REAR_INITRD_FILENAME ) bytes) in $needed_seconds seconds"
        else
            # No need to clean up things (like 'popd') because Error exits directly:
            Error "Failed to create recovery/rescue system $REAR_INITRD_FILENAME"
        fi
        ;;

[jsmeix]

@will-code-for-pizza
thank you for your analysis what the root cause is
and for your solution!

I will add your solution to the ReaR code.

[will-code-for-pizza]

@jsmeix: You´re welcome. This is how OpenSource should work ;-)
Thanks for your support.
Cheers!

[will-code-for-pizza]

Unfortunatelly, the error occurs again.
I have to investigate deeper.

Booting into "Recover" succeeded most of the time. Booting in "Automatic recover" always failed.

Sorry.

[jsmeix]

@will-code-for-pizza
do you mean booting in "Automatic recover" always fails with

Loading kernel... ok
Loading initrd.cgz...ok
Decoding failed

as in your initial #2688 (comment)
or does booting in "Automatic recover" fail differently?

I ask because offhandedly I could not imagine
how booting in "Automatic recover" mode
could make a difference about initrd decoding
because one same initrd is used.

[jsmeix]

@will-code-for-pizza
what looks a bit strange is that we at ReaR upstream
do not have another same issue report from another user
but I know that there are several users with Ubuntu 20.04
so I wonder how ReaR with Ubuntu 20.04 works for them?

I.e. it could be something special in your environment
why no other ReaR users with Ubuntu 20.04
seem to also have this "Decoding failed" issue?

[will-code-for-pizza]

Hi @jsmeix and thanks for reply.
Sorry for the late answer from my site.

For me, the problem described in #2688 (comment) often occurs while choosing "Automatic recover".
I have to investigate further to be totally sure and will report my findings here.

Your arguments about Ubuntu 20.04 and a bunch of other users WITHOUT problems seems logical.
Probably it could depend on networking issues.

I´ll be back soon.

Cheers!

[DEvil0000]

I am running on ubuntu 20.04 and did not face such a issue. I however never tested auto recover. The command change however is unlikely to make a difference.
I use output=ISO and output=USB normally with backup=borg on HP and other hardware.

Edit:
did you try with rear 2.6 instead of 2.5? Or with master?
I don't think this is a rear issue at all. Maybe ubuntu but I don't think so. Did you update your HP firmware?
HP firmware tends to have all sorts of crazy bugs.
Thinking a bit more about it this may be bootloader config or kernel commandline options as well. Since systemd the commandline options and startup changed a bit. Not sure how automatic recover is done.

[github-actions]

Stale issue message