Failed to create recovery system (too small ESP on USB device - no space for .../EFI/BOOT/initrd.cgz)

[gaia]

Relax-and-Recover (ReaR) Issue Template

• ReaR version:
  Relax-and-Recover 2.4

• OS version:
  Debian 10 (buster)

• ReaR configuration:

OUTPUT=USB
BACKUP=BORG
USB_DEVICE=/dev/disk/by-label/REAR-000
###
USB_DEVICE_PARTED_LABEL=gpt
USB_DEVICE_FILESYSTEM=ext4
USING_UEFI_BOOTLOADER=1
UEFI_BOOTLOADER="/boot/efi/EFI/grubx64.efi"
###
SSH_UNPROTECTED_PRIVATE_KEYS="no"
SSH_FILES="avoid_sensitive_files"
###
EXCLUDE_MOUNTPOINTS=( '/zfs_containers2' '/mnt/storage' '/mnt/containers' '/mnt/usb' '/mnt/bindmounts' '/mnt/storage/swapfile' )
EXCLUDE_MD=( '/dev/md1' )
EXCLUDE_VG=( '/dev/mapper/lvmthin0' '/dev/mapper/lvmthin1' )
###
BORGBACKUP_REPO="/my_borg_backup"
BORGBACKUP_UMASK="0002"
BORGBACKUP_PRUNE_DAILY=7
BORGBACKUP_ENC_TYPE="keyfile"
BORGBACKUP_COMPRESSION="lz4"
BORGBACKUP_SHOW_PROGRESS="yes"
BORGBACKUP_SHOW_STATS="yes"
export BORG_PASSPHRASE='__________'
export BORG_RELOCATED_REPO_ACCESS_IS_OK="yes"
export BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK="yes"
###
PROGS_BORG=( )
BACKUP_PROG_INCLUDE=('/etc/pve/' )
COPY_AS_IS_BORG=( '/root/.config/borg/keys/' )
COPY_AS_IS_EXCLUDE=( "${COPY_AS_IS_EXCLUDE[@]}" 'home/*/.cache/*' 'var/cache/*' 'var/tmp/*' 'dev/*' 'media/*' 'proc/*' 'sys/*' 'tmp/*' 'var/run/*' 'var/lock/*' 'var/lib/apt/lists/*' '/var/cache/apt/*' )
COPY_AS_IS=( $SHARE_DIR $VAR_DIR )

• Hardware:
  Bare Metal Xeon 9th gen

• System architecture:
  x64

• Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or ELILO or Petitboot):
  UEFI + Grub2

• Storage (local disk or SSD) and/or SAN (FC or iSCSI or FCoE) and/or multipath (DM or NVMe):
  Spinning Rust + NVME disks

• Storage layout ("lsblk -ipo NAME,KNAME,PKNAME,TRAN,TYPE,FSTYPE,SIZE,MOUNTPOINT" or "lsblk" as makeshift):

• Description of the issue:

2020-11-24 04:30:34.623833405 ======================
2020-11-24 04:30:34.624841108 Running 'backup' stage
2020-11-24 04:30:34.626411395 ======================
2020-11-24 04:30:35.002765598 Including backup/default/005_valid_backup_methods.sh
2020-11-24 04:30:35.210903671 Including backup/default/010_pre_backup_script.sh
2020-11-24 04:30:35.261015771 Including backup/BORG/default/100_get_suffix.sh
2020-11-24 04:30:35.321998787 Including backup/BORG/default/250_mount_usb.sh
2020-11-24 04:30:35.335243934 Mounting with 'mount -v -o rw,noatime /dev/disk/by-label/REAR-000 /tmp/rear.60xzwC41QJxGTjm/borg_backup'
mount: /dev/sde2 mounted on /tmp/rear.60xzwC41QJxGTjm/borg_backup.
2020-11-24 04:30:35.654866407 Including backup/BORG/default/400_create_include_exclude_files.sh
2020-11-24 04:30:35.839167065 Including backup/BORG/default/500_make_backup.sh
2020-11-24 04:30:35.887096791 Creating archive rear_270 in repository /my_borg_backup
2020-11-24 04:44:10.968795129 ERROR: Failed to create backup
==== Stack trace ====
Trace 0: /usr/sbin/rear:543 main
Trace 1: /usr/share/rear/lib/mkbackuponly-workflow.sh:16 WORKFLOW_mkbackuponly
Trace 2: /usr/share/rear/lib/framework-functions.sh:101 SourceStage
Trace 3: /usr/share/rear/lib/framework-functions.sh:49 Source
Trace 4: /usr/share/rear/backup/BORG/default/500_make_backup.sh:34 source
Trace 5: /usr/share/rear/lib/_input-output-functions.sh:371 StopIfError
Message: Failed to create backup

• Workaround, if any:
  N/A, or at least IDH. It worked under this config for a while, then it stopped.

• Attachments, as applicable ("rear -D mkrescue/mkbackup/recover" debug log files):
  N/A

[gozora]

From the logs you've provided I can see that your USB Borg repository already contains some previous backups (archives), What I mean is that you've already run some successful backups prior this error. Is that correct?
If so, can it be that your USB disk is experiencing some IO errors/HW problems?
If not, can you please provide full log from rear -d -D mkbackup session ?

Thanks

V.

[gaia]

Relax-and-Recover 2.4 / Git
Using log file: /var/log/rear/rear.log
Using UEFI Boot Loader for Linux (USING_UEFI_BOOTLOADER=1)
Creating disk layout
Excluding RAID /dev/md1.
Excluding Volume Group /dev/mapper/____.
Excluding Volume Group /dev/mapper/____
Using guessed bootloader 'EFI' (found in first bytes on /dev/sda)
Creating root filesystem layout
Skipping 'vmbr1': not bound to any physical interface.
Cannot include keyboard mappings (no keymaps default directory '')
Using '/boot/efi/EFI/proxmox/grubx64.efi' as UEFI bootloader file
Copying logfile /var/log/rear/rear.log into initramfs as '/tmp/rear-partial-2020-11-25T13:39:29-08:00.log'
Copying files and directories
Copying binaries and libraries
Copying kernel modules
Copying all files in /lib*/firmware/
Creating recovery/rescue system initramfs/initrd initrd.cgz with gzip default compression
Created initrd.cgz with gzip default compression (205115772 bytes) in 24 seconds
ERROR: Could not copy /tmp/rear.JsEmyhSq6taUJk5/tmp/initrd.cgz to /tmp/rear-efi.B2USe//EFI/BOOT/initrd.cgz
Aborting due to an error, check /var/log/rear/rear.log for details
Exiting rear mkbackup (PID 12639) and its descendant processes
Running exit tasks
You should also rm -Rf /tmp/rear.JsEmyhSq6taUJk5
Terminated

Note I altered the actual block device names in EXCLUDE_VG, that's why it does not match the other logs.

[gozora]

Can you attach /var/log/rear/rear-node2.log ?

[gaia]

the pword to the zip file is your username

thanks

[gozora]

You need bigger ESP on your USB device:

Excerpt from the log:

++ cp -p -v /tmp/rear.JsEmyhSq6taUJk5/tmp/initrd.cgz /tmp/rear-efi.B2USe//EFI/BOOT/initrd.cgz
'/tmp/rear.JsEmyhSq6taUJk5/tmp/initrd.cgz' -> '/tmp/rear-efi.B2USe//EFI/BOOT/initrd.cgz'
cp: error writing '/tmp/rear-efi.B2USe//EFI/BOOT/initrd.cgz': No space left on device
...

ReaR controls size of ESP partition by USB_UEFI_PART_SIZE variable. Be aware that after updating of USB_UEFI_PART_SIZE, you need to reformat your USB storage (rear format ...), which will obviously cause DATA LOSS.

V.

[gdha]

@gozora @jsmeix perhaps it would be a nice idea to have something like https://github.com/gdha/upgrade-ux/blob/master/opt/upgrade-ux/scripts/postexecute/default/96_call_for_action_after_preview.sh to scan for errors?

[gozora]

Hello @gdha,

I personally don't think it is hard to locate error in ReaR log file, but if you think that others might benefit, I don't mind.
Just be aware that running code from 96_call_for_action_after_preview.sh on ReaR debug log still returns lot of errors:

$ ./check_error.sh rear-node2.log | wc -l
236

V.

[gdha]

Hello @gdha,

I personally don't think it is hard to locate error in ReaR log file, but if you think that others might benefit, I don't mind.
Just be aware that running code from 96_call_for_action_after_preview.sh on ReaR debug log still returns lot of errors:

$ ./check_error.sh rear-node2.log | wc -l
236

V.

That is indeed true, but out of space can be search out of array of the most common errors for example.

[jsmeix]

ReaR 2.4 is rather old and the terminal output in
#2526 (comment)
shows that ReaR 2.4. does not have our current ReaR
enhanced Error function behaviour that always shows
Some latest log messages since the last called script ..., cf.
https://github.com/rear/rear/blob/master/usr/share/rear/lib/_input-output-functions.sh#L557

I implemented this enhanced Error function behaviour via
2420688
and
#1877
in ReaR 2.5, see doc/rear-release-notes.txt

Now the Error function shows some last messages of the last sourced
script to the user (issues #1877 #1875)

https://github.com/rear/rear/blob/master/doc/rear-release-notes.txt#L1419

For an example how current ReaR Error function behaviour
looks like on the user's terminal see the rear -v mkrescue output at
#2524 (comment)

I assume if @gaia would have used current ReaR
there would have been some more useful output about the actual error
on the terminal where "rear mkrescue/mkbackup" was run
and then this issue report would perhaps have not been needed
because the root cause would have been directly visible to the user.

I think this issue can be closed and no further attention is needed.

[jsmeix]

@gdha
only for the fun of it:
Your script
https://github.com/gdha/upgrade-ux/blob/master/opt/upgrade-ux/scripts/postexecute/default/96_call_for_action_after_preview.sh
is a "no-go" because it contains the forbidden word WARNING, cf.
https://schlomo.schapiro.org/2015/04/warning-is-waste-of-my-time.html
;-))

[jsmeix]

@gozora
thank you for your time requesting ReaR debug log files
and inspecting them to show the user the root cause!