Cannot create rescue image on Ubuntu 23.04 and Debian/Bookworm - Confirmed bug in cp 9.x #2972

schlomo · 2023-04-23T10:47:38Z

Trying out ReaR on Ubuntu 23.04 and I stumble over a very strange error:

Context

my configuration contains
```
MODULES=( 'loaded_modules' )
```
otherwise this problem is not triggered.
running from source tree mounted via vboxsf but the problem is not related to that either, also not to REAR_VAR.

Observable Problem

root@rear-u2304:/src/rear# REAR_VAR=/tmp/rear_var usr/sbin/rear -C /etc/rear/local.conf -D mkrescue
Relax-and-Recover 2.7 / Git
Running rear mkrescue (PID 55846 date 2023-04-23 10:32:31)
Command line options: usr/sbin/rear -C /etc/rear/local.conf -D mkrescue
Using log file: /tmp/rear_var/log/rear/rear-rear-u2304.log
Using build area: /var/tmp/rear.Acrvl84FH5J2jWn
Sourcing additional configuration file '/etc/rear/local.conf'
...
... lots more
...

Copying binaries and libraries
Copying only currently loaded kernel modules (MODULES contains 'loaded_modules') and those in MODULES_LOAD
ERROR: Failed to copy '/lib/modules/6.2.0-20-generic/kernel/arch/x86/crypto/aesni-intel.ko /lib/modules/6.2.0-20-generic/kernel/arch/x86/crypto/crc32-pclmul.ko /lib/modules/6.2.0-20-generic/kernel/arch/x86/crypto/crct10dif-pclmul.ko /lib/modules/6.2.0-20-generic/kernel/arch/x86/crypto/ghash-clmulni-intel.ko /lib/modules/6.2.0-20-generic/kernel/arch/x86/crypto/polyval-clmulni.ko /lib/modules/6.2.0-20-generic/kernel/arch/x86/crypto/sha512-ssse3.ko /lib/modules/6.2.0-20-generic/kernel/arch/x86/events/rapl.ko /lib/modules/6.2.0-20-generic/kernel/arch/x86/kernel/msr.ko /lib/modules/6.2.0-20-generic/kernel/crypto/async_tx/async_memcpy.ko /lib/modules/6.2.0-20-generic/kernel/crypto/async_tx/async_pq.ko /lib/modules/6.2.0-20-generic/kernel/crypto/async_tx/async_raid6_recov.ko /lib/modules/6.2.0-20-generic/kernel/crypto/async_tx/async_tx.ko /lib/modules/6.2.0-20-generic/kernel/crypto/async_tx/async_xor.ko /lib/modules/6.2.0-20-generic/kernel/crypto/blake2b_generic.ko /lib/modules/6.2.0-20-generic/kernel/crypto/cryptd.ko /lib/modules/6.2.0-20-generic/kernel/crypto/crypto_simd.ko /lib/modules/6.2.0-20-generic/kernel/crypto/polyval-generic.ko /lib/modules/6.2.0-20-generic/kernel/crypto/xor.ko /lib/modules/6.2.0-20-generic/kernel/drivers/acpi/video.ko /lib/modules/6.2.0-20-generic/kernel/drivers/ata/ahci.ko /lib/modules/6.2.0-20-generic/kernel/drivers/ata/libahci.ko /lib/modules/6.2.0-20-generic/kernel/drivers/ata/pata_acpi.ko /lib/modules/6.2.0-20-generic/kernel/drivers/firmware/dmi-sysfs.ko /lib/modules/6.2.0-20-generic/kernel/drivers/firmware/efi/efi-pstore.ko /lib/modules/6.2.0-20-generic/kernel/drivers/gpu/drm/drm.ko /lib/modules/6.2.0-20-generic/kernel/drivers/gpu/drm/drm_kms_helper.ko /lib/modules/6.2.0-20-generic/kernel/drivers/gpu/drm/drm_ttm_helper.ko /lib/modules/6.2.0-20-generic/kernel/drivers/gpu/drm/ttm/ttm.ko /lib/modules/6.2.0-20-generic/kernel/drivers/gpu/drm/vmwgfx/vmwgfx.ko /lib/modules/6.2.0-20-generic/kernel/drivers/hid/hid-generic.ko /lib/modules/6.2.0-20-generic/kernel/drivers/hid/hid.ko /lib/modules/6.2.0-20-generic/kernel/drivers/hid/usbhid/usbhid.ko /lib/modules/6.2.0-20-generic/kernel/drivers/i2c/busses/i2c-piix4.ko /lib/modules/6.2.0-20-generic/kernel/drivers/input/input-leds.ko /lib/modules/6.2.0-20-generic/kernel/drivers/input/joydev.ko /lib/modules/6.2.0-20-generic/kernel/drivers/input/mouse/psmouse.ko /lib/modules/6.2.0-20-generic/kernel/drivers/input/serio/serio_raw.ko /lib/modules/6.2.0-20-generic/kernel/drivers/macintosh/mac_hid.ko /lib/modules/6.2.0-20-generic/kernel/drivers/md/dm-multipath.ko /lib/modules/6.2.0-20-generic/kernel/drivers/md/linear.ko /lib/modules/6.2.0-20-generic/kernel/drivers/md/multipath.ko /lib/modules/6.2.0-20-generic/kernel/drivers/md/raid0.ko /lib/modules/6.2.0-20-generic/kernel/drivers/md/raid1.ko /lib/modules/6.2.0-20-generic/kernel/drivers/md/raid10.ko /lib/modules/6.2.0-20-generic/kernel/drivers/md/raid456.ko /lib/modules/6.2.0-20-generic/kernel/drivers/net/ethernet/intel/e1000/e1000.ko /lib/modules/6.2.0-20-generic/kernel/drivers/platform/x86/wmi.ko /lib/modules/6.2.0-20-generic/kernel/drivers/powercap/intel_rapl_common.ko /lib/modules/6.2.0-20-generic/kernel/drivers/powercap/intel_rapl_msr.ko /lib/modules/6.2.0-20-generic/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko /lib/modules/6.2.0-20-generic/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko /lib/modules/6.2.0-20-generic/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko /lib/modules/6.2.0-20-generic/kernel/drivers/video/fbdev/core/syscopyarea.ko /lib/modules/6.2.0-20-generic/kernel/drivers/video/fbdev/core/sysfillrect.ko /lib/modules/6.2.0-20-generic/kernel/drivers/video/fbdev/core/sysimgblt.ko /lib/modules/6.2.0-20-generic/kernel/drivers/virt/vboxguest/vboxguest.ko /lib/modules/6.2.0-20-generic/kernel/fs/autofs/autofs4.ko /lib/modules/6.2.0-20-generic/kernel/fs/binfmt_misc.ko /lib/modules/6.2.0-20-generic/kernel/fs/btrfs/btrfs.ko /lib/modules/6.2.0-20-generic/kernel/fs/fscache/fscache.ko /lib/modules/6.2.0-20-generic/kernel/fs/lockd/lockd.ko /lib/modules/6.2.0-20-generic/kernel/fs/netfs/netfs.ko /lib/modules/6.2.0-20-generic/kernel/fs/nfs/nfs.ko /lib/modules/6.2.0-20-generic/kernel/fs/nfs/nfsv4.ko /lib/modules/6.2.0-20-generic/kernel/fs/nfs_common/grace.ko /lib/modules/6.2.0-20-generic/kernel/fs/nls/nls_iso8859-1.ko /lib/modules/6.2.0-20-generic/kernel/fs/vboxsf/vboxsf.ko /lib/modules/6.2.0-20-generic/kernel/lib/libcrc32c.ko /lib/modules/6.2.0-20-generic/kernel/lib/raid6/raid6_pq.ko /lib/modules/6.2.0-20-generic/kernel/net/ipv4/netfilter/ip_tables.ko /lib/modules/6.2.0-20-generic/kernel/net/netfilter/x_tables.ko /lib/modules/6.2.0-20-generic/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko /lib/modules/6.2.0-20-generic/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko /lib/modules/6.2.0-20-generic/kernel/net/sunrpc/sunrpc.ko /lib/modules/6.2.0-20-generic/kernel/net/wireless/cfg80211.ko /lib/modules/6.2.0-20-generic/kernel/sound/ac97_bus.ko /lib/modules/6.2.0-20-generic/kernel/sound/core/snd-pcm.ko /lib/modules/6.2.0-20-generic/kernel/sound/core/snd-timer.ko /lib/modules/6.2.0-20-generic/kernel/sound/core/snd.ko /lib/modules/6.2.0-20-generic/kernel/sound/pci/ac97/snd-ac97-codec.ko /lib/modules/6.2.0-20-generic/kernel/sound/pci/snd-intel8x0.ko /lib/modules/6.2.0-20-generic/kernel/sound/soundcore.ko '
Some latest log messages since the last called script 400_copy_modules.sh:
  '/lib/modules/6.2.0-20-generic/kernel/sound/core/snd-timer.ko' -> '/var/tmp/rear.Acrvl84FH5J2jWn/rootfs/lib/modules/6.2.0-20-generic/kernel/sound/core/snd-timer.ko'
  '/lib/modules/6.2.0-20-generic/kernel/sound/core/snd.ko' -> '/var/tmp/rear.Acrvl84FH5J2jWn/rootfs/lib/modules/6.2.0-20-generic/kernel/sound/core/snd.ko'
  /lib/modules/6.2.0-20-generic/kernel/sound/pci -> /var/tmp/rear.Acrvl84FH5J2jWn/rootfs/lib/modules/6.2.0-20-generic/kernel/sound/pci
  /lib/modules/6.2.0-20-generic/kernel/sound/pci/ac97 -> /var/tmp/rear.Acrvl84FH5J2jWn/rootfs/lib/modules/6.2.0-20-generic/kernel/sound/pci/ac97
  '/lib/modules/6.2.0-20-generic/kernel/sound/pci/ac97/snd-ac97-codec.ko' -> '/var/tmp/rear.Acrvl84FH5J2jWn/rootfs/lib/modules/6.2.0-20-generic/kernel/sound/pci/ac97/snd-ac97-codec.ko'
  cp: 'lib/modules/6.2.0-20-generic/kernel/sound/pci/ac97': No such file or directory
  '/lib/modules/6.2.0-20-generic/kernel/sound/pci/snd-intel8x0.ko' -> '/var/tmp/rear.Acrvl84FH5J2jWn/rootfs/lib/modules/6.2.0-20-generic/kernel/sound/pci/snd-intel8x0.ko'
  '/lib/modules/6.2.0-20-generic/kernel/sound/soundcore.ko' -> '/var/tmp/rear.Acrvl84FH5J2jWn/rootfs/lib/modules/6.2.0-20-generic/kernel/sound/soundcore.ko'
Aborting due to an error, check /tmp/rear_var/log/rear/rear-rear-u2304.log for details
Exiting rear mkrescue (PID 55846) and its descendant processes ...
Running exit tasks
To remove the build area you may use (with caution): rm -Rf --one-file-system /var/tmp/rear.Acrvl84FH5J2jWn
Terminated

Root Cause Analysis

A bit further digging leads to the observation that the cp utility fails even though it actually copied all files. Here is an extract to illustrate the problem:

root@rear-u2304:~# cat /etc/os-release 
PRETTY_NAME="Ubuntu 23.04"
NAME="Ubuntu"
VERSION_ID="23.04"
VERSION="23.04 (Lunar Lobster)"
VERSION_CODENAME=lunar
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=lunar
LOGO=ubuntu-logo

root@rear-u2304:~# rm -Rf /tmp/f && mkdir /tmp/f && cp --verbose -t /tmp/f -L --preserve=all --parents /etc/apt/sources.list && echo yes ; ls -lR /tmp/f/etc/apt/sources.list /etc/apt/sources.list
/etc -> /tmp/f/etc
/etc/apt -> /tmp/f/etc/apt
'/etc/apt/sources.list' -> '/tmp/f/etc/apt/sources.list'
cp: ‘etc/apt’: No such file or directory
-rw-r--r-- 1 root root 2437 Apr 23 09:53 /etc/apt/sources.list
-rw-r--r-- 1 root root 2437 Apr 23 09:53 /tmp/f/etc/apt/sources.list

root@rear-u2304:~#

Anybody has an idea what could be the problem here?

On an Ubuntu 22.04 system this works as expected:

schlomo@brho:~$ rm -Rf /tmp/f && mkdir /tmp/f && cp --verbose -t /tmp/f -L --preserve=all --parents /etc/apt/sources.list && echo yes ; ls -lR /tmp/f/etc/apt/sources.list /etc/apt/sources.list
/etc -> /tmp/f/etc
/etc/apt -> /tmp/f/etc/apt
'/etc/apt/sources.list' -> '/tmp/f/etc/apt/sources.list'
yes
-rw-r--r-- 1 root    root    263 Mär 26 15:20 /etc/apt/sources.list
-rw-r--r-- 1 schlomo schlomo 263 Mär 26 15:20 /tmp/f/etc/apt/sources.list
schlomo@brho:~$

I reported this bug to Ubuntu via https://bugs.launchpad.net/ubuntu/+source/coreutils/+bug/2017414 and upstream via https://debbugs.gnu.org/cgi/bugreport.cgi?bug=63245

The text was updated successfully, but these errors were encountered:

abbbi · 2023-05-03T08:31:33Z

this will hit with upcoming debian/bookworm release, too.
Works with debian/bullseye:

 rm -Rf /tmp/f && mkdir /tmp/f && cp --verbose -t /tmp/f -L --preserve=all --parents /etc/apt/sources.list && echo yes ; ls -lR /tmp/f/etc/apt/sources.list /etc/apt/sources.list
/etc -> /tmp/f/etc
/etc/apt -> /tmp/f/etc/apt
'/etc/apt/sources.list' -> '/tmp/f/etc/apt/sources.list'
yes
-rw-r--r-- 1 root root 47 May  3 10:29 /etc/apt/sources.list
-rw-r--r-- 1 root root 47 May  3 10:29 /tmp/f/etc/apt/sources.list

fails in bookworm and debian/sid:

rm -Rf /tmp/f && mkdir /tmp/f && cp --verbose -t /tmp/f -L --preserve=all --parents /etc/apt/sources.list && echo yes ; ls -lR /tmp/f/etc/apt/sources.list /etc/apt/sources.list
/etc -> /tmp/f/etc
/etc/apt -> /tmp/f/etc/apt
'/etc/apt/sources.list' -> '/tmp/f/etc/apt/sources.list'
cp: 'etc/apt': No such file or directory
-rw-r--r-- 1 root root 47 May  3 10:28 /etc/apt/sources.list
-rw-r--r-- 1 root root 47 May  3 10:28 /tmp/f/etc/apt/sources.list

schlomo · 2023-05-03T08:42:26Z

Any idea how to escalate this to the upstream projects (coreutils)

abbbi · 2023-05-03T08:48:43Z

i just tried with the latest coreutils source and the behavior is the same with the latest checkout, but theres a different error message, which suggests the issue is related to preserving

rm -Rf /tmp/f && mkdir /tmp/f && src/cp --verbose -t /tmp/f -L --preserve=all --parents /etc/apt/sources.list && echo yes ; ls -lR /tmp/f/etc/apt/sources.list /etc/apt/sources.list
/etc -> /tmp/f/etc
/etc/apt -> /tmp/f/etc/apt
'/etc/apt/sources.list' -> '/tmp/f/etc/apt/sources.list'
src/cp: preserving permissions for ‘/tmp/f/etc/apt’: No such file or directory
-rw-r--r-- 1 root root 278 Jun  7  2022 /etc/apt/sources.list
-rw-r--r-- 1 abi  abi  278 Jun  7  2022 /tmp/f/etc/apt/sources.list

maybe its best to report this to https://debbugs.gnu.org/cgi/pkgreport.cgi?pkg=coreutils or the debian project directly.

abbbi · 2023-05-03T10:17:56Z

i just attempted to mkrescue on debian/bookworm which ships rear 2.7 by default, and this worked like expected, so i guess this issue only hits if REAR_VAR=/tmp/rear_var is used or some other condition is met?

schlomo · 2023-05-03T10:22:46Z

We use cp --preserve=all in these three places

rear/usr/share/rear/build/GNU/Linux/400_copy_modules.sh

Line 138 in 54df969

    
           if ! cp $verbose -t $ROOTFS_DIR -L --preserve=all --parents $loaded_modules_files 1>&2 ; then

rear/usr/share/rear/build/GNU/Linux/400_copy_modules.sh

Line 223 in 54df969

    
           if ! cp $verbose -t $ROOTFS_DIR -L --preserve=all --parents $module_files 1>&2 ; then

rear/usr/share/rear/build/default/490_fix_broken_links.sh

Line 78 in 54df969

    
           cp $v --preserve=all $link_target .$link_target || LogPrintError "Failed to copy target of symlink '$broken_symlink' -> '$link_target'"

IMHO this code doesn't depend on REAR_VAR at all.

abbbi · 2023-05-03T10:25:07Z

yes, i missed the (MODULES contains 'loaded_modules') part. Default is all_modules.

jsmeix · 2023-05-03T11:15:18Z

At other places we use '--preserve=all' implicitly via cp ... -a:

# find usr/sbin/rear usr/share/rear -type f | xargs grep 'cp .* -a '

usr/share/rear/build/GNU/Linux/420_copy_firmware_files.sh:
    cp $verbose -t $ROOTFS_DIR -a --parents /lib*/firmware 2>>/dev/$DISPENSABLE_OUTPUT_DEV 1>&2

usr/share/rear/build/GNU/Linux/400_copy_modules.sh:
        if ! cp $verbose -t $ROOTFS_DIR -a -L --parents /lib/modules/$KERNEL_VERSION 2>>/dev/$DISPENSABLE_OUTPUT_DEV 1>&2 ; then

usr/share/rear/build/GNU/Linux/390_copy_binaries_libraries.sh:
    test -e $ROOTFS_DIR/$lib || cp $v -a -f $lib $ROOTFS_DIR/$lib 1>&2

usr/share/rear/finalize/GNU/Linux/250_migrate_disk_devices_layout.sh:
    cp $v -a --parents "$restored_file" $save_original_file_dir

I see right now that

# find usr/sbin/rear usr/share/rear -type f | xargs grep 'cp .*-a '

finds some more places:

usr/share/rear/output/RAWDISK/Linux-i386/270_create_grub2_efi_bootloader.sh:
    cp -a $v "$original_efi_root/." "$RAWDISK_BOOT_EFI_STAGING_ROOT" || Error "Could not copy EFI configuration"

usr/share/rear/output/RAWDISK/Linux-i386/270_create_grub2_efi_bootloader.sh:
            cp -a $v "$additional_grub_directory/$grub_modules_directory/$module" "$grub_target_directory/$grub_modules_directory"

usr/share/rear/build/Debian/i386/600_fix_debian_stuff.sh:
       cp -a $v /lib/libdevmapper* $ROOTFS_DIR/lib/ >&2

usr/share/rear/rescue/default/910_copy_logfile.sh:
cp -a $v $RUNTIME_LOGFILE $ROOTFS_DIR/tmp/${logfile_basename%.*}-partial-$(date -Iseconds).log >&2

usr/share/rear/restore/Fedora/050_copy_dev_files.sh:
cp -a /dev/. $TARGET_FS_ROOT/dev/

usr/share/rear/restore/YUM/default/950_grub_mkconfig.sh:
cp -a $TARGET_FS_ROOT/boot/grub/grub.conf $TARGET_FS_ROOT/boot/grub/grub.conf.bak

usr/share/rear/restore/YUM/default/400_restore_packages.sh:
cp -a /etc/yum.repos.d/* $TARGET_FS_ROOT/etc/yum.repos.d/

usr/share/rear/restore/YUM/default/410_restore_backup.sh:
                    cp -a $TMP_DIR/restore-exclude-list.txt $TARGET_FS_ROOT/tmp

usr/share/rear/restore/default/990_move_away_restored_files.sh:
        cp -a --parents $file_relative $move_away_dir || continue
usr/share/rear/restore/GALAXY11/default/500_migrate_commvault_certificates.sh:
    cp -a $certs "$TARGET_FS_ROOT/$GALAXY11_HOME_DIRECTORY" || Error "Could not copy $certs"
usr/share/rear/finalize/default/110_bind_mount_proc_sys_dev_run.sh:
            cp -a /dev/. $TARGET_FS_ROOT/dev || LogPrintError "Failed to copy /dev contents from ReaR recovery system to $TARGET_FS_ROOT/dev"

schlomo · 2023-05-03T11:29:02Z

It seems related to -t i think, couldn't reproduce with regular cp -a

schlomo · 2023-05-03T17:40:00Z

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=63245
is the upstream bugreport with a patch proposal
https://debbugs.gnu.org/cgi/bugreport.cgi?att=1;filename=cp-abs-parents.patch;msg=8;bug=63245

jsmeix · 2023-05-04T08:34:59Z

openSUSE Bugzilla report:
https://bugzilla.opensuse.org/show_bug.cgi?id=1211081

SUSE SLE15 and openSUSE Leap 15 are not affected because

# osc list SUSE:SLE-15:Update coreutils | grep tar
coreutils-8.29.tar.xz

So it seems only openSUSE Tumbleweed is affected because

# osc list openSUSE:Factory coreutils | grep tar
coreutils-9.3.tar.xz

jsmeix · 2023-05-04T08:43:35Z

@pcahyna
please have a look here regarding
whether or not RHEL or Fedora are affected
i.e. if RHEL or Fedora contain coreutils 9.1-9.3 inclusive.

pcahyna · 2023-05-04T08:49:20Z

RHEL does not (coreutils 8.*), but Fedora does (f37 has coreutils 9.1).

jsmeix · 2023-05-31T12:51:54Z

@rear/contributors
is there something simple, fail-safe, and backward compatible
that we could do with reasonable effort in ReaR
to avoid this issue in ReaR?

In general I think it does not belong to ReaR
to implement special stuff to avoid generic bugs
in standard programs that are called by ReaR.

github-actions · 2023-07-31T02:20:28Z

Stale issue message

abbbi · 2023-08-08T05:19:23Z

i think this should be reopened. Issue still not solved in debian/bookworm, related bugtracker:

https://bugs.debian.org/1035530
https://bugs.debian.org/1035531

jsmeix · 2023-08-08T05:52:23Z

@abbbi
is there something simple, fail-safe, and backward compatible
that we could do with reasonable effort in ReaR
to avoid this issue in ReaR?

In general I think it does not belong to ReaR
to implement special stuff to avoid generic bugs
in standard programs that are called by ReaR.

schlomo · 2023-08-08T08:39:56Z

I'd like to keep this open till it is fixed in the distros or in ReaR,
at least so that users can easily see that we are aware of this
(closed issues are hidden by default).

@jsmeix @abbbi we can of course
replace cp with tar -c | tar -x -i
which is the good old Unix way of copying file hierarchies.

jsmeix · 2023-08-08T09:51:29Z

Regarding using tar -c | tar -x -i to copy file hierarchies:

On my personal TODO list I have a "research project" which is
to investigate how to really properly copy "something"
also when there are symbolic links "in between"
so that the whole structure is copied to a new place
i.e. that after copying
the link target with its owner,group,permissions,...
and its name plus the symlink exist at the new place.

I think cp cannot do that.
I think with cp one gets
either a symlink that points to the original target
or the content of the target named as the symlink name
or the content of the target named as its original name.

In my quick experiments with tar -c | tar -x -i right now
I didn't see how to get cp --preserve=all results with 'tar'.
It seems 'tar' has various "preserve" options
but no general 'preserve=all' option?

For an example see the code when

 ... $lib is a symbolic link on the original system

in build/GNU/Linux/390_copy_binaries_libraries.sh
which is in current master code starting at
https://github.com/rear/rear/blob/master/usr/share/rear/build/GNU/Linux/390_copy_binaries_libraries.sh#L97

We have this kind of issue at several places in ReaR,
for a recent example see cp -L versus plain cp in
#3031 (comment)

I wished there was a generic function in ReaR that
"just works" to get "something" really properly copied.

My "research project" is to investigate (as time permits)
if such a generic function could be implemented properly
and with reasonable effort.

pcahyna · 2023-08-08T13:21:16Z

Regarding tar | tar, this is not without pitfalls:
see #3027
(of course, files shrinking during read are not typical).

abbbi · 2023-08-09T07:47:44Z

Regarding tar | tar, this is not without pitfalls: see #3027 (of course, files shrinking during read are not typical).

maybe cpio does its job right?

jsmeix · 2023-08-10T11:38:35Z

@pcahyna
thank you to mention #3027 in your
#2972 (comment)

I will replace all tar | tar in this issue
with more exact tar -c | tar -x -i where appropriate.

Done.

github-actions · 2023-10-10T02:03:07Z

Stale issue message

github-actions · 2023-12-10T02:11:39Z

Stale issue message

pcahyna · 2023-12-12T11:05:58Z

RHEL does not (coreutils 8.*), but Fedora does (f37 has coreutils 9.1).

Fedora 37 is EOL and coreutils in Fedora 38 has this in changelog:

* Thu May 04 2023 Pádraig Brady <P@draigBrady.com> - 9.1-12
- further fixes to backups; broken in 9.1
- fix some directory-relative syscalls; broken in 9.1

This is probably the fix, because I don't see a problem:

# echo 'MODULES=( 'loaded_modules' )' >> /etc/rear/local.conf
# rear mkrescue
Broken symlink '/usr/lib/firmware/ath10k/WCN3990/hw1.0/wlanmdsp.mbn.xz' in recovery system because 'readlink' cannot determine its link target
Broken symlink '/usr/lib/firmware/a300_pfp.fw.xz' in recovery system because 'readlink' cannot determine its link target
Broken symlink '/usr/lib/firmware/a300_pm4.fw.xz' in recovery system because 'readlink' cannot determine its link target
There are binaries or libraries in the ReaR recovery system that need additional libraries
/usr/lib64/systemd/libsystemd-core-253.12-1.fc38.so requires additional libraries
        libsystemd-shared-253.12-1.fc38.so => not found
ReaR recovery system in '/tmp/rear.GN21zGFD7tm6wUa/rootfs' needs additional libraries, check /var/log/rear/rear-vm-10-0-184-255.log for details

while when I downgrade coreutils to the previous build, I see a problem:

# dnf downgrade https://kojipkgs.fedoraproject.org//packages/coreutils/9.1/11.fc38/x86_64/coreutils-9.1-11.fc38.x86_64.rpm
# echo 'MODULES=( 'loaded_modules' )' >> /etc/rear/local.conf
# rear mkrescue
ERROR: Failed to copy '/lib/modules/6.6.3-100.fc38.x86_64/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko.xz
/lib/modules/6.6.3-100.fc38.x86_64/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko.xz
/lib/modules/6.6.3-100.fc38.x86_64/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko.xz
/lib/modules/6.6.3-100.fc38.x86_64/kernel/fs/nfs/nfsv4.ko.xz
/lib/modules/6.6.3-100.fc38.x86_64/kernel/net/dns_resolver/dns_resolver.ko.xz
/lib/modules/6.6.3-100.fc38.x86_64/kernel/fs/nfs/nfs.ko.xz
/lib/modules/6.6.3-100.fc38.x86_64/kernel/fs/lockd/lockd.ko.xz
/lib/modules/6.6.3-100.fc38.x86_64/kernel/fs/nfs_common/grace.ko.xz
/lib/modules/6.6.3-100.fc38.x86_64/kernel/fs/fscache/fscache.ko.xz
/lib/modules/6.6.3-100.fc38.x86_64/kernel/fs/netfs/netfs.ko.xz
/lib/modules/6.6.3-100.fc38.x86_64/kernel/net/rfkill/rfkill.ko.xz
/lib/modules/6.6.3-100.fc38.x86_64/kernel/drivers/powercap/intel_rapl_msr.ko.xz
/lib/modules/6.6.3-100.fc38.x86_64/kernel/drivers/powercap/intel_rapl_common.ko.xz
/lib/modules/6.6.3-100.fc38.x86_64/kernel/drivers/acpi/nfit/nfit.ko.xz
/lib/modules/6.6.3-100.fc38.x86_64/kernel/drivers/nvdimm/libnvdimm.ko.xz
/lib/modules/6.6.3-100.fc38.x86_64/kernel/arch/x86/kvm/kvm-intel.ko.xz
/lib/modules/6.6.3-100.fc38.x86_64/kernel/arch/x86/kvm/kvm.ko.xz
/lib/modules/6.6.3-100.fc38.x86_64/kernel/sound/drivers/pcsp/snd-pcsp.ko.xz
/lib/modules/6.6.3-100.fc38.x86_64/kernel/sound/core/snd-pcm.ko.xz
/lib/modules/6.6.3-100.fc38.x86_64/kernel/net/sunrpc/sunrpc.ko.xz
/lib/modules/6.6.3-100.fc38.x86_64/kernel/virt/lib/irqbypass.ko.xz
/lib/modules/6.6.3-100.fc38.x86_64/kernel/sound/core/snd-timer.ko.xz
/lib/modules/6.6.3-100.fc38.x86_64/kernel/sound/core/snd.ko.xz
/lib/modules/6.6.3-100.fc38.x86_64/kernel/drivers/net/virtio_net.ko.xz
/lib/modules/6.6.3-100.fc38.x86_64/kernel/drivers/net/net_failover.ko.xz
/lib/modules/6.6.3-100.fc38.x86_64/kernel/sound/soundcore.ko.xz
/lib/modules/6.6.3-100.fc38.x86_64/kernel/drivers/virtio/virtio_balloon.ko.xz
/lib/modules/6.6.3-100.fc38.x86_64/kernel/net/core/failover.ko.xz
/lib/modules/6.6.3-100.fc38.x86_64/kernel/drivers/i2c/busses/i2c-piix4.ko.xz
/lib/modules/6.6.3-100.fc38.x86_64/kernel/drivers/input/joydev.ko.xz
/lib/modules/6.6.3-100.fc38.x86_64/kernel/fs/fuse/fuse.ko.xz
/lib/modules/6.6.3-100.fc38.x86_64/kernel/drivers/block/loop.ko.xz
/lib/modules/6.6.3-100.fc38.x86_64/kernel/drivers/block/zram/zram.ko.xz
/lib/modules/6.6.3-100.fc38.x86_64/kernel/arch/x86/crypto/crct10dif-pclmul.ko.xz
/lib/modules/6.6.3-100.fc38.x86_64/kernel/arch/x86/crypto/crc32-pclmul.ko.xz
/lib/modules/6.6.3-100.fc38.x86_64/kernel/arch/x86/crypto/crc32c-intel.ko.xz
/lib/modules/6.6.3-100.fc38.x86_64/kernel/arch/x86/crypto/polyval-clmulni.ko.xz
/lib/modules/6.6.3-100.fc38.x86_64/kernel/crypto/polyval-generic.ko.xz
/lib/modules/6.6.3-100.fc38.x86_64/kernel/arch/x86/crypto/ghash-clmulni-intel.ko.xz
/lib/modules/6.6.3-100.fc38.x86_64/kernel/arch/x86/crypto/sha512-ssse3.ko.xz
/lib/modules/6.6.3-100.fc38.x86_64/kernel/arch/x86/crypto/sha256-ssse3.ko.xz
/lib/modules/6.6.3-100.fc38.x86_64/kernel/arch/x86/crypto/sha1-ssse3.ko.xz
/lib/modules/6.6.3-100.fc38.x86_64/kernel/drivers/block/virtio_blk.ko.xz
/lib/modules/6.6.3-100.fc38.x86_64/kernel/drivers/ata/ata_generic.ko.xz
/lib/modules/6.6.3-100.fc38.x86_64/kernel/drivers/gpu/drm/tiny/cirrus.ko.xz
/lib/modules/6.6.3-100.fc38.x86_64/kernel/drivers/ata/pata_acpi.ko.xz
/lib/modules/6.6.3-100.fc38.x86_64/kernel/drivers/input/serio/serio_raw.ko.xz
/lib/modules/6.6.3-100.fc38.x86_64/kernel/drivers/firmware/qemu_fw_cfg.ko.xz'
Some latest log messages since the last called script 400_copy_modules.sh:
  cp: 'lib/modules/6.6.3-100.fc38.x86_64/kernel/drivers/block': No such file or directory
  cp: 'lib/modules/6.6.3-100.fc38.x86_64/kernel/drivers/block/zram': No such file or directory
  cp: 'lib/modules/6.6.3-100.fc38.x86_64/kernel/arch/x86/crypto': No such file or directory
  cp: 'lib/modules/6.6.3-100.fc38.x86_64/kernel/crypto': No such file or directory
  cp: 'lib/modules/6.6.3-100.fc38.x86_64/kernel/drivers/ata': No such file or directory
  cp: 'lib/modules/6.6.3-100.fc38.x86_64/kernel/drivers/gpu/drm/tiny': No such file or directory
  cp: 'lib/modules/6.6.3-100.fc38.x86_64/kernel/drivers/input/serio': No such file or directory
  cp: 'lib/modules/6.6.3-100.fc38.x86_64/kernel/drivers/firmware': No such file or directory
Aborting due to an error, check /var/log/rear/rear-vm-10-0-187-232.log for details
Terminated

so, on Fedora, this is not a problem anymore.

pcahyna · 2023-12-12T11:06:34Z

i think this should be reopened. Issue still not solved in debian/bookworm, related bugtracker:

https://bugs.debian.org/1035530 https://bugs.debian.org/1035531

according to the links, the problem was fixed about a month ago.

jsmeix · 2023-12-12T12:49:29Z

@schlomo
I assigned it to you so that you may close it
when you think that it could or should be closed, cf.
#2972 (comment)

jsmeix added the external tool The issue depends on other software e.g. third-party backup tools. label May 3, 2023

schlomo changed the title ~~Cannot create rescue image on Ubuntu Server 23.04 - Bug in cp?~~ Cannot create rescue image on Ubuntu Server 23.04 - Confirmed bug in cp 9.x May 3, 2023

jsmeix added the not ReaR / invalid The root cause is not in the ReaR code or ReaR is misused. label May 31, 2023

github-actions bot added the no-issue-activity label Jul 31, 2023

github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Aug 8, 2023

schlomo reopened this Aug 8, 2023

schlomo changed the title ~~Cannot create rescue image on Ubuntu Server 23.04 - Confirmed bug in cp 9.x~~ Cannot create rescue image on Ubuntu 23.04 and Debian/Bookworm - Confirmed bug in cp 9.x Aug 8, 2023

github-actions bot removed the no-issue-activity label Aug 9, 2023

jsmeix added the discuss / RFC label Aug 10, 2023

github-actions bot added the no-issue-activity label Oct 10, 2023

jsmeix removed the no-issue-activity label Oct 10, 2023

jsmeix mentioned this issue Nov 10, 2023

Missing libraries in the recovery system for symlinks in COPY_AS_IS #3064

Closed

github-actions bot added the no-issue-activity label Dec 10, 2023

jsmeix removed the no-issue-activity label Dec 11, 2023

jsmeix added ready-to-close? and removed discuss / RFC labels Dec 12, 2023

jsmeix assigned jsmeix and schlomo and unassigned jsmeix Dec 12, 2023

Cannot create rescue image on Ubuntu 23.04 and Debian/Bookworm - Confirmed bug in cp 9.x #2972

Cannot create rescue image on Ubuntu 23.04 and Debian/Bookworm - Confirmed bug in cp 9.x #2972

schlomo commented Apr 23, 2023 •

edited

abbbi commented May 3, 2023

schlomo commented May 3, 2023

abbbi commented May 3, 2023 •

edited

abbbi commented May 3, 2023 •

edited

schlomo commented May 3, 2023

abbbi commented May 3, 2023

jsmeix commented May 3, 2023 •

edited

schlomo commented May 3, 2023

schlomo commented May 3, 2023 •

edited by jsmeix

jsmeix commented May 4, 2023 •

edited

jsmeix commented May 4, 2023

pcahyna commented May 4, 2023

jsmeix commented May 31, 2023

github-actions bot commented Jul 31, 2023

abbbi commented Aug 8, 2023

jsmeix commented Aug 8, 2023

schlomo commented Aug 8, 2023 •

edited by jsmeix

jsmeix commented Aug 8, 2023 •

edited

pcahyna commented Aug 8, 2023 •

edited by jsmeix

abbbi commented Aug 9, 2023

jsmeix commented Aug 10, 2023 •

edited

github-actions bot commented Oct 10, 2023

github-actions bot commented Dec 10, 2023

pcahyna commented Dec 12, 2023 •

edited

pcahyna commented Dec 12, 2023

jsmeix commented Dec 12, 2023

Cannot create rescue image on Ubuntu 23.04 and Debian/Bookworm - Confirmed bug in cp 9.x #2972

Cannot create rescue image on Ubuntu 23.04 and Debian/Bookworm - Confirmed bug in cp 9.x #2972

Comments

schlomo commented Apr 23, 2023 • edited

Context

Observable Problem

Root Cause Analysis

abbbi commented May 3, 2023

schlomo commented May 3, 2023

abbbi commented May 3, 2023 • edited

abbbi commented May 3, 2023 • edited

schlomo commented May 3, 2023

abbbi commented May 3, 2023

jsmeix commented May 3, 2023 • edited

schlomo commented May 3, 2023

schlomo commented May 3, 2023 • edited by jsmeix

jsmeix commented May 4, 2023 • edited

jsmeix commented May 4, 2023

pcahyna commented May 4, 2023

jsmeix commented May 31, 2023

github-actions bot commented Jul 31, 2023

abbbi commented Aug 8, 2023

jsmeix commented Aug 8, 2023

schlomo commented Aug 8, 2023 • edited by jsmeix

jsmeix commented Aug 8, 2023 • edited

pcahyna commented Aug 8, 2023 • edited by jsmeix

abbbi commented Aug 9, 2023

jsmeix commented Aug 10, 2023 • edited

github-actions bot commented Oct 10, 2023

github-actions bot commented Dec 10, 2023

pcahyna commented Dec 12, 2023 • edited

pcahyna commented Dec 12, 2023

jsmeix commented Dec 12, 2023

schlomo commented Apr 23, 2023 •

edited

abbbi commented May 3, 2023 •

edited

abbbi commented May 3, 2023 •

edited

jsmeix commented May 3, 2023 •

edited

schlomo commented May 3, 2023 •

edited by jsmeix

jsmeix commented May 4, 2023 •

edited

schlomo commented Aug 8, 2023 •

edited by jsmeix

jsmeix commented Aug 8, 2023 •

edited

pcahyna commented Aug 8, 2023 •

edited by jsmeix

jsmeix commented Aug 10, 2023 •

edited

pcahyna commented Dec 12, 2023 •

edited