Redhat recover failed: / cannot be remounted rw

[Framsfex]

• ReaR version ("/usr/sbin/rear -V"):

Relax-and-Recover 2.6 / 2020-06-17
(Installed on RHEL7 with "yum install rear")

• OS version ("cat /etc/os-release" or "lsb_release -a" or "cat /etc/rear/os.conf"):

NAME="Red Hat Enterprise Linux Server"
VERSION="7.9 (Maipo)"
ID="rhel"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="7.9"

• ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat /etc/rear/local.conf"):

OUTPUT=ISO
OUTPUT_URL=nfs://129.69.2.11/spnfs_data0/sp_i/Rear
BACKUP=NETFS
BACKUP_URL=nfs://129.69.2.11/spnfs_data0/sp_i/Rear
BACKUP_PROG_EXCLUDE=("${BACKUP_PROG_EXCLUDE[@]}" '/media' '/var/tmp' '/var/crash' '/mnt')
NETFS_KEEP_OLD_BACKUP_COPY=

• Hardware vendor/product (PC or PowerNV BareMetal or ARM) or VM (KVM guest or PowerVM LPAR):

Cisco Systems Inc UCSC-C240-M5S A0
Intel(R) Xeon(R) Gold 6132 CPU @ 2.60GHz, 56 x 3700 MHz

• System architecture (x86 compatible or PPC64/PPC64LE or what exact ARM device):

x86

• Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or ELILO or Petitboot):

GRUB BIOS

• Storage (local disk or SSD) and/or SAN (FC or iSCSI or FCoE) and/or multipath (DM or NVMe):

Local harddisk, Hardware RAID

• Storage layout ("lsblk -ipo NAME,KNAME,PKNAME,TRAN,TYPE,FSTYPE,LABEL,SIZE,MOUNTPOINT"):

NAME                      KNAME     PKNAME    TRAN TYPE FSTYPE      LABEL   SIZE MOUNTPOINT
/dev/sda                  /dev/sda                 disk                   557.9G 
|-/dev/sda1               /dev/sda1 /dev/sda       part xfs         boot      1G /boot
`-/dev/sda2               /dev/sda2 /dev/sda       part LVM2_member       556.9G 
  |-/dev/mapper/rhel-root /dev/dm-0 /dev/sda2      lvm  xfs                  32G /
  |-/dev/mapper/rhel-home /dev/dm-5 /dev/sda2      lvm  xfs                 256G /local
  |-/dev/mapper/rhel-swap /dev/dm-6 /dev/sda2      lvm  swap                 16G [SWAP]

• Description of the issue (ideally so that others can reproduce it):

sp-i is a Cisco UCS server with RHEL7.
I did a backup with ReaR which seems successful.
Then I did an upgrade to RHEL8, which is a bit complicated with Redhat, but it worked. The server bootet successfully RHEL8.
(1 week pause)
To reproduce and test the upgrade process I wanted to go back to RHEL7. For this I booted the ReaR Rescue CD (rear-sp-i.iso) and selected "recover", which seems successfully, too. But after rebooting the recovered RHEL7 system it was not able to remount the / filesystem from ro to rw state:

https://fex.rus.uni-stuttgart.de/fop/NtoxIEXY/X-20240404231512.jpg

I looked in the NFS rear directory and found:

-RW-     15,938,002 2024-03-25 14:07 backup.log
-RW-  4,321,746,328 2024-03-25 14:06 backup.tar.gz
-RW-            202 2024-04-04 01:31 README
-RW-    603,398,144 2024-04-04 01:31 rear-sp-i.iso
-RW-        102,804 2024-04-04 01:31 rear-sp-i.log
-RW-              0 2024-03-25 14:07 selinux.autorelabel
-RW-            268 2024-04-04 01:31 VERSION

rear-sp-i.iso is much younger than backup! And it was created at 1:30, a time where I am not working!
My assumption: Some unknown (cron?) job has recreated rear-sp-i.iso with RHEL8 and its newer xfs is incompatible with RHEL7 xfs.

Any idea what I can do in this situation?

I have second server (sp-j) with identical configuration, still running with RHEL7
Shall I run ReaR there and use this rear-sp-j.iso?

[Framsfex]

rear-sp-i.log

[abbbi]

rear-sp-i.iso is much younger than backup! And it was created at 1:30, a time where I am not working!
My assumption: Some unknown (cron?) job has recreated rear-sp-i.iso with RHEL8 and its newer xfs is incompatible with RHEL7 xfs.

attempting to restore a RHEL7 system with an recovery iso that contains already an RHEL8 system will not work and is not supported. REAR itself does not setup any cronjobs, you should clarify how the recovery iso was created after update to RHEL8. I dont see how REAR is at fault here.

One possible "dirty" solution would be:

1. create bootable Recovery iso image on the identical left over RHEL7 system
2. boot iso, extract the REAR disk layout config from the overwritten ISO Image based on RHEL8 and copy it to the running instance
3. run recovery

It should then re-create disk partitions based on the information from the original system using the tools compatible to the RHEL7 backup.

[jsmeix]

Only a guess but regarding "unknown (cron?) job" see
#3199 (comment)

@Framsfex
perhaps you also have an old cron job
as a leftover from ReaR 2.4 or before?

[Framsfex]

On Thu 2024-04-11 (04:00), Johannes Meixner wrote: Only a guess but regarding "unknown (cron?) job" see #3199 (comment)

That's it! We have the same time stamp of the new created ISO image! This is what I have already assumed. Too bad, our Redhat installation now is broken, we have had to reinstall it.

perhaps you also have an old cron job as a leftover from ReaR 2.4 or before?

We have installed ReaR 2.4 from the RHEL 7.9 repo and then made the upgrade to RHEL 8.3 (where the hidden cronjob run) which we tried to recover with ReaR back to RHEL7 I do not know which ReaR version comes with RHEL 8.3

…

-- Ullrich Horlacher Server und Virtualisierung Rechenzentrum TIK Universitaet Stuttgart E-Mail: ***@***.*** Allmandring 30a Tel: ++49-711-68565868 70569 Stuttgart (Germany) WWW: https://www.tik.uni-stuttgart.de/ ***@***.***>

[pcahyna]

Hi @Framsfex sorry to hear that. In RHEL 8.3 there is also ReaR 2.4, but the iso made on a different system may be incompatible.

I have second server (sp-j) with identical configuration, still running with RHEL7
Shall I run ReaR there and use this rear-sp-j.iso?

I think this may work, if the disk layout is identical (therefore you don't need to worry about copying the disklayout from the original server). But there may be some issues when bringing up the network. The rescue system will ask you to map MAC addresses of the second system to the MAC addresses of the first system, and then use the IP addresses of the second server (unless your IP addresses are configured by DHCP).

[pcahyna]

In RHEL 9 we deleted the cron job btw, thus users won't face this problem anymore.

[jsmeix]

@pcahyna
I don't know how ReaR's cron file was installed by Red Hat's RPM
but when Red Hat did it as we had in packaging/rpm/rear.spec via

%config(noreplace) %{_sysconfdir}/cron.d/rear

cf.
89a8f18
then according to the table in
https://www.cl.cam.ac.uk/~jw35/docs/rpm_config.html

The following table shows what we ended up with
after installing an RPM,
optionally editing the resulting files,
and then upgrading the RPM.

File marked as     | Changed in  | On-disk file     | On-disk file
                   | update RPM? | untouched        | edited
...
                   | No          | File from update | Edited file
%config(noreplace) |             |
                   | Yes         | File from update | Edited file,
                   |             |                  | file from the
                   |             |                  | update in .rpmnew

there might be the cron file left even when an RPM package update
does no longer contain that cron file?
The case when a config file is no longer provided by an
RPM package update is not described there so I don't know
what actually happens in this case.
I only liked to note that the RPM package update might not
have removed the unwanted cron file.

[gdha]

@jsmeix @pcahyna

there might be the cron file left even when an RPM package update does no longer contain that cron file? The case when a config file is no longer provided by an RPM package update is not described there so I don't know what actually happens in this case. I only liked to note that the RPM package update might not have removed the unwanted cron file.

Perhaps, we should add a %post action to remove the (old) cron entry in the spec file?

[gdha]

@Framsfex Did you succeed in the recovery with the similar RHEL 7 ISO rescue?

[pcahyna]

I only liked to note that the RPM package update might not
have removed the unwanted cron file.

Unfortunately, in RHEL 8 this file is still considered wanted (it is shipped in the package), so this reasoning does not apply here (the problem was related to a RHEL 7 -> RHEL 8 upgrade) and no way of removing obsolete config files would have helped the user here. I will check how does newer RHEL and Fedora (where the file is not wanted anymore) behave in this respect.

[Framsfex]

On Fri 2024-04-26 (02:05), gdha wrote: @Framsfex Did you succeed in the recovery with the similar RHEL 7 ISO rescue?

We used a dd image from a second server (identical hardware) and did not try it with ReaR again,.

…

-- Ullrich Horlacher Server und Virtualisierung Rechenzentrum TIK Universitaet Stuttgart E-Mail: ***@***.*** Allmandring 30a Tel: ++49-711-68565868 70569 Stuttgart (Germany) WWW: https://www.tik.uni-stuttgart.de/ ***@***.***>