900_create_missing_directories.sh fails calling chroot due to missing files in /mnt/local/ (with BACKUP=NSR)

[richardwheatley87]

Relax-and-Recover (ReaR) Issue Template

Fill in the following items before submitting a new issue
(quick response is not guaranteed with free support):

• ReaR version ("/usr/sbin/rear -V"):
  Relax-andRecover 2.4 / 2018-06-21

• OS version ("cat /etc/rear/os.conf" or "lsb_release -a" or "cat /etc/os-release"):
  
  RHEL Server 6.5

• ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat /etc/rear/local.conf"):
  

• Hardware (PC dor PowerNV BareMetal or ARM) or virtual machine (KVM guest or PoverVM LPAR):
  VMware Red Hat Enterprise Linux 6 32-bit, compatability ESXi 5.5 and later (VM version 10)

• System architecture (x86 compatible or PPC64/PPC64LE or what exact ARM device):
  x86 compatible

• Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or ELILO or Petitboot):
  BIOS firmware and GRUB bootloader

• Storage (lokal disk or SSD) and/or SAN (FC or iSCSI or FCoE) and/or multipath (DM or NVMe):
  Local disk 105GB Thick provisioned lazy zeroed

• Description of the issue (ideally so that others can reproduce it):
  rear recover fails during 900_create_missing_directories.sh with error:
  choot: failed to run command '/bin/bash' : No such file or directory <timestamp> Failed to 'chown root:root /mnt/cdrom'
  Error appears to be similar to closed issue Recover IBM Power "Failed to 'chown root:root"  #1715 and rear backup doesn't copy all libraries into chroot directory #879

• Workaround, if any: None

• Attachments, as applicable ("rear -D mkrescue/mkbackup/recover" debug log files):
  /var/log/rear/rear-dev-calbd-server.log extract:
  
  Debug log output:
  

[jsmeix]

When it fails during 900_create_missing_directories.sh
(that is run during 'rear recover' after the backup was restorted)
with

choot: failed to run command '/bin/bash' : No such file or directory

what actually happens is that
'chroot $TARGET_FS_ROOT' fails with '/bin/bash No such file or directory'.
which means:

After restore of the backup 'chroot $TARGET_FS_ROOT'
is not possible which indicates that the backup restore
was not correct (probably mandatory files are missing).

During 'rear recover' the backup restore happens from
within the running ReaR recovery system into TARGET_FS_ROOT
and TARGET_FS_ROOT is /mnt/local.

This looks as if no '/bin/bash' was restored which would mean
your backup restore did not work - perhaps even because
there is no '/bin/bash' in your backup?

Check what files are in your backup and inspect your
backup log that was created when you did "rear mkbackup" and
your backup restore log that was created during "rear recover".

In general regarding ReaR versus backup see the section
"Relax-and-Recover versus backup and restore" in
https://en.opensuse.org/SDB:Disaster_Recovery

For a more detailed analysis of issues see the section
"Debugging issues with Relax-and-Recover" in
https://en.opensuse.org/SDB:Disaster_Recovery
what information and files are needed when 'rear recover' failed
so that others (i.e. we at ReaR upstream) could do a more detailed analysis.

[jsmeix]

@richardwheatley87
in general regarding BACKUP=NSR:

I am not a EMC NetWorker (or Legato NetWorker) user
so that I cannot reproduce anything or even fix anything
that is specific regarding BACKUP=NSR.

In particular I do not have any proprietary third-party backup tool.

In general I do not use any of the third-party backup tools
so that in general I can neither reproduce any issue nor fix
or enhance things that are related to third-party backup tools.

When you have a Red Hat support contract, perhaps they could help?

[richardwheatley87]

Thank you for your prompt reply.
I shall read suggested information on disaster recovery, look at my EMC NetWorker backup configuration and check the logs (both ReaR and NSR) for info on any failures that occured during the NSR restore phase.

[gdha]

@richardwheatley87 Any update you want to share about the research you did?

[richardwheatley87]

Not yet, I have not had time to progress further. Since the problem appears to be that the backup has not been restored, I will change the NSR_DEFAULT_POOL_NAME=Default in the site.config file to be the actual EMC NetWorker pool where my machine backup is stored then re-create the rescue ISO, using rear mkrescue. I thought that the NSR agent on the machine would know which pool get the backup from.
Thank you for your interest, I will add an update once I have tried this and have more information.

[jsmeix]

No news is good news.