Recovery fails - Everything in disklayout file is commented out

[schlomo]

ReaR Snapshot build from today (and now it pains that we don't have the exact git revision in the RPM :-() on CentOS 7.3

The recovery disk has a different size from the source disk.

Interestingly this version did not ask me to map a disk while our last release (2.00) does ask me to map a disk (which I find stupid since there is only once in any case).

Am I the only one who sees this?

Click to open the Recovery Log

2017-05-09 17:53:54 Relax-and-Recover 2.00-git201705081501 / 2017-05-08
2017-05-09 17:53:54 Command line options: /bin/rear recover
2017-05-09 17:53:54 Using log file: /var/log/rear/rear-client.log
2017-05-09 17:53:54 Including /etc/rear/os.conf
2017-05-09 17:53:54 Including conf/Linux-i386.conf
2017-05-09 17:53:54 Including conf/GNU/Linux.conf
2017-05-09 17:53:54 Including /etc/rear/local.conf
2017-05-09 17:53:54 Including /etc/rear/rescue.conf
2017-05-09 17:53:54 ======================
2017-05-09 17:53:54 Running 'init' stage
2017-05-09 17:53:54 ======================
2017-05-09 17:53:54 Including init/default/010_set_drlm_env.sh
2017-05-09 17:53:54 Including init/default/030_update_recovery_system.sh
2017-05-09 17:53:54 Including init/default/050_check_rear_recover_mode.sh
2017-05-09 17:53:54 Running workflow recover within the ReaR rescue/recovery system
2017-05-09 17:53:54 Finished running 'init' stage in 0 seconds
2017-05-09 17:53:54 Using build area '/tmp/rear.qZsRSjsMC2Fu9HY'
mkdir: created directory '/tmp/rear.qZsRSjsMC2Fu9HY/rootfs'
mkdir: created directory '/tmp/rear.qZsRSjsMC2Fu9HY/tmp'
2017-05-09 17:53:54 Running recover workflow
2017-05-09 17:53:54 ======================
2017-05-09 17:53:54 Running 'setup' stage
2017-05-09 17:53:54 ======================
2017-05-09 17:53:54 Including setup/default/010_pre_recovery_script.sh
2017-05-09 17:53:54 Finished running 'setup' stage in 0 seconds
2017-05-09 17:53:54 ======================
2017-05-09 17:53:54 Running 'verify' stage
2017-05-09 17:53:54 ======================
2017-05-09 17:53:54 Including verify/default/020_cciss_scsi_engage.sh
2017-05-09 17:53:54 Including verify/default/020_translate_url.sh
2017-05-09 17:53:54 Including verify/default/030_translate_tape.sh
2017-05-09 17:53:54 Including verify/default/040_validate_variables.sh
2017-05-09 17:53:54 Including verify/NETFS/default/050_check_NETFS_requirements.sh
2017-05-09 17:53:54 Skipping 'ping' test for host 'server' in BACKUP_URL 'nfs://server/export/nfs'
2017-05-09 17:53:54 Including verify/default/050_create_mappings_dir.sh
2017-05-09 17:53:54 Including verify/GNU/Linux/050_sane_recovery_check.sh
2017-05-09 17:53:54 Including verify/NETFS/default/050_start_required_nfs_daemons.sh
2017-05-09 17:53:54 Starting required daemons for NFS: RPC portmapper (portmap or rpcbind) and rpc.statd if available.
2017-05-09 17:53:54 Started RPC portmapper 'rpcbind'.
2017-05-09 17:53:54 RPC portmapper 'rpcbind' available.
2017-05-09 17:53:54 Started rpc.statd.
2017-05-09 17:53:55 RPC status rpc.statd available.
2017-05-09 17:53:55 Started rpc.idmapd.
2017-05-09 17:53:55 Including verify/NETFS/default/060_mount_NETFS_path.sh
mkdir: created directory '/tmp/rear.qZsRSjsMC2Fu9HY/outputfs'
2017-05-09 17:53:55 Mounting with 'mount -v -t nfs -o rw,noatime server:/export/nfs /tmp/rear.qZsRSjsMC2Fu9HY/outputfs'
mount.nfs: timeout set for Tue May  9 17:55:55 2017
mount.nfs: trying text-based options 'vers=4,addr=192.168.33.15,clientaddr=192.168.33.10'
2017-05-09 17:53:55 Including verify/NETFS/default/070_set_backup_archive.sh
2017-05-09 17:53:55 Using backup archive '/tmp/rear.qZsRSjsMC2Fu9HY/outputfs/client/backup.tar.gz'
2017-05-09 17:53:55 Including verify/NETFS/default/090_set_readonly_options.sh
2017-05-09 17:53:55 Including verify/GNU/Linux/230_storage_and_network_modules.sh
2017-05-09 17:53:55 Including verify/GNU/Linux/260_recovery_storage_drivers.sh
2017-05-09 17:53:55 Will do driver migration (recreating initramfs/initrd)
2017-05-09 17:53:55 Including verify/NETFS/default/550_check_backup_archive.sh
2017-05-09 17:53:55 Calculating backup archive size
2017-05-09 17:53:55 Backup archive size is 399M	/tmp/rear.qZsRSjsMC2Fu9HY/outputfs/client/backup.tar.gz (compressed)
2017-05-09 17:53:55 Including verify/NETFS/default/600_check_encryption_key.sh
2017-05-09 17:53:55 Including verify/NETFS/default/980_umount_NETFS_dir.sh
2017-05-09 17:53:55 Unmounting '/tmp/rear.qZsRSjsMC2Fu9HY/outputfs'
umount: /tmp/rear.qZsRSjsMC2Fu9HY/outputfs (server:/export/nfs) unmounted
rmdir: removing directory, '/tmp/rear.qZsRSjsMC2Fu9HY/outputfs'
2017-05-09 17:53:55 Finished running 'verify' stage in 1 seconds
2017-05-09 17:53:55 ======================
2017-05-09 17:53:55 Running 'layout/prepare' stage
2017-05-09 17:53:55 ======================
2017-05-09 17:53:55 Including layout/prepare/default/010_prepare_files.sh
2017-05-09 17:53:55 Including layout/prepare/GNU/Linux/100_include_partition_code.sh
2017-05-09 17:53:55 Including layout/prepare/GNU/Linux/110_include_lvm_code.sh
2017-05-09 17:53:55 Including layout/prepare/GNU/Linux/120_include_raid_code.sh
2017-05-09 17:53:55 Including layout/prepare/GNU/Linux/130_include_filesystem_code.sh
2017-05-09 17:53:55 Including layout/prepare/GNU/Linux/130_include_mount_filesystem_code.sh
2017-05-09 17:53:55 Including layout/prepare/GNU/Linux/130_include_mount_subvolumes_code.sh
2017-05-09 17:53:55 Including layout/prepare/GNU/Linux/140_include_swap_code.sh
2017-05-09 17:53:55 Including layout/prepare/GNU/Linux/150_include_drbd_code.sh
2017-05-09 17:53:55 Including layout/prepare/GNU/Linux/160_include_luks_code.sh
2017-05-09 17:53:55 Including layout/prepare/GNU/Linux/170_include_hpraid_code.sh
2017-05-09 17:53:55 Including layout/prepare/default/200_recreate_hpraid.sh
2017-05-09 17:53:55 Including layout/prepare/GNU/Linux/210_load_multipath.sh
2017-05-09 17:53:55 Including layout/prepare/default/250_compare_disks.sh
2017-05-09 17:53:55 Comparing disks.
2017-05-09 17:53:55 Looking for sda...
2017-05-09 17:53:55 Device sda exists.
2017-05-09 17:53:55 Device sda has size 54223962112, 42949672960 expected
2017-05-09 17:53:55 Switching to manual disk layout configuration.
2017-05-09 17:53:55 Including layout/prepare/default/270_overrule_migration_mode.sh
2017-05-09 17:53:55 Including layout/prepare/default/300_map_disks.sh
2017-05-09 17:53:55 This is the disk mapping table:
2017-05-09 17:53:55 
2017-05-09 17:53:55 Including layout/prepare/default/310_remove_exclusions.sh
2017-05-09 17:53:55 Including layout/prepare/default/320_apply_mappings.sh
2017-05-09 17:53:55 Including layout/prepare/default/400_autoresize_disks.sh
2017-05-09 17:53:55 Including layout/prepare/default/500_confirm_layout.sh
2017-05-09 17:53:55 Please confirm that '/var/lib/rear/layout/disklayout.conf' is as you expect.
2017-05-09 17:54:58 User selected: 5) Continue recovery
2017-05-09 17:54:58 Including layout/prepare/default/510_list_dependencies.sh
2017-05-09 17:54:58 Including layout/prepare/default/520_exclude_components.sh
2017-05-09 17:54:58 Including layout/prepare/default/540_generate_device_code.sh
2017-05-09 17:54:58 Including layout/prepare/default/550_finalize_script.sh
2017-05-09 17:54:58 Including layout/prepare/default/600_show_unprocessed.sh
2017-05-09 17:54:58 Including layout/prepare/default/610_exclude_from_restore.sh
2017-05-09 17:54:58 Finished running 'layout/prepare' stage in 63 seconds
2017-05-09 17:54:58 ======================
2017-05-09 17:54:58 Running 'layout/recreate' stage
2017-05-09 17:54:58 ======================
2017-05-09 17:54:58 Including layout/recreate/default/100_ask_confirmation.sh
2017-05-09 17:54:58 Please confirm that '/var/lib/rear/layout/diskrestore.sh' is as you expect.
2017-05-09 17:55:47 User selected: 5) Continue recovery
2017-05-09 17:55:47 Including layout/recreate/default/200_run_script.sh
2017-05-09 17:55:47 Start system layout restoration.
  WARNING: Failed to connect to lvmetad. Falling back to device scanning.
+++ set +x
2017-05-09 17:55:47 Disk layout created.
2017-05-09 17:55:47 Including layout/recreate/default/250_verify_mount.sh
2017-05-09 17:55:47 ERROR: No filesystem mounted on '/mnt/local'. Stopping.
==== Stack trace ====
Trace 0: /bin/rear:518 main
Trace 1: /usr/share/rear/lib/recover-workflow.sh:17 WORKFLOW_recover
Trace 2: /usr/share/rear/lib/framework-functions.sh:87 SourceStage
Trace 3: /usr/share/rear/lib/framework-functions.sh:45 Source
Trace 4: /usr/share/rear/layout/recreate/default/250_verify_mount.sh:5 source
Message: No filesystem mounted on '/mnt/local'. Stopping.
== End stack trace ==
2017-05-09 17:55:47 Running exit tasks.
2017-05-09 17:55:47 Finished in 113 seconds
2017-05-09 17:55:47 Removing build area /tmp/rear.qZsRSjsMC2Fu9HY
removed directory: '/tmp/rear.qZsRSjsMC2Fu9HY'
2017-05-09 17:55:47 End of program reached

[schlomo]

@schabrolles can this be caused by dd65bd0? You seem to be the only author of any change in 300_map_disks.sh recently.

Test case is the recovery of the client VM from our rear-workshop. To test for yourself follow the instructions there to build the server and client VM, go to the client, install rear-workshop, install the latest Snapshot RPM, do cd /etc/rear ; cp workshop/local-with-nfs.conf local.conf and run rear -v mkbackup. Then start the recovery with vagrant up recover.

[schabrolles]

@schlomo Looks strange ... At a very quick look .... I think it is not related to my change (I just add multipath device to be proposed like a disk); but who know :) ....

I'm gonna try to run your rear-workshop later today and have look

[schabrolles]

@schlomo I quickly try rear-workshop, but it seems Vagrant use my KVM instead of Box to deploy centos_client ....
But I got an Error when deploying centos-server from bento/centos-7.3 (which seems to be a box VM which could explain why it failed).

Do you know how to force / configure vagrant to use box ?

[schabrolles]

@schlomo Ok I've found it ... vagrant up --provider virtualbox ...

Now I got a network problem (or a conflict with my virtualbox configuration)

==> server: Clearing any previously set network interfaces...
The specified host network collides with a non-hostonly network!
This will cause your specified IP to be inaccessible. Please change
the IP or name of your host only network so that it no longer matches that of
a bridged or non-hostonly network.

[schlomo]

Do you have a locally attached network with 192.168.33.0/24? Then please change 192.168.33 to some other range in all the files of rear-workshop

[schabrolles]

@schlomo ... It's running now :D ... I have to change the IP configuration in all Vagrant and scripts files from 192.168. to 192.167. to avoid conflict with my configuration.

[schabrolles]

@schlomo There is a Typo in usr/share/rear/layout/prepare/default/300_map_disks.sh... My mistake

There is a space before disk " ^disk (line 112).

done 3< <(grep -E " ^disk |^multipath " "$LAYOUT_FILE")

[schlomo]

Can you please submit an update? Thanks a lot.

[schabrolles]

Sure, I'm just finishing test before pushing the update.

[jsmeix]

@schlomo
does it work for you when you manually do the change in
736cb70

[schlomo]

Haven't had the time to try again, but the fix seems very much legitimate to me. In any case I don't see any reason why the blank could be correct there so that that fix is for sure an improvement.

[gdha]

@jsmeix @schlomo I can confirm the fix works fine.